I don't know how but the rubber grip on the xbox controller joysticks wears off. Especially, the left joystick.
Fortunately, Sugru can fix that.
Step 1: Apply Sugru
First make sure the joystick is clean. Remove any residue rubber.
Open a (mini) pack of sugru and roll it into a sphere.
Flatten it and place on joystick.
Smooth-en out the edges and the surface.
Step 2: Finish
The final result should look the the attached photo.
Make sure the Sugru dries before you use your controller.