can any one help me to distinguish between the wires of the encoder in this motor below?
It is a quadrature encoder which can tell direction of rotation.
Unlike a single interrupt detector.
You've got red, brown and green on the IR reciver side and the Orange and yellow on the IR emitting side. Put the Orange and yellow wires an a pair of AA batteries with a small resistor and look at it through your camera. That is if you camera is able to see IR light. You can check that by pointing a known good remote and your camera and pressing buttons to see if you can see the light. Swap wires if you don't see the light and mark the wires appropriately when you do see it. Often these IR receivers use the center wire as the sense wires while the Red and green go to the power source. Chances are Red is positive and green is negative.
But if it's still connected to a board that can power the setup then turn it one and put a meter on them. Also follow the traces back to the voltage source for verification.