Correct option is (b) 2R
There is no external force on the system of the small and big spheres in the horizontal direction. So the center of mass C (horizontal coordinate of COM) remains at the same position.
X coordinate of COM:
\(C = \frac{-5R \times M + 0\times 4M}{M + 4M}=- R\)
Thus the center of mass is a distance R away from the center of big sphere.
When the small sphere goes to the other acute extreme, the center of mass C remains at the same position horizontally. COM will still be at a distance R from the center of big sphere. But on the other side.
Then the center of big sphere moves by 2R to the other side of COM.
The change in the position of large sphere = R−(−R) = 2R