When the moving block A comes in contact with the spring fixed with the block B at rest ,the velocity of A is retarded and that of B is accelerated. As a result compression of spring goes on until maximum compression occurs.At this stage both the blocks attain same velocity (v) following conservation of momentum.
So we can write
mAvA+mBvB=(mA+mB)v
=> 5×8+10×0=(5+10)v
=v=8/3 m/s
Now net loss in KE during this interaction will go to increase the PE of the spring in its compressed state.. So we can write using conservation of mechanical energy
1/2mAvA2 -1/2(mA+mB)v2= 1/2 kx2 , where k is the force constant of the spring and x is its maximum compression
=> 5× 82-(5+10)(8/3)2 = 25x2
=>x2 =( 5x82×2)/3×1/25
=>x2= 128/15
=> x= 2.9m