(a) Since the potential at each point of a spherical surface (shell) is constant and is equal to `varphi=-(gammam)/(R)`, [as we have in Eq. (1) of solution of problem]
We obtain in accordance with the equation
`U=1/2intdmvarphi=1/2varphiintdm`
`=1/2(-(gammam)/(R))m=-(gammam^2)/(2R)`
(The factor 1/2 is needed otherwise contribution of different mass elements is counted twice.)
(b) In this case the potential inside the sphere depends only on r of the solution of problem)
`varphi=-(3gammam)/(2R)(1-(r^2)/(3R^2))`
Here `dm` is the mass of an elementary spherical layer confined between the radii `r` and `r+dr`:
`dm=(4pir^2drrho)=((3m)/(R^3))r^2dr`
`U=1/2intdmvarphi`
`=1/2underset(0)overset(R)int((3m)/(R^3))r^2dr{-(2gammam)/(2R)(1-((r^2)/(3R^2))}`
After integrating, we get
`U=-3/5(gammam^2)/(R)`