(a) Since the potential at each point of a spherical surface (shell) is constant and is equal to `varphi=-(gammam)/(R)`, [as we have in Eq. (1) of solution of problem]

We obtain in accordance with the equation

`U=1/2intdmvarphi=1/2varphiintdm`

`=1/2(-(gammam)/(R))m=-(gammam^2)/(2R)`

(The factor 1/2 is needed otherwise contribution of different mass elements is counted twice.)

(b) In this case the potential inside the sphere depends only on r of the solution of problem)

`varphi=-(3gammam)/(2R)(1-(r^2)/(3R^2))`

Here `dm` is the mass of an elementary spherical layer confined between the radii `r` and `r+dr`:

`dm=(4pir^2drrho)=((3m)/(R^3))r^2dr`

`U=1/2intdmvarphi`

`=1/2underset(0)overset(R)int((3m)/(R^3))r^2dr{-(2gammam)/(2R)(1-((r^2)/(3R^2))}`

After integrating, we get

`U=-3/5(gammam^2)/(R)`