In the first case, they are referring to the Giant's garden where they would play while returning from school. They had a great time whenever they played in the garden. In the second case, they are again referring to the garden; talking about how happy they were before the Giant stopped them from playing in his garden. In both the cases, the children are referring to the Giant's garden.
