This is due to the inertia of rest. When the branch is shaken swiftly, the branch suddenly moves a little distance, but the fruits attached to it by a slender stalk (petiole) tend to remain where they are (due to inertia). So, the stalk feels a jerk and breaks away and the fruit falls down.
