When no person is sitting on the mat, then the mat and floor are not pressed together.
Thus, there is no friction.
But, when a person is sitting on the mat.
Then due to the weight of the person, the mat and floor are pressed together harder, increasing the friction too much. Thus, then it is very difficult to drag the person.