Please return the pants that you borrowed from me.
In this question it sounds like you can also use the word 'which' to fill that gap as most people believe 'which' and 'that' can be used interchangeably. this is actually not the case. yes it can be confusing but there is a difference.
we use which when the clause in question can be removed or is not necessary but when it is absolutely necessary.
in the question, the person has given you a particular pair of pants and needs it back. you have many pants so we cannot remove the clause '.....you borrowed from me' as we are referring specifically to ones that you borrowed from me. because the clause is absolutely necessary to convey the proper meaning of the sentence we will use the relative pronoun 'that'.
