Answer is (A) AB and BA are square matrices
O(A) = m × (n + 5) and O(B) = m × (11 - n)
AB exists ⇒ n + 5 = m ⇒ m - n = 5 (1)
BA exists ⇒ 11 - n = m ⇒ m + n = 11 (2)
Solving Eqs. (1) and (2), we have
m = 8 and n = 3
Therefore, O(A) = 8 × 8 and O(B) = 8 × 8
Therefore, AB and BA both are square matrices of order 8 × 8.