Exam answer
Matrix multiplication is not commutative in general, so
(A+B)2=(A+B)(A+B)=A(A+B)+B(A+B)=A2+AB+BA+B2
Why?
Multiply out using the distributive law, keeping each product in its order:
A(A+B)B(A+B)=A2+AB=BA+B2
Watch out: for numbers, ab+ba=2ab. For matrices, AB and BA are usually different, so the middle terms cannot be combined into 2AB.
Given (A+B)2=A2+B2:
A2+AB+BA+B2⇒AB+BA⇒AB=A2+B2=O=−BA
Why?
Subtract A2 and B2 from both sides. What is left must be the zero matrix O.
For the other options, take
A=[0110], B=[100−1]
AB=[01−10], BA=[0−110]
Here AB=−BA, so the condition holds, but AB=O, BA=O and AB=BA. So (A), (C) and (D) need not be true.
Why?
Each entry of AB is a row of A times a column of B. For example, row 1 of A times column 2 of B is 0×0+1×(−1)=−1.
All four entries of AB (row of A times column of B):
(1,1)(1,2)(2,1)(2,2):(0)(1)+(1)(0)=0:(0)(0)+(1)(−1)=−1:(1)(1)+(0)(0)=1:(1)(0)+(0)(−1)=0
All four entries of BA (row of B times column of A):
(1,1)(1,2)(2,1)(2,2):(1)(0)+(0)(1)=0:(1)(1)+(0)(0)=1:(0)(0)+(−1)(1)=−1:(0)(1)+(−1)(0)=0
Change every sign of BA:
−BA=[01−10]=AB
One pair of non-zero matrices that satisfies the condition but breaks an option is enough to rule that option out.
Answer: (B) AB=−BA