Commuting matrices are simultaneously triangularizable

Question

Let $A$, $B$ be two $n\times n$ matrices over $\mathbb{C}$. If $AB=BA$ then there exists a basis $\mathcal B$ such that under this basis, the matrices of $A$, $B$ are both upper triangular. How to prove this?

I know how to prove the following: If $A$, $B$ are diagonalizable and commute, then they are simultaneously diagonalizable. Will the proof here be similar?

Moreover, give an example such that $A$, $B$ don't commute and are not simultaneously triangularizable. Also give an example such that $A$, $B$ commute but are not diagonalizable, then they are not simultaneously diagonalizable.

I will be very grateful if you post your answers and share with me. Helps is really in need urgently. Thanks a lot.

Answer 1

To show that $AB=BA$ implies that $A,B$ are simultaneously triangularizable, it is sufficient to prove that $A,B$ have a common eigenvector (after, reason by recurrence). Let $\lambda$ be an eigenvalue of $A$. Then $\ker(A-\lambda I)$ is a $B$-invariant subspace. Thus, there is an eigenvector of $B$ that is in $\ker(A-\lambda I)$. Clearly we may change $\mathbb{C}$ with any algebraically closed field.

EDIT: If $u$ is such a common eigenvector, then consider a basis $u,e_2,\cdots,e_n$. In this new basis, $A,B$ become $\begin{pmatrix}\lambda&K\\0&A_1\end{pmatrix},\begin{pmatrix}\mu&L\\0&B_1\end{pmatrix}$ with $A_1B_1=B_1A_1$ and you can conclude, using the recurrence hypothesis.

Ren, I think that the other questions are your business. Yet, you can easily prove that if $A,B$ are simult. triang., then $AB-BA$ is a nilpotent matrix.

Answer 2

Here's an answer, which is in principle same as the previous answer, but in the language of projections, which often turn out to be very useful.

Let $A$ and $B$ be operators on the Hilbert space $V$. Let $S$ be a subspace of $V$, and let $P$ be the orthoprojection onto $S$. Recall $P = P^*$. It is easy to show that the following are equivalent.

1. $S^\perp$ is invariant under A.
2. $S$ is invariant under $A^*$.
3. $PA^*P = A^*P$.

We will also use this definition of traingularization: $A$ has a traingularization if and only if $V$ has an orthonormal basis $y_1, \dots, y_n$ such that $Ay_j \in \text{span}[y_1, \dots, y_j]$ for $j = 1, \dots, n$.

Proof is by the induction on the dimension $n$ of the underlying space. Firstly, since $A$ and $B$ commute, (show) they share an (unit) eigenvector, say $y_1$. Now let $S = \text{span}[y_1]^\perp$ and let $P$ be the orthoprojection onto S. Since $\text{span}[y_1]$ is invariant under both $A$ and $B$, we have $$P A^* P = A^* P \text{ and } P B^* P = B^* P.$$

Now look at the operators $PA$ and $PB$. Note that $$ \begin{align*} (PAPB - PB PA)^* &= B^*P A^*P - A^*PB^*P\\ &= B^* A^*P - A^* B^* P\\ &= (AB - BA)^* P \\ &= 0. \end{align*} $$ So $PA$ and $PB$ commute, and they can be considered as operators from $S$ to $S$, which is an $n - 1$ dimensional space. By induction hypothesis, $S$ has an orthonormal basis $y_2, \dots, y_n$ which simultaneously triangularizes $PA$ and $PB$, that is, for $j = 2, \dots, n$ $$PAy_j \in \text{span}[y_2, \dots, y_j] \enspace \text{and} \enspace PBy_j \in \text{span}[y_2, \dots, y_j] .$$ Now $y_1, \dots, y_n$ is an orthonormal basis of $V$ and for $j = 1, \dots, n$ $$ Ay_j = (I - P) Ay_j + PAy_j \in \text{span}[y_1, \dots, y_j] $$ since $I - P$ is a projection to $\text{span}[y_1]$. The same is true for $B$. This completes the proof.

The argument also works when you have a family $\{A_\alpha\}$ of commuting matrices, instead of a pair.