What if there are two non-orthogonal invariant subspaces?

Question

Let $U_1,U_2\subseteq\Bbb R^n$ be two invariant subspaces w.r.t. some group $\Gamma\subseteq\mathrm O(\Bbb R^n)$ of orthogonal matrices.

I wonder the following:

Question: If $U_1$ and $U_2$ are not orthogonal (i.e. there are $u_i\in U_i$ with $\langle u_1,u_2\rangle \not=0$), then every $T\in\Gamma$ restricted to $U_1\oplus U_2$ is already either $\mathrm{Id}$ or $-\mathrm{Id}$.

If this is true, what are the weakest assumptions we need? Do we need that the $U_i$ are irreducible? Does $U_1\cap U_2 = \{0\}$ sussfice, or maybe just $U_1\not\subseteq U_2$ and $U_2\not\subseteq U_1$.

---

Update

I received some very helpful answers and the statement as written above is definitely wrong. I still wonder whether there is a counterexample under the following restrictions:

1. $\Gamma\subseteq\mathrm{O}(\Bbb R^n)$ is a finite group.
2. The subspaces $U_i$ are irreducible invariant subspaces of $\Gamma$.

Answer 1

Take the natural action of $O_2(\mathbb R)$ on the space of two-by-two matrices by left multiplication. This space has an invariant inner product $\langle M,N\rangle = \operatorname{tr}(MN^T).$ Take

$$U_1=\Big\{\begin{pmatrix}a&0\\b&0\end{pmatrix}\mid a,b\in\mathbb R\Big\}$$ $$U_2=\Big\{\begin{pmatrix}a&a\\b&b\end{pmatrix}\mid a,b\in\mathbb R\Big\}.$$

These spaces are not orthogonal - set $a=b=1$ in both.

Answer 2

The statement as it stands is certainly not true. Just pick any $n>1$, set $ U_1=U_2=\mathbb R^n$ and pick any $\Gamma$ that contains at least one non-diagonalisable orthogonal matrix over $\mathbb R$.

The assertion is true, however, if every nonzero vector in $U_1\ne0$ is not orthogonal to every nonzero vectors in $U_2\ne0$. In this case, pick a nonzero vector $u\in U_1$. If $U_2$ contains two linearly independent vectors $v,w$, then by assumption, $c_1=\langle u,v\rangle$ and $c_2=\langle u,w\rangle$ are nonzero. But then we have $\langle u,\ c_2v-c_1w\rangle=0$, which is a contradiction to our non-orthogonality assumption.

Therefore $U_2$ must be one-dimensional. Similarly for $U_1$. Hence $U_1,U_2$ are eigenspaces. As eigenspaces for different eigenvalues of an orthogonal matrix are orthogonal to each other, $U_1$ and $U_2$ must be eigenspaces for the same eigenvalues. Now, over $\mathbb R$, the eigenvalues of an orthogonal matrix can only be $1$ or $-1$. Hence the restriction of each orthogonal transform in the group on $U_1\oplus U_2$ is $\pm I$.

Answer 3

With the help of the provided answers, I was able to find a counterexample with a finite group acting on $\Bbb R^3$, and invariant subspaces with $U_1\cap U_2=\{0\}$.

Choose $U_1=\mathrm{span}\{e_1,e_2\}$ and $U_2=\mathrm{span}\{e_1+e_3\}$. These are obviously non-orthogonal and only intersect in $\{0\}$. Moreover, they are invariant subspaces of

$$T:=\begin{pmatrix} 1 & \phantom+0 & 0 \\ 0 & -1 & 0 \\ 0 & \phantom+0 & 1\end{pmatrix}\not\in\{\mathrm{Id},-\mathrm{Id}\}.$$

We can simply take the group $\Gamma=\{\mathrm{Id}, \,T\}$.

However, $U_1$ is not an irreducible invariant subspaces of $\Gamma$ as it contains the invariant subspaces $\mathrm{span}\{e_1\}$ and $\mathrm{span}\{e_2\}$.

---

Here is an example for which the $U_i$ are irreducible invariant subspaces. Let $\bar \Gamma\subseteq\mathrm{O}(\Bbb R^2)$ be some (sufficiently large) finite group of orthogonal transformations in the plane, e.g. $\bar\Gamma=D_3$. We consider the group

$$\Gamma:=\left\{ \begin{pmatrix} T & 0 \\ 0 & T\end{pmatrix} \;\middle\vert\; T\in\bar\Gamma\,\right\}\subseteq\mathrm{O}(\Bbb R^4).$$

This group has the following irreducible invariant subspaces

$$\mathrm{span}\{e_1,e_2\},\quad\mathrm{span}\{e_3,e_4\},\quad\mathrm{span}\{e_1+e_3,e_2+e_4\}.$$

While the first two are orthogonal, none of these is orthogonal to the last one.