Maximal Closed Subgroups of $ SO_5(\mathbb{R}) $

Question

$ SO_2(\mathbb{R}) $ does not have any maximal closed subgroups.

The maximal closed subgroups of $ SO_3(\mathbb{R}) $ are $ O_2(\mathbb{R}) $ as well as the two finite groups $ S_4 $ and $ A_5 $.

$ SO_4(\mathbb{R}) $ double covers $ SO_3(\mathbb{R}) \times SO_3(\mathbb{R}) $. So the six maximal closed subgroups of $ SO_4(\mathbb{R}) $ are the lifts through this double cover of the three subgroups of the form $ H \times SO_3(\mathbb{R}) $ and the three subgroups of the form $ SO_3(\mathbb{R}) \times H $, where $ H $ is one of the three maximal closed subgroups of $ SO_3(\mathbb{R}) $.

What are the maximal closed subgroups of $ SO_5(\mathbb{R}) $?

https://arxiv.org/pdf/math/0605784.pdf classifies all maximal closed subgroups of $ SO_5 $ whose identity component is not simple (here trivial counts as simple). According to this paper, pages 1024-1025, the maximal closed subgroups of $ SO_5 $ of this type are $$ S(O_3 \times O_2 ) $$ with two connected components and $$ S(O_4 \times O_1) \cong O_4 $$ also with two connected components.

Next, we consider maximal closed subgroups with nontrivial simple connected component.

By dimension, such a subgroup would be isogeneous to $ SU_2$ or $ SU_3 $.

There are no 5d irreps of $ SU_3 $ since the dimension of $ SU_3 $ irreps are given by the formula $$ \frac{(m_1+1)(m_2+1)(m_1+m_2+2)}{2} $$

I think that the irreducible real 5 dimensional irrep of $$ SO_3(\mathbb{R}) $$ is the identity component of a maximal closed subgroup of $ SO_5 $. Perhaps it is the entire maximal group? Or perhaps the normalizer $ N(SO_3(\mathbb{R})) $ is slightly larger?

Finally, what (if any) are the finite maximal closed subgroups of $ SO_5(\mathbb{R}) $? $$ S_6 $$ of order $ 720 $ is a finite maximal closed subgroup. There is a faithful 4d complex irrep of $ 2.S_6 \cong Aut(SL_2(9)) $ which is quaternionic (Schur indicator -1) so $ 2.S_6 $ is a subgroup of $ Sp_2 $. Thus $ S_6 $ is a subgroup of $ Sp_2/\pm I \cong SO_5(\mathbb{R}) $. One can also verify directly that $ S_6 $ has a faithful 5d real irrep (it is called the "standard representation" here https://groupprops.subwiki.org/wiki/Linear_representation_theory_of_symmetric_group:S6 and is obtained as the unique non trivial subrep of the defining permutation rep). Since this $ S_6 $ subgroup is irreducible and not a subgroup of $ SO_3 $ it it not contained in any infinite closed group. Since $ S_6 $ is maximal among the finite subgroups that is enough to conclude that $ S_6 $ is a finite maximal closed subgroup of $ SO_5 $.

The other finite maximal closed subgroup of $ SO_5 $ is the group of signed permutation matrices of determinant 1, which I denote $ W_5 $ see here Signed Permutations and Coxeter Groups $$ W_5 \cong W(D_5) $$ of order $ 5!2^5/2=1920 $ also known as the Weyl group of $ D_5 $. This is related, in a slightly convoluted way, to the PerfectGroup(1920,6) subgroup of $ Sp_2 $.

This group is irreducible and not a subgroup of $ SO_3 $. Thus all that remains to show is that it is maximal among the finite subgroups of $ Sp_2 $.

$ 2.W(D_5) $ is the intersection of $ Sp_2 $ with group XXVII from https://arxiv.org/abs/hep-th/9905212. XXVII isn't quite maximal among the finite subgroups of $ SU_4 $, since it is contained in group XXX. However $ 2.W(D_5) $ is maximal finite in $ Sp_2 $ because group XXX cannot be contained in $ Sp_2 $ by part (i) of theorem 8 of https://core.ac.uk/download/pdf/82740228.pdf (where $ 2.W(D_5) $ is referred to as calligraphic B*)

To summarize, the full list of maximal closed subgroups of $ SO_5 $ is \begin{align*} & S(O_3 \times O_2) \\ & S(O_4 \times O_1)\cong O_4 \\ & SO_3 \\ & S_6 \\ & W(D_5) \end{align*}

I'm interested in any references or thoughts on the correctness of this list. I'm especially curious about the maximality of the 5d irrep of $ SO_3 $, and thoughts on the finite maximal subgroups. Note that $ SO_3 \times SO_2 $ in $ SO_5 $ lifts to $ U_2 $ in $ Sp_2 $ and $ O_4 $ in $ SO_5 $ lifts to $ O_4 $ in $ Sp_2 $.

Note on $ n=6 $ case: Since $ SO_6(\mathbb{R})\cong SU_4/\pm I $ then the maximal closed subgroups of $ SO_6 $ are exactly the quotient by $ \pm I $ of the groups given in Maximal Closed Subgroups of $ SU_4 $ Namely, the maximal closed subgroups of $ SU_4 $ and their corresponding subgroups in $ SO_6 $ are \begin{align*} & N(T) \twoheadrightarrow N(T) \cong (SO_2 \times SO_2 \times SO_2) \rtimes S_3 \\ & S(U_3 \times U_1) \cong U_3 \twoheadrightarrow U_3 \\ & N(S(U_2 \times U_2)) \twoheadrightarrow S(O_2 \times O_4) \\ & N(SU_2 \otimes SU_2) \twoheadrightarrow N((SO_3 \times SO_3)) \\ & N(Sp_2) \twoheadrightarrow S(O_5 \times O_1)\cong O_5 \\ &4 \circ_2 2.A_7 \twoheadrightarrow 2 \times A_7 \\ &4 \circ_2 Sp_4(\mathbb{F}_3) \twoheadrightarrow 2 \times PSp_4(\mathbb{F}_3) \\ &N(2^{2(2)+1}) \twoheadrightarrow W_6 \end{align*} where in the second line $ U_3 $ is the image of the standard embedding of $ U_n $ into $ SO_{2n}(\mathbb{R}) $, the third group has cyclic 2 component group generated by minus the identity, and for the fourth group the normalizer has cyclic 4 component group generated by $$ \begin{bmatrix} 0 & I\\ -I & 0 \end{bmatrix} $$ where here $ I $ is the $ 3 \times 3 $ identity matrix.

Note that throughout we have used freely the commuting diagram where $$ Sp_2 \hookrightarrow SU_4 $$ double covers $$ SO_5 \hookrightarrow SO_6 $$

Also not that $ W_n $ and $ W(D_n) $ are isomorphic only for odd $ n $ so here $ W_6 $ truly is not isomorphic to $ W(D_6) $, for details see Signed Permutations and Coxeter Groups .

Answer 1

The final answer is that $SO(3)$ (embedded into $SO(5)$ as an irrep) is maximal.

I'll first prove that $SO(3)$ is maximal among connected Lie groups, and then I'll worry about components after that.

So, let's assume $SO(3)\subseteq K\subseteq SO(5)$ with $K$ connected. We can assume $SO(3)\neq K\neq SO(5)$. Then counting dimensions and rank shows that $K$ is, up to cover, one of $SO(3)\times S^1, SO(3)\times SO(3)$, or $SU(3)$. You have already shown $SU(3)$ is not contained in $SO(4)$, so it must be one of $SO(3)\times S^1$ or $SO(3)\times SO(3)$. In any case, $K$ has a cover of the form $K' = SU(2)\times S^1$ or $SU(2)\times SU(2)$.

The irreps of $K'$ are tensor products of irreps of the factors. The factor $SU(2)$ has a unique irrep in each (complex) dimension, which alternates between real and quaternionic. The irreps of $S^1$ are all one dimensional, and classified by integers. They are complex, except the trivial rep is real.

Let's start with $K' = SU(2)\times S^1$. Since the rep a) be $5$-dimensional and b) contain $(\mathbf{5}\otimes \rho)$ as a sub rep (where $\mathbf{5}$ denotes the $5$-dim irrep of $SU(2)$ and $\rho$ denotes an arbitrary irrep of $S^1$), it follows that the rep of $K'$ given by $K'\rightarrow K\subseteq SO(5)$ is of the form $\mathbf{5}\otimes \rho$. If $\rho$ is trivial, then the covering $K'\rightarrow K$ has kernel containing $\{e\}\times S^1$, so is not finite, a contradiction. If $\rho$ is non-trivial, the rep is a tensor product of an orthogonal rep with a complex rep, so is complex (not orthgonal), again, giving a contradiction. Thus, this case cannot occur.

Let's next assume $K' = SU(2)\times SU(2)$. Unlike the previous case, we cannot simply assert that such a rep must contain $\mathbf{5}\otimes \rho$ because the $SO(3)$ we care about a priori could be diagonally embedded, or something like that.

So, let's just find all almost-faithful $5$-dim real reps of $K'$. If such a rep includes a sub rep of the form $\mathbf{1}\otimes \mathbf{n}$, of dimension $n\geq 2$, it must also include one of the form $\mathbf{m}\otimes \mathbf{n'}$ with $m\neq 1$ and $mn' = 5-n$. From here, it's easy to see that $n' = 1$. From here, it follows that precisely one of $n,m$ is odd, which then implies the rep is complex. (A real reap is a sum of real reps together with a sum of other irreps paired with their duals.)

So, the only reps we need to consider have the form $\mathbf{m}\otimes \mathbf{n}$ with both $m,n\geq 2$. Then it's clear the only option is $(\mathbf{2}\otimes \mathbf{2}) \oplus (\mathbf{1}\otimes\mathbf{1}).$ This rep (which corresponds to the usual $SU(2)\times SU(2)\rightarrow SO(4)\subseteq SO(5)$ is obviously reducible. This contradicts the fact that the $SO(3)\subseteq K$ action is irreducible. This concludes the case where $K$ is connected.

So, $SO(3)$ is maximal among connected groups, but what about components? Well, as in your previous questions, it follows now that $N:=N_{SO(5)}(SO(3))$ is a maximal subgroup of $SO(5)$ containing $SO(3)$. Let's see why $N = SO(3)$.

So, suppose $g\in N$. Then conjugation by $g$ induces an isomorphism of $SO(3)$. All isomorphisms of $SO(3)$ are inner, so there is an $h\in SO(3)$ with $hAh^{-1} = gAg^{-1}$ for all $A\in SO(3)$. Said another way, $h^{-1}g$ centralizes $SO(3)$.

Now, think of $SO(3)$ as acting on $\mathbb{C}^5$ as a complex rep, Schur's lemma tells us the the isomorphisms of the rep are $\mathbb{C}$-multiples of the identity matrix $I$. But anything which centralizes $SO(3)$ is an isomorphism of the rep. Thus, $h^{-1}g = \lambda I$ for some $\lambda \in \mathbb{C}$. As the only multiple of the identity in $SO(5)$ is the identity, we conclude $h^{-1}g$ is the identity, That is, $h = g$. This proves $N\subseteq SO(3)$, so $SO(3)$ is, in fact, a maximal subgroup of $SO(5)$.