If $M$ is maximal subgroup of a finite group $G$, if $M$ is normal in $G$, then $|G:M|$ is prime.

Question

I want to show that if $M$ is maximal subgroup of a finite group $G$, if $M$ is normal in $G$, then $|G:M|$ is prime. I feel pretty close to getting this one, but I'm getting stuck

Suppose $M \leq G$ is maximal in the finite group $G$. Suppose also $M \lhd G$ but toward a contradiction, suppose $M$ does not have prime index in $G$. Then $|G:M| = pm$ for $p$ prime and $m \in \mathbb{N}, m \geq 2$. Now, $G/M$ is a group with $|G/M| = pm$. By Cauchy's theorem, there exists $A \leq G/M$ with $|A| = p$. By lattice correspondence theorem, $A \cong \frac{B}{M}$ for some $B \in \{X: M \leq X \leq G\}$. Then, we can write

$$|G:M| = |G:B||B:M| \implies |G:B| = \frac{|G:M|}{|B:M|}$$

If I can show that $|B:M| \geq 2$, then I will have a contradiction since the index of $B$ will be strictly less than $|G:M|$, implying $M < B$; at the same time, if I knew that $|B:M| \geq 2$, I would already have contradicted the maximality of $M$, completing the proof. Any ideas here? Is there a better approach to this problem? Any hints are greatly appreciated.

Answer 1

It is false that every maximal subgroup of a finite group has prime index (original post). It is true if the subgroup is both maximal and normal.

For a counterexample to the general claim, consider $A_3$ as a subgroup of $A_4$. The index of $A_3$ is $4$. If it were not maximal, it would be contained in a subgroup of order $6$, but $A_4$ has no subgroups of order $6$. So $A_3$ is maximal in $A_4$, but its index is not prime.

This is the smallest counterexample. You can get examples with any index by considering $S_{n-1}$ as a subgroup of $S_{n}$

For the case where $M$ is maximal and normal, the quotient cannot have any proper nontrivial subgroup by the Isomorphism Theorems, so the quotient must be cyclic of prime order.

(It is also false that a maximal normal subgroup, i.e., a subgroup maximal among normal subgroups, must have prime index. Any nonabelian simple group provides counterexamples.)

Answer 2

This is not true if $M$ is not normal (as stated in the original version of your post): the simple group of order $168$ has maximal subgroups of index $7$ and of index $8$. Your assertion is however true if $G$ is supersolvable. The converse is then also true, a famous theorem of Bertram Huppert dating from $1954$. If $G$ is solvable it is not difficult to prove that your $M$ must have prime-power index.