Subgroups of smallest possible index in a solvable group

Question

The following question appears in Isaacs' Finite Group Theory:

3B.15) (Berkovich) Let $G$ be solvable, and let $H<G$ be a proper subgroup having the smallest possible index in $G$. Show that $H\lhd G$.

If we assume that $G$ is a minimal counterexample, then it is possible to show that $H$ has trivial core in $G$, that $H$ is a complement of the Fitting subgroup of $G$, and that the Fitting subgroup is a unique minimal normal subgroup of $G$. But this mostly follows from the fact that $H$ must be a maximal subgroup of $G$, rather than using the fact that $H$ has smallest possible index. It also doesn't seem to lead to any contradiction.

I cannot find the solution. Problem 3B.12 from the same chapter is wrong, so there is the possibility that this question is wrong too. If it is wrong, are there any counterexamples?

Answer 1

So, using the action of $G$ on the cosets of $H$, we can regard $G$ as a subgroup of $S_n$, where $H=G_\alpha$ is a point stabilizer and $n=|G:H|$. The minimal normal subgroup $N$ is regular and abelian, so $|N|=n$.

Since $H \ne 1$, there is a point $\beta$ with $H_\beta \ne H$. Then $|H:H_\beta|$ is the length of the orbit of $H$ on $\beta$, and hence is less than $n$, so $|G:NH_\beta| < n$, contradicting the minimality of $n$.

Answer 2

Although all credits go to Derek Holt, here is a more elaborate proof using the theory developed in the preceding chapters of Isaacs' book.

I.M. Isaacs - Finite Group Theory, Problem 3B.15

(Berkovich) Let $G$ be a solvable group and $H$ a proper subgroup of smallest possible index. Then $H$ is a normal subgroup.

Proof (Any reference to a Theorem, Lemma, Corollary or Problem will be from the aforementioned book) Let us assume that $H$ is not normal. We will derive a contradiction by constructing a proper subgroup with smaller index than that of $H$.
By induction on $|G|$ we can assume that core$_G(H)=1$ (see also Theorem 1.1). Note that $|H|$ is a non-normal maximal subgroup, and hence $H=N_G(H)$. Put index$[G:H]=n$. Let $N$ be a minimal normal subgroup of $G$. By Lemma 3.11, $N$ is an elementary abelian $p$-group (that is $N \cong C_p \times \cdots \times C_p$, $p$ prime).

Consider the subgroup $HN$. Since $H$ is maximal, we either have $H=HN$ or $G=HN$. The first case is impossible since this is equivalent to $N \subseteq H$, contradicting core$_G(H)=1$. So $G=HN$ and since $N$ is normal and abelian it follows that $H \cap N$ is normalized by both $H$ and $N$, which gives $H \cap N \lhd G$. Again, since core$_G(H)=1$, we get $H \cap N=1$.This implies that $|N|=$index$[G:H]=n$, a prime-power (and solves en passant Problem 3B.1)

Now let us have a look at the action of $H$ on $N$ by conjugation. Of course the orbit of $1 \in N$, $\mathcal{O}_1=\{1\}$ and this implies that all other orbits must have a length $\leq n-1$. We claim that if $g \in N-\{1\}$, then the length of its orbit $\mathcal{O}_g$ equals index$[H:H \cap H^g]$. By the Fundamental Counting Principle (Theorem 1.4) this is equivalent to showing that the stabilizer subgroup of $g$ equals $H \cap H^g$. Let $h \in H$ and suppose $g^h=h^{-1}gh=g$. Then $h=g^{-1}hg \in H^g$, whence $h \in H \cap H^g$. Conversely, assume that $h \in H \cap H^g$, say $h=g^{-1}kg$ for some $k \in H$. Then $hk^{-1}=g^{-1}(kgk^{-1})$. The left-hand side, $hk^{-1} \in H$, the right-hand side $\in N$, since $N$ is normal. But $H \cap N=1$, so $h=k$ and $h$ and $g$ commute. This proves the claim.

Now fix a $g \in N-\{1\}$ and put $K=(H \cap H^g)N$. We will show that $K$ is a proper subgroup of $G$ and that index $[G:K] \lt n$, finally contradicting the choice of $H$. Suppose $G=(H \cap H^g)N$. Note that $(H \cap H^g) \cap N=1$ and this gives by comparing indices, index$[H: H \cap H^g]=1$, so $H=H \cap H^g$, implying $H=H^g$. But then $g \in N_G(H) \cap N= H \cap N=1$, contradicting the choice of $g$. So $K$ is a proper subgroup.
Now using Corollary X.11, $|G|=|HN|=\frac{|H||N|}{|H \cap N|}=|H||N|$. Similarly, $|K|=|(H \cap H^g)N|=|H \cap H^g||N|$. This shows that index$[G:K]=$index$[H:H \cap H^g]=\#\mathcal{O}_g \lt n$. The proof is now complete.

Corollary Let $G$ be a solvable group and $H$ a proper subgroup of largest possible order. Then $H$ is a normal subgroup.

Answer 3

More of the same, perhaps shorter.

Since the smallest possible index corresponds to the greatest possible order, $H$ is maximal.

Let $C=\text{core}_G(H)$. If $C\ne\langle1\rangle$, induction on $|G|$ implies that $H/C\triangleleft G/C$ because $|G/C:H/C|=|G:H|$ is the smallest possible index in $G/C$. Hence, $H\triangleleft G$ and we are done.

Now suppose, toward a contradiction, that $C=\langle1\rangle$. Take $N$ minimal normal in $G$. Then we have: (1) $N\cap H\triangleleft N$ because $N$ is (elementary) abelian, and (2) $N\cap H\triangleleft H$ because $N\triangleleft G$. In consequence, $N\cap H\triangleleft NH=G$. Thus, $N\cap H=1$ and $|N|=|G:H|$.

Consider the action of $H$ on $N$ by conjugation. By the fundamental counting principle, $$ |{\cal O}_y| = |H:H_y|\qquad (y\in N). $$ Since ${\cal O}_1=\{1\}$ we deduce that $|{\cal O}_y|\le |N|-1$ because orbits are disjoint subsets of $N$. According to Problem 3B.12, there exists $K$ with $|G:K|=|H:H_y|$. Then, $$ |G:K| = |H:H_y| = |{\cal O}_y| \le |N|-1 = |G:H|-1, $$ which, as anticipated, contradicts the hypothesis on $|G:H|$ when $y\ne1$.