Maximal subgroup contains either the center or the commutator subgroup

Question

Here's the problem:

Suppose $G$ is a group and $H$ is a maximal subgroup of $G$. Show that either $Z(G) \leq H$ or $G' \leq H$.

I know that if $G$ is abelian, then $G = Z(G)$, $G' = \{e\}$, and $G' \leq H$.

Furthermore, if $G$ is not abelian but $H$ is abelian, then $Z(G) \leq H$, because otherwise the subgroup $\langle Z(G), H \rangle$ would be an abelian subgroup of $G$ that contains $H$, and because of the maximality of $H$, is equal to $G$, contradicting the fact that we took $G$ to be non-abelian.

I'm unsure of what happens when both $G$ and $H$ are non-abelian. Any hints would be appreciated!

Answer 1

A short answer. We can suppose that $Z=Z(G)$ is not contained in $H$, and we have to prove that $G'\le H$. So $HZ$ properly contains $H$, hence by maximality equals $G$. The subgroup $N=Z\cap H$ is central hence normal, and thus $G/N=H/N \times Z/N$. So $(G/N)'=(H/N)'$, that is, $G'N=H'N$. Thus $G'\subset G'N=H'N=H'(H\cap Z)\subset H$.

Note that I'm not assuming $G$ to be finite.

Answer 2

Here's an alternative approach (without using the nice indication of @SteveD in the comments) for when G is finite.

We work by induction over $|G|$. Suppose that there exists a minimal normal subgroup $N$ of $G$ included in $H$. Then $H/N$ is a maximal subgroup of $G/N$. By induction $Z(G/N)\leq H/N$ or $(G/N)'=G'N/N\leq H/N$. Since $Z(G)N/N\leq Z(G/N)$ then we conclude that $Z(G)\leq H$ or $G'\leq H$. Thus we may assume that $H$ does not have any nontrivial subgroup which is normal in $G$.

If $Z(G)\nleq H$ and $G'\nleq H$ then $G=Z(G)H=G'H$ by the maximality of $H$. Now, since $H,Z(G)\leq N_G(Z(G)\cap H)$ and $H,Z(G)\leq N_G(G'\cap H)$ then both $Z(G)\cap H$ and $G'\cap H$ are normal in $G$, and therefore these subgroups are trivial. Then

$$G/Z(G)\cong H/Z(G)\cap H=H/G'\cap H\cong G/G'$$ is abelian. Since $G/Z(G)$ is nilpotent then so is $G$, and therefore $H\unlhd G$ (since it is a maximal subgroup of the nilpotent group $G$). As a consequence $G/H$ is a cyclic group of prime order and hence $G'\leq H$ (since $G/H$ is abelian), which is a contradiction.

Answer 3

If $Z(G) \nsubseteq H$, then by the maximality of $H$, it follows that $G=HZ(G)$. This easily implies $G'=H' \subseteq H$.

You also can draw the conclusion that always $G' \cap Z(G) \subseteq \Phi(G)$, the Frattini subgroup. If you are familiar with upper and lower central series, then even $\gamma_{n+1}(G) \cap \zeta_n(G) \subseteq \Phi(G)$ for all non-negative numbers $n$.