I'm trying to find the subgroups of $D_4$, the group of symmetries of the square.
One way this could be done is to go through each element and every possible subset and check the axioms: the identity rotation belongs in each, it must have inverses, it must be closed. But I'm trying to find a more systematic way to go about this, as I'm sure that was the intention in the problem.
First, there are $8$ symmetries of the square: $4$ rotations, including the identity, and $4$ reflections. Let $r$ be a rotation through $\frac{\pi}{2}$ radians and $s$ a reflection through a line of symmetry. (I don't think I technically need to specify precisely which line of symmetry because any will do, but for concreteness, let's just say this is a line of symmetry through vertex $1$, the upper right corner, and I've numbered the vertices counterclockwise about that vertex.)
$D_4$ is generated by $r$ and $s$, so I can write: $$ D_4 = \{e, r, r^2, r^3, s, rs, r^2 s, r^3 s\}. $$ By Lagrange's theorem, the order of any subgroup of $D_4$ must divide $8$. That leaves us with $1$, $2$, $4$, and $8$. The only subgroup of order $1$ is $H_1 = \{e\}$. The only subgroup of order $8$ is $H_8 = D_4$. Any element of order $2$ then generates a cyclic subgroup of order $2$. These elements are $r^2$ and all of the reflections, so that gives subgroups: $$ H_{21} = \{e,r^2\}, H_{22} = \{e, s \}, H_{23} = \{e, rs\}, H_{24} = \{e, r^2 s\}, H_{25} = \{e, r^3 s\}. $$ Finally, we move to groups of order $4$, which is where I get stuck. There are two groups of order $4$, up to isomorphism. They are the Klein $4$-group and the cyclic group of order $4$. The latter must of course be generated by an element of order $4$. We have exactly two of those, $r$ and $r^3$, but both generate the subgroup consisting of all of the rotations: $$ H_4 = \{e, r, r^2, r^3\}. $$ I need only find the remaining groups of order $4$ that are isomorphic to the Klein $4$-group and not cyclic. The Klein $4$-group is surely abelian, so any group isomorphic to it must be abelian, which limits my choices somewhat. It must contain the identity, so I'm restricted to three additional elements. If I include $r$, it will include all of the rotations, so it can't include $r$.
I'm not sure of a systematic way to find the remaining subgroups of order $4$, other than just brute force. I can certainly do that, but it isn't very instructive, and I'm surely missing something. Is there a better way to do this?