Here's another reason for $\mathfrak{S}_3$:
Proposition: If $H$ is a group with a characteristic subgroup $K$ with abelian automorphism group, but $K$ is not contained in the center of $H$, then $H$ is not the derived subgroup of any group.
Proof: Since $K$ is characteristic in $H$, $K$ is normal in $G$. There is a homomorphism from $G$ to the automorphism group of $K$. Since the automorphism group of $K$ is abelian, we must have that $H$ is in the kernel of this homomorphism. However, that just means $K$ is contained in the center of $H$. $\square$
This shows that dihedral groups (of order at least 6) are not derived subgroups. Their subgroup $K$ of rotations are characteristic and cyclic, so the proposition applies.