In any non-abelian $p$-group, the derived subgroup is of index at least $p^2$.
It is a natural question to ask about structure of groups for which equality holds.
The answer is completely known if $p=2$. What are such $2$-groups? There are at most four non-abelian $2$-groups which have a cyclic subgroup of index $2$: they are dihedral ($D_{2^n}$), semi-dihedral $SD_{2^n}$, and quaternion ($Q_{2^n}$).
[$M_{2^n}$ was included in previous answer; but thanks to Holt for pointing out error.].
In these groups, derived subgroup is of index $4$.
A result of Tausky shows that these are the only non-abelian $2$-groups with derived subgroup of index $4$. Reference: See Groups of Prime Power Order, Vol. 1, by Y. Berkovich, its first chapter (titled Groups with a cyclic subgroup of index $p$).
For $p>2$: It seems a difficult question (to me at least) to classify $p$-groups with derived subgroups of index $p^2$. The $p$-groups of maximal nilpotency class is a family of examples of these types, whose classification, up to isomorphism is not known except for small orders (say $p^9$ or $p^{10}$ - sorry, I not remembered correctly).
On the other hand, the $2$-groups with $[G:G']=4$ have other characterizations also, and these are quite exceptional groups. You can find them in the chapter 1 of book mentioned above.