Consider the identity $|Z(G)|=|G|-\sum_{j=1}^m |C_{x_j}|$ where $C_1,...C_m$ are the conjugation classes in which $G\backslash Z(G)$ is partitioned into and $G$ is a non-abelian group with $p^m$ elements where $p$ is prime and $m \in \mathbb{N}$.
My goal is to prove $|Z(G)|>1$. Suppose it has only one element. How can I derive a contradiction then? In this case $|G|=\sum_{j=1}^m |C_{x_j}| + 1=p^m$ for some $p,m$. Since every $C_{x_j}$ has at least two elements the left hand side is bounded by $2m+1$ which is larger than $2^m$ in the case of $p=2$.
Is it enough to derive a contradiction for only one of the $p$'s?