Yes. Consider the action of $G$ by multiplication on the (left or right, whichever you prefer) cosets of $H$ in $G$. This action is transitive, and so the image of the action has order divisible by $p$.
I claim that this image has order exactly $p$, which implies that the action is regular and $H$ is normal in $G$. If not, then the stabilizers in the action are non-trivial. Since the action is transitive, we can choose $g$ in such a stabilizer that does not fix the coset $H$. So the cycle of $g$ on the coset $H$ has order $m$ with $1 < m < p$, and since $p$ is prime, $m$ does not divide $p$, so $g^p$ does not fix $H$. That is, $g^p \not\in H$, contrary to assumption.
I expect there is a more elegant way of writing that proof!