Note that we always have [$f(x)$ is reducible over $F$ $\Rightarrow$ $f(x^p)$ is reducible over $F$]. So it remains to show
[$\exists j,a_j\notin F^p$] iff [$f(x)$ is irreducible over $F$ $\Rightarrow$ $f(x^p)$ is irreducible over $F$].
($\Leftarrow$): Assume for a contradiction that $\forall j,a_j\in F^p$ and $f(x)$ is irreducible. Then $f(x^p)=(\sum_i a_i^{1/p}x^i)^p$ is reducible, #.
($\Rightarrow$): Suppose $\exists j,a_j\notin F^p$ and $f$ is irreducible over $F$.
Let $\alpha$ be a root of $f(x^p)$ in an algebraic closure $\overline{F}$ of $F$, then $\alpha^p$ is a root of $f(x)$. Let $n=\deg f$ then $[F(\alpha^{p}):F]=\deg f=n$. Note that $[F(\alpha):F(\alpha^{p})]=1$ or $p$ depending whether or not $x^{p}-\alpha^{p}$ is irreducible over $F(\alpha^{p})$, or equivalently $\alpha^{p}\notin F(\alpha^{p})^{p}$, or equivalently $\alpha\notin F(\alpha^{p})$.
Suppose $\alpha\notin F(\alpha^{p})$, then $[F(\alpha):F(\alpha^{p})]=p$ and thus $[F(\alpha):F]=np$. Plus that $\deg f(x^{p})=np$, we must have $f(x^{p})$ is the minimal polynomial of $\alpha$ over $F$, so irreducible.
Otherwise $\alpha\in F(\alpha^{p})$ and thus $F(\alpha)=F(\alpha^{p})$. So $[F(\alpha):F]=n$. Let $h(x)=\sum_{i=1}^{n}c_{i}x^{i}$ be the minimal polynomial of $\alpha$ over $F$. Then $\sum_{i=1}^{n}c_{i}^{p}x^{i}$ annihilates $\alpha^{p}$ and is monic and of degree $n$, so it equals $f(x)$. Then all the coefficients of $f(x)$ are in $F^{p}$, #. The result follows.