Fact 1: If $n \in \mathbb{N}$, then $n \mid P(x)$ if and only if $n \mid P(x + n)$.
Fact 2: If $P$ is non-constant, then there are infinitely many primes which divide a term of the sequence $P(0), P(1), P(2), \ldots$.
For the second fact, the proof is the following:
Suppose for sake of contradiction that $p_1, \ldots, p_k$ are the only primes which divide a term of the sequence.
Let $N$ be large.
Then there are $< O((\log N)^k)$ numbers $< N$ which only have prime factors lying in $p_1, \ldots, p_k$.
But there are $\approx O(N^{\frac{1}{\deg P}})$ numbers in the sequence $P(0), P(1), \ldots$ which are $< N$.
The latter expression is much bigger than the former, provided $N$ is large enough.
This gives the desired contradiction.
Now we will prove that if $P$ is non-constant, then $P(0) = 0$.
We do this by showing $P(0)$ is divisible by infinitely many primes.
Let $p$ be a prime that divides a term of the sequence.
Let $x > 0$ be the smallest number such that $p \mid P(x)$.
By Fact 1, $x \le p$, but also $p \mid x!$ since $p \mid P(x)$ and $P(x) \mid x!$.
Hence $x = p$.
Then by Fact 1, $p \mid P(0)$.
Applying this for infinitely many primes gives $P(0) = 0$.
So assuming $P$ is non-constant, let $P(x) = xQ(x)$.
Note $Q$ satisfies the same property that $P$ did, so if $Q$ is non-constant then it is divisible by $x$.
But by your work so far, this is not possible.
Hence $Q$ would have to be constant.
This leaves us with the result that either $P$ is constant or $P(x) = cx$ for some constant $c$.
It is then easy to check that $P(x)$ must be one of $-1$, $1$, $x$ or $-x$.