Consider $n=5$ and $\{x_1,\cdots,x_5\}=\{1,2,4,6,7\}$, and let $P_i=\frac{1}{(n-1)!}\prod_{j\neq i}(x_j-x_i)$. Then
\begin{align*}
P_1 &= \frac{1\times3\times5\times6}{4!} = \frac{15}{4}, \\
P_2 &= \frac{(-1)\times2\times4\times5}{4!} = -\frac{5}{3}, \\
P_3 &= \frac{(-3)\times(-2)\times2\times3}{4!} = \frac{3}{2}, \\
P_4 &= \frac{(-5)\times(-4)\times(-2)\times1}{4!} = -\frac{5}{3}, \\
P_5 &= \frac{(-6)\times(-5)\times(-3)\times(-1)}{4!} = \frac{15}{4}
\end{align*}
The claim is true when $n \leq 4$. Since it is almost clear when $n \leq 3$, we prove the claim for $n=4$.
Let $x_1,\cdots,x_4$ be arbitrary integers. By the pigeonhole principle, there exist $a, b \in \{1,2,3,4\}$ such $a \neq b$ and $x_a \equiv x_b \text{ (mod 3)}$. Now
If $x_a \equiv x_b \text{ (mod 2)}$, then $x_b - x_a$ is a multiple of $6$.
If $x_a \not\equiv x_b \text{ (mod 2)} $, then for any $j \in \{1,2,3,4\}\setminus\{a,b\}$ either $x_j - x_a$ or $x_j - x_b$ is even.
In both cases, either $\prod_{j\neq a} (x_j-x_a)$ or $\prod_{j\neq b} (x_j-x_b)$ is a multiple of $6$.