$$P:=\forall x{\in} S \ \forall \color{cyan}y{\in} R\ (\color{cyan}y\preceq x\Rightarrow \color{cyan}y\in S)\tag1$$
$$Q:=\forall x{\in} S\ (\,\forall \color{green}y{\in} R\ (\color{green}y\preceq x)\Rightarrow \color\red y\in S\,)\tag2$$
$Q,$ but not $P,$ contains a free occurrence of $\color\red y$ (by convention, the scope of the green $\color{green}y$ is actually just ∀y∈R y⪯x). So, $P$ can't possibly be logically equivalent to $Q.$
$$\neg P = \exists x{\in} S \ \exists \color{cyan}y{\in} R\ (\color{cyan}y\preceq x\,\land\, \color{cyan}y\notin S)\tag3$$
$$\neg Q=\exists x{\in} S\ (\ \forall \color{green}y{\in} R\ (\color{green}y\preceq x)\,\land\, \color{red}y\notin S\ )\tag4$$
Yes, you've negated $P$ and $Q$ correctly.
Are $P$ and $Q$ logically equivalent?
Now, \begin{align}Q&\equiv\forall x{\in} S \,\big((\forall \color\green z{\in} R\;\color\green z\preceq x)\Rightarrow \color\red y\in S\big)\tag2\\&\equiv \forall x{\in} S \,\exists \color{green}z{\in} R\,\big(\color{green}z\preceq x\Rightarrow \color\red y\in S\big)\tag{2A}\end{align}
\begin{align}\lnot Q&\equiv\exists x{\in} S \,\big((\forall \color\green z{\in} R\;\color\green z\preceq x)\,\land\, \color\red y\not\in S\big)\tag4\\&\equiv\exists x{\in} S \,\forall \color{green}z{\in} R\,\big(\color{green}z\preceq x\,\land\, \color\red y\not\in S\big)\tag{4A}\end{align}
(the second and fourth logical equivalences (≡) are explained in https://en.wikipedia.org/wiki/Prenex_normal_form#Conversion_to_prenex_form ).
Comparing (1) and (2A), and separately comparing (3) and (4A), confirms that $$P\not\equiv Q\\ \lnot P\not\equiv \lnot Q.$$
Where is my mistake?
Falsely believing that $(\forall y{\in} R\;y\preceq x)\Rightarrow y\in S\tag*{}$ is logically equivalent to $\forall y{\in} R\,\big(y\preceq x\Rightarrow y\in S\big)\tag*{}$ misled you, at the start, to think that $P$ is equivalent to $Q.$
Suppose that $(R,\prec)$ is a totally ordered set, and let $S\subseteq R.$
The logical equivalence of $P$ and $Q$ is actually indepdendent of this mathematics.
Reply to the OP's comment
Could you please explain why you're using a different color for y, and why you replaced y with z?
Sure. The green $\color{green}y,$ the cyan $\color{cyan}y$ and $x$ are distinct dummy/bound variables, so they can be renamed (though for each colour, any renaming must apply to every instance of the variable) to clarify this (instead of recycling the variables); in contrast, the red $\color{red}y$ is a free variable, so cannot be renamed. Analogously, $$\int_0^{\color{red}y} t\, \mathrm dt=\int_0^{\color{red}y} u\, \mathrm du\ne\int_0^x u\, \mathrm du.$$