Here $A^{\dagger}$ is the Moore-Penrose pseudo-inverse and $\geq$ denotes the Loewner partial order for positive semidefinite matrices.
I know that the statement is true for non-singular matrices, but cannot find whether this extends to singular matrices, and continuity arguments seem to not apply here.
Any help will be greatly appreciated.
It isn't true. The example could be the following: $$ A=\begin{pmatrix}1&0\\0&1\end{pmatrix},\quad B=\begin{pmatrix}1&0\\0&0\end{pmatrix}. $$ Their pseudoinverse matrices are $$ A^+=A,\quad B^+=B. $$ Which implies $$ A\ge B,\quad B^+\not\ge A^+. $$
It is true if and only if the matrices share the same nullspace.
Assume $null(A)=null(B)$. Let $P\in O(n)$ be the matrix of eigenvectors of $A$ so that the first eigenvectors correspond to the nullspace. Then $$A = P \begin{pmatrix} 0 & 0 \\ 0 & D\end{pmatrix} P^T, \qquad B = P \begin{pmatrix} 0 & 0 \\ 0 & \bar B\end{pmatrix} P^T $$ where $D$ is diagonal with nonzero elements and $\bar B$ is invertible, and they have the same size. Then $$A^\dagger = P \begin{pmatrix} 0 & 0 \\ 0 & D^{-1}\end{pmatrix} P^T, \qquad B^\dagger = P \begin{pmatrix} 0 & 0 \\ 0 & \bar B^{-1}\end{pmatrix} P^T. $$
Then, since $D$ and $\bar B$ are invertible, using the arguments of https://math.stackexchange.com/a/3018669/484640, \begin{align} A\ge B & \text{ iff } D \ge \bar B \\ & \text{ iff } I\ge D^{1/2} \bar B D^{1/2} \\ & \text{ iff } I\le D^{-1/2} \bar B^{-1} D^{-1/2} \\ & \text{ iff } D^{-1} \le \bar B^{-1} \\ & \text{ iff } A^\dagger \le B^\dagger. \end{align}
Now assume $A \ge B$ but that the nullspaces are different, say there exists a vector $x$ with $Ax \ne 0$ but $Bx = 0$. Then $x^T B^\dagger x =0$ by definition of the pseudo inverse of $B$, but $x^T A^\dagger x \ne 0$ by definition of the pseudo inverse of $A$. This implies $x^T A^\dagger x > x^T B^\dagger x = 0$ so $A^\dagger \le B^\dagger$ cannot hold.
