(I) Preliminaries
I will use the following notations (most of which are copied from this answer of mine to a related question).
Assumption
Let $a$, $b$, and $r$ be real numbers such that $r > 0$. (Intuitively $r$ represents the radius of a non-degenerate circle, but we will only ever use $r$ in the expression $r^2$, so we could have required instead that $r \neq 0$ without loss of generality.)
Definitions
$\Sigma$ is defined to be the unit sphere in the Euclidean space, i.e.
$$
\Sigma := \big\{(x,y,z)\in\mathbb{R}^3 : x^2+y^2+z^2 = 1\big\}.
$$
$N$ is defined to be the northern pole of the unit sphere, i.e. $N := (0,0,1)$.
$S$ is defined to be the stereographic projection, i.e. the function $S:\Sigma\setminus\{N\}\rightarrow\mathbb{R}^2$ satisfying
$$
S\big((x,y,z)\big) = \Big(\frac{x}{1-z},\frac{y}{1-z}\Big)
$$
for every $(x,y,z) \in \Sigma\setminus\{N\}$.
$R$ is defined to be the function $R:\mathbb{R}^2\rightarrow\mathbb{R}^3$ satisfying
$$
R\big((x,y)\big) = \Big(\frac{2x}{x^2+y^2+1}, \frac{2y}{x^2+y^2+1}, \frac{x^2+y^2-1}{x^2+y^2+1}\Big)
$$
for every $(x,y) \in \mathbb{R}^2$.
$\operatorname{Id}_\Sigma$ and $\operatorname{Id}_{\mathbb{R}^2}$ are defined to be the identity functions on $\Sigma$ and on $\mathbb{R}^2$, respectively. (We will only really need $\operatorname{Id}_\Sigma$.)
We define $k$ to be the circle $\big\{(x,y) \in \mathbb{R}^2 : (x-a)^2 + (y-b)^2 = r^2\big\}$.
Fact
As I did in the above mentioned answer, I take the following fact for granted, without proving it:
- $S$ and $R$ are inverses, i.e. (a) $R\circ S = \operatorname{Id}_\Sigma$, and (b) $\operatorname{Img} R \subseteq \Sigma\setminus\{N\}$ and $S\circ R = \operatorname{Id}_{\mathbb{R}^2}$. Note that (b) implies, in particular, that $R$ is injective. (The proof will only use the facts that $R\circ S = \operatorname{Id}_\Sigma$, that $\operatorname{Img} R \subseteq \Sigma$, and that $R$ is injective.)
(II) A formal statement of the problem
There is a plane $\Pi \subseteq \mathbb{R}^3$ not passing through $N$ and whose distance from the origin is strictly less than $1$, such that $R[k] = \Sigma\cap\Pi$.
(III) Proof
From Analytic Geometry, a plane $\Pi \subseteq \mathbb{R}^3$ is any set satisfying $\Pi = \big\{(x,y,z) \in \mathbb{R}^3 : Ax + By + Cz + D = 0\big\}$ for some $A, B, C, D \in \mathbb{R}$ such that $A^2 + B^2 + C^2 \neq 0$.
Define
$$
\begin{align*}
A &:= -a,\\
B &:= -b,
\end{align*}
$$
and define $C$ and $D$ to be the unique real numbers satisfying
$$
\begin{align*}
C + D &= 1\\
C - D &= r^2 - a^2 - b^2.
\end{align*}\tag{*}
$$
(Linear Algebra tells us that this system of equations has a unique solution $\begin{bmatrix}C\\D\end{bmatrix} \in \mathbb{R}_{2\times1}$, because the matrix of coefficients $\begin{bmatrix}1 & 1\\1 & -1\end{bmatrix}$ is invertible.)
Define $\Pi := \big\{(x,y,z) \in \mathbb{R}^3 : Ax + By + Cz + D = 0\big\}$.
Note:
$A^2 + B^2 + C^2 \neq 0$, since otherwise we would have $a = b = 0$, and $(*)$ would yield $r^2 = -1$, which is a contradiction, since, by assumption, $r \in \mathbb{R}$. Hence, by the first paragraph of this proof, $\Pi$ is a plane in $\mathbb{R}^3$.
$N \notin \Pi$, since otherwise the definition of $\Pi$ would yield $C + D = 0$, in contradiction to $(*)$.
I claim that in order to show that $\Pi$'s distance from the origin is strictly less than $1$, it suffices to show that $R[k] = \Sigma\cap\Pi$. Indeed, if $R[k] = \Sigma\cap\Pi$, then since $k$ is non-degenerate (since, by assumption, $r > 0$), the fact that $R$ is injective implies that the set $\Sigma\cap\Pi\ \big(=R[k]\big)$ includes at least two points, and therefore $\Pi$ passes through $\Sigma$'s interior, which implies, by $\Sigma$'s definition, that $\Pi$'s distance from the origin is strictly less than $1$.
It remains to show that $R[k] = \Sigma\cap\Pi$. We will do so by showing that $R[k] \subseteq \Sigma\cap\Pi$ and that $\Sigma\cap\Pi \subseteq R[k]$.
Firstly we show that $R[k] \subseteq \Sigma\cap\Pi$. By the fact that $\operatorname{Img} R \subseteq \Sigma$ it suffices to show that $R[k] \subseteq \Pi$. Let $(x,y) \in k$. Then, by the definitions of $R$ and of $\Pi$ it suffices to show that
$$
A\frac{2x}{x^2+y^2+1} + B\frac{2y}{x^2+y^2+1} + C\frac{x^2+y^2-1}{x^2+y^2+1} + D = 0,
$$
which, by algebraic manipulations, can be shown to be equivalent to
$$
2Ax + 2By + (C+D)(x^2+y^2) = C-D.
$$
Using the definitions of $A$, $B$, $C$, and $D$ the last equation reduces to
$$
-2ax -2by + x^2 + y^2 = r^2-a^2-b^2,
$$
which, by rearranging terms, can be shown to be equivalent to $(x-a)^2 + (y-b)^2 = r^2$. Since, by selection, $(x,y) \in k$, the last equation holds by the definition of $k$.
Secondly we show that $\Sigma\cap\Pi \subseteq R[k]$. Let $p \in \Sigma\cap\Pi$. Then, in particular, $p \in \mathbb{R}^3$, and we can denote its coordinates by $x$, $y$, and $z$, respectively, so that $p = (x,y,z)$. Note that since, by selection, $p \in \Sigma\cap\Pi$, and since, as we saw, $N \notin \Pi$, we have $p \in \Sigma\setminus\{N\}$. It follows from the definition of $S$ that $p \in \operatorname{dom} S$, and we can therefore define $c := S(p)$. By the fact that $R\circ S = \operatorname{Id}_\Sigma$, it follows that $p = R\big(S(p)\big) = R(c)$. Therefore, it suffices to show that $c \in k$, i.e., by the definition of $c$, that $S\big((x,y,z)\big) \in k$, i.e., by the definitions of $S$ and of $k$, that
$$
\Big(\frac{x}{1-z}-a\Big)^2 + \Big(\frac{y}{1-z}-b\Big)^2 = r^2,
$$
which, by algebraic manipulations can be shown to be equivalent to
$$
\big(x-a(1-z)\big)^2 + \big(y-b(1-z)\big)^2 = r^2(1-z)^2.\tag{**}
$$
(Since, as we saw above, $p \in \Sigma\setminus\{N\}$, it follows that $z \neq 1$.)
By algebraic manipulations and by using the definitions of $A$, $B$, $C$, and $D$, $(**)$ can be shown to be equivalent to
$$
x^2 + y^2 + 2(Ax+By)(1-z) - (C-D)(1-z)^2 = 0.
$$
By multiplying out and rearranging terms, this can be shown to be equivalent to
$$
\begin{multline*}
2(Ax+By+Cz+D) - 2(Ax+By+Cz+D)z\ +\\x^2 + y^2 + (C + D)z^2 - (C + D) = 0.
\end{multline*}\tag{***}
$$
Since, by $p$'s selection, $p \in \Pi$, it follows from the definition of $\Pi$ that $(*\!*\!*)$ is equivalent to $x^2 + y^2 + (C + D)z^2 - (C + D) = 0$, which, by $(*)$, is equivalent to $x^2 + y^2 + z^2 - 1 = 0$. The last equation is valid by the definition of $\Sigma$ and the fact that, by $p$'s selection, $(x,y,z) \in \Sigma$. Q.E.D.
Related posts
In this answer I proved that lines in $\mathbb{R}^2$ are mapped by the inverse of the stereographic projection to non-degenerate circles on the unit sphere that pass through the northern pole.
In this answer I clarified a step in a proof from this paper. That paper shows that non-degenerate circles on the unit sphere are mapped by the stereographic projection to either lines or to non-degenerate circles in $\mathbb{R}^2$, depending on whether the original circle passes or no through the northern pole, respectively.
