We want to find the intersection of the three quadrics:
$ r^T Q_1 r = 0 ,\hspace{5pt} r^T Q_2 r = 0 ,\hspace{5pt} r^T Q_3 r = 0 $
where $ r = [x, y, z, 1 ]^T $ and $Q_1, Q_2, Q_3$ are symmetric $4 \times 4 $ matrices.
Consider the matrix $Q_\alpha = Q_1 + \alpha Q_2 $
Solutions of $r^T Q_1 r = 0 , r^T Q_2 r = 0 $ are obviously solutions of $r^T Q_\alpha r = 0 $
Now by choosing $\alpha$ such that $| Q_a | = 0 $ we can simplify finding solution candidates. The determinant $|Q_\alpha|$ is a quartic polynomial in $\alpha$ and we can find its roots numerically or through a close-form formula. Let $\alpha_1$ be one of these roots, and let
$Q^* = Q_1 + \alpha_1 Q_2 $
Then $Q^*$ is rank deficient and being symmetric can be diagonalized into
$Q^* = R D R^T $
where
$ D = \begin{bmatrix} D_{11} && 0 && 0 && 0 \\ 0 && D_{22} && 0 && 0 \\ 0 && 0 && D_{33} && 0 \\ 0 && 0 && 0 && 0 \end{bmatrix} $
Now we want to solve
$ r^T R D R^T r = 0 $
Define $u = R^T r $, equivalently, $ r = R u $, then
$ u^T D u = 0 $
Depending on $D_{11} , D_{22}, D_{33} $ we have several cases to consider. I will discuss two important cases:
Case I: $D_{11} , D_{22} \gt 0 , D_{33} \lt 0 $
Case II: $D_{11} \gt 0 , D_{22} \lt 0 , D_{33} = 0 $
For Case I, the candidate solution for $u$ is
$ u = \begin{bmatrix} \dfrac{1}{\sqrt{D_{11}}} t \cos \theta \\ \dfrac{1}{\sqrt{D_{22}}} t \sin \theta \\
\dfrac{t}{\sqrt{-D_{33} }} \\ r \end{bmatrix} $
where $ t, \theta, r $ are real parameters.
For Case II, the candidate solution for $u$ is
$ u = \begin{bmatrix} \dfrac{t}{\sqrt{D_{11}}} \\
\dfrac{\pm t }{\sqrt{D_{22}}} \\ s \\ r \end{bmatrix} $
where $t, s, r $ are real parameters.
There are three conditions on the parameters that will determine their values
- The fourth entry of $r$ is $1$. This translates into
$$ [0, 0, 0, 1] R u = 1 $$
- $r^T Q_1 r = 0 $. This translates into
$$ u^T R^T Q_1 R u = 0 $$
- $r^T Q_3 r = 0 $. This translates into
$$ u^T R^T Q_3 R u = 0 $$
There is a lot of details in these three conditions.
Let's take Case I first. Applying the first condition to $u$ , we will get the equation
$ t ( a \cos \theta + b \sin \theta + c ) + d r = 1 \hspace{15pt} (I.1)$
where $a, b, c, d$ are constants that depend on the entries of the fourth row of matrix $R$ and the $D_{11} , D_{22}, D_{33} $ in a straight forward way.
The second of these conditions will result in the following equation
$ t^2 f_2(\theta) + tr f_1(\theta) + r^2 f_0 = 0 \hspace{15pt} (I.2)$
where
$f_2(\theta) = C_1 \cos \theta + C_2 \sin \theta + C_3 \cos 2 \theta + C_4 \sin 2 \theta + C_5 $
$f_1(\theta) = C_6 \cos \theta + C_7 \sin \theta + C_8 $
$f_0 = C_9 $
where $C_i , i = 1,.., 9 $ are constants that depend on the entries of the matrix $ R^T Q_1 R $, and the parameterization given above for $u$.
An equation with an identical structure to the above can be derived when applying the third condition, and we end up with
$ t^2 g_2(\theta) + t r g_1(\theta) + r^2 g_0 = 0 \hspace{15pt}(I.3)$
And we want to find $t, \theta, r$ that will satisfy $(I.1), (I.2)$ and $(I.3) $
Substituting from $(I.1)$ we can write $r $ in terms of $t$, and then substitute that into $(I.2) $ and $(I.3)$, and end up with the equations
$ t^2 F_2 (\theta) + t F_1(\theta) + F_0 = 0 \hspace{15pt}(I. 4) $
and
$ t^2 G_2 ( \theta) + t G_1 ( \theta ) + G_0 = 0 \hspace{15pt}(I.5)$
If $t$ satisfies both equations $(I.4)$ and $(I.5)$ then $\theta$ must satisfy the following condition,
$ (F_2 G_0 - F_0 G_2)^2 + (F_0 G_1 - G_0 F_1) (F_2 G_1 - F_1 G_2) = 0 \hspace{15pt}(I.6)$
To solve for $\theta$ using $(I.6)$, one can either transform it into a polynomial of degree $8$ in the variable $z = \tan \dfrac{\theta}{2} $ and find the real roots using one of the standard numerical methods for finding polynomial roots, or one could (and this what I did) scan the interval $[0, 2\pi) $ and use simple bracketing and the bisection method (for example) to find all the possible real roots. Once the roots $\theta$ are found, one can solve for the corresponding $t $ and $r$, thus determining the vector $u$ and consequently the vector $r$.
That's the summary of the method for Case I.
Case II is much simpler, as the first condition leads to an equation of a plane in the variables $t, s, r$, which can be parameterized using only two variables (because it is a two-dimensional space). Applying the second and third conditions leads to equations of conics in the two variables. So the problem reduces to finding the intersection of these two conics. The solution method can be found in the answer to this question