General Equations for Common Tangents to Two Ellipses

Question

This is a follow-up from one of my earlier questions here. I am attempting to find the general equation for common tangents to two ellipses:

$$ \left\{ \begin{array}{c} \frac{(x-h)^2}{a^2}+\frac{(y-k)^2}{b^2}-1=0 \\ \frac{(x-p)^2}{c^2}+\frac{(y-q)^2}{d^2}-1=0 \\ \end{array} \right. \tag{1} $$

However, the solve function of Maxima CAS does not give me an answer (it just runs and hangs). Am I doing something wrong, or is there no answer?

Method:

To solve, first we find the dual curves in line coordinates for the equations in $(1)$, which requires converting each to standard form and homogenizing

$$ \left\{ \begin{array}{c} f_1(x,y,z)=b^2x^2+a^2y^2-2(hb^2x+ka^2y)z+(h^2b^2+k^2a^2-a^2b^2)z^2=0 \\ f_2(x,y,z)=d^2x^2+c^2y^2-2(pd^2x+qc^2y)z+(p^2d^2+q^2c^2-c^2d^2)z^2=0 \\ \end{array} \right. \tag{2} $$

and then recognizing that since

1. the equation of a line in (homogeneous) line coordinates is $g(x,y,z)=Xx+Yy+Zz=0$, and
2. the tangent to a curve $f$ at a point $(x_0,y_0,z_0)$ is given by $f'(x_0,y_0,z_0)=f_x(x_0,y_0,z_0)+f_y(x_0,y_0,z_0)+f_z(x_0,y_0,z_0)$

finding dual curves amounts to solving an optimization problem subject to a constraint via the method of Lagrangian multipliers

$$ \nabla f=\lambda \nabla g \\ g = 0. $$

At point $(x_0,y_0,z_0)$, this becomes

$$ f_x(x_0,y_0,z_0) = \lambda X \\ f_y(x_0,y_0,z_0) = \lambda Y \\ f_z(x_0,y_0,z_0) = \lambda Z \\ Xx_0+Yy_0+Zz_0=0. $$

(NOTE: In truth, $\lambda$ can go on either side without loss of generality since it is a proportionality constant we wish to eliminate.)

Solving the above yields our dual curves $F_1$ and $F_2$

$$ \left\{ \begin{array}{c} F_1(X,Y) = (h^2-a^2)X^2 + (2hk)XY + (k^2-b^2)Y^2 + 2hX + 2kY + 1 = 0 \\ F_2(X,Y) = (p^2-c^2)X^2 + (2pq)XY + (q^2-d^2)Y^2 + 2pX + 2qY + 1 = 0 \\ \end{array} \right. \tag{3} $$

Second, we must find the line of incidence (i.e. the "point of intersection") of the system (3), which amounts to solving the system for $X$ and $Y$. Since this is a multivariate polynomial system, we can compute the reduced Gröbner bases of the l-elimination ideals of the ideal $I=<F_1,F_2>$ generated by $F_1$ and $F_2$ to solve.

Let $K=\mathbb Q [a,b,c,d,h,k,p,q]$, which is algebraically closed, and $R = K[x,y]$ be a polynomial ring under pure lex monomial ordering. Since $R$ is Noetherian by the Hilbert Basis Theorem (i.e. $R$ is finitely generated), the varieties $V(I)$ are affine (i.e. finite and equal to the varieties of the generators of $I$).

Using Macaulay2, the generators of the Gröbner basis of $I$ are found via

i1 : R = QQ[a,b,c,d,h,k,p,q];
i2 : S = R[x, y, MonomialOrder=>Lex];
i3 : I = ideal((h^2-a^2)x^2 + (2hk)xy + (k^2-b^2)y^2 + 2hx + 2ky + 1, (p^2-c^2)x^2 + (2pq)xy + (q^2-d^2)y^2 + 2px + 2q*y + 1);
i4 : G = gb I;
i5 : toString gens G

which yields a large $1 X 10$ matrix. The first elimination ideal, $G_1$, is the first entry of $G$, which is a quartic in $y$ alone. Since there are two variables in our problem, there is no second elimination ideal (it is the empty set).

Since the leading coefficients of $x$ in the other 9 entries in $G-G_1$ are constants (they do not involve $y$), the Elimination Theorem guarantees we should be able to extend our solution for $y$.

Problem:

Attempting to solve $G_1$ with any of the other entries of $G-G_1$ using solve in Maxima CAS does not give a solution. Here is my code (I call the first entry of $G-G_1$ the variable $H$):

(%i1) G1 : (b^4*c^4-2*a^2*b^2*c^2*d^2+a^4*d^4+2*b^2*c^2*d^2*h^2-2*a^2*d^4*h^2+d^4*h^4-2*b^2*c^4*k^2+2*a^2*c^2*d^2*k^2+2*c^2*d^2*h^2*k^2+c^4*k^4-2*b^4*c^2*p^2+2*a^2*b^2*d^2*p^2-2*b^2*d^2*h^2*p^2+4*b^2*c^2*k^2*p^2-2*a^2*d^2*k^2*p^2-2*d^2*h^2*k^2*p^2-2*c^2*k^4*p^2+b^4*p^4-2*b^2*k^2*p^4+k^4*p^4-4*b^2*c^2*h*k*p*q-4*a^2*d^2*h*k*p*q+4*d^2*h^3*k*p*q+4*c^2*h*k^3*p*q+4*b^2*h*k*p^3*q-4*h*k^3*p^3*q+2*a^2*b^2*c^2*q^2-2*a^4*d^2*q^2-2*b^2*c^2*h^2*q^2+4*a^2*d^2*h^2*q^2-2*d^2*h^4*q^2-2*a^2*c^2*k^2*q^2-2*c^2*h^2*k^2*q^2+2*a^2*b^2*p^2*q^2-2*b^2*h^2*p^2*q^2-2*a^2*k^2*p^2*q^2+6*h^2*k^2*p^2*q^2+4*a^2*h*k*p*q^3-4*h^3*k*p*q^3+a^4*q^4-2*a^2*h^2*q^4+h^4*q^4)*y^4+(-4*b^2*c^4*k+4*a^2*c^2*d^2*k+4*c^2*d^2*h^2*k+4*c^4*k^3-4*b^2*c^2*h*k*p-4*a^2*d^2*h*k*p+4*d^2*h^3*k*p+4*c^2*h*k^3*p+8*b^2*c^2*k*p^2-4*a^2*d^2*k*p^2-4*d^2*h^2*k*p^2-8*c^2*k^3*p^2+4*b^2*h*k*p^3-4*h*k^3*p^3-4*b^2*k*p^4+4*k^3*p^4+4*a^2*b^2*c^2*q-4*a^4*d^2*q-4*b^2*c^2*h^2*q+8*a^2*d^2*h^2*q-4*d^2*h^4*q-4*a^2*c^2*k^2*q-4*c^2*h^2*k^2*q-4*b^2*c^2*h*p*q-4*a^2*d^2*h*p*q+4*d^2*h^3*p*q+12*c^2*h*k^2*p*q+4*a^2*b^2*p^2*q-4*b^2*h^2*p^2*q-4*a^2*k^2*p^2*q+12*h^2*k^2*p^2*q+4*b^2*h*p^3*q-12*h*k^2*p^3*q-4*a^2*c^2*k*q^2-4*c^2*h^2*k*q^2+12*a^2*h*k*p*q^2-12*h^3*k*p*q^2-4*a^2*k*p^2*q^2+12*h^2*k*p^2*q^2+4*a^4*q^3-8*a^2*h^2*q^3+4*h^4*q^3+4*a^2*h*p*q^3-4*h^3*p*q^3)*y^3+(2*a^2*b^2*c^2-2*b^2*c^4-2*a^4*d^2+2*a^2*c^2*d^2-2*b^2*c^2*h^2+4*a^2*d^2*h^2+2*c^2*d^2*h^2-2*d^2*h^4-2*a^2*c^2*k^2+6*c^4*k^2-2*c^2*h^2*k^2-4*b^2*c^2*h*p-4*a^2*d^2*h*p+4*d^2*h^3*p+12*c^2*h*k^2*p+2*a^2*b^2*p^2+4*b^2*c^2*p^2-2*a^2*d^2*p^2-2*b^2*h^2*p^2-2*d^2*h^2*p^2-2*a^2*k^2*p^2-12*c^2*k^2*p^2+6*h^2*k^2*p^2+4*b^2*h*p^3-12*h*k^2*p^3-2*b^2*p^4+6*k^2*p^4-8*a^2*c^2*k*q-8*c^2*h^2*k*q+12*a^2*h*k*p*q+12*c^2*h*k*p*q-12*h^3*k*p*q-8*a^2*k*p^2*q+24*h^2*k*p^2*q-12*h*k*p^3*q+6*a^4*q^2-2*a^2*c^2*q^2-12*a^2*h^2*q^2-2*c^2*h^2*q^2+6*h^4*q^2+12*a^2*h*p*q^2-12*h^3*p*q^2-2*a^2*p^2*q^2+6*h^2*p^2*q^2)*y^2+(-4*a^2*c^2*k+4*c^4*k-4*c^2*h^2*k+4*a^2*h*k*p+12*c^2*h*k*p-4*h^3*k*p-4*a^2*k*p^2-8*c^2*k*p^2+12*h^2*k*p^2-12*h*k*p^3+4*k*p^4+4*a^4*q-4*a^2*c^2*q-8*a^2*h^2*q-4*c^2*h^2*q+4*h^4*q+12*a^2*h*p*q+4*c^2*h*p*q-12*h^3*p*q-4*a^2*p^2*q+12*h^2*p^2*q-4*h*p^3*q)*y+a^4-2*a^2*c^2+c^4-2*a^2*h^2-2*c^2*h^2+h^4+4*a^2*h*p+4*c^2*h*p-4*h^3*p-2*a^2*p^2-2*c^2*p^2+6*h^2*p^2-4*h*p^3+p^4$
(%i2) H : (2b^2c^6h^2-2a^2c^4d^2h^2+2c^4d^2h^4+2a^2c^4h^2k^2-2c^4h^4k^2-4a^2b^2c^4hp+4a^4c^2d^2hp+4b^2c^4h^3p-8a^2c^2d^2h^3p+4c^2d^2h^5p+2a^4b^2c^2p^2-2a^6d^2p^2-4a^2b^2c^2h^2p^2-6b^2c^4h^2p^2+6a^4d^2h^2p^2+4a^2c^2d^2h^2p^2+2b^2c^2h^4p^2-6a^2d^2h^4p^2-4c^2d^2h^4p^2+2d^2h^6p^2-2a^4c^2k^2p^2+2c^2h^4k^2p^2+8a^2b^2c^2hp^3-4a^4d^2hp^3-8b^2c^2h^3p^3+8a^2d^2h^3p^3-4d^2h^5p^3-2a^4b^2p^4+4a^2b^2h^2p^4+6b^2c^2h^2p^4-2a^2d^2h^2p^4-2b^2h^4p^4+2d^2h^4p^4+2a^4k^2p^4-2a^2h^2k^2p^4-4a^2b^2hp^5+4b^2h^3p^5-2b^2h^2p^6-4a^2c^4h^2kq+4c^4h^4kq+4a^4c^2kp^2q-4c^2h^4kp^2q-4a^4kp^4q+4a^2h^2kp^4q+2a^2c^4h^2q^2-2c^4h^4q^2-2a^4c^2p^2q^2+2c^2h^4p^2q^2+2a^4p^4q^2-2a^2h^2p^4q^2)x+(b^4c^6hk-2a^2b^2c^4d^2hk+a^4c^2d^4hk+2b^2c^4d^2h^3k-2a^2c^2d^4h^3k+c^2d^4h^5k-2b^2c^6hk^3+2a^2c^4d^2hk^3+2c^4d^2h^3k^3+c^6hk^5-3b^4c^4hkp^2+4a^2b^2c^2d^2hkp^2-a^4d^4hkp^2-4b^2c^2d^2h^3kp^2+2a^2d^4h^3kp^2-d^4h^5kp^2+6b^2c^4hk^3p^2-4a^2c^2d^2hk^3p^2-4c^2d^2h^3k^3p^2-3c^4hk^5p^2+3b^4c^2hkp^4-2a^2b^2d^2hkp^4+2b^2d^2h^3kp^4-6b^2c^2hk^3p^4+2a^2d^2hk^3p^4+2d^2h^3k^3p^4+3c^2hk^5p^4-b^4hkp^6+2b^2hk^3p^6-hk^5p^6-a^2b^4c^4pq+2a^4b^2c^2d^2pq-a^6d^4pq+b^4c^4h^2pq-4a^2b^2c^2d^2h^2pq+3a^4d^4h^2pq+2b^2c^2d^2h^4pq-3a^2d^4h^4pq+d^4h^6pq+2a^2b^2c^4k^2pq-2a^4c^2d^2k^2pq-6b^2c^4h^2k^2pq-4a^2c^2d^2h^2k^2pq+6c^2d^2h^4k^2pq-a^2c^4k^4pq+5c^4h^2k^4pq+2a^2b^4c^2p^3q-2a^4b^2d^2p^3q-2b^4c^2h^2p^3q+4a^2b^2d^2h^2p^3q-2b^2d^2h^4p^3q-4a^2b^2c^2k^2p^3q+2a^4d^2k^2p^3q+12b^2c^2h^2k^2p^3q+4a^2d^2h^2k^2p^3q-6d^2h^4k^2p^3q+2a^2c^2k^4p^3q-10c^2h^2k^4p^3q-a^2b^4p^5q+b^4h^2p^5q+2a^2b^2k^2p^5q-6b^2h^2k^2p^5q-a^2k^4p^5q+5h^2k^4p^5q+2a^2b^2c^4hkq^2-2a^4c^2d^2hkq^2-2b^2c^4h^3kq^2+4a^2c^2d^2h^3kq^2-2c^2d^2h^5kq^2-2a^2c^4hk^3q^2-2c^4h^3k^3q^2+4a^2b^2c^2hkp^2q^2+6a^4d^2hkp^2q^2-4b^2c^2h^3kp^2q^2-12a^2d^2h^3kp^2q^2+6d^2h^5kp^2q^2-4a^2c^2hk^3p^2q^2+12c^2h^3k^3p^2q^2-6a^2b^2hkp^4q^2+6b^2h^3kp^4q^2+6a^2hk^3p^4q^2-10h^3k^3p^4q^2-2a^4b^2c^2pq^3+2a^6d^2pq^3+4a^2b^2c^2h^2pq^3-6a^4d^2h^2pq^3-2b^2c^2h^4pq^3+6a^2d^2h^4pq^3-2d^2h^6pq^3+2a^4c^2k^2pq^3+4a^2c^2h^2k^2pq^3-6c^2h^4k^2pq^3-2a^4b^2p^3q^3+4a^2b^2h^2p^3q^3-2b^2h^4p^3q^3+2a^4k^2p^3q^3-12a^2h^2k^2p^3q^3+10h^4k^2p^3q^3+a^4c^2hkq^4-2a^2c^2h^3kq^4+c^2h^5kq^4-5a^4hkp^2q^4+10a^2h^3kp^2q^4-5h^5kp^2q^4-a^6pq^5+3a^4h^2pq^5-3a^2h^4pq^5+h^6pq^5)y^3+(-b^4c^6h+2a^2b^2c^4d^2h-a^4c^2d^4h-2b^2c^4d^2h^3+2a^2c^2d^4h^3-c^2d^4h^5-2b^2c^6hk^2+2a^2c^4d^2hk^2+2c^4d^2h^3k^2+3c^6hk^4+a^2b^4c^4p-2a^4b^2c^2d^2p+a^6d^4p-b^4c^4h^2p+4a^2b^2c^2d^2h^2p-3a^4d^4h^2p-2b^2c^2d^2h^4p+3a^2d^4h^4p-d^4h^6p-2a^2b^2c^4k^2p+2a^4c^2d^2k^2p-2b^2c^4h^2k^2p-4a^2c^2d^2h^2k^2p+2c^2d^2h^4k^2p+a^2c^4k^4p+3c^4h^2k^4p+3b^4c^4hp^2-4a^2b^2c^2d^2hp^2+a^4d^4hp^2+4b^2c^2d^2h^3p^2-2a^2d^4h^3p^2+d^4h^5p^2+6b^2c^4hk^2p^2-4a^2c^2d^2hk^2p^2-4c^2d^2h^3k^2p^2-9c^4hk^4p^2-2a^2b^4c^2p^3+2a^4b^2d^2p^3+2b^4c^2h^2p^3-4a^2b^2d^2h^2p^3+2b^2d^2h^4p^3+4a^2b^2c^2k^2p^3-2a^4d^2k^2p^3+4b^2c^2h^2k^2p^3+4a^2d^2h^2k^2p^3-2d^2h^4k^2p^3-2a^2c^2k^4p^3-6c^2h^2k^4p^3-3b^4c^2hp^4+2a^2b^2d^2hp^4-2b^2d^2h^3p^4-6b^2c^2hk^2p^4+2a^2d^2hk^2p^4+2d^2h^3k^2p^4+9c^2hk^4p^4+a^2b^4p^5-b^4h^2p^5-2a^2b^2k^2p^5-2b^2h^2k^2p^5+a^2k^4p^5+3h^2k^4p^5+b^4hp^6+2b^2hk^2p^6-3hk^4p^6+4a^2b^2c^4hkq-4a^4c^2d^2hkq-4b^2c^4h^3kq+8a^2c^2d^2h^3kq-4c^2d^2h^5kq-4a^2c^4hk^3q-4c^4h^3k^3q+4a^2b^2c^4kpq-4a^4c^2d^2kpq-4b^2c^4h^2kpq+4c^2d^2h^4kpq-4a^2c^4k^3pq+12c^4h^2k^3pq+4a^4d^2hkp^2q-8a^2d^2h^3kp^2q+4d^2h^5kp^2q+16c^2h^3k^3p^2q-8a^2b^2c^2kp^3q+4a^4d^2kp^3q+8b^2c^2h^2kp^3q-4d^2h^4kp^3q+8a^2c^2k^3p^3q-24c^2h^2k^3p^3q-4a^2b^2hkp^4q+4b^2h^3kp^4q+4a^2hk^3p^4q-12h^3k^3p^4q+4a^2b^2kp^5q-4b^2h^2kp^5q-4a^2k^3p^5q+12h^2k^3p^5q-2a^2b^2c^4hq^2+2a^4c^2d^2hq^2+2b^2c^4h^3q^2-4a^2c^2d^2h^3q^2+2c^2d^2h^5q^2-2a^2c^4hk^2q^2-2c^4h^3k^2q^2-2a^4b^2c^2pq^2+2a^6d^2pq^2+4a^2b^2c^2h^2pq^2-6a^4d^2h^2pq^2-2b^2c^2h^4pq^2+6a^2d^2h^4pq^2-2d^2h^6pq^2+2a^4c^2k^2pq^2+12a^2c^2h^2k^2pq^2-14c^2h^4k^2pq^2+4a^2b^2c^2hp^2q^2+2a^4d^2hp^2q^2-4b^2c^2h^3p^2q^2-4a^2d^2h^3p^2q^2+2d^2h^5p^2q^2-12a^2c^2hk^2p^2q^2+20c^2h^3k^2p^2q^2-2a^4b^2p^3q^2+4a^2b^2h^2p^3q^2-2b^2h^4p^3q^2+2a^4k^2p^3q^2-20a^2h^2k^2p^3q^2+18h^4k^2p^3q^2-2a^2b^2hp^4q^2+2b^2h^3p^4q^2+14a^2hk^2p^4q^2-18h^3k^2p^4q^2+4a^4c^2hkq^3-8a^2c^2h^3kq^3+4c^2h^5kq^3+4a^4c^2kpq^3-4c^2h^4kpq^3-12a^4hkp^2q^3+24a^2h^3kp^2q^3-12h^5kp^2q^3+4a^4kp^3q^3-16a^2h^2kp^3q^3+12h^4kp^3q^3-a^4c^2hq^4+2a^2c^2h^3q^4-c^2h^5q^4-3a^6pq^4+9a^4h^2pq^4-9a^2h^4pq^4+3h^6pq^4-3a^4hp^2q^4+6a^2h^3p^2q^4-3h^5p^2q^4)y^2+(a^2b^2c^4hk+b^2c^6hk-a^4c^2d^2hk-a^2c^4d^2hk-b^2c^4h^3k+2a^2c^2d^2h^3k+c^4d^2h^3k-c^2d^2h^5k-a^2c^4hk^3+3c^6hk^3-3c^4h^3k^3-2a^2b^2c^4kp+2a^4c^2d^2kp+2b^2c^4h^2kp-4a^2c^2d^2h^2kp+2c^2d^2h^4kp+2a^2c^4k^3p+6c^4h^2k^3p-2a^2b^2c^2hkp^2-3b^2c^4hkp^2+a^4d^2hkp^2+2a^2c^2d^2hkp^2+2b^2c^2h^3kp^2-2a^2d^2h^3kp^2-2c^2d^2h^3kp^2+d^2h^5kp^2+2a^2c^2hk^3p^2-9c^4hk^3p^2+6c^2h^3k^3p^2+4a^2b^2c^2kp^3-2a^4d^2kp^3-4b^2c^2h^2kp^3+4a^2d^2h^2kp^3-2d^2h^4kp^3-4a^2c^2k^3p^3-12c^2h^2k^3p^3+a^2b^2hkp^4+3b^2c^2hkp^4-a^2d^2hkp^4-b^2h^3kp^4+d^2h^3kp^4-a^2hk^3p^4+9c^2hk^3p^4-3h^3k^3p^4-2a^2b^2kp^5+2b^2h^2kp^5+2a^2k^3p^5+6h^2k^3p^5-b^2hkp^6-3hk^3p^6-2a^2b^2c^4hq+2a^4c^2d^2hq+2b^2c^4h^3q-4a^2c^2d^2h^3q+2c^2d^2h^5q-6a^2c^4hk^2q-2c^4h^3k^2q+a^4b^2c^2pq+a^2b^2c^4pq-a^6d^2pq-a^4c^2d^2pq-2a^2b^2c^2h^2pq-b^2c^4h^2pq+3a^4d^2h^2pq+2a^2c^2d^2h^2pq+b^2c^2h^4pq-3a^2d^2h^4pq-c^2d^2h^4pq+d^2h^6pq-a^4c^2k^2pq-5a^2c^4k^2pq+10a^2c^2h^2k^2pq+9c^4h^2k^2pq-9c^2h^4k^2pq+4a^2b^2c^2hp^2q-2a^4d^2hp^2q-4b^2c^2h^3p^2q+4a^2d^2h^3p^2q-2d^2h^5p^2q-4a^2c^2hk^2p^2q+20c^2h^3k^2p^2q-a^4b^2p^3q-2a^2b^2c^2p^3q+a^4d^2p^3q+2a^2b^2h^2p^3q+2b^2c^2h^2p^3q-2a^2d^2h^2p^3q-b^2h^4p^3q+d^2h^4p^3q+a^4k^2p^3q+10a^2c^2k^2p^3q-10a^2h^2k^2p^3q-18c^2h^2k^2p^3q+9h^4k^2p^3q-2a^2b^2hp^4q+2b^2h^3p^4q+10a^2hk^2p^4q-18h^3k^2p^4q+a^2b^2p^5q-b^2h^2p^5q-5a^2k^2p^5q+9h^2k^2p^5q+5a^4c^2hkq^2+a^2c^4hkq^2-10a^2c^2h^3kq^2-c^4h^3kq^2+5c^2h^5kq^2+6a^4c^2kpq^2+4a^2c^2h^2kpq^2-10c^2h^4kpq^2-9a^4hkp^2q^2-10a^2c^2hkp^2q^2+18a^2h^3kp^2q^2+10c^2h^3kp^2q^2-9h^5kp^2q^2+2a^4kp^3q^2-20a^2h^2kp^3q^2+18h^4kp^3q^2+9a^2hkp^4q^2-9h^3kp^4q^2-2a^4c^2hq^3+4a^2c^2h^3q^3-2c^2h^5q^3-3a^6pq^3+a^4c^2pq^3+9a^4h^2pq^3-2a^2c^2h^2pq^3-9a^2h^4pq^3+c^2h^4pq^3+3h^6pq^3-6a^4hp^2q^3+12a^2h^3p^2q^3-6h^5p^2q^3+3a^4p^3q^3-6a^2h^2p^3q^3+3h^4p^3q^3)y-a^2b^2c^4h+b^2c^6h+a^4c^2d^2h-a^2c^4d^2h+b^2c^4h^3-2a^2c^2d^2h^3+c^4d^2h^3+c^2d^2h^5-a^2c^4hk^2+c^6hk^2-3c^4h^3k^2+a^4b^2c^2p-a^2b^2c^4p-a^6d^2p+a^4c^2d^2p-2a^2b^2c^2h^2p+b^2c^4h^2p+3a^4d^2h^2p-2a^2c^2d^2h^2p+b^2c^2h^4p-3a^2d^2h^4p+c^2d^2h^4p+d^2h^6p-a^4c^2k^2p+a^2c^4k^2p+2a^2c^2h^2k^2p+3c^4h^2k^2p-c^2h^4k^2p+2a^2b^2c^2hp^2-3b^2c^4hp^2-a^4d^2hp^2+2a^2c^2d^2hp^2-2b^2c^2h^3p^2+2a^2d^2h^3p^2-2c^2d^2h^3p^2-d^2h^5p^2+2a^2c^2hk^2p^2-3c^4hk^2p^2+6c^2h^3k^2p^2-a^4b^2p^3+2a^2b^2c^2p^3-a^4d^2p^3+2a^2b^2h^2p^3-2b^2c^2h^2p^3+2a^2d^2h^2p^3-b^2h^4p^3-d^2h^4p^3+a^4k^2p^3-2a^2c^2k^2p^3-2a^2h^2k^2p^3-6c^2h^2k^2p^3+h^4k^2p^3-a^2b^2hp^4+3b^2c^2hp^4-a^2d^2hp^4+b^2h^3p^4+d^2h^3p^4-a^2hk^2p^4+3c^2hk^2p^4-3h^3k^2p^4-a^2b^2p^5+b^2h^2p^5+a^2k^2p^5+3h^2k^2p^5-b^2hp^6-hk^2p^6+2a^4c^2hkq-2a^2c^4hkq-4a^2c^2h^3kq+2c^4h^3kq+2c^2h^5kq+2a^4c^2kpq-2a^2c^4kpq+4a^2c^2h^2kpq+2c^4h^2kpq-6c^2h^4kpq-2a^4hkp^2q-4a^2c^2hkp^2q+4a^2h^3kp^2q+4c^2h^3kp^2q-2h^5kp^2q-2a^4kp^3q+4a^2c^2kp^3q-4a^2h^2kp^3q-4c^2h^2kp^3q+6h^4kp^3q+6a^2hkp^4q-6h^3kp^4q-2a^2kp^5q+2h^2kp^5q-a^4c^2hq^2+a^2c^4hq^2+2a^2c^2h^3q^2-c^4h^3q^2-c^2h^5q^2-a^6pq^2+a^4c^2pq^2+3a^4h^2pq^2-2a^2c^2h^2pq^2-3a^2h^4pq^2+c^2h^4pq^2+h^6pq^2-3a^4hp^2q^2-2a^2c^2hp^2q^2+6a^2h^3p^2q^2+2c^2h^3p^2q^2-3h^5p^2q^2+3a^4p^3q^2-6a^2h^2p^3q^2+3h^4p^3q^2+a^2hp^4q^2-h^3p^4q^2$
(%i3) solve([G1,H],[x,y]);

Answer 1

I do not know why Maxima CAS won't solve, but this SO answer provides a simpler method for finding common tangents.

From Perspectives on Projective Geometry by Jürgen Richter-Gebert (credit to @Jan-MagnusØkland for this insight):

A conic consists of all points p that satisfy an equation $p^TAp = 0.$ The set of all tangents to this conic can be described as $\{Ap | p^TAp = 0\}.$

The proof is that since $\forall p,q \in \mathbb R^3,p \cdot q := p^T q$ and $p \cdot q = 0$ implies orthogonality, the equations $q = Ap$ define tangent lines (in line coordinates) to the conic at points $p$.

He goes on to show the dual to the conic in line coordinates can be found from

$$ p^T A p = p^T A A^{-1} Ap = p^T A^T A^{-1} Ap = (Ap)^T A^{-1} (Ap) = q^T A^{-1} q $$

where we have used the fact that $A = A^T$ since $A$ is symmetric.

From the SO answer and Wikipedia, for two real non-degenerate conics $f=0$ and $g=0$, the conics $g+tg=0$ form a pencil containing one or three degenerate conics. Hence, the conic

$$ q^T(A^{-1} + tB^{-1})q=0 \tag{1} $$

has degenerates where

$$ det(A^{-1} + tB^{-1})=0 \tag{2} $$

because that would make it factorable. Solving (2) yields a general cubic in $t$, for which Cardan gives a closed-form solution. Solving for $t$, plugging into

$$ (A^{-1} + tB^{-1})q=0 \tag{3} $$

and finding the roots of each linear equation gives the pairs of points forming the tangents to the ellipses.