I suppose the formulas you can use are the ones on this page:
http://www.diracdelta.co.uk/science/source/s/p/spherical%20triangles/source.html
(except for the area formula, which is not of any use here anyway).
Label the following points on the sphere: $A$ at London,
$B$ at New York, $C$ at the north pole.
Labeling the opposite sides of the spherical triangle $a$, $b$, $c$,
assuming those are all angular measurements in radians,
then $A$ (London) is $b$ radians from $C$ (the north pole)
so the latitude of London is $lat_A = \frac\pi2 - b$;
so $\sin(lat_A) = \cos b$ and $\cos(lat_A) = \sin b$.
We can see then that the great circle formula is just one of the cosine formulas, which you correctly applied.
Once you have the distance $c$ from $A$ to $B$, it is relatively
quick and easy to get the departure angles at New York and London by
applying the sine law to $\triangle ABC$:
$$
\frac{\sin A}{\sin a} = \frac{\sin B}{\sin b} = \frac{\sin C}{\sin c}.
$$
Keep this in mind about the sine law: it's not one big equation, it's actually
three equations, $\frac{\sin A}{\sin a} = \frac{\sin B}{\sin b}$,
$\frac{\sin A}{\sin a} = \frac{\sin C}{\sin c}$,
and $\frac{\sin B}{\sin b} = \frac{\sin C}{\sin c}$.
We usually use only one of the three equations at a time.
You already know $a$, $b$, $C$, and $c$, so you can get $A$ from
$\frac{\sin A}{\sin a} = \frac{\sin C}{\sin c}$ (one equation, one unknown)
and $B$ from $\frac{\sin B}{\sin b} = \frac{\sin C}{\sin c}$.
Either angle $A$ or angle $B$ can help you find the maximum latitude of the great circle, but just one of them will be enough. Suppose we just figured out the angle at $A$.
The trick then is to draw a new spherical triangle with vertices at $A$
and $C$, with the third vertex somewhere along the great-circle path from
$A$ to $B$. Possible choices for the third vertex are the point
where the great circle crosses the equator, or the point of maximum latitude
itself.
Call the new vertex $B'$ and let $a'$ be the length of the arc
from $B'$ to $C$. The side of $\triangle AB'C$ opposite $B'$ is still
$b$ (the arc from $A$ to $C$ has not changed),
and the side opposite $A$ is $a'$. The angle at $A$ also either has not
changed, or it's $\frac\pi2$ minus the old angle; either way, its sine
is still $\sin A$. So the sine law applied to $\triangle AB'C$ says that
$$
\frac{\sin A}{\sin a'} = \frac{\sin B'}{\sin b}.
$$
If you chose to put $B'$ at the maximum latitude, the angle of
$\triangle AB'C$ at $B'$ must be a right angle; so you have three knowns
($\sin A$, $B'$, and $b$) and one unknown $a'$ to solve for.
Then the latitude of $B'$ is $\frac\pi2 - a'$.
If you chose to put $B'$ at the crossing of the great circle and the equator,
the arc $a'$ is $\frac\pi2$ (the arc from the equator to the pole);
so you have three knowns ($\sin A$, $a'$, and $b$) and one unknown $B'$ to solve for.
Then $\frac\pi2 - B'$ is the angle at which the great circle crosses
the equator, which is also the maximum latitude on the great circle.
