Consider the following 2D autonomous system of ODEs: $$ \left\{ \begin{array}{ll} \dot{x} = x^2 + 2y - x \\ \dot{y} = 3xy/2 - 3x^2 - y + 2x \end{array} \right. $$
How can we prove the existence and uniqueness of a homoclinic orbit (i.e. a solution $X$ of the system for which $\lim_{t \to \infty} X(t) = \lim_{t \to -\infty} X(t) = x_0,$ where $x_0$ is an equilibrium point of the system) for the system?
It is not hard to determine the equilibrium points: $(0,0), (-3, -6)$ and $(2/3, 1/9)$. The latter two are asymptotically stable equilibrium points, so there cannot be a homoclinic orbit for those points.
However, $(0,0)$ is a saddle equilibrium, so perhaps there is a homoclinic orbit here. We can also see that the function $H(x,y) = - y^2 + x^2(1-x)$ is constant on the solutions of the system. The set $H(x,y) = 0$ seems to be comprised of three solutions of the system, one of which is indeed a homoclinic orbit, so this proves existence.
However, how do we prove that this is unique? I don't really know how to approach this part.