Exercise :
Determine the stability of $O(0,0)$ for the system of differential equations: $$x' = -x + y + x^2 + ax^3$$ $$y' = x - y + x^2 + bxy + y^3$$
Discussion :
Finding the matrix of the linearized system (the Jacobian), we get :
$$J(x,y) = \begin{bmatrix} -1 + 2x + 3ax^2 &1 \\ 1 + 2x + by& -1 + bx +3y^2 \end{bmatrix}$$
and the matrix at $O(0,0)$ is :
$$J(0,0) = \begin{bmatrix} -1 & 1 \\ 1 & -1\end{bmatrix}$$
Finding the eigenvalues of the matrix to proceed with determining its stability, we get :
$$\det(J(0,0)-λI) = 0 \Rightarrow\dots \Rightarrow λ(λ+2) = 0 \Leftrightarrowλ=-2 \space \text{or} \space λ =0$$
My question is, how does one proceed in the case of a zero eigenvalue, which means that $λ_1 \cdot λ_2 = 0$ ?
I know a way bypassing such issue is converting to polar coordinates, which can be done rather simply but it doesn't lead anywhere clear on this specific example.
The system also cannot be solved so it can't be worked around freely by this way (obviously).
Finally, I've figured out 1-2 theorems working over the implicit function theorem to prove a strong case via it, but they include hypothesis and an assisting function that is a total mess.
I would really appreciate anyone's help on how to proceed on such problems where we are stuck with a zero eigenvalue.