I'm working on my thesis on prey-predator system. I have 3 species here, 1 prey and 1 predator with 1 species having a mutualism with the prey and protecting the prey from predators. Here is the system :
$$\frac{dx}{dt}=r_1x(1-\frac{x}{\gamma})-\frac{xz}{1+\alpha N+x}+\theta_1xy$$
$$\frac{dy}{dt}=r_2y(1-\frac{y}{\gamma})+\theta_2xy-\Delta_1yz$$
$$\frac{dz}{dt}=\frac{Bz(x+N)}{1+\alpha N+x}-\Delta_2yz-Mz$$
Finding equilibria without the presence of predator is pretty easy. So far 2 equilibrium $E_0=(0,0,0)$ and $E_1=(x_1^*,y_1^*,0)$ has been found. The problem is for the third equilibria where all species live together. It is really hard to find the closed form solutions of the third equilibria. So, I stop here at these equations by $f_i(x,y,z)=0$ we get :
$$r_1x(1-\frac{x}{\gamma})-\frac{xz}{1+\alpha N+x}+\theta_1xy=0$$
$$\frac{B(x+N)}{1+\alpha N+x}-\Delta_2y-M=0$$
$$\frac{r_2}{\Delta_1}(1-\frac{y}{\gamma})+\frac{\theta_2}{\Delta_1}x=z$$
From here I don't know how to proceed to analyze the stability of the third equilibrium since how can you input the equation to the jacobian matrix of the system. How shall I approach on this problem? Can I do Newton-Raphson or any numerical approximation in my problem here? I wish I could somehow make it shorter.
From above, i try to make $x$ as a parameter and solving for $y$ and $z$ that gives me polynomial of $x$ in degree of 3. I tried using software Mathematica to work on the solution of this system. Mathematica give me 3 non-trivial equilibrium (this is because the polynomial in 3rd degree) which is a very large output. I couldn't find online how to analyze the stability of a non linear system like this. Any suggestions are very much appreciated. Thank you.
$r_i$ is the rate of growth species-i. $\gamma$ is the system capacity that can hold how much of the species. $alpha$ is the additional food quality. $N$ is the quantity of the additional food. $B$ is maximum growth of predator. $\theta_i$ is the amount of profit from mutualism to the species. $\Delta_i$ is the rate of death because interaction of predator and species 2. And $M$ is the rate of death of predator.
Edit : all symbols are all constant parameters, the only variables are $x$, $y$, and $z$
Edit 2 : Adding additional info on the parameters.