Please explain the last step of this newton method for system of equations

Question

The step of working out x$^1$. I know the above is the formula but do they actually work out the inverse of the derivative matrix, is there a quicker way to do this?

Answer 1

I am using Solving a set of equations with Newton-Raphson for background.

The regular Newton-Raphson method is initialized with a starting point $x_0$ and then you iterate $$x_{n+1}=x_n-\dfrac{f(x_n)}{f'(x_n)}$$

In higher dimensions, there is an exact analog. We define:

$$F\left(\begin{bmatrix}x_1\\x_2\\x_3\end{bmatrix}\right) = \begin{bmatrix}f_1(x_1, x_2, x_3) \\ f_2(x_1, x_2, x_3) \\ f_3(x_1, x_2, x_3)\end{bmatrix} = \begin{bmatrix}\begin{array}{rcl} x_1^3 x_2-x_3 \\ 5x_1^3 x_3-5x_2&\\ x_2^3x_3^2-32 \end{array} \end{bmatrix}$$

The derivative of this system is the $3x3$ Jacobian given by:

$$J(x, y, z) = \begin{bmatrix} \dfrac{\partial f_1}{\partial x_1} & \dfrac{\partial f_1}{\partial x_2} & \dfrac{\partial f_1}{\partial x_3}\\ \dfrac{\partial f_2}{\partial x_1} & \dfrac{\partial f_2}{\partial x_2} & \dfrac{\partial f_2}{\partial x_3} \\ \dfrac{\partial f_3}{\partial x_1} & \dfrac{\partial f_3}{\partial x_2} & \dfrac{\partial f_3}{\partial x_3}\end{bmatrix} = \begin{bmatrix} 3 x_1^2 x_2 & x_1^3 &-1 \\ 15 x_1^2 x_3 & -5 & 5 x_1^3 \\ 0 & 3 x_2^2 x_3^2 & 2 x_2^3 x_3 \end{bmatrix}$$

The function $G$ is defined as:

$$G(x) = x - J(x)^{-1}F(x)$$

and the functional Newton-Raphson method for nonlinear systems is given by the iteration procedure that evolves from selecting an initial $x^{(0)}$ and generating for $k \ge 1$,

$$x^{(k)} = G(x^{(k-1)}) = x^{(k-1)} - J(x^{(k-1)})^{-1}F(x^{(k-1)}).$$

We can write this as:

$$\begin{bmatrix}x_1^{(k)}\\x_2^{(k)}\\x_3^{(k)}\end{bmatrix} = \begin{bmatrix}x_1^{(k-1)}\\x_2^{(k-1)}\\x_3^{(k-1)}\end{bmatrix} + \begin{bmatrix}y_1^{(k-1)}\\y_2^{(k-1)}\\y_3^{(k-1)}\end{bmatrix}$$

where:

$$\begin{bmatrix}y_1^{(k-1)}\\y_2^{(k-1)}\\y_3^{(k-1)}\end{bmatrix}= -\left(J\left(x_1^{(k-1)},x_2^{(k-1)},x_3^{(k-1)}\right)\right)^{-1}F\left(x_1^{(k-1)},x_2^{(k-1)},x_3^{(k-1)}\right)$$

Here is a starting vector:

$$x^{(0)} = \begin{bmatrix}x_1^{(0)}\\x_2^{(0)}\\x_3^{(0)}\end{bmatrix} = \begin{bmatrix}0.1\\2.0\\3.0\end{bmatrix}$$

Using this starting value, the iterates are given by:

• $x_0 = (0.1, 2.0, 3.0)$
• $x_1 = (25.475, 2.28528, 1.52479)$
• $x_2 = (16.7237, 2.35526, 1.57156)$
• $x_3 = (11.1596, 2.35117, 1.56911)$
• $x_4 = (7.44242, 2.3506, 1.56967)$
• $x_5 = (4.96763, 2.34871, 1.57156)$
• $x_7 = (3.32525, 2.34244, 1.57786)$
• $x_8 = (2.24692, 2.32233, 1.59819)$
• $x_9 = (1.56361, 2.26486, 1.65773)$
• $x_{10} = (1.17742, 2.14634, 1.78976)$
• $x_{11} = (1.02391, 2.03045, 1.94396)$
• $x_{12} = (1.00041, 2.00083, 1.99775)$
• $x_{13} = (1., 2., 2.)$
• $x_{14} = (1., 2., 2.)$

As you can see, after fourteen iterations, we have:

$$\begin{bmatrix}x_1\\x_2\\x_3\end{bmatrix} = \begin{bmatrix} 1. \\ 2.\\ 2. \end{bmatrix}$$

Of course, for a different starting vector you are going to get a different solution and perhaps no solution at all.

If we started at $x_0 = (1, 2, 3)$, we converge much faster (three iterations), for example.

Update The problem was changed in totality after posting the answer above!

$$F\left(\begin{bmatrix}x_1\\x_2\\x_3\end{bmatrix}\right) = \begin{bmatrix}f_1(x_1, x_2, x_3) \\ f_2(x_1, x_2, x_3) \\ f_3(x_1, x_2, x_3)\end{bmatrix} = \begin{bmatrix}\begin{array}{rcl} x_1 + x_2+ x_3 \\ x_1^2 + x_2^2 + x_3^2 - 5\\ e^{x_1} + x_1 x_2 - x_1 x_3 -1 \end{array} \end{bmatrix}$$

$$J(x, y, z) = \begin{bmatrix} \dfrac{\partial f_1}{\partial x_1} & \dfrac{\partial f_1}{\partial x_2} & \dfrac{\partial f_1}{\partial x_3}\\ \dfrac{\partial f_2}{\partial x_1} & \dfrac{\partial f_2}{\partial x_2} & \dfrac{\partial f_2}{\partial x_3} \\ \dfrac{\partial f_3}{\partial x_1} & \dfrac{\partial f_3}{\partial x_2} & \dfrac{\partial f_3}{\partial x_3}\end{bmatrix} = \begin{bmatrix} 1 & 1 & 1 \\ 2 x_1 & 2 x_2 & 2 x_3 \\ e^{x_1} + x_2 - x_3 & x_1 & -x_1 \end{bmatrix}$$

Here is a starting vector:

$$x^{(0)} = \begin{bmatrix}x_1^{(0)}\\x_2^{(0)}\\x_3^{(0)}\end{bmatrix} = \begin{bmatrix}2\\13\\12\end{bmatrix}$$

Note: The author has two obvious errors in his first calculation. The first is $2 x_3 = 24$, not $20$ as shown in $a_{23}$. Also, the Jacobian has an incorrect partial derivative term in $J_{31}$, both of which I have corrected in my answer.

To calculate $x_1$, we have:

$$x_1 = x_0 - J^{-1}(x_0)F(x_0) = \begin{bmatrix}2\\13\\12\end{bmatrix} - \begin{bmatrix}1 & 1 & 1\\ 4 & 26 & 24\\ 1 + e^2 & 2 & -2\end{bmatrix}^{-1}\begin{bmatrix}27\\312\\1 + e^2\end{bmatrix} = \begin{bmatrix}-12.5744\\35.2562\\-22.6819\end{bmatrix}$$

Using this starting value, the iterates are given by:

• $x_0 = (2., 13., 12.)$
• $x_1 = (-12.5744,35.2562,-22.6819)$
• $\ldots$
• $x_{11} = (0. , 1.58114 , -1.58114 )$

We converge to:

$$\begin{bmatrix}x_1\\x_2\\x_3\end{bmatrix} = \begin{bmatrix} 0. \\ 1.58114 \\ -1.58114 \end{bmatrix}$$