0

Let us consider the centered finite difference approximation of the first derivative of a smooth function

$$f'(x_i) = \frac{f(x+h) - f(x-h)}{2h}$$

It's well known that if we do a $\text{loglog}$ plot of the error vs. the stepsize, we will have that after a certain value of $h \approx 10^{-5}$, the error will start growing. What I wanted to understand, is how such value can be predicted.

I found this answer by user @LutzLehmann, where he wrote that, since function evaluations will produce noise, the error is a combination of the approximation error and that noise, i.e. something like

$$\frac{M_0\mu}h+M_3h^2$$

where $M_0$ is the magnitude of the function evaluation and $M_3$ the magnitude of the evaluation of the third derivative.

Then, he wrotes that the error will be minimal if $h \approx \mu^{\frac{1}{3}}$ (i.e. about $10^{-5}$ if $\mu$ is the machine double precision), under the hypothesis that $M_0$ and $M_3$ are about equal.

I can't understand "how" he find that value: I mean, let's say $M_0=M_3=M$. Then we have that the error is about $$M( \mu + h^3)$$

If I want to minimize, I would say $h^3 = - \mu$, which implies a negative step, which makes no sense. Why doesn't he have that minus sign?

andereBen
  • 705

1 Answers1

1

The minimum of the error term is where both contributions to $\frac{\mu}h+h^2$ are about equal. Or more precisely, where the derivative is zero, $$ 0=-\frac{\mu}{h^2}+2h\iff h=\sqrt[3]{\frac\mu2}. $$ For a general qualitative argument, factors close to $1$ do not change the overall magnitude, so $h\sim \mu^{1/3}$.

Lutz Lehmann
  • 131,652
  • Many thanks for the quick reply! It's all clear, except one thing: you state

    $$\frac{M_0\mu}h+M_3h^2$$

    so it seems that the noise is just in the evaluation of the function, not in its derivatives. Why is that?

     – andereBen Aug 17 '20 at 15:23
  • The errors in the derivative give a correction of the size $M_3h^2\mu$ and thus are negligible against the other terms. – Lutz Lehmann Aug 17 '20 at 17:09
  • I'm really sorry but I'm still missing how you find that correction term. Why do you say it has that size? Or do you have any reference for that? I can't find those arguments – andereBen Aug 17 '20 at 17:17
  • 1
    Properly seen, there are no evaluation errors in the derivatives. The estimate of the leading error terms is composed of the evaluation error of the difference quotient formula used, and the analytical error of the difference quotient to the exact derivative. Note that $M_k$ are supremum norms for some small interval around the evaluation point, in $M_0$ one has to use a function where all sums and differences in its computation are replaced by sums of absolute values. – Lutz Lehmann Aug 17 '20 at 17:24
  • I think I got it. For instance, in $D_2(f,h)$ you have $\frac{f(x+h) - f(x-h)}{2h}$. So in some sense, you have $\frac{2 M \mu }{2h} = \frac{M \mu}{h}$ from the rounding and $M_3 h^2$ from the truncation error. ($M_3 h^2 \mu$ is negligible so it does not appear).

    Thank you really much.

    The really last point: when I think about rounding errors in the evaluation of the function $f(x)$, I always write $f(x)(1+\delta)$, with $|\delta| \leq \mu$. Is this the same think you do when you write $f(x)=f(x) \mu$?

     – andereBen Aug 17 '20 at 17:30
  • 1
    This is valid at points where catastrophic cancellation does not occur. For instance for a polynomial near a root the near-zero value can only be obtained via cancellation of large terms, thus the sum of the absolute values of these terms is a better coefficient for the machine constant for a valid error bound. – Lutz Lehmann Aug 17 '20 at 17:46
  • Perfect, I got it. So in your computations, you are actually using $f_{\varepsilon} (x)=\text{fl}(f(x)) = f(x)(1+\delta)$ and the idea described here in Christoph's answer [https://math.stackexchange.com/questions/3671517/error-term-in-finite-differences-using-taylors-theorem ]

    because the sums and differences are replaced by absolute value. Then, he uses $$|f(x)- f(x)(1+\delta)| = |f(x) (- \delta)| \leq |f(x)| \mu$$

    I should have all the details now, thanks for the fruitful discussion

     – andereBen Aug 17 '20 at 17:55