Why Huber loss has its form?

Question

Huber loss formula is

$\hspace{3.0cm} L_\delta(a) = \begin{cases} \frac{1}{2} a^2 && |a| \leq \delta \\ \delta (|a| - \frac{1}{2} \delta) && |a| > \delta\end{cases}$ where $a = y - f(x)$

As I read on Wikipedia, the motivation of Huber loss is to reduce the effects of outliers by exploiting the median-unbiased property of absolute loss function $L(a) = |a|$ while keeping the mean-unbiased property of squared loss function $L(a) = a^2$ within the typical region (i.e. region without outliers)

I don't know why don't Huber loss has the form:

$\hspace{3.0cm} L_\delta(a) = \begin{cases} \frac{1}{2} a^2 && |a| \leq \delta \\ c |a| && |a| > \delta\end{cases}$ where $c$ is some constant which controls the error magnitude of outliers

or similar forms ?

Answer 1

Let's check for continuity for the new proposed function. For convenience, let me call it $L_{\delta, c}$.

$$\lim_{a \to \delta^+}L_{\delta, c}(a)=c\delta$$ and $$\lim_{a \to \delta^-}L_{\delta, c}(a)=\frac{\delta^2}2$$

If we want the function to be continuous, we would want $c$ to take value $\frac{\delta}2$.

Now, let's check for diffentiability of $L_{\delta, \frac{\delta}2}$.

$$\lim_{h \to 0^+} \frac{L_{\delta, \frac{\delta}2}(\delta+h)-L_{\delta, \frac{\delta}2}(\delta)}{h}=\lim_{h \to 0^+} \frac{\frac{\delta(\delta+h)}2-\frac{\delta^2}2}{h}=\frac{\delta}2$$

$$\lim_{h \to 0^-} \frac{L_{\delta, \frac{\delta}2}(\delta+h)-L_{\delta, \frac{\delta}2}(\delta)}{h}=\lim_{h \to 0^-} \frac{\frac{(\delta+h)^2}2-\frac{\delta^2}2}{h}=\delta$$

which shows that we can't have $L_{\delta,c}$ that is differentiable.

Now, let's examine the differentiable of the original huber loss at $\delta$.

$$\lim_{a \to \delta^+}L_\delta (a)=\delta(\delta-\frac{\delta}2)=\lim_{a \to \delta^-}L_{\delta}(a)=L_{\delta}(\delta)$$

Also,

$$\lim_{h \to 0^+} \frac{L_{\delta}(\delta+h)-L_{\delta}(\delta)}{h}=\lim_{h \to 0^+} \frac{\delta(\frac\delta2+h)-\frac{\delta^2}2}{h}=\delta=\lim_{h \to 0^-} \frac{L_{\delta}(\delta+h)-L_{\delta}(\delta)}{h}.$$

The huber loss function is constructed such that it is differentiable.