0

This question has been bothering me for a while: In calculating the standard deviation, why do we square the difference from the mean, as opposed to cubing the differences (and then taking the cube root at the end)? Is this just a random choice, since eventually you have to choose some exponent, and "2" was sufficient? Or is there a deeper mathematical reason?

Thanks in advance!

David Richerby
  • 2,480
JMS
  • 103

3 Answers3

7

An intuitive way of thinking about it is that standard deviation is a measure of spread. So you need some way of saying for every value in the data set, on average how far away is that value from the mean.

So you take the differences $d_i = x_i - \bar{x}$. What do you do with the differences? If you simply add them up, you'll get zero. You want to give equal weight to differences that are positive $d_i=+k$ as negative $d_i=-k$ so the first thing you think of is to take the absolute value: spread = $\sum|d_i|$. This is called the average absolute deviation, and it is, along with related measures, an accepted way to measure spread.

Unfortunately, taking the absolute value doesn't play well with calculus, and you want to use calculus to differentiate the spread to be able to minimize it. Minimizing it is important when for example you want to fit a line to a set of data points. So what function gives equal weight to $\pm d_i$ and is easy to differentiate? The simplest function is taking the square of each difference. The average of squared differences, the variance, is easy to differentiate and we can scale back to the size of our original data items by taking the square root of the sum to get standard deviation.

So, at last, why not take the cubed differences? It's because taking the cube of a negative difference gives a different result to taking the cube of a positive difference of the same magnitude. I.e, if $d_i=-k$ and $d_j=k$ then $d_i^3 \neq d_j^3$. And in fact, with a symmetric distribution, taking the average of the cubed differences gives you zero, which clearly isn't a measure of spread. What taking the cubed differences does is tell you how skewed the distribution is. (If you go further and look at taking fourth powers, you get into something called kurtosis, which measure how fat the tails of the distribution are.)

TooTone
  • 6,443
  • 1
    Thanks, that's interesting. Do you have a simple, intuitive explanation for why higher powers measure different things? Is it similar to how taking the second derivative tell us the degree at which change is occurring, while the third derivative tells us the degree at which the change of the change is occurring, etc ? – JMS Feb 22 '14 at 00:54
  • good q (you might even want to ask as a new one, referencing this one + the one it duplicated and explaining why your questions is different. Please reply to this comment if you do!). My opinion is it's not analogous to derivatives. Rather you need to think about the effect of taking higher powers. (1) The higher the power you take the more you promote larger differences at the expense of smaller differences. So kurtosis, the 4th moment emphasises what happens at the tails more than variance the 2nd moment (2) Odd vs even powers. Odd powers incorporate some measure of location, see my answer. – TooTone Feb 22 '14 at 12:33
1

Technically the smallest power that will accomplish smoothness at zero is 1+ϵ, as the derivative will be left with x^ϵ which then vanishes, basically minimally blunting the sharp 'tip' of the absolute value function. I mention this not to be pedantic, but because it can be useful if you really do want something that largely behaves like the absolute value function but want to have less bad behavior near zero. Another approach is to use something like the Huber loss function

user130344
  • 97
1

The sum of cubes of differences with the mean (or integral, in the case of a continuous distribution) gives the third central moment, known as a measure of skewness, rather than variance in the second moment case.

More technically, the skewness $\gamma_1$ is defined by:

$$ \gamma_1 = E[(X-\mu)^3]/\sigma $$

where $\mu$ is mean of the distribution of random variable $X$ and $\sigma$ is the standard deviation (square root of the variance).

hardmath
  • 37,715
  • Thanks! Is there an intuitive reason for why cubing gives "skewness" and not difference from the mean? At first glance, it seems like there is no intuitive difference... – JMS Feb 22 '14 at 00:52
  • 1
    The cube is an odd power, so negatives cubed are negative and positives cubed are positives. In this way the skewness can indicate a longer "tail" spreading out from the mean on one side than the other. – hardmath Feb 22 '14 at 01:10
  • Perhaps, to state this as an advantage for squaring when we define variance, the even power makes a positive contribution equally for data above or below the mean, where cubing preserves the signs and causes cancellations between positive and negative contributions. So variance does not measure "lopsidedness" of a distribution, but skewness will. – hardmath Feb 22 '14 at 15:27