It doesn't have to be an inversion about the mean.
Let $R$ be the "reflect-a-vector operator", meaning
$$R(v) = I - 2 |v\rangle \langle v|$$
Grover's algorithm works by starting in some state $|d\rangle$ and then alternating two reflection operations, $R(s)$ and $R(d)$, where $s$ is the solution vector and $d$ is a "diffusion vector". The choice of $d$ affects the speed of the algorithm. Basically, the more $d$ aligns with $s$ (the closer they are to parallel), the faster you will go. The problem is that you don't know what $s$ is, so you need to pick a $d$ that works okay for any possible $s$.
The simplest $d$ that works equally well for every possible $s$, and the $d$ that Grover happened to use, is the normalized sum of each possible $s$. That is to say, you set $d= \frac{1}{\sqrt{N}}\sum_{k=0}^{N-1} |k\rangle = |+\rangle^{\otimes \lg N}$. This $d$ is the average of all the solutions, so it inverts about the average.
Another perfectly acceptable choice of $d$ is $d = \frac{1}{\sqrt{N}}\sum_{k=0}^{N-1} (-1)^{\text{HammingWeight}(k)} |k\rangle = |-\rangle^{\otimes \lg N}$. For example, this is the state used in Quirk's example Grover circuit. Yet another perfectly acceptable choice of $d$ is the Fourier transform of any $|k\rangle$, e.g. $d = \text{QFT} \cdot |1\rangle = \frac{1}{\sqrt{N}}\sum_{k=0}^{N-1} e^{i k / N} |k\rangle$.
More generally, any $d$ that can be written in the form $\frac{1}{\sqrt{N}}\sum_{k=0}^{N-1} e^{i \theta_k}|k\rangle$ will work. As long as $|\langle d|k \rangle|^2 = 1/N$ for all $k$, you're good to go ... except that not all choices have a nice compact circuit. For that reason, you should stick to values of $\theta_k$ that factor across the qubits, i.e. states that can be factorized into the form $\otimes_{q=0}^{\lg N - 1} Z^{\phi_q}|+\rangle$.
