Okay, I finally worked it out. I'll give a complete solution to the exercise:
First of all we have the trivial character, which we'll denote $\chi_1$. We have $\left<\chi_1,\alpha\right>=\left<\chi_1,\beta\right>=\left<\chi_1,\gamma\right>=0$, so it isn't present in any of the characters we've been given.
Computing the inner product of each character with itself gives $\left<\alpha,\alpha\right>=2$, $\left<\beta,\beta\right>=2$ and $\left<\gamma,\gamma\right>=4$, so $\alpha$ and $\beta$ are each the sum of two distinct irreducible characters, and $\gamma$ is either the sum of four distinct irreducible characters or two copies of the same irreducible character. In fact it must be the latter, because $\left<\beta,\gamma\right>=2$, which would mean that both the irreducible characters in $\beta$ would have to appear in $\gamma$. But then the remaining two irreducible characters in $\gamma$ would only have dimension $1$ between them, contradiction. So $\chi_2=\gamma/2$ is an irreducible character of dimension $8$.
Then it turns out that $\left<\chi_2,\alpha\right>=1$ and $\left<\chi_2,\beta\right>=1$, so we can find two more irreducible characters: $\chi_3=\alpha-\chi_2$ and $\chi_4=\beta-\chi_2$, of dimensions $6$ and $7$. Using the fact that the squares of the dimensions of the irreducible representations must sum to 168, we can see that the two remaining irreducible characters must both have dimension $3$.
So our character table so far is given by:
\begin{array}{c r r r r r r} |[g]|&1&21&42&56&24&24\\ \hline
\chi_1&1&1&1&1&1&1\\
\chi_2&8&0&0&-1&1&1\\
\chi_3&6&2&0&0&-1&-1\\
\chi_4&7&-1&-1&1&0&0\\
\chi_5&3&x_2&x_3&x_4&x_5&x_6\\
\chi_6&3&y_2&y_3&y_4&y_5&y_6\\
\end{array}
To work out the $x$s and $y$s we'll use column orthonormality. Using orthonormality of the second column with the first and with itself gives
$x_2+y_2=-2$ and $|x_2|^2+|y_2|^2=2$, which imply $x_2=y_2=-1$. Likewise, $x_3+y_3=2$ and $|x_3|^2+|y_3|^2=2$ so $x_3=y_3=1$, and $x_4+y_4=0$ and $|x_4|^2+|y_4|^2=0$ so $x_4=y_4=0$.
In the fifth column we get $x_5+y_5=-1$ and $|x_5|^2+|y_5|^2=4$. These equations have multiple solutions. But since the complex conjugate of an irreducible character is an irreducible character, we know that either $x_5$ and $y_5$ are both real, or they are each other's complex conjugate. In the real case we have $x_5+y_5=-1$ and $x_5^2+y_5^2=4$, and hence $x_5=\frac{-1\pm\sqrt 7}2$ (without loss of generality we can take the "$+$" case, and then $y_5=\frac{-1-\sqrt 7}2$). In the unreal case we have $2\mathrm{Re}(x_5)=-1$ and $2|x_5|^2=4$, and hence $x_5=\frac{-1\pm\sqrt 7i}2$ (again we will take the "$+$" case, and hence $y_5=\frac{-1-\sqrt 7i}2$).
It's clear that $x_6$ and $y_6$ must take the same values as $x_5$ and $y_5$, but the other way around. So we have narrowed down the possibilities for the character table to two remaining choices, both of which fully satisfy all the orthogonality relations. The character table is either
\begin{array}{c r r r r r r} |[g]|&1&21&42&56&24&24\\ \hline
\chi_1&1&1&1&1&1&1\\
\chi_2&8&0&0&-1&1&1\\
\chi_3&6&2&0&0&-1&-1\\
\chi_4&7&-1&-1&1&0&0\\
\chi_5&3&-1&1&0&\frac{-1+\sqrt 7}2&\frac{-1-\sqrt 7}2\\
\chi_6&3&-1&1&0&\frac{-1-\sqrt 7}2&\frac{-1+\sqrt 7}2\\
\end{array}
or
\begin{array}{c r r r r r r} |[g]|&1&21&42&56&24&24\\ \hline
\chi_1&1&1&1&1&1&1\\
\chi_2&8&0&0&-1&1&1\\
\chi_3&6&2&0&0&-1&-1\\
\chi_4&7&-1&-1&1&0&0\\
\chi_5&3&-1&1&0&\frac{-1+\sqrt 7i}2&\frac{-1-\sqrt 7i}2\\
\chi_6&3&-1&1&0&\frac{-1-\sqrt 7i}2&\frac{-1+\sqrt 7i}2\\
\end{array}
and it remains to rule out one of these possibilities.
Here's the intended solution: By the Orbit-Stabilizer theorem the elements in the conjugacy classes of size $24$ must have centralizers of size $7$. Since these elements are in their own centralizers and aren't the identity, they must therefore have order $7$. The eigenvalues of a representation of an element of order $7$ must be $7$th roots of unity. So since $\boldsymbol{\sqrt{7}\notin\mathbb{Q}(e^{2\pi i/7})}$, the first character table above cannot be correct, and the answer must be the second one.
Of course I think it's actually easier to not use the extra assumption. For example you can calculate that if we take $\chi_5$ and $\chi_6$ as in the first character table we get $\left<\chi_5,\chi_5\otimes\chi_6\right>=1/2$, which is a contradiction since this value should be a natural number. This answers the question using only properties of characters, and without having to delve into the properties of the group.
