The fact that the matrix is symmetric follows by the symmetry of the scalar product; e.g. consider some basis vectors $\{e_i\}$, then
$$
M_{ij} = \langle e_i | e_j \rangle
$$
such that $M_{ij} = M_{ji}$.
The fact that it's invertible follows since $M_{ij}$ has trivial kernel, which is immediate from your definition of nondegeneracy. Allow there to exist some $v = \sum_i v^i e_i \ne 0$ such that $v\in \ker M$, that is
$$
M_{ij}v^j = 0, \quad\forall i
$$
but, for any $w = \sum_j w^je_j$ we have $M_{ij}v^jw^i = 0 = \langle w|v\rangle = \langle v|w\rangle$, yet $v\ne 0$ by assumption, which immediately implies the above since $w$ was arbitrary. By contrapositive, then, $M$ has a trivial kernel and is therefore invertible.