The true meaning of eigenvalues and eigenvectors, and positive (semi) definite matrix?

Question

After studying some linear algebra, the true meaning of eigenvalues, eigenvectors, and positive definite matrix are still ambiguous. So, could you answer my questions or explanations about those things? Tell me if I am wrong or correct, please.

First, as I understood, 1x1 matrix $a$ such as 2, -2, 1+j can be easily classified into several fields.

• +2 is Positive Real
• -2 is Negative Real
• 1+j is Complex

This classification can be done by the following observation, alternatively. If we multiply $a$ to $1$, then

• $2\cdot 1= 1$ : same sign
• $-2\cdot 1 = -2$ : opposite sign
• $1+j \cdot 1 = 1+j$ : complex

Likewise, $n\times n$ matrix $A$ can be classified. The way is to observe the result $Ax$ for an arbitrary vector $x$.

• If we multiply the real vector $x$ to $A$ and the result $Ax$ is also real, then we can classify $A$ as "real" matrix. (Hermitian)
• If we multiply vector $x$ to $A$ and the result $Ax$ has still the same direction as $x$, then we can classify $A$ as "real and positive" matrix.
• If we multiply vector $x$ to $A$ and the result $Ax$ has the opposite direction to $x$, then we can classify $A$ as "real and negative" matrix.

Moreover, for ease of classification, we do the projection $Ax$ onto $x$, i.e., $x^HAx$ is clearer metric to decide whether the matrix $A$ is positive definite, real (Hermitian), negative definite, etc. Thus, I think the definition of the three matrices deal with $x^H A x$.

Also, eigenvalue means

$Ax=\lambda x$

where $x$ is the corresponding eigenvectors to $\lambda$. Now if $A$ has $k$ eigenvectors, then, an arbitrary vector $x$ can be expressed by $k$ eigenvectors. (If $k<n$, the other $n-k$ components of $x$ is meaningless in transformation $Ax$.) Then the followings make sense.

• If $x$ is real and the $k$ eigenvalues are real, then $Ax$ must be real.
• If $x$ is real and the $k$ eigenvalues are positive definite, then $Ax$ must be real and has the same direction as $x$.
• If $x$ is real and the $k$ eigenvalues are negative definite, then $Ax$ must be real and has the opposite direction as $x$.

So, the two things can be combined as follows;

• The eigenvalues are real values = real matrix (like Hermitian) = $Ax$ is real for a real vector $x$ = $x^H A x$ is real.
• The eigenvalues are real and positive values = positive definite = $Ax$ has the same direction as $x$ = $x^H A x > 0$.
• The eigenvalues are real and negative values = negative definite = $Ax$ has the opposite direction as $x$ = $x^H A x < 0$.

Am I right?

Answer 1

A $1\times 1$ matrix $a$ is called a scalar (a single number). This scalar can be any number: natural ($a\in \mathbb{N}$), integer ($a\in \mathbb{Z}$), rational ($a\in \mathbb{Q}$), real ($a\in \mathbb{R}$), complex ($a\in \mathbb{C}$), etc. Particularly, you have the following

• If $a \in \mathbb{N}$: $a$ is always positive (or zero if $a=0$).
• If $a \in \mathbb{Z}$ or $a \in \mathbb{Q}$ or $a \in \mathbb{R}$: $a$ is positive if $a > 0$, negative if $a<0$, and zero if $a=0$ (basically a comparison with the number $0$).
• If $a \in \mathbb{C}$: we write $a=x+iy$ where $x$ and $y$ are real numbers called the real-part and imaginary-part of $a$, respectively, and $i^2=-1$. In this case, however, you cannot qualify $a$ as either positive or negative (see Positive and negative complex numbers?), but you can still say $a$ is zero if both $x=0$ and $y=0$.

With the above in mind, we say that a $n\times n$ matrix $A$ is

• real-valued if all its elements $a_{ij}$ are real numbers$^1$.
  • positive: if all its elements are positive ($a_{ij}>0, \forall i,j$).
  • negative: if all its elements are negative ($a_{ij}<0, \forall i,j$).
• complex-valued if at least one of its elements is a complex number$^2$.

Note that there is no need to have $Ax$ for some arbitrary vector $x$ to say the above about $A$.

Now, a matrix $A$ being positive or negative (semi) definite is different from $A$ being positive or negative. This positive/negative definiteness is usually defined for Hermitian matrices$^3$ and is a very useful thing to know about a matrix (see What is the importance of definite and semidefinite matrices?).

A Hermitian matrix $A$ is positive definite if for any nonzero (possibly complex) $x$ we have $$x^H A x > 0,$$ where superscript $H$ denotes conjugate transpose.

Similarly, $A$ will be positive semi-definite, negative definite, or negative semi-definite if $x^H A x $ is $\ge 0, <0$, or $\le 0$, respectively.

Moreover, a Hermitian matrix $A$ has real eigenvalues since

\begin{matrix} \lambda x = A x \Rightarrow x^H \lambda x = \lambda ||x||^2 = x^H A x\\ \text{then}\\ (\lambda ||x||^2)^H = (x^H A x)^H = x^H A^H x \Rightarrow \lambda^* ||x||^2 = x^H A x\\ \text{hence}\\ \lambda = \lambda^* \Rightarrow \lambda \text{ is real-valued (superscript $*$ denotes the complex conjugate).} \end{matrix}

Finally, from the above we have (the Rayleigh quotient) $$ \lambda = \frac{x^H A x}{||x||^2}, $$ and since $||x||^2$ is positive the eigenvalues are

• (real) positive if $A$ is positive-definite.
• (real) negative if $A$ is negative-definite.

  ---

$1$: This has nothing to do with a Hermitian matrix, which is a complex-valued matrix defined as a matrix $A$ for which $A^H= A$, where $A^H$ is the conjugate transpose of $A$ (see Hermitian matrix).

$2$: Assuming the sets mentioned in the beginning of the answer.

$3$: Otherwise $x^H A x = \sum_{i=1}^{n} \sum_{j=1}^{n} x_i^* a_{ij} x_j$ gives a complex scalar and we cannot say it's positive or negative!

If $A$ is Hermitian then we have $x^H A x = \sum_{i=1}^{n} a_{ii} |x_i|^2+\sum_{i=1}^{n} \sum_{j=1, j\ne i}^{n} x_i^* a_{ij} x_j = \sum_{i=1}^{n} a_{ii} |x_i|^2+\sum_{i=1}^{n(n-1)/2} \sum_{j=1, j\ne i}^{n(n-1)/2} x_i^* a_{ij} x_j + x_j^* a_{ij}^* x_i = \sum_{i=1}^{n} a_{ii} |x_i|^2+\sum_{i=1}^{n(n-1)/2} \sum_{j=1, j\ne i}^{n(n-1)/2} 2 Re(x_i^* a_{ij} x_j)$,

which is a real scalar since $a_{ii}$ and $|x_i|^2$ are real, and $Re(\cdot)$ is the real-part operator.