Since $\operatorname{tr}\mathbf I=4$ for a $4\times 4$ matrix, this is equivalent to showing that $\operatorname{tr}(\mathbf A+\mathbf A^2)\geq-1$.
As you know, the trace is related to the sum of the eigenvalues. Let's say we know the eigenvalues of $\mathbf A$ - call them $\{\lambda _i\}_{i\in\{1,\dots ,4\}}$.
We know that for the eigenvectors $\{\boldsymbol v_i\}_{i\in\{1,\dots ,4\}}$ that
$$\mathbf A\boldsymbol v_i=\lambda_i\boldsymbol v_i$$
So,
$$\mathbf A\mathbf A\boldsymbol v_i=\mathbf A\lambda_i\boldsymbol v_i \\ \mathbf A^2\boldsymbol v_i=\lambda_i(\mathbf A\boldsymbol v_i) \\ \mathbf A^2\boldsymbol v_i={\lambda_i}^2\boldsymbol v_i$$
Therefore, the eigenvalues of $\mathbf A^2$ are $\{{\lambda_i}^2\}_{i\in\{1,\dots,4\}}$. Hence
$$\operatorname{tr}(\mathbf A+\mathbf A^2)=\sum_{i=1}^4 \lambda_i+{\lambda_i}^2$$
But, for any $\lambda\in\mathbb R$, we have $\lambda+\lambda^2\geq -1/4$, and hence
$$\sum_{i=1}^4\lambda_i+{\lambda_i}^2\geq \sum_{i=1}^4 \frac{-1}{4}=-1$$
So $\operatorname{tr}(\mathbf A+\mathbf A^2)\geq-1$, and the desired result follows.