No, it is not a manifold.
First note that $X$ is path connected: If $A\in X$, then $tA\in X$ for all real $t$, so we can join any element $A\in X$ to $0 = \begin{bmatrix} 0 & 0\\ 0 & 0\end{bmatrix}\in X$. In addition, this implies that $X$ is a union of lines through $0\in M(2,\mathbb{R})$.
Now, assume for a contradiction that $X$ is a manifold. And consider the $0$ matrix $0\in X$ as well as the four curves $\gamma_1(t) = \begin{bmatrix} t & 0 \\ 0 & 0\end{bmatrix}$, $\gamma_2(t) =\begin{bmatrix} 0 & 0\\ 0 & t\end{bmatrix}$, $\gamma_3(t) = \begin{bmatrix} t & t\\ 0 & 0\end{bmatrix}$, and $\gamma_4(t) = \begin{bmatrix} t & 0\\ t & 0\end{bmatrix}.$
I will leave it to you to verify that $\gamma_i(0) = 0$ for any $i$, and that $\gamma_i(t)\in X$ for all $t$ and for any $i$.
Thus, $\gamma_i'(0)\in T_0 X$, which easily implies that $T_0 X$ is $4$-dimensional. Thus, there is a neighborhood $U$ (in $M(2,\mathbb{R})$Q) about $0$ which lies entirely in $X$. But $X$ is a union of lines through $0$, and every such line hits $U$. Thus, we conclude $X = M(2,\mathbb{R})$, giving an obvious contradiction. Thus, $X$ cannot be a manifold at the point $0$.
One may object that $0$ is quite special here. I agree. In fact, we have the following result.
If $Y = X\setminus \{0\}$, then $Y$ is a manifold.
To see this, first consider $Z = \{A\in M(2,\mathbb{R}): A\neq 0, \det A = 0\}$. As shown in Robert Israel's comment to Georges Elencwagj's answer to this MSE question, $Z$ is a smoothly embedded submanifold in $M(2,\mathbb{R})$.
We then claim that $Y$ is the intersection of $Z$ with an open subset $V$ of $M(2,\mathbb{R})$, which will establish the fact that $Y$ is a smoothly embedded submanifold.
Let $V = \left\{\begin{bmatrix} a&b\\c&d\end{bmatrix}\in M(2,\mathbb{R}): a+d\neq 0\right\}$. Then we claim that $Y = Z\cap V$.
To see this, suppose $A = \begin{bmatrix} a&b\\c&d\end{bmatrix}\in Z$ but $A\notin V$. Then $A$ has an eigenvalue of $0$ with algebraic multiplicity $2$. However, since $A\in Z$, $A$ has rank at least $1$, so its Jordan form must also have rank at least $1$. This implies the Jordan form is $\begin{bmatrix} 0 & 1\\ 0 & 0\end{bmatrix}$, so $A$ is not diagonalizable. That is, $A\notin Y$.
On the other hand, assume that $A=\begin{bmatrix} a&b\\c&d\end{bmatrix}\in Z\cap V$. Then $a+d\neq 0$, so $A$ has distinct eigenvalues of $0$ and $a+d$. Thus, $A$ is diagonalizable, so $A \in Y$.