Is $\mathbb{Z}(\sqrt{11})$ a UFD?
I read about real quadratic filed and about algebraic integers. In all of books that I read tehy show that imaginary filed doesn't have unique factorization such as $$6=2\cdot3=(1+\sqrt{-5})\cdot(1-\sqrt{-5})\,,$$ but not for real.
The main reason that you won't find anything in the literature on the fact that $\mathbb{Z}[\sqrt{D}]$ is a UFD or not is that we don't know.
For $D>0$ squarefree such that $D\not\equiv 1 \mod 4$, this is equivalent to ask whether $\mathbb{Z}[\sqrt{D}]$ is a PID or not (because this ring is a Dedekind domain in this case).
It is conjectured that this will be the case for infinitely many values of $D$, but at this point, this is wide open.
Concerning your specific example, the answer is YES. The ring $R=\mathbb{Z}[\sqrt{11}]$ is indeed a PID (hence a UFD).
To show it, you need the machinery of algebraic number theory. First, you have to realize that $R$ is the ring of integers of $K=\mathbb{Q}(\sqrt{11})$.
Minkowski's bound then says that any element of the class group of $K$ may be represented by an ideal $I$ of norm $\leq 3$. Since $I$ is the product of prime ideals and the norm is multiplicative, it is enough to show that prime ideals of norm $2$ and $3$ are principal. Such ideals are exactly those which appear into the decomposition of $2R$ and $3R$ into a product of prime ideals. By a celebrated theorem of Dirichlet, these decomposition are reflected by the decomposition of $X^2-11$ mod $2$ and $3$ into irreducible factors.
First, $X^2-11=X^2+1$ mod $3$, which is irreducible mod $3$ since $-1$ is not a square mod $3$. Hence $3R$ is the decomposition we are looking for. We conclude that the only prime ideal lying above $3$ is $3R$, which is principal. Note that $3R$ has in fact norm $9$, and that we can even discard it...
Now $X^2-11=(X-1)(X+1)$ mod $2$. Hence $2R=(2, -1+\sqrt{11})(2,1+\sqrt{11})$ (using Dirichlet's theorem). Set $\mathfrak{p}=(2, -1+\sqrt{11})$. Then $(2,1+\sqrt{11})=(2,-1-\sqrt{11})=\mathfrak{p}^*$, where $*$ is the unique non trivial $\mathbb{Q}$-automorphism of $K$. Thus, if $\mathfrak{p}$ is principal with generator $\alpha$, so is $\mathfrak{p}^*$ (generated by $\alpha^*$).
It remains to show that $\mathfrak{p}$ is principal. Since this ideal has norm $2$, it will be generated by an element of norm $2$ . Let $\alpha=3-\sqrt{11}$. Then $\alpha=2 -(-1+\sqrt{11})\in\mathfrak{p}$. Hence $(\alpha)\subset\mathfrak{p}$. Now $N((\alpha))=\vert 3^2-11(-1)^2\vert =2=N(\mathfrak{p})$, so $\mathfrak{p}=(\alpha)$. Finally, $\mathfrak{p}=(\alpha)$, and we are done.
