I'm have difficulty solving this problem in my textbook.
Does $x^2+x+1 \equiv 0\pmod{997}$ have solutions? Why or why not?
I guess the first step would be $$ \begin{array}{l} (2x+1)^2 \equiv (-3)\pmod{997} \\ y^2 \equiv -3\pmod{997} \end{array} $$ Is that mean no solution ?
From this, using $\displaystyle\left(\frac{ab}p\right)=\left(\frac ap\right)\left(\frac bp\right)$
$$\left(\frac{-3}p\right)=\left(\frac{-1}p\right)\left(\frac 3p\right)$$
As $\displaystyle997\equiv1\pmod4,\left(\frac{-1}p\right)=1 $ (See $-1$ is a quadratic residue modulo $p$ if and only if $p\equiv 1\pmod{4}$)
Now use Quadratic Reciprocity Theorem, $$\left(\frac 3{997}\right)\left(\frac{997}3\right)=1$$
As $997\equiv1\pmod3,$ $$\left(\frac{997}3\right)=\left(\frac13\right)$$
Now $\displaystyle\left(\frac1p\right)=(1)^{\dfrac{p-1}2}=1$ for all odd prime $p$
Hint $\,\ 997\, =\, 5^2\! + 3(18)^2\,\Rightarrow\, -3 \equiv (5/18)^2\pmod{997}.\ $ Or, use quadratic reciprocity.
OK, let's try some crude brute-force arithmetic. We're looking at $$ x^2 + x + 1 \pmod{997}. $$ Quadratic equations are solved by completing the square, and that means taking half the coefficient of the first-degree term and squaring it, and adding that. The coefficient of the first-degree term is $1$, so what's half of $1$ in $\bmod997$? Since $2\times499\equiv1$, half of $1$ is $499$. The square of that is $499^2\equiv748\pmod{997}$. So we have $$ (x^2 + x + 748) +(1-748) \equiv (x + 499)^2 + 250 \equiv 0. $$ $$ (x+499)^2 \equiv-250\equiv747 $$ Now the question is whether $747$ has a square root. Brute-force number crunching via computer tells me that $194^2\equiv747$, so of course $(-194)^2\equiv747$, and $-194\equiv803$. So $x+449 \equiv\pm 194$, which means $x+449\equiv(194\text{ or }803)$.
You wrote $(2x+1)\equiv-3$. Since $388^2\equiv-3$, that would mean there is a solution.
Moral of the story: The "hard" part is finding square roots. That can be done by brute force, but it would be desirable to have an intelligent way to do it. The nature of the "intelligent" was to do it has been addressed in answers posted by others. Half of the numbers in $\{1,\ldots,996\}$ have square roots mod $997$ and half don't.
