Let $f(x)=ax^2+bx+c$ be a quadratic. The floor function can be written as
$$\lfloor x \rfloor=\sum_{n=1}^\infty \theta(x-n)-\sum_{n=0}^\infty \theta(-x-n)$$
where $\theta(x)=1$ if $x\ge0 $ and $\theta(x)=0$ otherwise.
This transforms the integral into a sum
$$\int_\alpha^\beta \lfloor f(x) \rfloor dx= \sum_{n=1}^\infty \int_\alpha^\beta\theta(f(x)-n) dx-\sum_{n=0}^\infty \int_\alpha^\beta\theta(-f(x)-n) dx$$
Each integral appearing in this sum measures the size of the solution set (in $[\alpha,\beta]$) to $f(x)\ge n$ or $f(x)\le -n$ for $n>0$. For a quadratic, $f(x)=ax^2+bx+c$ let $r_1,r_2$ be the two roots of $f(x)-n=0$. Then the solutions to $f(x)\ge n$ or $f(x)\le -n$ are either $[r_1,r_2]$ or $\mathbb{R}\setminus[r_1,r_2]$ according to the sign of $a$. The roots can of course be written in terms of the coefficients $a,b,c$ using the quadratic formula.
Let's see how this works in your example, $f(x)=-8x^2+6x-1$. In this case, the stationary point is at $(\frac{3}{8},\frac{17}{8})$ so $f(x)\ge n$ has no solutions for $n>0$. Moreover, for $n>2$, $f(x)\le -n$ has no solutions in $[\alpha,\beta]=[0,1]$. Therefore, the whole thing is a sum of just a few terms:
\begin{align*}
-8x^2+6x-1\le 0 & \iff x\in [0,1]\setminus(\frac{1}{4},\frac{1}{2})\\
-8x^2+6x-1\le -1 & \iff x\in [0,1]\setminus(0,\frac{3}{4})\\
-8x^2+6x-1\le -2 & \iff x\in [0,1]\setminus(\frac{3-\sqrt{17}}{8},\frac{3+\sqrt{17}}{8})\\
\end{align*}
To get the answer, you add the negative length of each solution-set. So,
\begin{align*}
\int_\alpha^\beta \lfloor -8x^2+6x-1 \rfloor dx&=-\left(1-\left(\frac{1}{2}-\frac{1}{4}\right)\right)-\left(1-\left(\frac{3}{4}-0\right)\right)-\left(1-\left(\frac{3+\sqrt{17}}{8}-\frac{3-\sqrt{17}}{8}\right)\right)\\
&=-\frac{3}{4}-\frac{1}{4}-\frac{4-\sqrt{17}}{4}\\
&=\frac{-8+\sqrt{17}}{4}
\end{align*}