You say that you got stuck when trying $k=\frac13$. That’s likely because that’s not the slope of either asymptote. If a line has direction vector $(l,m)$, then its slope is equal to $\frac ml$, not $\frac lm$. So, what you found by solving $3k^2+2k-1=0$ was the reciprocals of the slopes. You can, of course, get the slopes directly using your method, but the equation to be solved is instead $A+2Bk+Ck^2=0$.
The reciprocal of $-1$ is $-1$ again, so your method of adjusting $b$ so that there is no intersection worked accidentally. On the other hand, there is an infinite number of lines with slope $\frac13$ that don’t intersect this hyperbola, and that’s why you didn’t get a definitive result for that. Using the correct slope value of $3$, we end up with $(38-4b)x=b^2-10b-14$, and setting $b=\frac{19}2$ guarantees no intersection with the hyperbola.
There’s another method of finding the constant terms for the asymptotes that may or may not be easier to use. If you vary $F$ in the general equation, you get a family of hyperbolas with common asymptotes. For some value of $F$ the hyperbola degenerates into intersecting lines—the asymptotes themselves. Now, a joint equation of the asymptotes is $$(k_1x-y+b_1)(k_2x-y+b_2)=0.$$ Per the preceding observation, if you expand this expression, the coefficients of all but the constant term must be proportional to the corresponding coefficients in the original hyperbola equation. Comparing coefficients will result in a simple system of equations to solve for $b_1$ and $b_2$. (Indeed, when the quadratic part of the conic equation factors easily, as it does in this problem, you can go straight to comparing coefficients without going through solving the work of solving a quadratic equation.)
In this problem, we have $$(-x-y+b_1)(3x-y+b_2) = -3x^2-2xy+y^2+(3b_1-b_2)x-(b_1+b_2)y+b_1b_2.$$ Comparing this to the original system gives $$3b_1-b_2=-8 \\ b_1+b_2=10.$$ I trust that you can solve this system.
As for why you can find the direction vectors of the asymptotes by considering only the quadratic part of the equation, I’ll offer a projective-geometric explanation. A hyperbola intersects the line at infinity at two points; its asymptotes are the tangents to the hyperbola at those points. On the other hand, the point at infinity on a line corresponds to its direction vector. We can find these intersection points by homogenizing the equation to $$Ax^2+2Bxy+Cy^2+2Dxw+2Eyw+Fw^2=0$$ and then setting $w=0$.
Another way to understand this method of finding the asymptote slopes is to translate the hyperbola so that its center is at the origin. This doesn’t affect any of the second-degree terms, so the resulting equation is of the form $Ax^2+2Bxy+Cy^2=F'$. Per the earlier discussion, varying the constant term on the right-hand side produces a hyperbola with the same asymptotes, and it should be fairly obvious that at $F'=0$ you get intersecting lines. Since these lines both pass through the origin, the coordinates of any point on one of these lines also give a direction vector of that line.
eq1:3*x^2+2*x*y-y^2+8*x+10*y+14;solve([diff(eq1,x),diff(eq1,y)],[x,y]);[[x = -9/4,y = 11/4]]factor((-y^2)+2*x*y+10*y+3*x^2+8*x+14-((-(11/4)^2)+2*(-9/4)*(11/4)+10*(11/4)+3*(-9/4)^2+8*(-9/4)+14));-((2*y-6*x-19)*(2*y+2*x-1))/4– Jan-Magnus Økland May 19 '20 at 19:11