You've made a good start. However, rather than trying to solve for $a$ as a quadratic equation, as suggested by J.G.'s comment, we instead get from your reduced equation that
$$\begin{equation}\begin{aligned}
a^2 + b^2 &= 17^2(a - b) \\
a^2 - 17^{2}a + b^2 + 17^{2}b & = 0 \\
4a^2 - 4\times 17^{2}a + 4b^2 + 4\times 17^{2}b & = 0 \\
(2a - 17^2)^2 + (2b + 17^2)^2 &= 2(17^2)^2
\end{aligned}\end{equation}$$
As shown in this answer, the positive integer solutions for $u^2 + v^2 = 2w^2$ come from the positive coprime integers $m$ and $n$, and a positive integer $k$, where
$$u = k\,\lvert n^2 + 2mn - m^2\rvert, \; v = k\,\lvert m^2 + 2mn - n^2\rvert, \; w = k(m^2 + n^2)$$
Due to the symmetry, we can WLOG have $m \ge n$. In our case, we have $k(m^2 + n^2) = 17^2$, so $k = 1, 17 $ or $17^2$, but $k = 17^2$ requires $n = 0$, so it's not allowed. Also, since $b \gt 0$, we require either $u$ or $v$ to be $\gt 17^2$.
First, with $k = 1$, we have $m^2 + n^2 = 17^2$. This is a primitive Pythagorean triple, so since $17 = 4^2 + 1^2$, we have the only case being $m = 4^2 - 1^2 = 15$ and $n = 2\times 4\times 1 = 8$. Thus,
$$2b + 289 = 15^2 + 2\times 15\times 8 - 8^2 = 401 \;\;\to\;\; b = 56$$
$$2a - 289 = \pm(8^2 + 2\times 15\times 8 - 15^2) = \pm 79 \;\;\to\;\; a = 105, 184$$
Next, with $k = 17$, we have $m^2 + n^2 = 17$, so $m = 4$ and $n = 1$. This then gives
$$2b + 289 = 17(4^2 + 2\times 4\times 1 - 1^2) = 391 \;\;\to\;\; b = 51$$
$$2a - 289 = \pm 17(1 + 2\times 4\times 1 - 4^2) = \pm 119 \;\;\to\;\; a = 85, 204$$
Multiplying the $a$ and $b$ results by $7$ gives that the positive solutions of the original equation are
$$(x,y) \in \lbrace(735, 392), (1288, 392), (595, 357), (1428, 357)\rbrace$$