Take your example of trying to reflect the line $y=x$ in the parabola $y=x^2$.
Consider a point on the parabola $(t,t^2)$. The tangent there has gradient $2t$, so its normal can be expressed as $y=-\frac1{2t}x+t^2+\frac12$. This intersects the line $y=x$ at the point $\left(\dfrac{2t^3+t}{2t+1},\dfrac{2t^3+t}{2t+1}\right)$. So its reflected point has the parametric form $$\left(\dfrac{-2t^3+4t^2+t}{2t+1},\dfrac{2t^3+2t^2-t}{2t+1}\right)$$ which as far as I am concerned is the solution. I suspect trying to write this in the form $y=f(x)$ may involve solving at least a messy cubic equation.
The red curve below is what I think is the reflection of the green line in the blue parabola looks like. Much of it may seem counterintuitive, but at least the red curve passes through the points $(0,0)$ and $(1,1)$ as expected.
Points on the green line above about $(3.0354,3.0354)$ get reflected three times in the parabola: in the right of the parabola onto the part of the higher red curve going to the top left, in the left of the parabola onto the lower red curve going to the top left, and in the part of the parabola roughly between $(-0.802, 0.644)$ and $(-0.5,0.25)$ onto the part of the red curve going to the bottom left. Points on the green line below about $(3.0354,3.0354)$ get reflected once near the centre of the parabola roughly between $(-0.5,0.25)$ and $(1.834, 3.363)$ onto the central loop and the part of the red curve going off to the top right.
Anyways, I can confirm that the curve provided by @Henry is indeed correct.
– Edoardo Serra Jul 16 '23 at 18:42