The edge of the circle is on a plane that cuts the sphere normal to the radius, call it the cplane.
The line segment is on a circle disk that passes through the two points on the sphere and passes through the origin. This circle is on a plane, call it the lplane.
The circle and line segment are on the sphere.
The strategy is to find the equations of the planes and solve them along with the sphere equation.
If real solutions exist then check that they are on the line segment.
Three points are required to determine the equation of a plane.
$$(R-R_0) \cdot N = 0 \tag{1}$$
$R = (x,y,z)$.$R_0$ is a fixed point on the plane. $N$ is the normal to the plane. Three points are required to find the normal.
cplane
The center point $(R,\theta_c,\phi_c)$ is not on the cplane.
Edit $1$ start:
The normal to the cplane is the vector from the origin $\mathbf{O}$ to the center of the circle, $N_c = (R,\theta_c,\phi_c)$.
$C_1 = \left(R,\theta_c +\frac{r}{R},\phi_c\right)$ and $C_2 = \left(R,\theta_c -\frac{r}{R},\phi_c\right)$ are two points on the circle radius $r$.
Convert $N_c$ and $C_1$ into Cartesian coordinates. Take $N_c$ and $R_0 = C_1$ plug them into $(1)$ this will give the equation of the cplane of the form:
Edit $1$ end:
$$ax + by + cz = d \tag{2}$$
lplane
The origin $\mathbf{O}$, $P_1$ and $P_2$ are three points that determine the lplane.
Convert $\mathbf{O}$,$P_1$ and $P_2$ into Cartesian coordinates.
Take $N_l = P_1 \times P_2$, $R_0 = \mathbf{O}$ plug them into $(1)$. This will give an equation of the form:
$$ex + fy + gz = 0 \tag{3}$$
The Sphere
$$x^2 + y^2 + z^2 = R^2 \tag{4}$$
Solution
Solve equations $(2),(3),(4)$. It forms a quadratic in one variable.
If the solutions are complex then there is no solution, (complex conjugates).
If there is one real solution or a real double root then the lplane touches the cplane on the sphere at one point.
If there are two real solutions then the lplane cuts the cplane on the sphere at two points.
The line segment $P_1P_2$ is only part of the lplane.
Are the solutions inside the line segment $P_1P_2$?
Take the path from $P_1$ to $P_2$ in the direction of $P_2-P_1$. There is the pathological case where they are $\pi$ degrees opposite each other.
It requires a parameterized equation of the path.
Let $\widehat{V}$ denote the unit vector in the direction of $V$.
Find the unit vector $\widehat{Q}$ in the lplane normal to $\widehat{P_1}$.
$\widehat{Q} = \widehat{(\widehat{P_2}-(\widehat{P_2} \cdot \widehat{P_1})\widehat{P_1})} \tag{5}$
The $P_1$ $P_2$ path can be parameterized as:
$$P = \widehat{P_1}R\cos(\alpha) + \widehat{Q}R\sin(\alpha) \tag{6}$$
At $\alpha_1 = 0$ then $P = \widehat{P_1}R = P_1$.
At $\alpha_2$ let $P_2 = \widehat{P_1}R\cos(\alpha_2) + \widehat{Q}R\sin(\alpha_2)$.
Solve for $\alpha_2$ in the positive sense $[0,2\pi)$.
At the solution $S(S_x,S_y,S_z)$
$$S = \widehat{P_1}R\cos(\alpha_s) + \widehat{Q}R\sin(\alpha_s) \tag{7}$$
Solve for $\alpha_s$ in the positive sense $[0,2\pi)$.
If $0 \le \alpha_s \le \alpha_2$ then the solution is on the line segment.
Is the line segment inside the circle?
There must be two real solutions for the line segment to be inside the circle.
Parameterize the circular arc starting from solution $1$ $S_1$ to solution $2$ $S_2$. Determine the angle of $S_2$ relative to $S_1$. Determine the angle of the points $P_1$ and $P_2$ wrt $S_1$. If the angles of $P_1$ and $P_2$ are less than the angle of $S_2$ then they are inside the curve. Note the arc from $S_1$ to $S_2$ is the same curve as from $P_1$ to $P_2$ but the angle starts at $S_1$. The direction from $S_1$ to $S_2$ is key.This method avoids complicated $2\pi - angle$ shortest angle scenarios that I could not resolve.
Let $\widehat{T}$ be normal to $\widehat{S_1}$ in the $\mathbf{O}S_1S_2$ plane.
$$\widehat{T} = \widehat{ \widehat{S_2} - (\widehat{S_2}\cdot \widehat{S_1}) \widehat{S_1} } \tag{8} $$
$$SPath = \widehat{S_1}R\cos(\beta) + \widehat{T} R \sin(\beta) \tag{9}$$
Solve $\beta$ for $S_2$,$P_1$ and $P_2$ on the SPath in a positive sense $[0,2\pi)$.
$$S_2 = \widehat{S_1}R\cos(\beta_{S2}) + \widehat{T} R\sin(\beta_{S2}) \tag{10}$$
$$P_1 = \widehat{S_1}R\cos(\beta_1) + \widehat{T} R\sin(\beta_1) \tag{11}$$
$$P_2 = \widehat{S_1}R\cos(\beta_2) + \widehat{T} R\sin(\beta_2) \tag{12}$$
If $0 \le \beta_1,\beta_2 \le \beta_{S2}$ then the line segment is inside the circle.