$$
r\ddot \theta+2\dot r \dot \theta = 0 \implies 2\frac{\dot r}{r} + \frac{\ddot\theta}{\dot\theta} = 0 \implies \ln(r^2\dot\theta) = C \implies r^2\dot\theta = C \tag{1}
$$
By the condition $F=mr(0) \dot \theta(0)^2$ and the equation
$$
\ddot r-r\dot \theta^2+\frac{F}{m}=0 \tag{2}
$$
at $t= 0$ you can see that $\ddot r(0) = 0$. Substituting $(1)$ in $(2)$ to eliminate $\dot \theta$ yields
$$
\ddot r -C^2\frac{1}{r^3}+\frac{F}{m}=0 \tag{3}
$$
Now taking all derivatives of $(3)$ at $t = 0$, knowing that $\ddot r(0) = \dot r(0) = 0$, should show recursively that $\frac{d^n}{dt^n}r(0) = 0, n\ge 1$, so the Taylor expansion of $r(t)$, (if it has one,) will have only one nonvanishing term, namely the constant one $r(0) \ne 0$, (it is essential for this method that this term to be nonzero).
This result and the same idea can be use to find $\frac{d^n}{dt^n}\theta(0) = 0, n \ge 2$ from the equation ($1$) rewritten as
$$
\dot \theta = \frac{C}{r^2}
$$
Uniqueness
Set $\dot r = u$ and $\dot \theta = v$, then
\begin{align}
\dot r &= u \\
\dot \theta &= v \\
\dot u &= rv^2-\frac{F}{m} \\
\dot v &= -2\frac{u}{r} v \\
\end{align}
The function
$$
f:\mathbb R^4\backslash\{(0,\theta,u,v)\} \to \mathbb R^4, \qquad (r,\theta,u,v) \mapsto \left( u,v,rv^2-\frac{F}{m},-2\frac{u}{r} v \right)
$$
is locally Lipschitz in every neighborhood with $r \ne 0$ because it is differentiable. So the uniqueness of the solution follows, as the "mathcounterexamples.ne" noted, from the Picard-Lindelöf Theorem.
For my solution above I didn't like invoking the Taylor series, but I couldn't find other ways in the time being
– Physor Oct 23 '20 at 15:04