It would be nice if this was listed in some table of Fourier transforms. I looked at Messiah's book in QM and Pauli's book on Chemistry. Neither of those had anything. Mathematica does not simplify the integral. I did not see much in Stein and Weiss. You could try to calculate the integral of all the spherical harmonics over that set, and try to obtain a series from that. In principle, I believe that you can get it down to a 1d integral to put into Mathematica, by direct calculus techniques. But I think Mathematica would not do that integral. Before I state those calculations, it feels to me like a physics approach would be to try to do some Gaussian integrals.
Define $F(\kappa,\lambda) = |\mathbb{S}^+|^{-1} \int_{\mathbb{S}^+} e^{i (\kappa x + \lambda z)}\, d^2S(x,y,z)$ for $\mathbb{S}^+$ being the restriction of the 2d sphere $\mathbb{S}$ to the upper half-space $\{(x,y,z) : z>0\}$ and $d^2S(x,y,z)$ being the surface measure on the sphere. So $|\mathbb{S}^+|=2\pi$. Then something that would be easy to calculate is the average $\int_0^{\infty} \sqrt{\frac{2}{\pi}}\, F(r\kappa,r\lambda) r^2 e^{-r^2/2}\, dr$ because that will give Gaussian integrals
$$
\frac{2}{(2\pi)^{3/2}}\,
\int_{\mathbb{R}^3} \left(e^{i \kappa x} e^{-x^2/2}\right) e^{-y^2/2} \left(e^{i \lambda z} e^{-z^2/2} \mathbf{1}_{(0,\infty)}(z)\right)\, dx\, dy\, dz
$$
which equals $e^{-(\kappa^2+\lambda^2)/2} \left(1 + \operatorname{erf}\left(\frac{i \lambda}{\sqrt{2}}\right)\right)$ according to Mathematica. Now by symmetry and scaling, you have $\int_0^{\infty} \frac{\sqrt{2}}{\sqrt{\pi}\, \sigma^3} F(r\kappa,r\lambda) r^2 e^{-r^2/(2\sigma^2)}\, dr$ equals $e^{-\sigma^2(\kappa^2+\lambda^2)/2} \left(1 + \operatorname{erf}\left(\frac{i \lambda \sigma}{\sqrt{2}}\right)\right)$ for each $\sigma>0$.
But I do not see how to go back from there to get the Dirac-delta function in $r$, which is what I feel a physicist would try to do. (It might be basic and I am forgetting.)
You can analytically continue the Fourier transform to elements $\underline{\xi} \in \mathbb{C}^3$. But generally for an analytic function on $\mathbb{C}$ there is no unique way to go from the real part of the function on $\mathbb{R}$ to recover the whole function.
In terms of getting down to 1 integral, part of it uses the incomplete Gamma function, which is just another way of writing a particular integral, and part of it uses the Bessel function (which is another way of writing a different integral). But in case it is useful, here is part of it:Set $R=1$. You can rescale the radius of the ball into the modulus of $\underline{\xi}$.
In terms of the Bessel function, $\pi^{-1} \int_0^{\pi} e^{i t \cos(\theta)}\, d\theta = J_0(t)$. Therefore, using cylindrical coordinates $\xi_{\rho} = \sqrt{\xi_1^2+\xi_2^2}$ for $\underline{\xi}=(\xi_1,\xi_2,\xi_3)$, your integral is
$$
\int_0^1
\left(
\int_0^{\pi/2} J_0\big(\xi_\rho r \sin(\theta)\big) \exp\left(i \xi_3 r \cos(\theta)\right)
\sin(\theta)\, d\theta\right)\, dr
$$
But Mathematica does not give me an integral for this. If we define $w=r \sin(\theta)$ and use the coordinates $r$ and $w$, so that $\sin(\theta)=w/r$, $dw=r \cos(\theta)\, d\theta$ (holding $r$ fixed) and $r \cos(\theta) = \sqrt{r^2-w^2}$ then we get
$$
\int_0^1
\left(
\int_0^{r} J_0\big(\xi_\rho w\big) \exp\left(i \xi_3 \sqrt{r^2-w^2}\right)
\frac{w/r}{\sqrt{r^2-w^2}}\, dw\right)\, dr
$$
Then we could interchange the order of the integral
$$
\int_0^1
J_0\big(\xi_\rho w\big) \left(
\int_w^{1} \exp\left(i \xi_3 \sqrt{r^2-w^2}\right)
\frac{w/r}{\sqrt{r^2-w^2}}\, dr\right)\, dw
$$
Now Mathematica will evaluate $\int_0^1 e^{i a x} (b^2+x^2)^{-1}\, dx$ for $a,b \in \mathbb{R}$ in terms of elementary functions and $\Gamma(0,z)=\int_z^{\infty} t^{-1} e^{-t}\, dt$. The expression is somewhat complicated. But using $t = \sqrt{r^2-w^2}/\sqrt{1-w^2}$, you can get
$$
\int_w^{1} \exp\left(i \xi_3 \sqrt{r^2-w^2}\right)
\frac{w/r}{\sqrt{r^2-w^2}}\, dr\,
=\, \frac{w}{\sqrt{1-w^2}}\, \int_0^1 \frac{e^{i a t}}{b^2+t^2}\, dt\, ,
$$
where $a = \xi_3 \sqrt{1-w^2}$ and $b = w/\sqrt{1-w^2}$. Then you would multiply the result of whatever comes out of Mathematica by $J_0(\xi_\rho w)$ and you would have a single integral over $w$.
