You've made an excellent start with integration by parts!
Let the integral be $\color{navy}{\mathcal{I}}$. We want to evaluate
$$\bbox[15px, #E0F8FF, border:5px groove #007BFF]{\color{navy}{\mathcal{I}} := \int_{0}^{\infty} \frac{\arctan(x^3)}{x^2} \ dx}$$
Using integration by parts with $u_0 = \arctan(x^3)$ and $dv_0 = \frac{1}{x^2} dx$.
Then $du_0 = \frac{3x^2}{1+x^6} dx$ and $v_0 = -\frac{1}{x}$.
So,
$$\begin{align*}
\color{navy}{\mathcal{I}} &= \left[ -\frac{\arctan(x^3)}{x} \right]_0^\infty - \int_0^\infty \left(-\frac{1}{x}\right) \frac{3x^2}{1+x^6} \ dx \\
&= \left[ -\frac{\arctan(x^3)}{x} \right]_0^\infty + 3 \int_0^\infty \frac{x}{1+x^6} \ dx
\end{align*}$$
Let's evaluate the boundary term:
- As $x \to \infty$: $\arctan(x^3) \to \frac{\pi}{2}$. So, $-\frac{\arctan(x^3)}{x} \to -\frac{\pi/2}{x} \to 0$.
- As $x \to 0^+$: We can use the Taylor series for $\arctan(y) \approx y$ for small $y$. So, $\arctan(x^3) \approx x^3$. Thus, $-\frac{\arctan(x^3)}{x} \approx -\frac{x^3}{x} = -x^2 \to 0$.
Therefore, the boundary term is $\require{cancel}\cancelto{0}{\left[ -\frac{\arctan(x^3)}{x} \right]_0^\infty} = 0 - 0 = 0$.
The integral simplifies to:
$$\color{navy}{\mathcal{I}} = 3 \int_0^\infty \frac{x}{1+x^6} \ dx$$
Let $\color{teal}{\mathcal{J}_1} = \int_0^\infty \frac{x}{1+x^6} \ dx$. So, $\color{navy}{\mathcal{I}} = 3 \color{teal}{\mathcal{J}_1}$.
Hints for solving $\displaystyle \color{teal}{\mathcal{J}_1} = \int_0^\infty \frac{x}{1+x^6} \ dx$:
Substitution: The term $x^6$ suggests a substitution. Try $u = x^2$ (or $x \mapsto \sqrt{u}$). How does the integral transform? You should arrive at an integral of the form $C \int_0^\infty \frac{1}{1+u^3} du$. Let's call this $J_2 = \int_0^\infty \frac{1}{1+u^3} du$.
Symmetry/Reciprocal Substitution for $J_2$: For integrals of the form $\int_0^\infty f(u) du$, the substitution $u \mapsto 1/t$ (so $du = -1/t^2 dt$) is often very powerful. Apply this to $J_2$. You should find that $J_2$ is also equal to $\int_0^\infty \frac{t}{1+t^3} dt$.
Combine and Simplify: Using the result from hint 2, you have two expressions for $J_2$. Consider $2J_2 = \int_0^\infty \frac{1}{u^3+1} du + \int_0^\infty \frac{u}{u^3+1} du = \int_0^\infty \frac{1+u}{u^3+1} du$. Simplify the integrand $\frac{1+u}{u^3+1}$ by factoring the denominator.
Evaluate the Simplified Integral: The simplified integral should be of the form $\int_0^\infty \frac{1}{au^2+bu+c} du$. Complete the square in the denominator. This will lead to an $\arctan$ function.
Back-substitute: Once you find $J_2$, remember that $I = 3 J_1$ and $J_1 = \frac{1}{2} J_2$.
Try working through these hints. The full solution follows below.
Full Solution:
Following the hints:
We have $\color{navy}{\mathcal{I}} = 3 \color{teal}{\mathcal{J}_1} = 3 \int_0^\infty \frac{x}{1+x^6} \ dx$.
Let $u = x^2$. Then $du = 2x \ dx$, so $x \ dx = \frac{1}{2} du$.
The limits remain $0$ to $\infty$.
$$\color{teal}{\mathcal{J}_1} = \int_0^\infty \frac{1}{1+u^3} \cdot \frac{1}{2} du = \frac{1}{2} \int_0^\infty \frac{1}{1+u^3} \ du$$
Let
$$\displaystyle \bbox[10px, #F0FFF0, border:3px dashed #28A745]{\color{teal}{\mathcal{J}_2} := \int_0^\infty \frac{1}{1+u^3} \ du}$$
so that $\color{navy}{\mathcal{I}} = 3 \color{teal}{\mathcal{J}_1} = \frac{3}{2} \color{teal}{\mathcal{J}_2}$.
To evaluate $\color{teal}{\mathcal{J}_2}$, let $u \mapsto 1/t$, so $du = -1/t^2 dt$.
When $u=0$, $t=\infty$. When $u=\infty$, $t=0$.
$$\color{teal}{\mathcal{J}_2} = \int_\infty^0 \frac{1}{1+(1/t)^3} \left(-\frac{1}{t^2}\right) dt = \int_0^\infty \frac{1}{\frac{t^3+1}{t^3}} \frac{1}{t^2} dt = \int_0^\infty \frac{t^3}{t^3+1} \frac{1}{t^2} dt = \int_0^\infty \frac{t}{t^3+1} dt$$
Since this is a definite integral, the variable name doesn't matter, so $\color{teal}{\mathcal{J}_2} = \int_0^\infty \frac{u}{u^3+1} du$.
Now we have two expressions for $\color{teal}{\mathcal{J}_2}$:
$$\color{teal}{\mathcal{J}_2} = \int_0^\infty \frac{1}{u^3+1} du \quad \text{and} \quad \color{teal}{\mathcal{J}_2} = \int_0^\infty \frac{u}{u^3+1} du$$
Adding them gives:
$$2\color{teal}{\mathcal{J}_2} = \int_0^\infty \frac{1}{u^3+1} du + \int_0^\infty \frac{u}{u^3+1} du = \int_0^\infty \frac{1+u}{u^3+1} du$$
We know $u^3+1 = (u+1)(u^2-u+1)$. So, for $u \ge 0$ (and $u \neq -1$, which is true on our domain of integration):
$$\frac{1+u}{u^3+1} = \frac{\cancel{(1+u)}}{\cancel{(u+1)}(u^2-u+1)} = \frac{1}{u^2-u+1}$$
Thus,
$$2\color{teal}{\mathcal{J}_2} = \int_0^\infty \frac{1}{u^2-u+1} du$$
Complete the square in the denominator: $u^2-u+1 = \left(u - \frac{1}{2}\right)^2 - \frac{1}{4} + 1 = \left(u - \frac{1}{2}\right)^2 + \frac{3}{4}$.
$$2\color{teal}{\mathcal{J}_2} = \int_0^\infty \frac{1}{\left(u - \frac{1}{2}\right)^2 + \left(\frac{\sqrt{3}}{2}\right)^2} du$$
This is a standard arctangent integral: $\int \frac{1}{y^2+a^2} dy = \frac{1}{a}\arctan\left(\frac{y}{a}\right)$.
Here $y = u-1/2$ and $a = \sqrt{3}/2$.
$$\begin{align*}
2\color{teal}{\mathcal{J}_2} &= \left[ \frac{1}{\sqrt{3}/2} \arctan\left(\frac{u-1/2}{\sqrt{3}/2}\right) \right]_0^\infty \\
&= \left[ \frac{2}{\sqrt{3}} \arctan\left(\frac{2u-1}{\sqrt{3}}\right) \right]_0^\infty \\
&= \frac{2}{\sqrt{3}} \left( \lim_{u\to\infty} \arctan\left(\frac{2u-1}{\sqrt{3}}\right) - \arctan\left(\frac{-1}{\sqrt{3}}\right) \right) \\
&= \frac{2}{\sqrt{3}} \left( \frac{\pi}{2} - \left(-\frac{\pi}{6}\right) \right) \\
&= \frac{2}{\sqrt{3}} \left( \frac{\pi}{2} + \frac{\pi}{6} \right) = \frac{2}{\sqrt{3}} \left( \frac{3\pi+\pi}{6} \right) = \frac{2}{\sqrt{3}} \left(\frac{4\pi}{6}\right) = \frac{2}{\sqrt{3}} \left(\frac{2\pi}{3}\right) = \color{maroon}{\frac{4\pi}{3\sqrt{3}}}
\end{align*}$$
So, $\color{teal}{\mathcal{J}_2} = \frac{1}{2} \cdot \color{maroon}{\frac{4\pi}{3\sqrt{3}}} = \color{maroon}{\frac{2\pi}{3\sqrt{3}}}$.
We can state this intermediate result beautifully:
$$\bbox[15px, #E6FFF3, border:5px groove #1E90FF]{\color{teal}{\mathcal{J}_2} = \int_0^\infty \frac{1}{1+u^3} \ du = \color{maroon}{\frac{2\pi}{3\sqrt{3}}}}$$
Finally, substitute back into $\color{navy}{\mathcal{I}} = \frac{3}{2} \color{teal}{\mathcal{J}_2}$:
$$\color{navy}{\mathcal{I}} = \frac{3}{2} \cdot \color{maroon}{\frac{2\pi}{3\sqrt{3}}} = \color{maroon}{\frac{\pi}{\sqrt{3}}}$$
Rationalizing the denominator gives $\color{maroon}{\frac{\pi\sqrt{3}}{3}}$.
Thus,
$$\bbox[20px, #FFF0F5, border:5px groove #DB7093 ]{\int_{0}^{\infty} \frac{\arctan(x^3)}{x^2} \ dx = \color{maroon}{\frac{\pi}{\sqrt{3}}}}$$
Alternative for $\displaystyle \color{teal}{\mathcal{J}_2} = \int_0^\infty \frac{1}{1+u^3} du$ using Partial Fractions:
One could also evaluate $\color{teal}{\mathcal{J}_2}$ using partial fractions. We set up the decomposition:
$$\frac{1}{u^3+1} = \frac{1}{(u+1)(u^2-u+1)} = \frac{\color{purple}{A}}{u+1} + \frac{\color{purple}{B}u+\color{purple}{C}}{u^2-u+1}$$
Multiplying by $(u+1)(u^2-u+1)$ gives $1 = \color{purple}{A}(u^2-u+1) + (\color{purple}{B}u+\color{purple}{C})(u+1)$.
To find $\color{purple}{A}, \color{purple}{B}, \color{purple}{C}$:
- For $u=-1$: $1 = \color{purple}{A}(1-(-1)+1) + (\color{purple}{B}(-1)+\color{purple}{C})(-1+1) = \color{purple}{A}(3) + 0 \implies 3\color{purple}{A} = 1 \implies \boldsymbol{\color{purple}{A}=1/3}$.
- Comparing coefficients of $u^2$: $0 \cdot u^2 = \color{purple}{A} u^2 + \color{purple}{B} u^2 \implies \color{purple}{A}+\color{purple}{B} = 0 \implies \color{purple}{B} = -\color{purple}{A} \implies \boldsymbol{\color{purple}{B} = -1/3}$.
- Comparing constant terms: $1 = \color{purple}{A}(1) + \color{purple}{C}(1) \implies \color{purple}{A}+\color{purple}{C} = 1 \implies \color{purple}{C} = 1-\color{purple}{A} = 1-1/3 \implies \boldsymbol{\color{purple}{C} = 2/3}$.
So, $\displaystyle \frac{1}{u^3+1} = \frac{1}{3(u+1)} + \frac{-\frac{1}{3}u+\frac{2}{3}}{u^2-u+1} = \frac{1}{3}\left(\frac{1}{u+1} - \frac{u-2}{u^2-u+1}\right)$.
The indefinite integral is:
$$\begin{align*}
\int \frac{1}{1+u^3}du &= \frac{1}{3}\int \left(\frac{1}{u+1} - \frac{u-1/2-3/2}{(u-1/2)^2+3/4}\right)du \\
&= \frac{1}{3}\left(\ln|u+1| - \int \frac{u-1/2}{(u-1/2)^2+3/4}du + \frac{3}{2}\int \frac{1}{(u-1/2)^2+(\sqrt{3}/2)^2}du\right) \\
&= \frac{1}{3}\left(\ln|u+1| - \frac{1}{2}\ln(u^2-u+1) + \frac{3}{2} \frac{1}{\sqrt{3}/2}\arctan\left(\frac{u-1/2}{\sqrt{3}/2}\right)\right)+K \\
&= \frac{1}{3}\left(\ln|u+1| - \frac{1}{2}\ln(u^2-u+1) + \sqrt{3}\arctan\left(\frac{2u-1}{\sqrt{3}}\right)\right)+K \\
&= \frac{1}{3} \left[ \ln\left(\frac{|u+1|}{\sqrt{u^2-u+1}}\right) + \sqrt{3}\arctan\left(\frac{2u-1}{\sqrt{3}}\right) \right]+K
\end{align*}$$
Evaluating the definite integral $\color{teal}{\mathcal{J}_2} = \int_0^\infty \frac{1}{1+u^3}du$:
$$\color{teal}{\mathcal{J}_2} = \frac{1}{3} \left[ \ln\left(\frac{u+1}{\sqrt{u^2-u+1}}\right) + \sqrt{3}\arctan\left(\frac{2u-1}{\sqrt{3}}\right) \right]_0^\infty$$
For the logarithmic term:
As $u \to \infty$, $\ln\left(\frac{u+1}{\sqrt{u^2-u+1}}\right) = \ln\left(\frac{u(1+1/u)}{\sqrt{u^2(1-1/u+1/u^2)}}\right) = \ln\left(\frac{\cancel{u}(1+1/u)}{\cancel{u}\sqrt{1-1/u+1/u^2}}\right) \to \ln(1) = 0$.
At $u=0$, $\ln\left(\frac{1}{\sqrt{1}}\right) = \ln(1) = 0$.
So the logarithmic term contributes $\cancelto{0}{0-0}=0$ to the definite integral.
For the arctangent term:
$\sqrt{3}\left(\lim_{u\to\infty}\arctan\left(\frac{2u-1}{\sqrt{3}}\right) - \arctan\left(\frac{-1}{\sqrt{3}}\right)\right) = \sqrt{3} \left(\frac{\pi}{2} - \left(-\frac{\pi}{6}\right)\right) = \sqrt{3}\left(\frac{2\pi}{3}\right)$.
So,
$$\color{teal}{\mathcal{J}_2} = \frac{1}{3} \left(0 + \sqrt{3} \frac{2\pi}{3}\right) = \color{maroon}{\frac{2\pi\sqrt{3}}{9}} = \color{maroon}{\frac{2\pi}{3\sqrt{3}}}$$
This matches the result obtained by the symmetry trick.
Regarding your substitution $\boldsymbol{x^3 = \tan t}$ (Beta Function Approach):
This is also an elegant and valid approach. If $x^3 = \tan t$, then $x = (\tan t)^{1/3}$.
This implies $dx = \frac{1}{3}(\tan t)^{(1/3)-1} \sec^2 t \ dt = \frac{1}{3}(\tan t)^{-2/3}\sec^2 t \ dt$.
The limits of integration change: as $x \to 0$, $\tan t \to 0 \implies t \to 0$. As $x \to \infty$, $\tan t \to \infty \implies t \to \pi/2$.
The integral becomes:
$$\color{navy}{\mathcal{I}} = \int_0^{\pi/2} \frac{t}{((\tan t)^{1/3})^2} \cdot \frac{1}{3}(\tan t)^{-2/3}\sec^2 t \ dt = \frac{1}{3} \int_0^{\pi/2} t (\tan t)^{-4/3} \sec^2 t \ dt$$
We integrate by parts. Let $u_{part}=t$ and $dv_{part}=(\tan t)^{-4/3}\sec^2 t \ dt$.
Then $du_{part}=dt$. To find $v_{part}$, let $w = \tan t$, so $dw = \sec^2 t \ dt$.
$v_{part} = \int w^{-4/3} dw = \frac{w^{-4/3+1}}{-4/3+1} = \frac{w^{-1/3}}{-1/3} = -3(\tan t)^{-1/3}$.
$$\begin{align*}
\color{navy}{\mathcal{I}} &= \frac{1}{3} \left( \left[ t \cdot \left(-3(\tan t)^{-1/3}\right) \right]_0^{\pi/2} - \int_0^{\pi/2} \left(-3(\tan t)^{-1/3}\right) dt \right) \\
&= \cancelto{0}{\frac{1}{3} \left[ -3t(\tan t)^{-1/3} \right]_0^{\pi/2}} + \int_0^{\pi/2} (\tan t)^{-1/3} dt
\end{align*}$$
The boundary term vanishes:
- As $t \to (\pi/2)^-$, $t \to \pi/2$ and $\tan t \to \infty$, so $(\tan t)^{-1/3} \to 0$. The expression $-t(\tan t)^{-1/3} \to 0$.
- As $t \to 0^+$, $-t(\tan t)^{-1/3} = -t\left(\frac{\sin t}{\cos t}\right)^{-1/3} = -t (\cos t)^{1/3} (\sin t)^{-1/3}$. Since $\cos t \to 1$ and $\sin t \approx t$ for small $t$, this is $\approx -t \cdot 1 \cdot t^{-1/3} = -t^{2/3} \to 0$.
Thus, $\color{navy}{\mathcal{I}} = \int_0^{\pi/2} (\tan t)^{-1/3} dt = \int_0^{\pi/2} \cot^{1/3}(t) dt$.
This integral can be evaluated using the Beta function. Recall that $\int_0^{\pi/2} (\sin t)^a (\cos t)^b dt = \frac{1}{2} B\left(\frac{a+1}{2}, \frac{b+1}{2}\right)$.
We have $\cot^{1/3}(t) = (\cos t)^{1/3} (\sin t)^{-1/3}$. So $a=-1/3$ and $b=1/3$.
$$\begin{align*}
\color{navy}{\mathcal{I}} &= \int_0^{\pi/2} (\sin t)^{-1/3} (\cos t)^{1/3} dt \\
&= \frac{1}{2} B\left(\frac{-1/3+1}{2}, \frac{1/3+1}{2}\right) \\
&= \frac{1}{2} B\left(\frac{2/3}{2}, \frac{4/3}{2}\right) = \frac{1}{2} B\left(\frac{1}{3}, \frac{2}{3}\right)
\end{align*}$$
Using the relationship $B(x,y) = \frac{\Gamma(x)\Gamma(y)}{\Gamma(x+y)}$ and Euler's reflection formula $\Gamma(z)\Gamma(1-z) = \frac{\pi}{\sin(\pi z)}$:
$$\color{navy}{\mathcal{I}} = \frac{1}{2} \frac{\Gamma(1/3)\Gamma(2/3)}{\Gamma(1/3+2/3)} = \frac{1}{2} \frac{\Gamma(1/3)\Gamma(1-1/3)}{\Gamma(1)}$$
Since $\Gamma(1)=1$ and taking $z=1/3$ in the reflection formula:
$$\color{navy}{\mathcal{I}} = \frac{1}{2} \cdot \frac{\pi}{\sin(\pi/3)} = \frac{1}{2} \cdot \frac{\pi}{\sqrt{3}/2} = \color{maroon}{\frac{\pi}{\sqrt{3}}}$$
This confirms the result, arriving at the same beautiful conclusion:
$$\bbox[15px, #F5EEFF, border:5px groove #8A2BE2 ]{\color{navy}{\mathcal{I}} = \color{maroon}{\frac{\pi}{\sqrt{3}}}}$$
And we're done! $\blacksquare$
Cheers :)