Here are a few approaches. First let you integral be:
\begin{equation}
I = \int_0^\infty e^{-t^2}\cos\left(t^2\right)\:dt
\end{equation}
Method 1:
Introduce
\begin{equation}
J(a) = \int_0^\infty e^{-t^2}\cos\left(at^2\right)\:dt
\end{equation}
Observe that $I = J(1)$ and $J(0) = \int_0^\infty e^{-t^2} = \frac{\sqrt{\pi}}{2}$. From here you can
Method 1.1:
Use Laplace Transforms:
\begin{align}
J(a) &= \mathscr{L}_{s \rightarrow a}^{-1} \left[ \mathscr{L}_{a \rightarrow s}\left[J(a) \right] \right] = \mathscr{L}_{s \rightarrow a}^{-1}\left[ \mathscr{L}_{a \rightarrow s}^{-1}\left[ \int_0^\infty e^{-t^2}\cos\left(at^2\right)\:dt \right]\right]\\
&= \mathscr{L}_{s \rightarrow a}^{-1}\left[ \int_0^\infty e^{-t^2}\mathscr{L}_{a \rightarrow s}\left[\cos\left(at^2\right)\right]\:dt \right]= \mathscr{L}_{s \rightarrow a}^{-1}\left[ \int_0^\infty \frac{se^{-t^2}}{s^2 + t^2}\:dt \right]
\end{align}
So you need to address the following integral:
\begin{equation}
\int_0^\infty \frac{se^{-t^2}}{s^2 + t^2}\:dt
\end{equation}
Method 1.2
Instead of using the Laplace Transform operator, you can differentiate instead to form a differential equation. Thus you create a DE of the form
\begin{equation}
J'(a) = F(J(a), a)
\end{equation}
Here we have
\begin{equation}
J'(a) = \int_0^\infty e^{-t^2} \frac{\partial}{\partial a} \left[\cos\left(at^2\right)\right]dt = \int_0^\infty e^{-t^2} \cdot -t^2 \sin\left(a t^2\right)\:dt
\end{equation}
From here use integration by parts to form $F(J(a), a)$.
Please note that for both 1.1,1.2 you could use the alternative parametisation:
\begin{equation}
J(a) = \int_0^\infty e^{-at^2}\cos\left(t^2\right)\:dt
\end{equation}
Where $I = J(1)$ and $J(0) = \int_0^\infty \cos\left(t^2\right)\:dt = \frac{1}{2}\sqrt{\frac{\pi}{2}}$
Method 2:
Employ Complex representations. We know that:
\begin{equation}
\cos\left(t^2\right) = \Re \left[ e^{-it^2}\right]
\end{equation}
And thus,
\begin{equation}
I = \int_0^\infty e^{-t^2}\cos\left(t^2\right)\:dt = \Re \left[ \int_0^\infty e^{-t^2} e^{-it^2}\:dt \right] = \Re \left[ \int_0^\infty e^{-(1 + i)t^2} \:dt \right]
\end{equation}
This becomes simple to evaluate using:
\begin{equation}
\int_0^\infty e^{-at^2}\:dt = \frac{1}{2}\sqrt{\frac{\pi}{a}}
\end{equation}
Method 3:
An extension of Method 1 (and a much better approach IMO) is to employ a two parameter form of Feynman's Trick:
\begin{equation}
J(a,b) = \int_0^\infty e^{-at^2}\cos\left(bt^2\right)\:dt
\end{equation}
We observe that $I = J(1,1)$
From here, my personal recommendation is to employ whichever integral / complex representation / differential / etc operators you need to make the integral simple to evaluate. Now if we look at both functions of the integrand they both have rational functions as their Laplace Transformations, i.e.
\begin{equation}
\mathscr{L}_{a \rightarrow s} \left[ e^{-at^2}\right] = \frac{1}{s + t^2}, \qquad \mathscr{L}_{b \rightarrow w} \left[ \cos\left(bt^2\right)\right] = \frac{t^2}{w^2 + t^4}
\end{equation}
And so in applying both to $J(a,b)$:
\begin{equation}
\mathscr{L}_{a \rightarrow s} \mathscr{L}_{b \rightarrow w} \left[ J(a,b) \right] = \int_0^\infty \frac{1}{s + t^2} \cdot \frac{t^2}{w^2 + t^4} \:dt
\end{equation}
Which can be evaluated easily using a Partial Fraction Decomposition combined with the Integral (that I cover here):
\begin{equation}
\int_0^\infty \frac{t^k}{\left(t^n + a\right)^m}\:dt = \frac{1}{n}a^{\frac{k + 1}{n} - m} B\left(m - \frac{k + 1}{n}, \frac{k + 1}{n}\right)
\end{equation}
From there, it's a matter of taking two inverse Laplace Transforms for resolve