I too was unable to derive a direct proof of the formula you have given; the following proof therefore is somewhat indirect. It rests on the following claim.
Claim. Given functions $H(t), \, f(t)$, the renewal equation $g(t) = H(t) + \int_{0}^t g(t-x) f(x) \mathrm d x$ has a unique solution.
Henceforth I will use the normal notation for convolution $\phi*\psi(t) = \int_0^t \phi(t-x)\psi(x) \mathrm d x$ so simplify notation. The equation above then reads: $g = H + g*f$.
My proof proceeds in two steps:
- Derive a renewal type equation for the variance.
- Prove that it is solved by the right hand side of the formula you gave.
Part 1 Instead of working with the variance, i will consider $g(t) = \textbf{E}[N(t)^2]$. Exactly as though we were deriving the usual renewal equation, we condition on the first jump time
\begin{align*}
g(t) & = \int_{0}^\infty \textbf{E}[N(t)^2 \, | \, X_1 = x] f(x) \mathrm d x \\
& = \int_0^t \textbf{E}[N(t)^2 \, | \, X_1 = x] f(x) \mathrm d x + \int_t^\infty 0 \mathrm \, d x \\
& = \int_0^t \textbf{E}[(1 + N(t-x) )^2] f(x) \mathrm d x \\
& = \int_0^t f(x)\Big(1 + 2 \textbf{E}[N(t-x)] + \textbf{E}[N(t-x)^2] \Big ) \mathrm d x \\
& = F(t) + 2(m * f)(t) + (g*f)(t).
\end{align*}
In particular, $g$ satisfies a renewal type equation with $H = F +2 m*f$.
Part 2 According to the formula you gave, we want to show $g(t) = 2(m*m')(t) + m(t)$, note that this is exactly the right hand side of your formula but with the $m(t)^2$ term removed, since we are not working with the variance. Substituting this formula into the right hand side of the renewal equation we have
\begin{align}
H + (2(m*m') + m)*f & = F + 2(m*f) +2(m*m')*f + m*f\\
& = (m - m*f) + 2(m*f) +2(m*m')*f + m*f\\
& = m + 2( m*f + m*m'*f),
\end{align}
where in the second line we substituted in the renewal equation $F = m - m*f$. It remeains then to show
\begin{align} m*f + m*m'*f =m*m'. \tag{1}
\end{align}
But again using the renewal equation, and differentiating it
\begin{align}
m' = (F + m*f)' = f +m'*f,
\end{align}
which when rearranged is
$$m'*f =m'-f $$
Substituting into (1)
\begin{align} m*f + m*m'*f & = m*f + m*(m' -f) \\
& =m*m',
\end{align}
as required.