Here's an example that might help: Let $X$ have order $7$ and $Y$ have order $3$. These groups are cyclic; write $x$ for a generator of $X$ and $y$ for a generator of $Y$. Let $\phi\colon X \to X$ and $\psi\colon X \to X$ be the automorphisms $\phi(x) = x^2$, and $\psi(x) = x^4$.
Notice that $\phi^3(x) = x^{2^3} = x^8 = x$, and $\psi^3(x) = x^{4^3} = x^{64} = x$, so as automorphisms of $X$, $\phi$ and $\psi$ each have order three.
I claim that $X\rtimes_\phi Y \cong X\rtimes_\psi Y$. Indeed, (abusing notation slightly) $X\rtimes_\phi Y$ and $X\rtimes_\psi Y$ are generated by $x$ and $y$, so to specify a homomorphism, I just need to tell you where $x$ and $y$ go, and then check that the map I've written down indeed defines a homomorphism. I claim that the map $x \mapsto x$, $y \mapsto y^{-1}$ is such a homomorphism and moreover that it is an isomorphism. I'll leave it to you to check this.
So if the answer has to be "sometimes" and not "never," maybe we should reconsider what should be true based on this example. Aut$(X)$ is generated by $x \mapsto x^3$. There is an automorphism of Aut$(X)$ that sends the automorphism $x \mapsto x^3$ to the automorphism $x \mapsto x^5$. Under this automorphism of Aut$(X)$, $\phi$ is mapped to $\psi$.
So some questions to investigate: if we have $\phi$ and $\psi$ in Aut$(X)$, will it be the case that $X\rtimes_\phi Y\cong X\rtimes_\psi Y$ whenever there is some automorphism $f\colon$ Aut$(X) \to $ Aut$(X)$ such that $f(\phi) = \psi$? Is this the only condition?