Assume that $p\equiv1\pmod{31}$. I am fairly sure that these primes give you the largest number of non-isomorphic groups.
Assume first that the Sylow $p$-subgroup is isomorphic to $P=\Bbb{Z}_p^2$, that is, a $2$-dimensional vector space over the prime field of $p$ elements.
In this case $Aut(P)\cong GL_2(\Bbb{Z}_p)$. Because $31\mid p-1$, there is a multiplicative group $\mu_{31}$ of $31$st roots of unity in $\Bbb{Z}_p^*$. Let us fix a generator $g$ of $\mu_{31}$.
Consider the homomorphisms $\phi_j:C_{31}\to Aut(P)$ gotten by mapping a generator $c$ of $C_{31}$ to the diagonal matrix $\mathrm{diag}(g,g^j)$. Here $j$ takes values in the range $0\le j<31$. We can then form the semidirect product
$$
G_j=P\rtimes_{\phi_j}C_{31}.
$$
Observe that if $j>0$ and $j'$ is the multiplicative inverse of $j$ modulo $31$, i.e. $jj'\equiv1\pmod{31}$, then $\phi_j(c^{j'})=\mathrm{diag}(g^{j'},g)$ - a matrix conjugate to $\phi_{j'}(c)$. This implies that $G_j\cong G_{j'}$. On the other hand, if $j''\notin\{0,j,j'\}$ then it seems to me that $G_{j''}$ is not isomorphic to $G_j$ (see the next paragraph).
For if $j\neq0$ then $c$ does not commute with any non-identity element of $P$. This is because $\phi_j(c)$ does not have $1$ as an eigenvalue. The same applies to all non-trivial powers of $c$. It follows that there are no element of order $31p$ in $G_j$, so all the elements of $G_j\setminus P$ have order $31$. Thus $G_j$ has $p^2$ Sylow $31$-subgroups. All of those are conjugate to each other, and each contains two elements with an eigenvalue $g$ on $P$, namely the conjugates of $c$ and $c^{j'}$.
The other eigenvalues of those elements are thus $g^j$ and $g^{j'}$ respectively.
Any isomorphism $f:G_j\to G_{j''}$ would have to preserve this pair of eigenvalues, implying the claim $j''\in\{j,j'\}$.
Similarly, we see that $G_0$ is not isomorphic to any other $G_j$. This is because in $G_0$ we have elements of order $31p$ as any eigenvector of $\phi_0(c)$ belonging to eigenvalue $1$ commutes with $c$.
Let's take stock. $j=j'$ if and only if $j\equiv\pm1\pmod{31}$. The remaining $28$ choices of $j$ split into $14$ pairs $(j,j')$. Altogether we get $17$ non-abelian pairwise non-isomorphic semidirect product $(C_p\times C_p)\rtimes C_{31}$.
In addition to two non-isomorphic abelian groups of order $31p^2$ we also have a semidirect product $C_{p^2}\rtimes C_{31}$ coming from embedding $C_{31}$ into $Aut(C_{p^2})\cong C_{p(p-1)}$.
Barring mistakes and/or oversights I arrived at twenty non-isomorphic groups of order $31p^2$ for any prime $p\equiv1\pmod{31}$.
The order of $GL_2(\Bbb{Z}_p)$ is $p(p-1)^2(p+1)$. The subgroups of order $p+1$ are cyclic, so I doubt we will get as many non-isomorphic semidirect products when $31\mid p+1$.
See this thread and others linked to it for more information about when semidirect products of two given groups are isomorphic.