As promised in comments, here is a full answer:
The box product of infinitely many copies of the Khalimsky line is a counterexample. Indeed, the Khalimsky line is finitely generated and connected, but the box product is not connected - the collection of bounded sequences is a proper clopen subset.
Now, I want to provide an equivalent condition for when $\square_i X_i$ is connected. For this purpose, recall that the specialization preorder on a topological space is defined by $x \leq y$ iff $x \in \overline{\{y\}}$. Recall that in a finitely generated space $X$, the specialization preorder fully determines open and closed sets - open (resp., closed) sets are exactly sets that are upward (resp., downward) closed under the specialization preorder. We impose a simple graph structure on $X$ by declaring $x, y \in X$, $x \neq y$ to be connected iff $x \leq y$ or $y \leq x$. Let $d(x, y)$ denote the length of the shortest path between $x$ and $y$ and let $\text{diam}(X) = \sup_{x, y \in X} d(x, y)$.
Now, since clopen sets are exactly sets that are both upward and downward closed, we see that a subset of $X$ is clopen iff it is a union of connected components of $X$ as a graph. Thus, $X$ is connected as a topological space iff it is connected as a graph, i.e., iff $d(x, y) < \infty$ for all $x, y \in X$. Now, in $\square_i X_i$, it is easy to check that $(x_i) \leq (y_i)$ iff $x_i \leq y_i$ for all $i$. From this, one can see that,
$$\sup_i d(x_i, y_i) \leq d((x_i), (y_i)) \leq \sup_i d(x_i, y_i) + 1$$
(Whether $d((x_i), (y_i))$ equals $\sup_i d(x_i, y_i)$ or is larger by $1$ depends on whether the directions, i.e., upwards or downwards in the specialization preorders, of the first steps in the shortest paths from $x_j$ to $y_j$, among those $j$ s.t. $d(x_j, y_j) = \sup_i d(x_i, y_i)$, are the same.) Hence, we immediately obtain that,
$\square_i X_i$ is disconnected iff there exist distinct $i_1, i_2, \cdots$ s.t. $\text{diam}(X_{i_n}) \geq n$ for all $n$. Equivalently, $\square_i X_i$ is connected iff there are only finitely many $X_i$ with infinite diameters and all the remaining $X_i$ have uniformly bounded finite diameters.
For the Khalimsky line, its diameter is infinite, so we see that infinite box products of those are disconnected. Any finite box product clearly satisfies the condition for connectedness above, so we retrieve the result that finite box products of finitely generated connected spaces are connected. Note that every hyperconnected or ultraconnected finitely generated space has diameter at most $2$, so any box product of hyperconnected and ultraconnected finitely generated spaces is connected. The same goes for any box product of connected finite spaces with uniformly bounded cardinalities, etc.
