It turns out that it is indeed decidable in polynomial time in the number of states whether a given machine is both generic and distinguishable. Here is a sketch of an algorithm and an outline of the reason it works.
Distinguishability
Define the relation $
\renewcommand\SS{\mathcal{S}}
\newcommand\OO{\mathcal{O}}
\newcommand\oo{\mathfrak{o}}
\newcommand\N{\mathbb{N}}
\sim_k \subset \SS\times \SS$ for $k \in \N$ as follows.
- $s_1 \sim_0 s_2$ if and only if $\oo(s_1) = \oo(s_2)$
- For $k \in \N$, $s_1 \sim_{k+1} s_2$ if and only if both
- $s_1\sim_k s_2$ and
- For all $\sigma\in \Sigma$ there exist $\tilde{s}_i\in \tau(s_i, \sigma)$ such that $\tilde{s}_1\sim_k \tilde{s}_2$.
I.e. $\sim_k$ is the relation of states being indistinguishable in $k$ steps. Note that $\sim_{k+1}$ is a refinement of $\sim_k$, and each $\sim_k$ is reflexive and symmetric, but not necessarily transitive (cf. [Moore56, Theorem 6]). Note also that if $\sim_{k+1}$ coincides with $\sim_k$ for some $k$, then all the higher relations are the same too, so $\sim_k$ must stabilize for $k\geq {|\SS|(|\SS|-1)}/2 = O(|\SS|^2)$. This provides a polynomial algorithm to verify distinguishability.
(Aside conjecture: $\sim_k$ stabilizes for $k \geq |\SS| - 1$. I can prove this for $\Sigma = 1$, but not for higher.)
Genericness
Deciding whether a given machine is "generic" in the above sense is actually PSPACE hard (see remark at the end). However, if a machine is distinguishable, then deciding genericness becomes easy because of the following.
Definition.
Call a machine complete if it satisfies the following
- If $s_1, s_2 \in \SS$ with $\oo(s_1) = \oo(s_2)$, then there is some $\sigma\in \Sigma$ such that $\tau(s_1, \sigma) \cap \tau(s_2, \sigma) = \varnothing$.
- We have for each $o\in \OO$, $$\SS^\Sigma = \bigcup_{s\in \oo^{-1}(o)} \tau(s, -),$$ where
$$\tau(s, -) = \prod_{\sigma\in \Sigma}\tau(s, \sigma) \subset \SS^\Sigma.$$
We can write conditions 1 and 2 together concisely as
$$
\OO\times\SS^\Sigma = \bigsqcup_{s\in \SS} \oo(s)\times\tau(s, -).
$$
Lemma. A complete machine is generic.
Proof.
A complete machine comes with a map
$$
\rho: \OO\times \SS^\Sigma \to \SS.
$$
Iterating, we get maps
$$
\rho^{\leq k}: \OO^{\Sigma^{\leq k}} \times \SS^{\Sigma^{k+1}} \to \SS.
$$
For $\beta: \Sigma^{\leq k} \to \OO$, we have $\beta \in C^{\leq k}(s)$ for $s = \rho^{\leq k}(\beta, \bar{s})$, and any $\bar{s} \in \SS^{\Sigma^{k+1}}$, where $C^{\leq k}(s)\subset \OO^{\Sigma^{\leq k}}$ is the subset of behaviors compatible with initial state $s$ as in the question, but restricted to $\Sigma^{\leq k}$. Since each $C(s) \subset \OO^{\Sigma^*}$ is closed in the product topology, this is sufficient to see that the machine is generic.
Lemma. A machine that's both distinguishable and generic must be complete.
Proof. Condition 1 in the definition of a complete machine clearly follows from distinguishability. Condition 2 requires a bit more work but turns out to also hold.
Thus a machine is both generic and distinguishable if and only if it's complete and distinguishable, and the latter can be checked in polynomial time, as outlined so far (note that Condition 2 of completeness is a priori exponential in the size of the input alphabet, but in the presence of Condition 1 it can be verified by simply summing the sizes of the $\tau(s, -)$).
Remark
To see that genericness in isolation is PSPACE hard, consider the following reduction, due to Vanessa.
By [KRS09, Lemma 2], universality for all-accepting NFAs with at least 2 input characters is PSPACE hard.
Given an all-accepting NFA with input alphabet $\OO$ (note: the inputs to the NFA are the outputs of our Moore machine), set of states $\SS_{NFA}$ and transition $\tau_{NFA}: \SS_{NFA}\times \OO \to \SS_{NFA}$, we construct a unary non-deterministic Moore machine with output alphabet $\OO$ as follows.
Let the set of states of the Moore machine bet $\SS = \SS_{NFA}\times \OO$ with $\oo$ given by projection onto the $\OO$ coordinate. Let the transition of the Moore machine be given by $$\tau(s) = \tau_{NFA}(s_{NFA}, o) \times \OO \subset \SS_{NFA}\times \OO,$$ where $s = (s_{NFA}, o) \in \SS$. Remove iteratively any states of the Moore machine with empty transitions. It is easy to verify that the resulting Moore machine is generic if and only if the original NFA was all-accepting.
[Moo56] Moore, Edward F., Gedanken-experiments on sequential machines. In
C. E. Shannon and J. McCarthy, editors, Automata Studies, pages
129–154. Princeton University Press, Princeton, 1956.
[KRS09] Jui-Yi Kao, Narad Rampersad, and Jeffrey Shallit, On NFAs Where All States are Final, Initial, or Both. https://arxiv.org/abs/0808.2417, 2009.