I don't know if this helps you because you seem to be interested in recursion, but there's a way to do this using gradings on monoids that doesn't require recursion. I'll expand a bit on what I mean with gradings and then construct your function. It's a bit verbose but it has the merit of being constructive. With an ordering on the alphabet, this could even be turned into an algorithm to compute your function.
Consider your alphabet $\Sigma$ and the alphabet $\Pi \overset{def}= \Sigma \cup \{(, )\}$ with the two extra symbols for each parenthesis. We consider $\Sigma^*$ to be the set of (finite) sequence of elements of $\Sigma$ called words (including the empty word) and we define $\Pi^*$ similarly. Together with the operation of concatenation, those sets form monoids, and $\Sigma^*$ is a submonoid of $\Pi^*$ because $\Sigma \subseteq \Pi$.
A grading on a monoid $M$ is just a morphism of monoids $g : M \to \mathbb Z$. We can write $M = \bigcup_{n \in \mathbb Z} M_n^g$ where $M_n^g \overset{def}= g^{-1}(n)$ is the set of elements of $g$-degree $n$, also called the $g$-graded component of degree $n$ of $M$. It's because $g$ is a morphism of monoids that for $m \in M_n^g$ and $m' \in M_{n'}^g$, we have $mm' \in M_{n+n'}^g$.
Of interest to us is a particular class of gradings. For a subset $\Gamma \subseteq \Sigma$, we get a morphism of monoids $\pi_{\Gamma}: \Sigma^* \to \Gamma^*$ given by taking a word $x = a_1 \cdots a_n \in \Sigma^*$ and removing the symbols $a_i$ belonging to $\Sigma \setminus \Gamma$, keeping those belonging to $\Gamma$ in the same order. Then, we consider the length grading $\ell : \Sigma^* \to \mathbb Z$ which just takes a word and returns the number of symbols in it (counting repetitions of course). We can compose morphisms of monoids, so for a given $\Gamma \subseteq \Sigma$, the composition $\ell_{\Gamma} \overset{def}= \ell \circ \pi_{\Gamma}$ is called the $\Gamma$-length grading; it counts the number of symbols appearing in a word that belong to $\Gamma$.
Note that gradings can be added and subtracted; that is because the map $\mathbb Z \to \mathbb Z$ multiplying by $-1$ is a morphism of abelian groups, so that the negative of a grading is also a grading (use composition).
The few gradings we will need to consider:
- For $\Gamma = \Sigma \subseteq \Pi$, the $\Sigma$-length grading on $\Pi^*$ determines the number of symbols in a word belonging to $\Sigma$. We denote it by $\gamma : \Pi^* \to \mathbb Z$.
- For $\Gamma = \Pi$, the $\Pi$-length grading on $\Pi^*$ is just the full word length $\ell : \Pi^* \to \mathbb Z$.
- For $\Gamma = \{(\}$, we get the left parenthesis grading $L$; similarly, we get the right parenthesis grading $R$ for $\Gamma = \{)\}$. These count the number of left (resp. right) parentheses in a word in $\Pi^*$.
- Let $b = b_1 \cdots b_n \in \Pi^*$ (where $b_1,\cdots,b_n \in \Pi$). For $m \ge 0$, the $m$-truncation of $b$, denoted by $[b]_m$, is just the word $b_1 \cdots b_m$ for $m < n$, and $b$ for $m \ge n$. To restrict our attention to words with bracketings of interest (using Dyck paths as an analogy), we need to consider words $b \in \Pi^*$ satisfying $L([b]_m) \ge R([b]_m)$ for $m \ge 0$ and $L(b) = R(b)$ (so that each right parenthesis can match with a left parenthesis coming before it). This can be reformulated with the grading $P \overset{def}= L - R$.
A word $b \in \Pi^*$ is called coherently bracketed if $P(b) = 0$
and $P([b]_m) \ge 0$ for all $m \ge 0$. Coherently bracketed words in $\Pi^*$ form a submonoid $\mathrm{CB}(\Sigma) \subseteq \Pi^*$. To see this, note that for $b,b' \in \Pi^*$ and $m \ge 0$, we have
$$
[bb']_m = \begin{cases}
[b]_m & \text{ if } m < \ell(b) \\
b[b']_{\ell(x) - m} & \text{ if } m \ge \ell(b).
\end{cases}
$$
Suppose $b,b'$ are coherently bracketed. Then $P(bb') = P(b) + P(b') = 0 + 0 = 0$. For $m \ge 0$, we'd have
$$
P([bb']_m) = \begin{cases}
P([b]_m) & \text{ if } m < \ell(b) \\
P(b) + P([b']_{\ell(x) - m}) & \text{ if } m \ge \ell(b).
\end{cases}
$$
which is non-negative in both cases. Since $\mathrm{CB}(\Sigma)$ is a submonoid, it also inherits all the gradings we put on $\Pi^*$ by composing a grading $g : \Pi^* \to \mathbb Z$ with the inclusion map $\mathrm{CB}(\Sigma) \subseteq \Pi^*$.
Now what we are interested in are not all coherently bracketed words in $\Pi^*$ because we want them to satisfy the following property: if "$(b_1 \cdots b_k)$" is a subword of $b \in \mathrm{CB}(\Sigma)$ with $b_1,\cdots,b_k \in \mathrm{CB}(\Sigma)$, then $k=2$ (so we can apply composition later to those subwords). We denote this property by (*).
Coherently bracketed words satisfying (*) determine a submonoid of $\mathrm{CB}(\Sigma)$ because if $x,y \in \Pi^*$ are coherently bracketed and the word $z$ given by "$(b_1 \cdots b_k)$" is a subword of the concatenation $xy$, we can consider the function $m \mapsto P([z]_m)$ which satisfies $P([z]_m) = 0$ if and only if $m = 0$ or $m \ge \ell(z)$; that's because the subwords $b_1, \cdots, b_k$ are coherently bracketed. So if we write $xy = vzw$ for some (possibly empty) words $v,w \in \Pi^*$, since $P(x) = 0$ and $P([z]_m) > 0$ for $0 < m < \ell(z)$, we deduce that $z$ is a subword of $x$ or $y$, and in both cases we deduce that $k=2$ since $x$ and $y$ satisfy (*), which means $xy$ does too. We denote this submonoid by $\mathrm{CB}_*(\Sigma)$.
So, if we want to consider all the bracketings of the word $a_1 \cdots a_n \in \Sigma^*$, what we want is the set of words $b \in \Pi^*$ such that
- the word $b$ has to be coherent, i.e. $P(b) = 0$ and $P([b]_m) \ge 0$ for all $m \ge 0$
- $L(b) = R(b) = \gamma(b)-2$
- $b$ satisfies (*), i.e. if "$(b_1 \cdots b_k)$" is a subword of $b$ with $b_1,\cdots,b_k \in \mathrm{CB}(\Sigma)$ and $k \ge 0$, then $k=2$.
Those are precisely the sets $\mathrm{CB}_*(\Sigma)_n^{\gamma} \cap \mathrm{CB}_*(\Sigma)_{n-2}^{L}$; they contain all the bracketings of all the words in $\Sigma^*$ of $\gamma$-length $n$.
Now we go back to the question of determining the function $\Sigma^* \to \mathcal P(\Sigma)$ that you wanted. We are given a binary operation $\circ : \Sigma^2 \to \Sigma$. Given a word $x \in \Sigma^*$, first we want to have the set of all bracketings in $\mathrm{CB}_*(\Sigma)$ which when we strip the parentheses from them, give us $x$ back. Then we want to use those parentheses to determine the order in which we should perform the compositions with $\circ$.
Recall from earlier that since $\Sigma \subseteq \Pi$, we have a morphism of monoids $\pi_{\Sigma} : \Pi^* \to \Sigma^*$ that simply removes parentheses from words. We can restrict this morphism to $\mathrm{CB}_*(\Sigma)$ and get the morphism $\pi_{\mathrm{CB}_*} : \mathrm{CB}_*(\Sigma) \to \Sigma^*$ (which does the same thing: remove parentheses).
Let $x \in \Sigma^*$ with $\ell(x) \ge 3$. A bracketing of $x$ is a coherently bracketed word $b \in \mathrm{CB}_*(\Sigma)$ satisfying $\gamma(b) = \ell(x)$, $L(b) = R(b) = \gamma(b) - 2$ and $\pi_{\mathrm{CB}_*}(b) = x$ (in particular, $b$ satisfies (*)). Therefore, if we set $n \overset{def}= \ell(x)$, the set
$$
B(x) \overset{def}= \pi_{\mathrm{CB}_*}^{-1}(x) \cap \mathrm{CB}_*(\Sigma)_n^{\gamma} \cap \mathrm{CB}_*(\Sigma)_{n-2}^L
$$
consists of all the bracketings of $x$. If $0 < \ell(x) < 3$, we set $B(x) = \{x\}$, and if $\ell(x) = 0$, we set $B(x) = \varnothing$.
Let $b$ be a bracketing of $x \in \Sigma^*$ with $\gamma(b) \ge 3$. We work iteratively with two steps: we first find a canonical choice of sub-word of $b$ of the form "$(a_1a_2)$" with $a_1,a_2 \in \Sigma$, and then we use that sub-word and replace it by $a_1 \circ a_2$. This new word is now a word in $\Sigma^*$ of length $\ell(x) - 1$, so we can iteratively repeat this process until $\ell(x) = 2$, in which case we don't need the brackets and just compose to get an element of $\Sigma$.
Step 1: There exists a sub-word of $b$ of the form "$(a_1a_2)$" for some $a_1,a_2 \in \Sigma$. To see this, consider the function $m \mapsto P([b]_m)$. Since it takes only finitely many values, it attains a maximal value, say $\alpha$. Suppose $m_0$ satisfies $P([b]_{m_0}) = \alpha$. Because $m \mapsto P([b]_m)$ only varies in increments of $1$, the values taken by that function surrounding $m_0$ must look like
$$
\cdots, P([b]_{m_0-1}), P([b]_{m_0}), P([b]_{m_0+1}), \cdots, P([b]_{m_0+k}), P([b]_{m_0+k+1}) \cdots \qquad = \qquad \cdots, \alpha-1, \alpha, \alpha, \cdots, \alpha, \alpha-1, \cdots.
$$
with the value $\alpha$ repeated $k$ times for some $k \ge 0$ ($k=0$ would make the sequence look like $\cdots, \alpha-1, \alpha, \alpha-1, \cdots$). The first occurence of $\alpha$ as the value of $P([b]_{m_0})$ implies that $[b]_{m_0}$ ends with a left parenthesis, and the occurence of $\alpha-1$ after the sequence of $\alpha$'s implies that $[b]_m$ ends in "$(a_1 \cdots a_k)$" with $k \ge 0$ and $a_i \in \Sigma$ for $i=1,\cdots,k$. Since $b$ satisfies (*), this means $k=2$ and "$(a_1a_2)$" is a subword of $b$.
Step 2: Let $x'$ be the word $x$ where the subword $a_1a_2$ is replaced by the composition $a_1 \circ a_2$. Also, let $b'$ be the word $b$ where the subword "$(a_1a_2)$" is replaced by the single symbol $a_1 \circ a_2 \in \Sigma$. We claim that $b'$ is a bracketing of $x'$.
It is clear that $b'$ is coherently bracketed, since the values $P([b']_m)$ taken are also values taken by $P([b]_m)$, so they are non-negative. Also, $P(b') = P(b) = 0$, and $L(b') = L(b) - 1 = \gamma(b) - 3 = \gamma(b') - 2$ (and similarly for $R(b')$).
As for the property (*) of $b'$, if "$(c_1 \cdots c_k)$" with $c_1,\cdots,c_k \in \mathrm{CB}(\Sigma)$ is a subword of $b'$, then either $a_1 \circ a_2$ is not involved and this is just a subword of $b$ (so that $k=2$) or $a_1 \circ a_2$ appears as a subword of one of the $c_i$'s. But then replacing $a_1 \circ a_2$ by "$(a_1 a_2)$" in that $c_i$ still keeps it coherently bracketed, and so $k=2$ because $b$ satisfies (*).
By taking the first available instance of a subword of the form "$(a_1a_2)$" in $b$ (first in the sense that you read $b$ from left to right), for any $n \ge 3$, we produced a function
$$
f_n : \mathrm{CB}_*(\Sigma)_n^{\gamma} \cap \mathrm{CB}_*(\Sigma)_{n-2}^L \to \mathrm{CB}_*(\Sigma)_{n-1}^{\gamma} \cap \mathrm{CB}_*(\Sigma)_{n-3}^L
$$
which sends a coherently bracketed word $b$ with $\gamma(b) = n$ and $L(b) = n-2$ satisfying (*) to $f_n(b) \overset{def}= b'$, and $\gamma(b') = n-1$ and $L(b') = n-3$. Let
$$
\mathrm B(\Sigma) = (\Sigma^*)_1^{\ell} \cup (\Sigma^*)_2^{\ell} \cup \left( \bigcup_{n \ge 3} \mathrm{CB}_*(\Sigma)_n^{\gamma} \cap \mathrm{CB}_*(\Sigma)_{n-2}^L \right).
$$
Also, write $\mathrm B(\Sigma)_1 \overset{def}= (\Sigma^*)_1^{\ell} = \Sigma$ (words of length one are just the alphabet), then $\mathrm B(\Sigma)_2 \overset{def}= (\Sigma^*)_2^{\ell}$ (the words of length 2), and finally $\mathrm B(\Sigma)_n \overset{def}= \mathrm{CB}_*(\Sigma)_n^{\gamma} \cap \mathrm{CB}_*(\Sigma)_{n-2}^L$ for $n \ge 3$ (the bracketings of $n \ge 3$ symbols). By construction, we have $\mathrm B(\Sigma) = \bigcup_{n \ge 1} \mathrm B(\Sigma)_n$, and the subsets $\mathrm B(\Sigma)_n$ are all pairwise disjoint subsets of $\Pi^*$, so they partition $\mathrm B(\Sigma)$.
We define $f : \mathrm B(\Sigma) \to \mathrm B(\Sigma)$ as follows: for $n=1$ and $x \in \mathrm B(\Sigma)_1$, we set $f(x) = x$. For $n=2$ and $x \in \mathrm B(\Sigma)_2$, we have $x = a_1a_2$ for some $a_1,a_2 \in \Sigma$, and we let $f(a_1a_2) = a_1 \circ a_2$. For $n \ge 3$, we set $f(x) = f_n(x)$ where $n = \ell(x)$.
Putting all the pieces together: suppose $x \in \Sigma^*$. Then $B(x)$ contains all its bracketings, and $f^{\ell(x)-1}(B(x))$ (which is the set of all elements of $B(x)$ on which we apply the function $f$, $\ell(x) - 1$ times) is the composition of all those bracketings. So I would define the map as
$$
\Lambda : \Sigma^* \to \mathcal P(\Sigma), \qquad x \mapsto \begin{cases} \varnothing & \text{ if } x \text{ is the empty word } \\
f^{\ell(x) - 1}(x) & \text{ otherwise.}
\end{cases}
$$
Hope that helps,
But this is really a question about Recursion theorems, not really about bracketings. Bracketings are just the example that demonstrates the trickiness with the recursion theorem that's giving me trouble. That is, I'm able to write down other recursive definitions like the factorial function or syntax formation rules. But this one has something tricky for me.
– Jagerber48 Jan 22 '23 at 16:23