5

Consider a set of words where you want to divide the set into subsets of words, where all members of each subset are anagrams (same letters, different arrangement). You can do this computationally in several ways, the "best" way is sorting the letters from A through Z in each word. If two words are equal when their respective letters are sorted, then they must be anagrams. Otherwise, they are not. For the general case we cannot do any better. The sorting, which is $\mathcal{O}(M \log M)$ for each word of length $M$, is the computationally hardest part.

But here is something I've been discussing. Suppose we know there is a longest word with $M$ characters. Then imagine we have a so-called "hash function" $f$ which maps a word to a positive integer. Let each letter A,B,C,...,Z have a specific weight $a_1,a_2,a_3,\dots,a_{26}$. For a word $x$, the value $f(x)$ is the sum of the corresponding weights without regard to their position within the word. $f(ZABC) = a_{26} + a_1 + a_2 + a_3$. This has linear complexity, $\mathcal{O}(M)$ for each word.

If we can select these weights such that only words that are anagrams map to the same value, then a computer can easily calculate whether two words are anagrams by looking at their respective function value. The question is of course, how do we choose the weights?

I can formulate the problem as such:

For fixed $M$ and $N$, find $\{a_1,a_2,\dots,a_{N}\}$, where $a_1 < a_2 < \dots < a_{N}$, such that $$c_ia_i \neq c_1a_1 + c_2a_2 + \dots + c_{i-1}a_{i-1} + c_{i+1}a_{i+1} + \dots + c_{N}a_{N}$$ for any combination of $c_j \in \{0,1,2,\dots,M\}$ except $c_1=c_2=\dots=c_{N}=0$.

Basically, we want to find a linearly independent $N$-subset of $\Bbb Z_{>0}$ subject to a constraint on the coefficients. Is there a clever method for this?

A very simple solution is to let $a_1 = 1$ and recursively define $a_{i+1} = Ma_i + 1$ but this grows very quickly and $a_{26}$ is too large for use in computers. For small $M$ and $N$ I have been able to find better solutions.

Is there a better solution for $N=26$ and say $M \approx 15$? What is the smallest possible $a_{26}$?

naslundx
  • 9,896

2 Answers2

1

A higher density of information will be to use $$f(W) :=1+\sum_{i=1}^M N^{W_i-1} \in [1, N^M]$$ This will use more natural numbers than you set and each can be reversed to a unique (sorted) representation of the anagram (it's a bijection on the sorted words).

I think a better solution is to use a specialized sorting algorithm to obtain the "normalized word", since a real computer will not be able to quickly compute with such large numbers and it will be extensible. Given $N$ and $M$ you can map a word to its histogram $h(W) \in [0,M]^N$. Comparison of these is $O(N)$ and creation is $O(M+N)$. Thus a full comparison will run in $O(3N+2M)$ where the $O$-constant does not depend on $M$ or $N$ at all.

AlexR
  • 25,110
  • $N^M$ gives me all possible words, not just all sorted words. – naslundx Aug 07 '15 at 11:18
  • @naslundx Sorry about that, you're right. Maybe a stars-and-bars like enumeration gives better density. But do you really need this? It will be very hard to maintain. – AlexR Aug 07 '15 at 11:32
  • I'm not yet sure if it has any merit practically, but I'm still interested in trying to find an upper limit on $a_{26}$ and an algorithm to find all $a_j$, if possible. – naslundx Aug 07 '15 at 12:31
  • The "specialized sorting algorithm" is a bucket sort, i.e., each word gets a vector counting the A, B, ..., Z in it. That is clearly linear in the length of the word. And you can't possibly do better, you need to look at each letter in the word. – vonbrand Aug 07 '15 at 22:47
  • @vonbrand It's an optimized version of bucket sort to be precise - usually you'd need to allow multiple passes to sort numers larger than the number of buckets. – AlexR Aug 08 '15 at 11:15
1

In this answer I consider how you can compactly represent the equivalence classes as natural numbers, without worrying about linearity.

You want a different result for every different multiset of cardinality at most $M$ taken from a set of size $N$. This number equals the number of different multisets of cardinality exactly $M$ taken from a set of size $N+1$ (we fill the missing places with a null letter). For $N=26$ and $M=15$ this gives

$$\left(\!\!{N+1\choose M}\!\!\right) = \binom{M+N}{N} \approx 2^{35.9}$$

You can accomplish the most efficient mapping by employing the so called combinatorial number system.

Algorithmically, you could compute the histogram for each word and then reduce it to

$$f(x) = \sum_{k=1}^{N} \binom{c_k}{k}$$

where $c_k = \sum_{i=1}^{k} (a_i + 1)-1$ is roughly the cumulative sum of the frequencies $(a_i)$.

The density of this representation could be sacrificed to get the simpler mapping

$$g(x) = \sum_{k=1}^{N}2^{c_k}$$

In this case, we would need $M+N = 41$ bits, which is still in the practical range between 32 and 64.

This last representation corresponds to viewing the cumulative histogram as a general subset of $\{0,\ldots,M+N-1\}$ and representing it directly in binary. The previous one takes advantage of the knowledge that this set has exactly $M$ elements.

filipos
  • 1,572