21

Given $n$ items with weight $w_n$ each -- what is the probability that item $i$ is chosen in a $k$-out-of-$n$ "weighted random sampling without replacement" experiment? Can a closed-form solution that depends only on $w_i / w_\cdot$ be derived ($w_\cdot = \sum_j w_j$.)?

EDIT: A solution that depends only on $w_i / w_\cdot$ is impossible. Assume $n=3$, $i=1$, $w_1 = 1$, and two cases: (1) $w_2 = 1, w_3 = 1 + \varepsilon$, (2) $w_2 = 2, w_3 = \varepsilon > 0$. In case (1) the probability is almost $2/3$, in case (2) it is almost $1$, but in both cases $w_1 = 1$ and $w_\cdot = 3 + \varepsilon$.

What I have tried so far to solve the problem:

Let $P^n_k(w, i)$ be the probability that item $i$ is chosen in an $k$-out-of-$n$ experiment with weight vector $w$. In the first draw, the item is chosen with probability $w_i / w_\cdot$. Otherwise, we are looking for the probability to choose this item in a $k-1$-out-of-$n-1$ experiment, with the same weight vector except for the item that has been selected. Hence:

$$ P^n_k(w, i) = w_i / w_\cdot + \sum_{j \neq i} w_j / w_\cdot \cdot P^{n-1}_{k-1}(w - j, i) $$ $$ = w_\cdot^{-1} \left( w_i + \sum_{j \neq i} w_j \cdot P^{n-1}_{k-1}(w - j, i) \right) $$

with $w-j$ being "the vector $w$ without the $j$th element".

How to solve this recurrence relation? (If it is correct at all...)

krlmlr
  • 640
  • I don't think it's reasonable to expect a "pretty" result for a general weight function. – Guest 86 Feb 27 '13 at 21:41
  • @Guest86: I was just hoping that some magic cancellation would happen, but so far I don't see how... – krlmlr Feb 27 '13 at 21:43
  • This algorithm appears to essentially enumerate all subsets of size k containing element i, and therefore will run in exponential time in the worst case: O(2^n). I'm curious to know whether I'm missing something that can make this run in polynomial time. (To be fair, exponential time is better than the brute force, factorial runtime of enumerating all permutations containing element i and scoring them.) – dsg Sep 28 '16 at 05:57

1 Answers1

9

I'm pulling this from Pavlos S. Efraimidis, Paul G. Spirakis, Weighted random sampling with a reservoir, Information Processing Letters, Volume 97, Issue 5, 16 March 2006, Pages 181-185, ISSN 0020-0190, 10.1016/j.ipl.2005.11.003.

There, the authors begin by describing a basic weighted random sampling algorithm with the following definition:

Input: A population $V$ of $n$ weighted items

Output: A set $S$ with a WRS of size m

  1. Repeat Steps 2 and 3 for $k=1,2,\ldots,m$
  2. The probability of $v_i$ to be selected is:$$p_i(k) = \frac{w_i} {\sum_{S_j \in V - S} {w_j}}$$
  3. Randomly select an item $v_k \in V - S$ and insert it into $S$

The authors go on to explain how they arrive at the probability, but I'll summarize. Starting with the first item, the probability that $w_n$ is selected is its own weight divided by the sum of all weights. $$\frac{w_n}{w_1 + w_2+w_3+\ldots+w_n}$$ Easy enough. Now, the probability that each subsequent item will be chosen is its own weight divided by the sum of the remaining weights. If we do this calculation for each weight $w_i$ in order with $i=[1,n]$, we arrive at the authors' final summary equation for any permutation $\Pi$:

$$ P(\Pi) = \prod_{i=1}^{n} {\frac{w_i} {w_1 + w_2 +\ldots+w_i}} $$

Which is to say, the probability that an item is chosen can be defined as indexes of an array $w$ that contains all weights, like so: $$\frac{w(i)}{w([1,i])}$$

This isn't particularly difficult work, but I didn't want to create the proof myself, hence the quoting of Efraimidis & Spirakis.

Community
  • 1
Dinre
  • 236
  • The algorithm by Pavlos S. Efraimidis and Paul G. Spirakis is by far the most beautiful thing I've seen for a long time, just for it's simplicity. It's just as sweet as implementing convolution through FFT, not sure which wins though... – krlmlr Mar 04 '13 at 19:06
  • 1
    However: Just by reading your argument I don't see how this answers my question. You have lost me at "Which is to say..." -- especially I don't see how $i$ is bound in your last equation, and what $w([1,i])$ means. While the algorithm can be used to implement weighted r.s. without replacement, I'm not yet sure that it can be used to derive direct selection probabilities. ? – krlmlr Mar 04 '13 at 19:10
  • That part about expressing the probability as arrays is more for those who prefer matrix math over the element expression used in the paper. The weight of a single item is its own weight divided by the weights of the remaining elements. The equation works, because we are not assuming the order of elements is arbitrary. Rather, we are assuming the elements are chosen starting with the element $w_n$ and going down to $w_{n-m}$. The order of these elements mean something in a practical space, but in theoretical space, the order is irrelevant. – Dinre Mar 04 '13 at 19:15
  • Also, $w([1,i])$ is the elements array $w$ using the indexes in the set $[1,i]$ or $(1,2,...,i)$. It's just a different way of expressing the same thing but with intervals and indexes. – Dinre Mar 04 '13 at 19:17
  • 1
    In your notation: What is the probability that $v_i \in S$ when sampling $m$ out of $n$ items, without replacement, using weights $w$, for any given $i$? Can you find a closed-form solution? – krlmlr Mar 04 '13 at 19:23
  • 1
    @krlmlr, I can give you the algorithmic expression in math terms, but there is no closed-form solution for a weighted random selection process. The hitch is that the solution of the problem must calculate the probability of all relevant permutations in order to give you an exact answer. Otherwise, the answer is a range of probabilities (which ceases to be closed-form). I'm starting to think my answer doesn't suit your needs very well, since it sounds more like you want a practical solution rather than a mathematical expression. – Dinre Mar 05 '13 at 13:46