If there are 2 competing products on Amazon, and one has an average rating of 4.5 (by 2 people) and one has an average rating of 4.49 (by 10000 people), I'll obviously choose the second product with the better sample size. A simple rule of thumb that takes into account sample size is to take the average rating, and subtract off some constant times $\frac{1}{\sqrt{n}}$ (where $n$ is the sample size).
Similarly, as an extreme ranked-choice voting example: there are 4 choices (A, B, C, D). 9997 people vote that A > B > C (and don't know where to put D). 1 person votes A > D. 2 people vote that D > A. And so the final ordering you end up with is D > A > B > C. But the number of votes that compared D and anything else were hardly any, so this isn't exactly an ideal ranking, as it doesn't take into account sample size/statistical significance.
Is there a more "robust" ranked choice voting scheme that gives some kind of weight to sample size?
