Why use mean revenue in a split test?

Question

I asked a data science question regarding how to decide on the best variation of a split test on the Statistics section of StackExchange. I hope I will have better luck here. The question is basically, "Why is mean revenue per user the best metric to make your decision on in a split test?"

The original question is here: https://stats.stackexchange.com/questions/107599/better-estimator-of-expected-sum-than-mean

Since it was not well received/understood I simplified the problem to a discrete set of purchases and phrased it as a classical probability problem. That question is here: https://stats.stackexchange.com/questions/107848/drawing-numbered-balls-from-an-urn

The mean may be the best metric for such a decision but I am not convinced. We often have a lot of prior information so a Bayesian method would likely improve our estimates. I realize that this is a difficult question but Data Scientists are doing such split tests everyday.

Answer 1

If you've got prior information then you should certainly not use simple mean in a split test. I assume you're trying to just predict which group will produce the greatest amount of revenue overall, by trying to emulate the underlying distribution.

Firstly, it's worth noting that any metrics you choose will actually reduce to mean in a pretty trivial way. Eventually mean will necessarily work out, though using a standard bayesian method to estimate the mean is probably your best bet.

If you've got a prior then using a standard bayesian approach to update the prior on your mean revenue is probably the best way to do it. Basically, just take the individual results you get and update a multinomial distribution representing your prior in each case.

If you want some more full background on multinomial distributions as bayesian priors are pretty well, this Microsoft paper does a pretty good job of outlining it. In general, I wouldn't care so much about the fact that your distribution is technically discrete, as a multinomial distribution will effectively interpolate across your solution space, giving you a continuous distribution that is a very good approximation of your discrete space.