Let's suppose I have three friends, all remote. We want to randomly and fairly pick one person to be the first player in a game. In order to do so, we'll each be assigned a unique number in {1, 2, 3, 4} and then roll a fair four-sided die (1d4). This works in person because we can all verify that the correct die was rolled only once, but not remotely. Suppose I am the one to roll the die - the others all have to trust that I'm not lying. Or perhaps I record a video of me rolling the die. This still doesn't work because I could simply have rolled until I got the result I wanted, then use that video.
One way I thought of doing this was to have a service that would generate a number. Each person would verify to the service that they would accept the result, then after all of us have done so, the service will generate a random number for us. Then we could retrieve the random number, which would be the same for us all. The issues with this is that we must trust the third-party service.
Is there a simple, trustless way to generate small random numbers by hand (with dice)? Such that even if some subset of people are malicious, they can't affect the outcome in a meaningful way? I suppose we can rule out collusion, or perhaps limit collusion to m of n people (i.e., 2 of 4 people can collude without affecting the outcome but not more than that).
I'm specifically looking for solutions that will work with "small" numbers - numbers found on standard dice. Less than 20 would be ideal, but if the only viable solution needs a d100 that would probably work.