While testing some capabilites of GPT-4o through the chat interface recently, I noticed a bias toward certain numbers in the responses: when phrases suggesting percentages with two significant figures were included in the prompt, it did return two significant figures in seven out of seven trials—but it also returned numbers ending in 5 in all of the trials, despite there being no suggestion of this in the prompt.
Of course, these results are likely to depend heavily on the prompt: obviously, a prompt that has a math problem or type of math problem heavily present in the training data might exhibit a heavier bias toward the correct answer than to any particular digits. Still, in cases where the model's responses are closer to a "guess," there might be expected to be a bias: neither are digit distributions uniform in a lot of real-world data, nor do humans, who provide much of the textual training data, tend to provide uniform random numbers.
This question has particular relevance for the unbiased estimator properties of numbers returned by ChatGPT if people try to use it for certain tasks instead of purpose-built or fine-tuned models, e.g. zero-shot regression on images or estimates of textual similarity (though I doubt it currently performs well competitively or even provides adequately formatted data with an acceptable probability for production).
Has anyone done any publicly available research in numerical biases in the digits returned by large language models or large multimodal models, either across samples from plausible real-world distributions of prompts or in particular classes of prompts? If so, what does it indicate?
