Probability is Strange: Example 1

Really, this is just Part 2 of Why The Real Numbers are Cool (name adapted). Let’s start by stating a strange fact.

There is no way to uniformly sample values from an unbounded set (like ). To get an appreciation for this, let’s first define the uniform probability distribution.

Definition: A continuous probability distribution is uniform if its probability density function is given by , when and when or .

Definition: A discrete probability distribution is uniform if its probability mass function is given by .

We’ll first prove that there is no continuous uniform probability density function on an unbounded set, which we’ll do by contradiction.

Suppose that we could define a uniform probability distribution over . This function would have to satisfy . Since we are supposing that is constant, we could rewrite this as . The region over which we’d like to integrate has an infinitely long bottom side (given by the integration bounds). Since the height should be the same everywhere, any infinitely long rectangle (no matter how short), will have an infinite area.

This means that we won’t be able to define a function that is constant over all of and integrates to 1. So, any probability density function over will have to assign a higher probability to some regions than others. We specify regions because for continuous probability density functions, there’s no notion of the probability that a random variable is equal to a certain value (only that it is in the range of particular values).

In the discrete case, let’s try to define a uniform probability mass function over . Since every integer should be equally likely, this would mean that for any , , for some . Since there are infinitely many natural numbers, we would have to be able to evaluate . But, no matter how small is, this sum will always grow without bound!

This means that we won’t be able to assign an equal probability to every element in

The thing that’s so strange about this is that when someone asks you to pick a random positive integer, even if you think you’re being totally unbiased, you are never giving every number an equal chance of being picked.

The real numbers were used as an example for a continuous random variable, but any continuous-valued random variable will have the same problem. Similarly, just because we showed an example with the natural numbers, you’ll have the same problem if you use the integers or some other countably infinite set.

What’s interesting about this is that it kind of shows how even if you try to choose integers or real numbers in a way is consistent with intuition (like choosing which chair to sit in), it’s not possible!