Monthly Archives: August 2011

Deriving the Poisson Distribution

The Poisson distribution can be derived from the binomial distribution by doing two steps:

  1. substitute \frac{\mu}{n} for p
  2. Let n increase without bound

Step one is possible because the mean of a binomial distribution is \mu = np. So another way of expressing p, the probability of success on a single trial, is \frac{\mu}{n} . This has some intuition. Recall that a binomial distribution gives the probability of a number of successes (x) in a fixed number of trials (n) for some probability (p). So if an event has a probability of 0.2, and we observe 10 trials (where the trials are independent), the expected value of seeing the event occur is 10(0.2) = 2. On average we would see the event happen twice in 10 trials. Which means we can state the probability as a rate, \frac{2}{10}. Success occurs, on average, 2 times per 10 trials.

The next step leads to the Poisson. We let the number of trials increase without bound, which means the probability shrinks. Since n is in the denominator of \frac{\mu}{n} , this gives a small chance of success in a huge number of trials. So whereas the binomial deals with a fixed number of trials, the Poisson accounts for an unlimited number of trials in time or space. Taking the limit as n tends to infinity gives us the Poisson. Here’s how.

Recall the binomial probability mass function:
f(x) = \frac{n!}{x!(n-x)!}p^{x}(1-p)^{n-x}

Substitute \frac{\mu}{n} for p:
f(x) = \frac{n!}{x!(n-x)!}(\frac{\mu}{n})^{x}(1-\frac{\mu}{n})^{n-x}

Now the fun begins. Let n grow very large:
\lim_{n \to \infty}\frac{n!}{x!(n-x)!}(\frac{\mu}{n})^{x}(1-\frac{\mu}{n})^{n-x}

But wait! Don’t let n blow up just yet. Let’s do some rearranging first to help us take the limit:
\lim_{n \to \infty}\frac{n!}{(n-x)!n^{x}}\frac{\mu^{x}}{x!}(1-\frac{\mu}{n})^{n}(1-\frac{\mu}{n})^{-x}

And let’s also simplify \frac{n!}{(n-x)!n^{x}}:

\frac{n!}{(n-x)!n^{x}} = \frac{n(n-1)(n-2)\dots(n-x+1)}{n^{x}} =1(1-\frac{1}{n})\dots(1-\frac{x-2}{n})(1-\frac{x-1}{n})

Before we flip the switch on the limit-to-infinity machine, let’s assess what we got:
\lim_{n \to \infty}1(1-\frac{1}{n})\dots(1-\frac{x-2}{n})(1-\frac{x-1}{n})\frac{\mu^{x}}{x!}(1-\frac{\mu}{n})^{n}(1-\frac{\mu}{n})^{-x}

Notice this is basically four things multiplied together:

  1. 1(1-\frac{1}{n})\dots(1-\frac{x-2}{n})(1-\frac{x-1}{n})
  2. \frac{\mu^{x}}{x!}
  3. (1-\frac{\mu}{n})^{n}
  4. (1-\frac{\mu}{n})^{-x}

We can make life easier by taking the limit of each of those four pieces individually:

  1. \lim_{n \to \infty}1(1-\frac{1}{n})\dots(1-\frac{x-2}{n})(1-\frac{x-1}{n})=1
  2. \lim_{n \to \infty}\frac{\mu^{x}}{x!}=\frac{\mu^{x}}{x!} (no n to blow up!)
  3. \lim_{n \to \infty}(1-\frac{\mu}{n})^{n}=e^{-\mu}
  4. \lim_{n \to \infty}(1-\frac{\mu}{n})^{-x}=1

1, 2, and 4 are easy. Number 3, though, requires a long forgotten formula you probably learned in calculus just long enough to take an exam:

\lim_{n \to \infty}(1+\frac{b}{n})^{n}=e^{b}

Set b = -\mu and you get the result.

Now put it all back together and you have the probability mass function of the Poisson distribution:

f(x) = \frac{\mu^{x}e^{-\mu}}{x!}

Often you see \mu as \lambda in this formula.

This new formula is approximately the binomial probability mass function for a large number of trials and small probability. That’s basically what we did, right? Substitute an expression for p involving n and let n grow large. That means the probability got small while n got large. Which further means that before we had modern computers, the Poisson was very handy for approximating binomial probabilities since the binomial coefficient can be difficult to compute for large n.

For example, say we have 100 independent trials with probability of success 0.05. What is the probability of observing 7 successes?

Using the binomial:
f(7) = \frac{100!}{7!(100-7)!}0.05^{7}(1-0.05)^{100-7} = 0.106026

Using the Poisson as an approximation (where \mu = np = 100(0.05) = 5):
f(7) = \frac{5^{7}e^{-5}}{7!} = 0.104445

Very close indeed!

By the way, how did I calculate those probabilities? I used Excel functions:

Binomial: =BINOMDIST(7,100,0.05,0)
Poisson: =POISSON(7,5,0)

The numbers in the parentheses should be self-explanatory. The final 0 means “false, no cumulative”. If you set it to 1, you get the probability of 7 or fewer successes in 100 trials. You can also use a TI-83 Plus. Just press 2nd – DISTR and choose binompdf or poissonpdf. If want to do cumulative probabilities, select binomcdf or poissoncdf. The parameters are in a different order from Excel:

binomcdf/binompdf(n,p,x)
poissoncdf/poissonpdf(mu,x)

Of course “real statisticians” would use R:

> dbinom(7,100,0.05)
[1] 0.1060255

> dpois(7,5)
[1] 0.1044449

I would be careless not to mention that the Poisson distribution is not the same as the binomial. The binomial describes the distribution of events (or “successes”) over a fixed number of trials for a fixed probability of success on each trial. The Poisson describes the distribution of events in space or time for a given rate of occurrence. The binomial is specified by n (number of trails) and p (probability of success on each trial). The Poisson is specified by \mu, the rate at which events occur per unit of space or time.

And finally, and no doubt most important, the way to pronounce Poisson is \pwa-sawn\. Say it correctly and you earn instant credibility at cocktail parties.

Fume Cupboards

I was reading one of my favorite statistics books, Principles of Statistics, and came across a fun problem I decided to tackle. It involves fume cupboards, big hoods chemists sometimes work under to suck up noxious fumes. The problem isn’t that complicated, but it tripped me up and I thought I should write it about it for posterity so I don’t let it get me again. Here’s the problem (slightly condensed to save typing):

In an extensive survey of chemical research workers, it was found that on average each worker required no fume cupboard 60% of the time, one fume cupboard 30% of the time, and two fume cupboards 10% of the time. Three or more were never required. If a group of four chemists work independently of one another, what’s the minimum number of fume cupboards required to provide adequate facilities at least 95% of the time?

The plan of attack came to me right away. We need to determine the probability the four chemists will – all together – need 0 fume cupboards, 1 fume cupboard, 2 fume cupboards, all the way up to 8 fume cupboards, and find the minimum number that exceeds 95%. The tricky part was counting all the different ways the four chemists can use fume cupboards.

First let’s determine how many combinations of fume-cupboard-needs can arise. One way to contemplate this is to think of a combination lock with four dials, kind of like an old bicycle lock. In this case the dials would have values of 0, 1, and 2. So let’s say chemists 1 and 2 needed one fume cupboard, and chemists 3 and 4 needed none, then our combination would look like this:

1 1 0 0

That sums to 2 and means we would need two fume cupboards to keep the chemists happy, well-ventilated and working. How many possible combinations can we have? The quick answer is 3^{4}=81. That’s three possible states of need for each chemist, all multiplied together.

Next let’s consider the minimum and maximum number of fume cupboards these chemists could possibly need. It’s possible they could all be at their desk and need 0 fume cupboards. On the other hand, they could all simultaneously need two fume cupboards for their latest experiment and thus require a total of eight. So at any given time when all four chemists are at work, they could need anywhere from 0 to 8 fume cupboards.

The probability they all need 2 fume cupboards at the same time is 0.1^{4}=0.0001. So if their company provided just 7 cupboards, they would have adequate facilities 99.99% of the time. However, the problem asks us to find the minimum number such that they have what they need 95% of the time. So we know the answer is not 8 or 7.

What’s the probability they need 7 fume cupboards? Well the first, second and third chemists could need 2, and the fourth one. The probability of that happening is 0.1^{3}\times 0.3 . But the first chemist could need one and the chemists 2-4 could need two. Or the 2nd chemist could need one and chemists 1, 3, and 4 need two. Or the 3rd chemist could need one and chemists 1, 2, and 4 need two. That’s four different ways the chemists could need 7 fume cupboards. So our probability becomes 0.1^{3}\times 0.3 \times 4 = 0.0012.

Since the events “need 7 fume cupboards” and “need 8 fume cupboards” are mutually exclusive, we can add them to obtain the probability the chemists would need 7 or more fume cupboards. That comes to 0.0013. That means if we had 6 fume cupboards (let’s call them FCs, I’m tired of typing fume cupboard), we would have adequate coverage 99.87% of the time.

Now we’re hitting the tricky part of this problem: counting the various combinations of FCs. The extremes aren’t so bad. There’s one way to need 0 and 8 FCs. There are four ways to need 1 and 7 FCs. But when we get to 2-6, we have to be careful how we count. If we undercount our probability is too low. Overcount and we go too high. One way to check ourselves is to see if we have come up with 81 possible scenarios. There are 10 possibilities accounted for when we consider 0, 1 , 7 , and 8 FCs. That means there are 71 ways for the four chemists to need 2 – 6 FCs.

What stymied me was counting the possible ways the chemists could need 3, 4 and 5 FCs. It would help if I could draw a picture in real time and explain it. But I can’t, so I’ll just show you the answer:

P(0 FCs) =0.6^{4} = 0.1296
P(1 FCs) =0.3*(0.6^{3})*4 = 0.2592
P(2 FCs) =(0.3^{2})*(0.6^{2})*6+0.1*(0.6^{3})*4 = 0.2808
P(3 FCs) =0.1*0.3*(0.6^{2})*12+(0.3^{3})*0.6*4 = 0.1944
P(4 FCs) =(0.1^{2})*(0.6^{2})*6+(0.3^{4})+0.1*(0.3^{2})*0.6*12 = 0.0945
P(5 FCs) =(0.1^{2})*0.3*0.6*12+0.1*(0.3^{3})*4 = 0.0324
P(6 FCs) =(0.1^{3})*0.6*4+(0.1^{2})*(0.3^{2})*6 = 0.0078
P(7 FCs) =(0.1^{3})*0.3*4 = 0.0012
P(8 FCs) =0.1^{4} = 0.0001

Notice the probabilities sum to 1:
0.1296 + 0.2592 + 0.2808 + 0.1944 + 0.0945 + 0.0324 + 0.0078 + 0.0012 + 0.0001 = 1

Also notice the multiples of the various combinations sum to 81:
Number of ways to require 0 or 8 FCs = 2
Number of ways to require 1 and 7 FCs = 8
Number of ways to require 2 and 6 FCs = 20
Number of ways to require 3 and 5 FCs = 32
Number of ways to require 4 FCs = 19

All that’s left now is to find the minimum number of FCs that’s at least 95%. It turns out to be 4:

P(4 or fewer FCs needed) = 0.1296 + 0.2592 + 0.2808 + 0.1944 + 0.0945 = 0.9585

Since a good industrial FC costs well over $10,000, one can see how a problem such as this has use in real life (provided our initial estimates of FC use is reasonably accurate).