Why is the funnel on a funnel plot shaped like a funnel?
How can we explain the relationship between sample size and variation?
Let’s look at a funnel plot that shows the rates at which patients from different GP practices were admitted to hospital as emergencies last year.
There are 44 GP practices here. Each one of them is represented by a grey square on the chart.
The vertical axis measures the admission rate. So that the practices can be easily compared, it’s a rate per 1,000 practice population. The higher up the axis the grey square is, the higher that practice’s admission rate is.
The horizontal axis measures the size of the practice population. The bigger practices are on the right (the biggest practice has a list size of just under 14,000); the smaller practices are on the left.
There are some lines on the chart as well. The solid black horizontal line running through the middle of the grey squares is the overall admission rate for all of the 44 practices combined: 105 emergency admissions per 1,000 practice population.
All of this chart content is relatively easy to understand. And relatively easy to explain, too. It’s the bit of the chart that we haven’t yet explained that’s hard: the two curved, dotted lines that make this a funnel plot. They aren’t horizontal. They’re wider apart on the left than they are on the right. And you have to be able to explain why.
And it’s no good thinking that you can just say something like: “These dotted lines are funnel-shaped because well, obviously, everybody knows that there’s an inverse relationship between sample size and standard error.” Because everybody doesn’t know that. And, in fact, it’s sentences like that that give information analysts a bad name. No, you’re going to have to explain it step by laborious step.
Well, as a way into this, let’s imagine that someone looking at this chart has just said:
“Here’s what I don’t understand. There’s a practice on the far right of this chart, the practice with the largest practice population, and it’s got an admission rate that’s just a bit higher than the overall rate [it’s actually 116 admissions per 1,000 population compared with the overall rate of 105 admissions per 1,000 population]. That value of 116 is showing as outside the dotted line control limits. So we therefore have to conclude that it’s got an unusually high admission rate. Yet further to the left on the chart there’s a practice with an admission rate of 121 per 1,000 population—a rate that’s higher than the practice we’ve just been talking about. And yet this higher value is within the control limits. How can this be?”
And like I’ve already said, it’s no good replying that this has come about because there’s an inverse relationship between standard error and sample size (though there is and you’d be sort of right to say it). No, you’ll have to come up with something else. Something a bit more user-friendly. You need to compile a "repertoire" of examples, analogies and stories that you can draw upon whenever you’re in this position of having to explain funnel plots (and other complex charts for that matter).
If you don’t have such a repertoire, here is one way of visualising it to get your imagination working. It’s a bit contrived (to say the least!) but it might help get the idea across. It might even help you think of a better example.
You are going to imagine a scenario in which these 44 GP practices are about to play a game of dice. The object of the game is to throw as many sixes as possible. But we aren’t going to measure the score by the number of sixes you throw; instead we’re going to measure it by the percentage of sixes that you throw. So—for example—if you were to throw six dice and one of them was a six, you’d have scored 16.7%; if you had 12 dice and you threw one six your score would be 8.3%; if you had 100 dice and you threw 19 sixes your score would be 19%. I’m sure you’ve got the hang of this by now.
The reason we’re using percentages as our scores is because each of the 44 teams will be given a different number of dice to play with. In fact, we’re going to give each teams a number of dice that is proportional to the size of their practice population.
The first team is the smallest GP practice (a list size of just 300) and it will be given just six dice. The second team is the second smallest GP practice (a list size of 434), and it will be given nine dice. The third practice (list size: 517), they'll get ten dice. And so on, all the way through the GP practices until you get to the largest practice (with a practice population of 13,308), and you'll give that team 266 dice.
Before you start the game you should probably do a quick check to make sure that you got your arithmetic right. The ratio between the smallest practice and the largest practice (300: 13,308) ought to be the same as the ratio between the smallest number of dice and the largest number of dice (6: 266).
So now we’re ready. The teams throw the dice and they calculate their percentage score.
Here are the results of that game shown as a funnel plot:
How does this help us answer the earlier question? How does it help us explain why the funnel on a funnel plot is shaped like a funnel?
Here’s how. There is more variation in the scores on the left hand side of the chart (the teams that were only given a few dice to throw) than there is on the right (the teams that were given lots of dice to throw). Which, when you think about it, is obvious.
Look, for example, at the score obtained by the first team, the GP practice with the smallest population (list size = 300). We only gave that team six dice and they got a score of 33.3%. So they must have thrown two sixes out of six dice thrown. Nothing unusual about that, you think. But it would be unusual if that same team had maintained that level of performance if we'd given them lots of dice to throw. If we’d given that team 266 dice instead of just six dice, they’d have had to throw 89 sixes (33.3% of 266) in order to maintain that initial score of 33.3%. There’s no way on earth that would have happened. When you do something lots of times you’re far more likely to get the "long-run" result, not a freak result. And that is exactly what has happened in this elaborate dice game. The "long-run" result is 1/6 or 16.7%. The more dice you throw, the closer you will be to this outcome. The fewer dice you throw, the more variable will be your outcome.
And what that also means is that if you were the biggest team, the team of GPs from the largest practice, and you were given 266 dice, any score of less than 10% or more than 23% would have counted as being unusual. Significantly high or significantly low.
So there's an example of how you might want to help people visualize a funnel plot. It's contrived, it's elaborate, it's a bit ridiculous. But if the clunkiness of my example inspires you to invent one of your own that's more elegant and concise, then this blog article will have served its purpose.
[18 November 2011]
|© Kurtosis 2011. All Rights Reserved.|