Understanding Chi Square
Chi Square lets you know whether two groups have significantly different opinions, which makes it a very useful statistic for survey research. It's applied to cross-tabulations (AKA pivot tables) which are simply breakdowns like this:
| Yes | No | Total | |
|---|---|---|---|
| Female | 45 | 5 | 50 |
| Male | 15 | 35 | 50 |
| Total | 60 | 40 | 100 |
This article starts with the theory, and then has guidelines for using the statistic:
Understanding the calculations
When we eyeball our table above, it looks like women are much more likely to answer Yes, but is it random variation or something we can count on? What Chi Square does is compare the actual or Observed data we have from respondents with an Expected value. In our two questions, the total answers are:
| Female | 50 |
|---|---|
| Male | 50 |
| Yes | 60 |
| No | 40 |
If there were no relationship between the questions, then you would Expect a table that allocates those totals to look like this:
| Yes | No | Total | |
|---|---|---|---|
| Female | 30 | 20 | 50 |
| Male | 30 | 20 | 50 |
| Total | 60 | 40 | 100 |
The formula for the upper-left cell is:
(TotalYes * TotalFemale) / TotalTable
( 60 * 50 ) / 100
In less tidy examples, the Expected values often have a decimal or two. Once we have all the Expected values, we need to find the difference squared (so they're all positive) between the individual cells' Expected and Observed values:
D = ((O - E)2 / E)
| Yes | No | Total | |
|---|---|---|---|
| Female | E: 30 O: 45 D: 7.50 |
E: 20 O: 5 D: 11.25 |
E&O: 50 |
| Male | E: 30 O: 15 D: 7.50 |
E: 20 O: 35 D: 11.25 |
E&O: 50 |
| Total | E&O: 60 | E&O: 40 | E&O: 100 D: 37.5 |
Adding all the differences, we get a Total Chi Square of 37.5—which is yet another interim value in this calculation. So on to the next stage.
Many statistics rely on a concept called Degrees of freedom. The details vary stat to stat, but it's based on the number of variables involved in a calculation. For Chi Square, the degrees of freedom are:
df = (# rows - 1) * (# columns - 1)
= ( 2 - 1) * ( 2 - 1) = 1
In our cast we now have:
- Assorted Observed and Expected values
- Total Chi Square = 37.5
- Degrees of freedom = 1
We have two more players, and those are the Probability and Critical Value.
Any time you have a statistic designed to "predict" for a larger population or tell you a value's validity or reliability, part of the calculation is a level of confidence. Sometimes you'll see this indicated as the level of risk such as 5%, and at other times it will be noted as the level of certainty, 95%. For Chi Square, the tables are based on the level of risk, with common thresholds of 10%, 5%, 2.5%, 1% and 0.1%. Each one of those risk levels has a Critical Value associated with it:
| Probability | Critical Value when df = 1 |
|---|---|
| 10.0% | 2.71 |
| 5.0% | 3.84 |
| 2.5% | 5.02 |
| 1.0% | 6.64 |
| 0.1% | 10.83 |
| (More values—see the "Upper" table) | |
Our final step to calculate Chi Square is to compare our Total to the Critical Values. In our case, 37.5 > 10.83 which means it's even more than 99.9% significant. If instead we only came up with a Total of 4.5, that's > 3.84 so we'd say it was 95% significant.
Calculating Chi Square in real life
If you're lucky, you have a survey software or statistics program which will take your Observed values and crunch everything for you—some won't even make you specify a probability first.
If you don't have an application which makes this easy, try the on-line calculator Kristopher J. Preacher has posted on his site.
While Microsoft Excel has a CHITEST function, it takes a bit of hand work. You have to manually generate all the Expected values, and all it does is give you the Total Chi Square (our 37.5). To get the probability, you have to pair it with the CHIDIST function, manually giving it the degrees of freedom.
Applying Chi Square to surveys
Question types:
Chi square can be used with any pair of single answer discrete questions. This includes:
- Demographics
- Likert scales
- Cities, product names, instructor names, etc.
- Dates once they've been grouped into periods
- Numbers once they've been grouped into ranges
The answers do not need to be ordered, equal or symmetrical—just discrete. This is part of what makes Chi Square a handy statistic for surveys.
"Mark all that apply" questions cannot be used as an individual respondent cannot exist in more than one cell of our table. For example, a woman answering the survey can't appear in both the Yes and No columns.
Presenting the information:
While the statistic has to be calculated on the counts, that's not necessarily the best approach for our brains to spot patterns. For example in this table we have over 3 times the number of In Store respondents as On-line:
| Excellent | Good | Fair | Poor | Total | |
|---|---|---|---|---|---|
| On-line | 325 | 597 | 216 | 52 | 1,190 |
| In Store | 1,527 | 1,712 | 304 | 96 | 3,639 |
| Total | 1,852 | 2,309 | 520 | 148 | 4,829 |
In a report, it's easier for our brains to compare percentages:
| Excellent | Good | Fair | Poor | Total | |
|---|---|---|---|---|---|
| On-line | 27.3% | 50.2% | 18.2% | 4.4% | 100.0% 1,190 |
| In Store | 42.0% | 47.0% | 8.4% | 2.6% | 100.0% 3,639 |
| Total | 38.4% 1,852 |
47.8% 2,309 |
10.8% 520 |
3.1% 148 |
100.0% 4,829 |
You still want to keep the count totals in the report so that readers know the relative sizes of the groups.
Cross-tabs can also be well suited to graphical views, including stacked bar charts, bar graphs and line/profile graphs.
Low count cells:
The guidelines on this vary, but if you have more than one cell with 5 or fewer respondents, the final calculation may overstate your level of probability. If you do have this situation, either wait on this statistic until you have more data, or combine categories.
Dropping answer options:
In our original example, our column scale might have been "Yes/Uncertain/No." If the Uncertain column totaled 0, we would have to drop it as the Expected values for it would have all been 0. This means the difference calculation would be attempting to divide by 0, which is challenging.
Completely empty rows or columns are the only answers you should ever drop. Even if there was just 1 response in the Uncertain column, you need to include that individual in the table for the statistic to be reliable. We can, however, combine Uncertain with Yes or No if needed.
Combining categories:
This is used to increase the counts of cells when you have too many with infrequent responses, or simply to clarify the relationships for your analysis.
With an ordered scale such as a 5 level Likert, this could take the form of combining the upper and lower categories into a 3 level "Agree/Neither/Disagree" breakdown.
With unordered data such as product names, you might combine into categories. With city names you might group the information into geographic regions or urban/rural classifications.
The main issue is to make sure the categories are sufficiently related that you're not masking a relationship. When in doubt, first run the cross-tabulation and Chi Square on an expanded table, then start combining.
Questions left blank:
In surveys respondents will often skip one or both of the questions in your comparison. If this represents more than a couple people, you may want to add a "No Answer" or "Empty" row and column. Just as with non-response sampling errors, sometimes there's a relationship in the people who don't give an answer.
And that's Chi Square in a nutshell! (Or as close to nutshells as inferential statistics get.)
10 Comments
Note: New comments disabled for a few days while debugging.
Home
Articles
Need a Hand?
A little help can add a lot of polish—or just save hours and headaches:
Ann worked with us on an accelerated project... helping us ‘soup to nuts’ in defining our research objectives, methodology, survey design and ‘out of the box’ survey promotion ideas. She was a very fast learn, interacted beautifully with our client, and truly delivered on all commitments... exceeding expectations in a very short time period.
Bruce Scheer
Partner, CMO
MAARK
a simple to use chi square website that i have used before is www.jspearson.com/Science/chiSquare.html
Excellent post thanks
Very helpful. Thanks!
Thank you very much for a very helpful page. It is informative and well presented.
I intended to draft you the little bit of remark so as to thank you over again considering the remarkable knowledge you've featured on this page. It's quite strangely open-handed of you giving extensively what exactly most of us would've marketed for an e-book to help with making some cash for themselves, most notably considering that you could have tried it in case you considered necessary. The creative ideas likewise acted to become a easy way to be sure that some people have the same dreams much like my personal own to realize whole lot more with respect to this issue. I believe there are lots of more pleasurable moments ahead for people who looked at your website.
Thanks! I'll get the ebook out one of these days, but the blog will always be best for specialized topics like this one. Plus I'd rather write an article than pay for advertising or attend a networking luncheon ;-).
Wondering if you maybe wanted to start writing my blog for me cuz DAMN that was good!
Thanks! Need to get back to the keyboard for my own stuff first :-)
Great information! Your explanations are fabulous.
Interactive Chi Square table:
http://www.quantpsy.org/chisq/chisq.htm
Thank you so much for the link! I knew there had to be another calculator floating around somewhere. :-)