Understanding Chi Square

Chi Square lets you know whether two groups have significantly different opinions, which makes it a very useful statistic for survey research. It's applied to cross-tabulations (AKA pivot tables) which are simply breakdowns like this:

  Yes No Total
Female 45 5 50
Male 15 35 50
Total 60 40 100

This article starts with the theory, and then has guidelines for using the statistic:

Understanding the calculations

When we eyeball our table above, it looks like women are much more likely to answer Yes, but is it random variation or something we can count on? What Chi Square does is compare the actual or Observed data we have from respondents with an Expected value. In our two questions, the total answers are:

Female 50
Male 50
Yes 60
No 40

If there were no relationship between the questions, then you would Expect a table that allocates those totals to look like this:

  Yes No Total
Female 30 20 50
Male 30 20 50
Total 60 40 100

The formula for the upper-left cell is:

(TotalYes * TotalFemale) / TotalTable
( 60 * 50 ) / 100

In less tidy examples, the Expected values often have a decimal or two. Once we have all the Expected values, we need to find the difference squared (so they're all positive) between the individual cells' Expected and Observed values:

D = ((O - E)2 / E)

  Yes No Total
Female E: 30
O: 45
D: 7.50
E: 20
O: 5
D: 11.25
E&O: 50
Male E: 30
O: 15
D: 7.50
E: 20
O: 35
D: 11.25
E&O: 50
Total E&O: 60 E&O: 40 E&O: 100
D: 37.5

Adding all the differences, we get a Total Chi Square of 37.5—which is yet another interim value in this calculation. So on to the next stage.

Many statistics rely on a concept called Degrees of freedom. The details vary stat to stat, but it's based on the number of variables involved in a calculation. For Chi Square, the degrees of freedom are:

df = (# rows - 1) * (# columns - 1)
= ( 2 - 1) * ( 2 - 1) = 1

In our cast we now have:

  • Assorted Observed and Expected values
  • Total Chi Square = 37.5
  • Degrees of freedom = 1

We have two more players, and those are the Probability and Critical Value.

Any time you have a statistic designed to "predict" for a larger population or tell you a value's validity or reliability, part of the calculation is a level of confidence. Sometimes you'll see this indicated as the level of risk such as 5%, and at other times it will be noted as the level of certainty, 95%. For Chi Square, the tables are based on the level of risk, with common thresholds of 10%, 5%, 2.5%, 1% and 0.1%. Each one of those risk levels has a Critical Value associated with it:

Probability Critical Value
when df = 1
10.0% 2.71
5.0% 3.84
2.5% 5.02
1.0% 6.64
0.1% 10.83
(More values—see the "Upper" table)

Our final step to calculate Chi Square is to compare our Total to the Critical Values. In our case, 37.5 > 10.83 which means it's even more than 99.9% significant. If instead we only came up with a Total of 4.5, that's > 3.84 so we'd say it was 95% significant.

Calculating Chi Square in real life

If you're lucky, you have a survey software or statistics program which will take your Observed values and crunch everything for you—some won't even make you specify a probability first.

If you don't have an application which makes this easy, try the on-line calculator Kristopher J. Preacher has posted on his site.

While Microsoft Excel has a CHITEST function, it takes a bit of hand work. You have to manually generate all the Expected values, and all it does is give you the Total Chi Square (our 37.5). To get the probability, you have to pair it with the CHIDIST function, manually giving it the degrees of freedom.

Applying Chi Square to surveys

Question types:

Chi square can be used with any pair of single answer discrete questions. This includes:

  • Demographics
  • Likert scales
  • Cities, product names, instructor names, etc.
  • Dates once they've been grouped into periods
  • Numbers once they've been grouped into ranges

The answers do not need to be ordered, equal or symmetrical—just discrete. This is part of what makes Chi Square a handy statistic for surveys.

"Mark all that apply" questions cannot be used as an individual respondent cannot exist in more than one cell of our table. For example, a woman answering the survey can't appear in both the Yes and No columns.

Presenting the information:

While the statistic has to be calculated on the counts, that's not necessarily the best approach for our brains to spot patterns. For example in this table we have over 3 times the number of In Store respondents as On-line:

  Excellent Good Fair Poor Total
On-line 325 597 216 52 1,190
In Store 1,527 1,712 304 96 3,639
Total 1,852 2,309 520 148 4,829

In a report, it's easier for our brains to compare percentages:

  Excellent Good Fair Poor Total
On-line 27.3% 50.2% 18.2% 4.4% 100.0%
In Store 42.0% 47.0% 8.4% 2.6% 100.0%
Total 38.4%

You still want to keep the count totals in the report so that readers know the relative sizes of the groups.

Cross-tabs can also be well suited to graphical views, including stacked bar charts, bar graphs and line/profile graphs.

Low count cells:

The guidelines on this vary, but if you have more than one cell with 5 or fewer respondents, the final calculation may overstate your level of probability. If you do have this situation, either wait on this statistic until you have more data, or combine categories.

Dropping answer options:

In our original example, our column scale might have been "Yes/Uncertain/No." If the Uncertain column totaled 0, we would have to drop it as the Expected values for it would have all been 0. This means the difference calculation would be attempting to divide by 0, which is challenging.

Completely empty rows or columns are the only answers you should ever drop. Even if there was just 1 response in the Uncertain column, you need to include that individual in the table for the statistic to be reliable. We can, however, combine Uncertain with Yes or No if needed.

Combining categories:

This is used to increase the counts of cells when you have too many with infrequent responses, or simply to clarify the relationships for your analysis.

With an ordered scale such as a 5 level Likert, this could take the form of combining the upper and lower categories into a 3 level "Agree/Neither/Disagree" breakdown.

With unordered data such as product names, you might combine into categories. With city names you might group the information into geographic regions or urban/rural classifications.

The main issue is to make sure the categories are sufficiently related that you're not masking a relationship. When in doubt, first run the cross-tabulation and Chi Square on an expanded table, then start combining.

Questions left blank:

In surveys respondents will often skip one or both of the questions in your comparison. If this represents more than a couple people, you may want to add a "No Answer" or "Empty" row and column. Just as with non-response sampling errors, sometimes there's a relationship in the people who don't give an answer.

And that's Chi Square in a nutshell! (Or as close to nutshells as inferential statistics get.)


Note: New comments disabled for a few days while debugging.

a simple to use chi square website that i have used before is www.jspearson.com/Science/chiSquare.html

Very helpful. Thanks!

Thank you very much for a very helpful page. It is informative and well presented.

I intended to draft you the little bit of remark so as to thank you over again considering the remarkable knowledge you've featured on this page. It's quite strangely open-handed of you giving extensively what exactly most of us would've marketed for an e-book to help with making some cash for themselves, most notably considering that you could have tried it in case you considered necessary. The creative ideas likewise acted to become a easy way to be sure that some people have the same dreams much like my personal own to realize whole lot more with respect to this issue. I believe there are lots of more pleasurable moments ahead for people who looked at your website.

Thanks! I'll get the ebook out one of these days, but the blog will always be best for specialized topics like this one. Plus I'd rather write an article than pay for advertising or attend a networking luncheon ;-).

Wondering if you maybe wanted to start writing my blog for me cuz DAMN that was good!

Thanks! Need to get back to the keyboard for my own stuff first :-)

Great information! Your explanations are fabulous.

Interactive Chi Square table:

Thank you so much for the link! I knew there had to be another calculator floating around somewhere. :-)

Need a Hand?

A little help can add a lot of polish—or just save hours and headaches:

(206) 399-2344 Download VCard LinkedIn Profile

Thanks so much for such a terrific training class. I really enjoyed it and learned a lot. Most importantly, I've got a lot more confidence now in using SurveyPro.

Thomas Hier
Biddison Hier