Fleiss’ kappa
Agreement among three or more raters.
When to use it
Use Fleiss’ kappa when three or more raters each code the same cases. Cohen’s kappa is the two-rater version.
Assumptions
Each column is one rater. Every rater uses the same categories. Tensr needs at least three rater columns.
Running it in Tensr
Options
Prop
Type
Reading the output
Three raters on the same 80 essays. Agreement beyond chance should be moderate, not perfect.
| Category | Proportion |
|---|---|
| 1 | 0.279 |
| 2 | 0.417 |
| 3 | 0.304 |
Fleiss' κ for 3 raters across 80 subjects. The primary result is significant (p = < .001). This is large enough, in this sample, to treat the comparison this page is about as a real association rather than noise. Metrics: κ = 0.447; SE = 0.065; z = 6.93; p-value = < .001; 95% CI = [0.321, 0.574].
Reporting (APA 7)
Fleiss' κ for 3 raters across 80 subjects. This result is significant (p = < .001). The effect is moderate, so report its size with the p-value.
Coming from SPSS
Analyze → Scale → Fleiss Kappa.