How to Measure Inter-Rater Reliability When Subjects Fall into Multiple Categories
Explainer video | Generalizing Fleiss' kappa: Measuring agreement when subjects have multiple categories
👉 This explainer video is based on the article:
Moons, F. & Vandervieren, E. (2025). Measuring agreement among several raters classifying subject into one or more (hierarchical) categories: A generalization of Fleiss' kappa. Behavior Research Methods. https://doi.org/10.3758/s13428-025-02...
📥 Download the Excel add-in for applying the generalized Fleiss’ kappa:
Add-in: https://osf.io/q5nft/files/osfstorage...
Worksheet used in this video: https://osf.io/q5nft/files/osfstorage...
❗ Remark: First install the add-in, next open the worksheet.
📚 Table of contents of this explainer video
1:12 When to use it?
1:34 Example
3:22 When do Cohen's & Fleiss' kappa fail?
3:59 How to calculate it?
6:09 How to interpret it?
8:29 Advanced topics for possible future videos
If you found this video helpful, don’t forget to subscribe and hit the 🔔 to get notified about future tutorials on inter-rater reliability, agreement statistics, and practical tools for researchers.