Comparative Evaluation of Statistical Methods for Detecting Rater Bias in Ordinal Categorical Data
Rachelle P. Tapio *
La Salle University, Ozamiz City, Philippines.
*Author to whom correspondence should be addressed.
Abstract
Aims: This study seeks to examine four statistical methods Modified McNemar Test, Single Binomial Test, Marginal Homogeneity Test, and Bias Index for identifying bias between two raters utilizing ordinal categorical data.
Study Design: This study employs a simulation-based comparative approach, using hypothetical 3×3 contingency tables to illustrate low, moderate, and high levels of agreement.
Place and Duration of Study: Executed by a teaching member of a private university in the Philippines, from January 2024 to April 2025.
Methodology: Three simulated contingency tables representing varying agreement levels were analyzed using the four statistical methods. Test statistics and p-values were computed for each, and the Bias Index was calculated to assess directional bias. The outcomes were evaluated comparatively.
Results: Under low agreement conditions, all methods detected substantial bias, as reflected in high test statistics and low p-values. For moderate agreement, only the Marginal Homogeneity Test and Bias Index showed strong sensitivity. In the high agreement scenario, all approaches yielded non-significant results, and the Bias Index confirmed the absence of directional bias. Across all simulations, the Bias Index consistently produced interpretable and stable results.
Conclusion: While each method offers useful insights, the Bias Index emerged as the most robust and interpretable tool for assessing rater bias in ordinal data. A combined use of complementary statistical tests is recommended to determine both the direction and magnitude of bias in rater agreement studies.
Keywords: Statistical methods, rater bias, ordinal categorical data, bias index, modified McNemar test