Evaluator compliance is important in clinical research and cohens Kappa is a widely used method of assessing inter-rate reliability; However, there are well-documented statistical problems with measurement. To assess its usefulness, we compared it to Gwets` AC1 and compared the results. There are a number of statistics that can be used to determine reliability between evaluators. Different statistics are adapted to different types of measures. Some options are the common probability of an agreement, cohens Kappa, Scotts Pi and the related fleiss-Kappa, inter-rater correlation, concordance correlation coefficient, intraclass correlation and Krippendorffs Alpha. Gisev N, Bell JS, Chen TF: Interrater Agreement and Interrater Reliability: Key concepts, approaches, and applications. Res Social Adm Pharm. Im Druck, Ingenhoven TJ, Duivenvoorden HJ, Brogtrop J, Lindenborn A, van den Brink W, Passchier J: Interrater Reliability for kernberg`s structural interview on the assessment of personality organization. J Pers Disord.
2009, 23: 528-534. 10.1521/pedi.2009.23.5.528. There are several formulas that can be used to calculate compliance limits. The simple formula given in the previous paragraph, which works well for sample sizes greater than 60, is as follows: there are actually two categories of reliability when it comes to data collectors: reliability beyond multiple data collectors, which is interracter reliability, and reliability of a single data collector called intrarater reliability. .