You gave the same answer. B for four of the five participants. So you accepted 80% of the opportunities. Your approval percentage in this example was 80%. The number of your pair of workshops may be higher or lower. So on a scale from zero (chance) to a (perfect), your approval in this example was about 0.75 – not bad! The most important result here is %-agree, i.e. Your agreement as a percentage. The number also shows the number of subjects you have assessed and the number of people who have done evaluations. The bit that says tolerance is 0 refers to an aspect of the percentage agreement that is not dealt with in this course. If you`re curious about tolerance in a percentage chord calculation, enter the help file into the console and read the help file for that command. Number of successive rating categories to be considered a tying agreement (see details). For completely cross-referenced designs with three or more coders, Light (1971) proposes to calculate Kappa for all pairs of coders, and then use the arithmetic average of these estimates to provide an overall index of match.
Davies and Fleiss (1982) propose a similar solution using the average P (e) between all pairs of coders to calculate a cappa-type statistic for multiple coders. Light, Davies and Fleiss solutions are not available in most statistical packages; However, the light solution can be easily implemented by calculating Kappa for all pairs of coders using statistical software, and then manually calculating the arithmetic average. Cohens Kappa`s estimate, which is obtained by pair of coders, is 0.68 (estimates of the pair of kappa codex – 0.62 [codes 1 and 2], 0.61 [codes 2 and 3] and 0.80 [coder 1 and 3], indicating a substantial agreement according to landis and koch (1977). In SPSS, only Kappa seals and Castellans are provided, and Kappa, average on pairs of coders, is 0.56, indicating a moderate agreement (Landis-Koch, 1977). According to the more conservative cutoffs of Krippendorff (1980), Cohen`s kappa estimate might indicate that conclusions on coding fidelity should be discarded, while Siegel-Castellan`s Kappa estimate may indicate that preliminary conclusions will be drawn. Reports on these results should detail the specifics of the chosen kappa variant, provide a qualitative interpretation of the estimate, and describe all the effects of the estimate on statistical performance. For example, the results of this analysis may be reported as follows: Possible values for Kappa statistics range from 1 to 1, 1 indicating perfect match, 0 indicating a totally random match, and 1 indicating “perfect” differences. Landis and Koch (1977) provide guidelines for interpreting Kappa`s values, values between 0.0 and 0.2 being slightly consistent, 0.21 to 0.40, indicating fair consent, from 0.41 to 0.60, indicating moderate support, from 0.61 to 0.80 and from 0.81 to 1.0, indicating near-perfect or perfect consistency.
The use of these qualitative limit values, however, is under discussion and Krippendorff (1980) gives a more conservative interpretation that suggests that conclusions should be reduced for variables with values below 0.67, conclusions for values between 0.67 and 0.80, and final conclusions for values greater than 0.80. However, in practice, Kappa coefficients below Krippendorff`s conservative Cutoff values are often retained in research studies, and Krippendorff proposes these cutoffs on the basis of his own content analysis work, while acknowledging that acceptable estimates of IRR vary according to study methods and the research issue. The spSS and the R-pack require users to indicate a single or two-way model, an absolute type of match or consistency, as well as individual or medium units. The design of the hypothetical study provides information on the correct choice of ICC variants. Note that the SPSS, but not the R-irr package, allows a user to indicate random or mixed effects, the calculation and results are identical for random and mixed effects.