show percent agreement by item/unit when examining intercoder reliability

[sgenter]

tidycomm and irr both offer very nice options for calculating various statistics of intercoder reliability, but to my knowledge they do not offer options to show the intermediate calculations of agreement for each item or unit. The example below shows what I mean using percent agreement as an example. Having the intermediate calculations could be helpful for reviewing coding with a team because p_agree could be arranged easily to identify items with low agreement.

I've looked around quite a bit to find a function that could show agreement by item/unit for various statistics and that would take an arbitrary number of coders because the example approach below would be difficult to repurpose for different stats (e.g., Krippendorff's alpha) and for different numbers of coders. Does anyone know of a package or function out there that I've overlooked? If not, is there a simple solution to calculating percent agreement (to take a simple stat as a starting point) for each item for an arbitrary number of coders?

Thanks

 library(tidyverse)

 irr_data <- tibble(r1 = rep(c("yes", "no"), times = 5),
                    r2 = rep(c("yes", "no"), each = 5),
                    r3 = c(rep("yes", times = 3),
                           rep("no", times = 5),
                           rep("yes", times = 2)))
 
 irr_data %>% 
   mutate(r1_r2 = if_else(r1 == r2, 1, 0),
          r1_r3 = if_else(r1 == r3, 1, 0),
          r2_r3 = if_else(r2 == r3, 1, 0),
          n_agree = rowSums(across(r1_r2:r2_r3)),
          p_agree = n_agree/3) %>% 
   arrange(p_agree)
#> # A tibble: 10 x 8
#>    r1    r2    r3    r1_r2 r1_r3 r2_r3 n_agree p_agree
#>    <chr> <chr> <chr> <dbl> <dbl> <dbl>   <dbl>   <dbl>
#>  1 no    yes   yes       0     0     1       1   0.333
#>  2 no    yes   no        0     1     0       1   0.333
#>  3 yes   yes   no        1     0     0       1   0.333
#>  4 yes   no    no        0     0     1       1   0.333
#>  5 yes   no    yes       0     1     0       1   0.333
#>  6 no    no    yes       1     0     0       1   0.333
#>  7 yes   yes   yes       1     1     1       3   1    
#>  8 yes   yes   yes       1     1     1       3   1    
#>  9 no    no    no        1     1     1       3   1    
#> 10 no    no    no        1     1     1       3   1

^{Created on 2022-11-23 by the reprex package (v2.0.1)}