Can anyone help optimise this workflow? I'd like to skip the steps where I manually create "sensitivity" and "sens_hat" and somehow get these data from "cfx" and "summary(cfx)". Any ideas?

We are in the process of adding confidence intervals to yardstick. For any of the metrics that are in the form numer/denom, you can use basic R to get the interval:

> prop.test(cfx$table["1", "1"], sum(cfx$table[, "1"]))
1-sample proportions test with continuity correction
data: cfx$table["1", "1"] out of sum(cfx$table[, "1"]), null probability 0.5
X-squared = 0, df = 1, p-value = 1
alternative hypothesis: true p is not equal to 0.5
95 percent confidence interval:
0.23 0.85
sample estimates:
p
0.56

Thank you! Will there be options for different methods to calculate confidence intervals? E.g. if I'd like to use bootstrapping instead of a proptest?

I had a look at the code for the functions, although I'm very new to this I got the feeling I cant grab what I want from the output of yardstick. My assumption here might not even be correct, but if I wanted to do a bootstrap with infer, wouldn't I need a vector like the one above (sensitivity) for each metric?

One truth column and one estimate column merged to a new column representing the proportion of the metric in question?

My plan was to only use the bootstrap when there was no other alternative. Let me think about that.

I'm thinking that it would probably be easier to run infer on the original columns than on the table. My bootstrapping plan inside of yardstick was to do multinomial sampling on the table entries based on their cell proportions (maybe with an empty cell adjustment).

I haven't seen any infer examples where an arbitrary function can be passed to calculate (but I put an issue in).

If you want to estimate them, rsample can be used:

I had a look at rsample before I tried infer. I have no experience using list columns so I found it a bit difficult to understand what's going on, but your example could serve as a template going forward. Would you use the same method for getting the p value from permutations?

Thank you for taking the time to help a noob, I really appreciate it!