Get "raw" data from yardstick::conf_mat() for use with infer

Machine Learning and Modeling
, ,
1

Can anyone help optimise this workflow? I'd like to skip the steps where I manually create "sensitivity" and "sens_hat" and somehow get these data from "cfx" and "summary(cfx)". Any ideas?

library(tidymodels)
options(yardstick.event_first = FALSE) 

set.seed(2018)
x <- tibble(pred = ordered(rbinom(20, 1, 0.5)), 
            truth = ordered(rbinom(20, 1, 0.5)))

cfx <- conf_mat(x, truth, pred)
cfx
#>           Truth
#> Prediction 0 1
#>          0 5 4
#>          1 6 5

summary(cfx)
#> # A tibble: 12 x 2
#>    name         value
#>    <chr>        <dbl>
#>  1 accuracy    0.5   
#>  2 kappa      NA     
#>  3 sens        0.556 
#>  4 spec        0.455 
#>  5 prevalence  0.45  
#>  6 ppv         0.455 
#>  7 npv         0.556 
#>  8 mcc         0.0101
#>  9 j_index     0.0101
#> 10 precision   0.455 
#> 11 recall      0.556 
#> 12 F1          0.5

sensitivity <- tibble(sens = c("fn", "fn", "fn", "fn", 
                               "tp","tp", "tp", "tp", "tp"))
sens_hat <- sensitivity %>%
  specify(response = sens, success = "tp") %>% 
  calculate(stat = "prop")

sens_hat
#> # A tibble: 1 x 1
#>    stat
#>   <dbl>
#> 1 0.556

boot <- sensitivity %>% 
  specify(response = sens, success = "tp") %>% 
  generate(reps = 1000, type = "bootstrap") %>% 
  calculate(stat = "prop")

get_ci(boot)
#> # A tibble: 1 x 2
#>   `2.5%` `97.5%`
#>    <dbl>   <dbl>
#> 1  0.222   0.889

Created on 2018-09-09 by the reprex
package (v0.2.0).

2

We are in the process of adding confidence intervals to yardstick. For any of the metrics that are in the form numer/denom, you can use basic R to get the interval:

> prop.test(cfx$table["1", "1"], sum(cfx$table[, "1"]))

	1-sample proportions test with continuity correction

data:  cfx$table["1", "1"] out of sum(cfx$table[, "1"]), null probability 0.5
X-squared = 0, df = 1, p-value = 1
alternative hypothesis: true p is not equal to 0.5
95 percent confidence interval:
 0.23 0.85
sample estimates:
   p 
0.56
3

Thank you! Will there be options for different methods to calculate confidence intervals? E.g. if I'd like to use bootstrapping instead of a proptest?

I had a look at the code for the functions, although I'm very new to this I got the feeling I cant grab what I want from the output of yardstick. My assumption here might not even be correct, but if I wanted to do a bootstrap with infer, wouldn't I need a vector like the one above (sensitivity) for each metric?

One truth column and one estimate column merged to a new column representing the proportion of the metric in question?

4

I guess this is the column I want, preferably for all metrics in one go..:

y <- x %>% 
  transmute(senscol = case_when(
    pred == 1 & truth == 1 ~ 1,
    pred == 0 & truth == 1 ~ 0
  )) %>% 
  filter(!is.na(senscol))

Created on 2018-09-09 by the reprex
package (v0.2.0).

Does conf_mat() do anything similar do this at any point, or do you get the values from sums of positive/negative?

5

My plan was to only use the bootstrap when there was no other alternative. Let me think about that.

I'm thinking that it would probably be easier to run infer on the original columns than on the table. My bootstrapping plan inside of yardstick was to do multinomial sampling on the table entries based on their cell proportions (maybe with an empty cell adjustment).

I haven't seen any infer examples where an arbitrary function can be passed to calculate (but I put an issue in).

If you want to estimate them, rsample can be used:

> library(tidymodels)
> data("two_class_example")
> 
> get_sens <- function(split) {
+   analysis(split) %>% 
+   sens(truth = truth, estimate = predicted)
+ }
> 
> set.seed(1434)
> bt <- bootstraps(two_class_example, times = 2000) %>%
+   mutate(sens = map_dbl(splits,  get_sens))
> 
> sens_ci <- quantile(bt[["sens"]], probs = c(0.025, 0.975))
> sens_ci
     2.5%     97.5% 
0.8376233 0.9173309 
>

We have a devel version of rsample with better bootstrap confidence intervals but it's not quite ready yet.

6

It looks like you can use some infer functions with the code above. For example:

obs_sens <-
  tibble(
    stat = two_class_example %>% sens(truth = truth, estimate = predicted)
  )
visualize(bt, obs_stat = obs_sens)

works fine.

7

I had a look at rsample before I tried infer. I have no experience using list columns so I found it a bit difficult to understand what's going on, but your example could serve as a template going forward. Would you use the same method for getting the p value from permutations?

Thank you for taking the time to help a noob, I really appreciate it!