Investigating False Pos/Neg

I ask this question in the hope of being steered in the right direction.

I am running several M.L. models (RF, SVM, etc.) using caret. I would like to generate a confusion matrix which will allow me (or facilitate me) to investigate the actual observations that have been mis-classified. Is this possible?

For example, my dataset contains protein sequences as my observations. What I would like to find is a list of the proteins that have been mis-classified. My working hypothesis is quite simple. Do proteins of length less than X (eg < 25 polypeptides in length) have a higher rate of being miss-classified?

The savePredictions option can be used to get the holdout predictions and combine them with the original data:

#> Loading required package: lattice
#> Loading required package: ggplot2
#> Registered S3 methods overwritten by 'ggplot2':
#>   method         from 
#>   [.quosures     rlang
#>   c.quosures     rlang
#>   print.quosures rlang
#> Registered S3 method overwritten by 'rvest':
#>   method            from
#>   read_xml.response xml2

ctrl <- trainControl(method = "cv", savePredictions = "final")
mod <- train(Species ~ ., data = iris, method = "lda", trControl = ctrl)

out_of_sample <- 
  mod %>% 
  pluck("pred") %>% 
  dplyr::filter(obs != pred)
#>   parameter       pred        obs rowIndex Resample
#> 1      none  virginica versicolor       71   Fold01
#> 2      none versicolor  virginica      134   Fold01
#> 3      none  virginica versicolor       84   Fold07

iris %>% 
  mutate(rowIndex = 1:nrow(iris)) %>% 
  inner_join(out_of_sample, by = "rowIndex")
#>   Sepal.Length Sepal.Width Petal.Length Petal.Width    Species rowIndex
#> 1          5.9         3.2          4.8         1.8 versicolor       71
#> 2          6.0         2.7          5.1         1.6 versicolor       84
#> 3          6.3         2.8          5.1         1.5  virginica      134
#>   parameter       pred        obs Resample
#> 1      none  virginica versicolor   Fold01
#> 2      none  virginica versicolor   Fold07
#> 3      none versicolor  virginica   Fold01

Created on 2019-04-11 by the reprex package (v0.2.1)

1 Like

Take a look at Sec 28.4.3 of

Basically, you divide your data into a test and training set; train your model on the training set and get a predicted outcome (y_hat), then run the model on the test set and use caret::confusionMatrix to get diagnostics

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.

If you have a query related to it or one of the replies, start a new topic and refer back with a link.