custom prediction handler for conformal inference with vetiver

Machine Learning and Modeling
, ,
1

I have a modeling workflow that uses conformal inference for prediction intervals and I'm trying to use the following workflow to deploy my models to Posit Connect:

  • tune/finalize models using tidymodels workflow
  • implement conformal inference using
    split_int <- int_conformal_split(fitted_workflow_object, validation)
  • pin models using vetiver_pin_write()
  • promote each pinned model to an API using vetiver_deploy_rsconnect()

The probably docs indicate i can just use predict(split_int, new data, level= 0.90) to get prediction and intervals. My initial thought was to pass split_int into the vetiver_model, but the vetiver docs state that the model argument in vetiver_model should be a "A trained model, such as an lm() model or a tidymodels workflows::workflow()."

As expected when using the conformal split object as the model, it fails on the server:

image
image882×378 14.7 KB

and when i call the endpoint from my session i get:

Error:
! lexical error: invalid char in json text.
                                       <!DOCTYPE html> <html lang="en-
                     (right here) ------^
Hide Traceback
    ▆
 1. ├─stats::predict(...)
 2. └─vetiver:::predict.vetiver_endpoint(...)
 3.   └─jsonlite::fromJSON(resp)
 4.     └─jsonlite:::parse_and_simplify(...)
 5.       └─jsonlite:::parseJSON(txt, bigint_as_char)
 6.         └─jsonlite:::parse_string(txt, bigint_as_char)

Do i need to define a custom handlers since the class of split_int is not a tuned workflow which vetiver_model expects? if so, how do i do this and where in the workflow does it fit? I check the vetiver docs on this but couldn't get very far with it.

Below is a pseudo-reprex - i wanted to mask our orgs connect endpoint:

library(workflows)
library(dplyr)
library(parsnip)
library(rsample)
library(tune)
library(modeldata)
library(probably)
library(vetiver)
library(pins)

set.seed(2)
sim_train <- sim_regression(500)
sim_cal <- sim_regression(200)
sim_new <- sim_regression(5) |> dplyr::select(-outcome)

mlp_spec <-
  mlp(hidden_units = 5, penalty = 0.01) |>
  set_mode("regression")

mlp_wflow <-
  workflow() |>
  add_model(mlp_spec) |>
  add_formula(outcome ~ .)

mlp_fit <- fit(mlp_wflow, data = sim_train)

mlp_int <- probably::int_conformal_split(mlp_fit, sim_cal)
mlp_int

predict(mlp_int, sim_new, level = 0.90)

# use the int_conformal_split as the model object
v <- vetiver_model(model = mlp_int, "mlp_fit_TEST_DELETE", description = "TEST", save_prototype = FALSE)
v

board <- board_connect(auth = "envvar")
board %>% vetiver_pin_write(v)

vetiver_deploy_rsconnect(
  board = board,
  name = "mlp_fit_TEST_DELETE",
  appTitle = "conf inf test",
  predict_args = list(debug = TRUE)
)

apiKey <- Sys.getenv("CONNECT_API_KEY")

#errors out with above error
preds <-
  predict(
    object = endpoint, #real endpoint masked, replace with real one for reprex to work
    type = "prob",
    level = 0.90,
    new_data = sim_new,
    httr::add_headers(Authorization = paste("Key", apiKey))
  )
2

Thank you for reporting this! What we need to do is to add support for probably objects to vetiver, similar to how we added support for stacks. This is not going to work until vetiver knows what probably objects need to be deployed.

Would you be up for opening a feature request for this?

3

Yes, happy to!

In the meantime i was able to sucessfully find a workaround that i'll describe here in case anyone comes across this. Definitely feels hack-y but it works.

The output of int_conformal_split contains a workflow, and 2 other objects: resid and n. I just split these up into 2 pins, one of which is a vetiver_model and the other is a just a "regular" pin of a list containing resid and n:


split_int <- int_conformal_split(fitted_workflow_object, validation)

# one pin for the workflow
model <- vetiver_model(split_int$wflow)
vetiver_pin_write(board, model)

# one pin for the conformal inference "metadata"
conformal_inference_metadata <-
  list(
    conf_inf_n = split_int$n,
    conf_inf_resid = split_int$resid
  )

pin_write(
  board = board,
  x = conformal_inference_metadata
)

Then to deploy the app, modify the plumber.R file:

  1. read in the conformal_inference_metadata pin
  2. add a confrmal prediction function
  3. add a custom /predict_conformal endpoint
# Generated by the vetiver package; edit with care

library(pins)
library(plumber)
library(rapidoc)
library(vetiver)
library(dplyr)

# Packages needed to generate model predictions
if (FALSE) {
    library(parsnip)
    library(ranger)
    library(recipes)
    library(workflows)
}

b <- board_connect(auth = "envvar")
v <- vetiver_pin_read(b, "username/model")
conf_inf <- pin_read(b, "username/conformal_inference_metadata")

conf_inf_n <- conf_inf$conf_inf_n
conf_inf_resid <- conf_inf$conf_inf_resid

predict_conformal <-
  function(model, new_data, level = 0.90, ...) {

    wf <- bundle::unbundle(model$model)

    new_pred <- predict(wf, new_data)

    alpha <- 1 - level
    q_ind <- ceiling(level * (conf_inf_n + 1))
    q_val <- conf_inf_resid[q_ind]

    new_pred$.pred_lower <- new_pred$.pred - q_val
    new_pred$.pred_upper <- new_pred$.pred + q_val
    new_pred
  }

#* @plumber
function(pr) {
  pr %>%
    vetiver_api(v) %>%

    # Add custom conformal prediction endpoint
    plumber::pr_post("/predict_conformal",
      function(req, res, level = 0.90) {
        # Parse the incoming JSON data
        new_data <- jsonlite::fromJSON(req$postBody)

        new_data <-
          new_data %>%
          mutate(across(ends_with("_date"), ~as.Date(.x)))

        new_data <- vetiver_type_convert(new_data, v$ptype)

        # for debugging in logs
        print(str(new_data))

        # Make conformal prediction
        result <-
          predict_conformal(
          v,
          new_data,
          level = 0.90
        )
        # Return pred w/ pred intervals
       return(result)
      }
    ) %>%
    vetiver_pr_docs(v)
}

then deploy as normal using:

deployAPI(
  plumber.R,
  appName = some_name,
  appTitle = "Some title",
)

When it comes time to predict use the /predict_conformal endpoint and specify the level:

predict(
 vetiver_endpoint(
  "https:some_big_url/predict_conformal"
  ),
  new_data = data,
  level = 0.90,
  httr::add_headers(Authorization = paste("Key", apiKey))
)