r + ML : predict()

[supermarco]

Hello

I run r shiny to do prediction on ML (adaboost, lasso, nnet, rf) and the (predict(fit_model, newdata, interval = "confidence", level = 0.95) works ONLY for OLS, for all others in ML it retuns no confidence interval. Is it normal?

Thanks a lot

[nirgrahamuk]

yes its normal.
OLS gives you confidence intervals through an analytic analysis, the validity being contingent on certain assumptions being true.
This isnt generally the case for modelling/ machine learning techniques in general.

the most general approach to confidence intervals would be achieved through bootstrapping methods

[Max]

@supermarco

Is it normal?

It is and it isn't specific to R. Most models can't produce confidence intervals; that requires the model to have some formal probability model. Models like boosted tree or glmnet do not.

Also, if you are doing prediction, it is a lot more relevant to use prediction intervals. These have the same issue though; not all models can produce them.

However, there are two tools that you can use to produce prediction intervals for any model.

Conformal inference is a method that you can use after you have a final fitted model. It doesn't do very well with small data sets but you can get intervals.

Boostrap prediction intervals are very good at making intervals but require you to refit the model thousands of times.

[supermarco]

Thanks a lot. Very clear. What i understand is

• for ML, preferable doing prediction with a same function like Caret to extract confidence interval (need to find how) instead of several different function (costly to see each particular case how to extract CI). I would enjoy to see which function call them? Again thanks a lot.

[supermarco]

If possible i have second issue that is surprising. I 'm testing ml models for prediction. On R all work. On R shiny, lasso and ridge do not provide prediction with the following error message:
"error evaluating argument 'x' when selecting a method for function 'as.matrix': method not yet implemented for <data.frame> %*% <dgCMatrix"
=> I saw that this issue happens to other people, i did a class () to my model fit and i receive this:

class(prediction_lasso)
[1] "matrix" "array"
class(predictions_ridge)
[1] "matrix" "array"
class(prediction_ols)
[1] "numeric"
=> my input data is built with : data.matrix()

Please any opinion on why it works on r but not shiny? Thanks a lot

[nirgrahamuk]

You are either making a shiny mistake, or a working with your models mistake.
We can't see your code so we can't say.
If you can make a plain script in R without shiny, that does what you want with the models; then you probably identify that you are making some shiny related mistake.

I'll link you to the shiny debugging and reprex guide all the same.

Shiny debugging and reprex guide

Shiny issues can be challenging to resolve relative to other problems with your code or statistical methods. Shiny apps are often large, complex projects with interacting files. When seeking help from others it is considered polite to: First, do your best to work through RStudio's debugging tools to diagnose your issue on your own. Often those shiny logs and tracebacks are useful to others trying to help out. Second, strive to minimize the effort required to replicate your issue. You can do this with a reproducible example ("reprex"). Shiny Debugging Errors in Shiny code can be difficult to track down. If you don't know where your problem is coming from, you can track it down with some o…

[supermarco]

Thank you very much for your message.
Sure I run the basic code in ML prediction under R and it works. I call this code under shiny and it works for several models except lasso and ridge. I saw there same issue with other people on line. It seems that it concerns the type of framework of the prediction output (as. Matrix, data.frame etc).

[supermarco]

Sorry I don’t have RStudio