Internal Server Error from Vetiver hosted Torch model

Posit Professional & Hosted Products Posit Connect
,
1

I'm trying to deploy a torch model using vetiver to host an API endpoint on our posit connect server. Everything runs smooth when I try to debug locally and the deployment to the server also goes fine without any errors.

The all endpoints of the API works except predict that returns "Internal Server Error" when trying with the dummy data, how should I proceed debugging and finding the cause for this?

This is my deployment script

import pandas as pd
import torch
import os
import vetiver
import pins
from mlp_model import MLPModel
from rsconnect.api import RSConnectServer

# Silence model_card warning
vetiver.utils.modelcard_options.quiet = True

# Path to state dict file
state_dict_path = r"mlp_model_state_dict.pt"

# Rebuild model and load weights
input_length, feats, hidden_sizes, output_size = 378, 1, (64, 8), 1
mlp = MLPModel(input_length, feats, hidden_sizes, output_size)
mlp.load_state_dict(torch.load(state_dict_path , map_location="cpu"))
mlp.eval()

# Define dummy data
INPUT_LEN = 378
FEATURES = [f"x{i}" for i in range(INPUT_LEN)]
X_proto = pd.DataFrame([[0.0]*INPUT_LEN], columns=FEATURES).astype("float32")

pin_name = "mlp_model"

# Vetiver model 
v = vetiver.VetiverModel(
    model=mlp, 
    model_name=pin_name, 
    prototype_data=X_proto, 
    handler_predict=vetiver.TorchHandler(
        model=mlp,
        prototype_data=X_proto
    ),
    versioned=True,
    description="Test of Torch model"
)

# Verify model, this runs OK
# y_vetiver = v.handler_predict(X_proto, check_prototype = True)
# print("OK vetiver:", y_vetiver)

#  Connect board and pin 
connect_server = RSConnectServer(
    url= os.environ["CONNECT_SERVER"], 
    api_key= os.environ["CONNECT_API_KEY"]
)

board = pins.board_connect(
    server_url=os.environ["CONNECT_SERVER"], 
    api_key=os.environ["CONNECT_API_KEY"], 
    allow_pickle_read=True
)

# Write pin
vetiver.vetiver_pin_write(board=board, model=v)

# Deploy
vetiver.deploy_connect(  
    connect_server=connect_server,
    board=board,
    pin_name=pin_name,
    title="VetiverTorchTest",
    extra_files=['requirements.txt','mlp_model.py'],  
    new=True,              
    app_id=None,            
    python=None,
    force_generate=False,  
    log_callback=None
)
2

Have you looked at the content logs? Those may provide more information about what the error looks like on the server.

3

Yes, should have posted that, its a lot of rows with internal lines. One that seems to be a issue I can understand is this one

2025/10/10 4:12:15 PM:
File "/opt/rstudio-connect/mnt/app/python/env/lib/python3.12/site-packages/vetiver/handlers/torch.py", line 48, in handler_predict
2025/10/10 4:12:15 PM:
prediction = self.model(torch.from_numpy(input_data))
2025/10/10 4:12:15 PM:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025/10/10 4:12:15 PM:
TypeError: can't convert np.ndarray of type numpy.object_. The only supported types are: float64, float32, float16, complex64, complex128, int64, int32, int16, int8, uint64, uint32, uint16, uint8, and bool.

4

I have build an alternative handler to test if there's a problem with the vetiver.TorchHandler that should be robust to different inputs. But the logs look the same so my interpretation is that the error is thrown before the input reaches the handler.

def predict_torch(model, new_data):
    import pandas as pd, numpy as np, torch

    # Accept: DataFrame, {"columns","data"}, or list[dict]
    if isinstance(new_data, pd.DataFrame):
        df = new_data.copy()
    elif isinstance(new_data, dict) and "data" in new_data and "columns" in new_data:
        df = pd.DataFrame(new_data["data"], columns=new_data["columns"])
    elif isinstance(new_data, list) and new_data and isinstance(new_data[0], dict):
        df = pd.DataFrame(new_data)
    else:
        # best-effort single-record dict
        df = pd.DataFrame([new_data]) if isinstance(new_data, dict) else pd.DataFrame(new_data)

    # Enforce schema & dtype
    df = df.reindex(columns=FEATURES, fill_value=0.0).astype("float32")  # <- key line

    X = torch.tensor(df.to_numpy(copy=False))  # already float32
    with torch.no_grad():
        y = model(X).detach().cpu().numpy().reshape(-1)

    return pd.DataFrame({"prediction": y})

# 4) Vetiver model → pin → deploy
v = vetiver.VetiverModel(
    model=mlp, 
    model_name=pin_name, 
    prototype_data=X_proto, 
    handler_predict=predict_torch,
    versioned=True,
    description="Test of Torch model"
)
5

Hi there! Thanks for trying out vetiver for your use case! This isn't quite working since vetiver is still trying to use the Torch handler for your mlp model. In this scenario, you'll have to make your own handler. You're most of the way there, actually! You can use the predict_torch function you've written as the handler_predict method on a custom class. Here are steps to write your own class. Then you'll instantiate your model like:

new_model = CustomHandler(model, prototype_data)

v = VetiverModel(new_model, "custom_model")

Let me know if you need any other help setting this up!

6

Based on the logs it looks like vetiver is still defaulting to the build in handler.

My custom class looks like this

# custom_torch_handler.py
import pandas as pd
import torch
from vetiver.handlers.base import BaseHandler

class TorchStateDictHandler(BaseHandler):

    def __init__(self, model, prototype_data=None, **kwargs):
         super().__init__(model, prototype_data)

    model_type = staticmethod(lambda: torch.nn.Module)
    pip_name = "torch" # package's installation name on pip

    def handler_predict(self, input_data, check_prototype: bool):

        if isinstance(input_data, pd.DataFrame):
            df = input_data.copy()
        elif isinstance(input_data, dict) and "data" in input_data and "columns" in input_data:
            df = pd.DataFrame(input_data["data"], columns=input_data["columns"])
        elif isinstance(input_data, list) and input_data and isinstance(input_data[0], dict):
            df = pd.DataFrame(input_data)
        elif isinstance(input_data, dict):
            df = pd.DataFrame([input_data])
        else:
            df = pd.DataFrame(input_data)
        
        df = df.apply(pd.to_numeric, errors="coerce").fillna(0.0).astype("float32")

        X = torch.tensor(df.to_numpy(copy=False))  # float32 already

        with torch.no_grad():         
            y = self.model(X).detach().cpu().numpy().reshape(-1)

        return pd.DataFrame({"prediction": y})

And I deploy using this

# deploy_from_state_dict_custom.py
import os
import pandas as pd
import torch
import vetiver
import pins
from rsconnect.api import RSConnectServer
from mlp_model import MLPModel
from custom_torch_handler import TorchStateDictHandler  

# quiet model card chatter if desired
vetiver.utils.modelcard_options.quiet = True

# path to stade dict file
state_dict_path = r""

# --- Rebuild MLP and load weights ---
input_length, feats, hidden_sizes, output_size = 378, 1, (64, 8), 1
mlp = MLPModel(input_length, feats, hidden_sizes, output_size)
mlp.load_state_dict(torch.load(state_dict_path, map_location="cpu"))
mlp.eval()

# --- Prototype / schema lock ---
INPUT_LEN = 378
FEATURES = [f"x{i}" for i in range(INPUT_LEN)]
X_proto = pd.DataFrame([[0.0]*INPUT_LEN], columns=FEATURES).astype("float32")

# --- Wrap the torch model in custom handler ---
wrapped = TorchStateDictHandler (mlp, prototype_data=X_proto)


pin_name = "mlp_wrapped_custm_handler"  
v = vetiver.VetiverModel(
    model=wrapped,                  
    model_name=pin_name,
    prototype_data=X_proto,
    versioned=True,
    description="Torch MLP wrapped by CustomTorchHandler"
)
print("Vetiver model uses:", type(v.model))

# --- Connect & pin ---
server_url  = os.environ["CONNECT_SERVER"]
api_key     = os.environ["CONNECT_API_KEY"]

connect_server = RSConnectServer(url=server_url, api_key=api_key)
board = pins.board_connect(server_url=server_url, api_key=api_key, allow_pickle_read=True)

vetiver.vetiver_pin_write(board=board, model=v)

vetiver.deploy_connect(
    connect_server=connect_server,
    board=board,
    pin_name=pin_name,
    title="VetiverTorchCustomHandler2",
    extra_files=[
        "requirements.txt",
        "mlp_model.py",
        "custom_torch_handler.py",
    ],
    new=True,
    force_generate=True,   
    log_callback=lambda m: print(m, end="")
)

and the last lines of the logs when testing the prototype.

2025/10/16 21:08:59.813296150   File "/opt/rstudio-connect/mnt/app/python/env/lib/python3.12/site-packages/fastapi/routing.py", line 288, in run_endpoint_function
2025/10/16 21:08:59.813296607     return await dependant.call(**values)
2025/10/16 21:08:59.813305254            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025/10/16 21:08:59.813305659   File "/opt/rstudio-connect/mnt/app/python/env/lib/python3.12/site-packages/vetiver/server.py", line 245, in custom_endpoint
2025/10/16 21:08:59.813314378     predictions = endpoint_fx(_to_frame, **kw)
2025/10/16 21:08:59.813314769                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025/10/16 21:08:59.813323514   File "/opt/rstudio-connect/mnt/app/python/env/lib/python3.12/site-packages/vetiver/handlers/torch.py", line 48, in handler_predict
2025/10/16 21:08:59.813323891     prediction = self.model(torch.from_numpy(input_data))
2025/10/16 21:08:59.813332630                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025/10/16 21:08:59.813333058 TypeError: can't convert np.ndarray of type numpy.object_. The only supported types are: float64, float32, float16, complex64, complex128, int64, int32, int16, int8, uint64, uint32, uint16, uint8, and bool.