I am trying to use pins to write to my company's Databricks Unity Catalog (UC) Volumes. I am using this documentation as a reference.
Environment:
- Local Windows 10 Enterprise machine
- R 4.4.1 RC
- RStudio 2024.04.2+764
- pins 1.4
I am able to write the iris dataset to my UC volume using board_databricks and pin_write. I can view the expected "data.txt" and "iris.csv" files in UC.
However, when I try to read a 16MB CSV file from disk into R and then use pin_write, it starts the process ("Creating new version...") and then hangs. It creates the folder structure and files ("data.txt" and "test_file.csv") in the UC volume, but the process never completes. I have let it run for an hour without any success. When I interrupt the R process and download the file from UC, it is only 64K. I have tried multiple files, and it always shows that the file size is 64K after interrupting it (no matter how long I let the process run). This makes me think it could be a buffer size issue with curl.
Does anyone have any advice on how to troubleshoot this issue?