VROOM_CONNECTION_SIZE needs increasing

tidyverse
1

running vroom and get this error message, but I can't find how to change setting:
Error: The size of the connection buffer (131072) was not large enough
to fit a complete line:

  • Increase it by setting Sys.setenv("VROOM_CONNECTION_SIZE")
2

@rbh Welcome to RStudio Community!

In general, you can define an environment variable in R using the pattern Sys.setenv("varname" = "value"), e.g. the code below defines a new environment variable named EXAMPLE:

Sys.getenv("EXAMPLE")
## [1] ""
Sys.setenv("EXAMPLE" = 3)
Sys.getenv("EXAMPLE")
## [1] "3"

Thus for the vroom error, I'd recommend trying to double the default buffer size by running the following prior to trying to read the file:

Sys.setenv("VROOM_CONNECTION_SIZE" = 131072 * 2)

And if you find yourself having to do this often, you can set this environment variable by default by creating the file .Renviron in your home directory with the following contents:

VROOM_CONNECTION_SIZE=262144

I don't actually know what a reasonable buffer size is for vroom, so you might need to experiment with different values.

2 Likes
3

Thanks! I kept increasing the size to 131072*5, but that leads to a new error message. I'm trying to import and merge 5 *.csv flies, each with 4 columns and a total of ~ 40K lines. Apparently vroom thinks at least one file has 256009 columns. However, if I read in the files individually with read.csv or all at once in fread, the d.f has 4 columns, as it should. May be a vroom issue.
Again, my thanks!

Error: Files must all have 256009 columns:

  • File 2 has 4 columns
1 Like
4

@rbh Since you've tested it with multiple other import functions, I'd recommend opening an Issue on the vroom repository.

5

Thanks -- good idea. I'll attempt to make a reprex that will duplicate the problem.

2 Likes
6

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.