Apache Arrow: shared in-memory data object

Yes, it is possible. This conversion from R to Arrow and from Arrow to R is what got implemented in sparklyr. That said, you can use the arrow package to read/write in a similar way.

The following example shows how to convert from a data frame from R into Arrow, then from Arrow back to R.

test-recordbatchreader.R#L21-L42

At this point, is up to you what to do with the Arrow representation. You can save it to disk, or send it to a different system/language over the network. For instance, you could read this back in Python using something similar to: python/data.html#record-batches.

1 Like