RStudio Cloud version and large datasets

[jrkrideau]

Welcome aboard.

Your question?

[jronsiek]

It looks like my first messages failed.

I have 2 initial questions:

1. Which RStudio product is best suited to handle large datasets? I do not know the actual file size but I do know the table contains 13 columns and several million rows.

2. I am currently working with the free (cloud) version of RStudio. I have figured out that I can not import a file into the RStudio workspace without uploading the file first. I have stepped through the upload process but have yet to complete a successful upload. I am sure that I am missing something simple. Does anyone have any suggestions on how to successfully upload a CSV file?

Thank you in advance.

[jrkrideau]

Hi, good to see you back.

I have no idea about Question 2. I have never worked in the Cloud.

As for Question 1, I think the answer depends very much on what you are doing. It depends on the available RAM, I believe.
Just having 13 columns and several million rows on my small laptop is not necessarily a problem. I just created a data.table of 54 million random numbers in 13 columns It is 944.3 MB in .csv format.

A couple of simple calculations work well.

library(data.table)
xx  <- data.table(matrix(rnorm(52000000), ncol = 13))
mean(xx$V1)
lm(V1 ~ V2, data = xx)

So you may not have any great problem in R with that size of data set. I have read that you may have storage problems in though Cloud but I don't know what those limitations are.

If you have a really large dataset that exceeds R's memory limits, the usual way is to store the data in a database and pull in the data as needed. R has packages that seem to allow one to connect to most available database programs. See Data Analysis and Visualisation in R for Ecologists for a discussion.

A relatively new one that seems to be gaining popularity is duckdb

[jronsiek]

Hello,
I have been trying to use write.csv() to export a CSV file my local (mac) computer. So far, I have been unsuccessful. ( I am still using cloud version of RStudio)
When I try and establish a working directory on my computer, I get the following error:

setwd("C:/Users")
Error in setwd("C:/Users") : cannot change working directory

I am able to import data from my computer into RStudio so I am hopeful that I can export a file from RStudio back to my local computer.

I apologize for messaging you directly but I am not able to send off a new message for some reason.
Wondering if you have any ideas on this?

Jeff

[jrkrideau]

I have seen that before and I am trying to remember what I did. I am also hampered by the fact have not used Windows close to 15-16 years.

Have a look at How To Fix R Error cannot change working directory

The suggestion about using getwd

I am not able to send off a new message for some reason.

That is really weird if you can send me a message. Are you getting any error messages?

[EconProf]

I do not know of any way to change the working directory in RStudio Cloud to a location on your local computer.

If you want to download a file to your computer, click on the Files tab and then More (with the blue gear icon). Select Export... from the drop down menu.

[startz]

Are you on a Mac? Do Mac's have a "c" drive?

[jronsiek]

Thanks again !! I will take a look at the link.