displaying and saving very large datasets and their correlations

I am computing correlations 3 different ways (packages easystats,correlation,Hmisc). I tried the following for this very large dataset choosing a subset of the variables.

corvars are columns (variables) of interest
e <- correlation(corvars)

c <- correlation(corvars, include_factors = TRUE)

h <- rcorr(as.matrix(corvars))

Typical output for easystats and correlation looks like:

Parameter1 | Parameter2 |    r |       95% CI | t(4158) |         p
X2         |         X4 | 0.78 | [0.77, 0.79] |   80.95 | < .001***
X2         |         X5 | 0.85 | [0.84, 0.86] |  103.32 | < .001***
X2         |         X6 | 0.78 | [0.77, 0.80] |   81.47 | < .001***

e,c and h are truncated due to their length. How can I get these datasets to display or save all rows?
I tried something like the following but it did not work right.
for (i in 1: comb(n,2)){
v <- paste(i)
This is an attempt to automate

... etc.

I also tried to save to text and csv files.

I am looking for a more general solution. I suspect I may be missing the obvious simple solution? Any suggestions? I am new to big data. Thank you.


Can you suggest similar data to use?

Here is a large example anyone can access

corvars <- map_dfc(c(letters,LETTERS),~tibble({{.x}} := rnorm(10)))


e <- correlation::correlation(corvars)

This is a 1,326 tall result.
By default if i ask for e to the console, I get all of it flooding my console, and allowing me to read the bottom; I suppose this might be consider truncating at the top.
You can use standard techniques to read so many lines from the top, or middle (using dplyr/tidyverse) i.e.
head(e,n=6) for the first 1 to 6 entries, slice(e,5:7) for rows 5:7
if I wanted to 'save it all' I would probably just saveRDS the object; and I could load it back into a future session.
For human readable formatted output, like a large HTML type deal. I think you can just throw the output at gt()


Then you can scroll around in your webrowser looking at the content; or else use Rstudio viewer

letting you pan around,  filter and column sort.
1 Like

This topic was automatically closed 42 days after the last reply. New replies are no longer allowed.

If you have a query related to it or one of the replies, start a new topic and refer back with a link.