About Big Matrix (more than 10w+ rows) SVD

[realdwang]

Hi all,

I used the SVD function to obtain the Singular values from a big matrix.
I got the errors from R as follows:

20 *** caught segfault ***
21 address 0x55724e6da770, cause 'memory not mapped'
22
23 Traceback:
24 1: La.svd(x, nu, nv)
25 2: svd(all, nu = svd_k)
26 An irrecoverable exception occurred. R is aborting now ...

I don't know what went wrong, and it's rare on the Internet. I hope someone can help me solve this problem.

Best
Eric

[AlexisW]

In the title, "10w+ rows" means 10k+ rows, right? That doesn't seem so big, and the error message is not what I would expect for running out of memory (segfault suggests an operation that should never have been attempted). Make sure your matrix is numerical without NA or anything strange.

I would suggest you first try some permutations in your matrix (with sample()). If the error disappears, it suggests there is something about that particular matrix that is a problem (in rank, singularity, ...). Then try to reduce the size by subsampling, if the problem disappears, that would point to a size/memory problem, that could be avoided using e.g. irlba.

Else, I could imagine some library problem, there is this post that solves a similar problem by reinstalling all packages, might be worth a try.