It really depends on what specifically you are doing (you are giving almost no details) but since you are working with a relatively large data set most likely your RAM is throttling at some point.
If you need more specific help, please provide a proper REPRoducible EXample (reprex) illustrating your issue.