pak::pkg_install() fails after downloading 2,000 packages from P3M

Posit Professional & Hosted Products Posit Package Manager
1

I'm running reverse dependency checks, which involves downloading a large number of packages (dependencies of reverse dependencies) in a batch. This workflow worked ok until recently.

After approximately 2,000 Ubuntu binary packages are downloaded from P3M, pak::pkg_install() begins to report "Failed to download" errors. I tried switching the IP address and the number of threads (10 and 50), but the issue persists consistently around the same threshold.

Could this be related to possible new rate limiting rules on P3M? If so, are there documented parameters I can follow to stay within the limits? I'm happy to adjust my workflow, for example, downloading in smaller batches or adding delays between them.

Minimal reproducible example:

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
sudo apt-get update && sudo apt-get install -y build-essential
cargo install revdeprun
ulimit -n 10240
revdeprun https://github.com/Rdatatable/data.table.git

This downloads 4,000+ packages from P3M. Failures start to happen at around 2,000 packages.

2

Hey @nanx thanks for bringing this to our attention. We've just recently updated our rate limiting rules on P3M to be ~2000 requests/5min, which would line up with the error you're hitting.

We wanted to start off as semi-aggressive with our limiting and learn about the legitimate use cases that come up (like yours) to see if we have to adjust our thresholds. As you've suggested, we're going to work on documenting the limits and how to handle them properly.

Do you know how many requests revdeprun attempts to fire off and in what timeframe? (I assume it's the 4k+ package install requests you've mentioned)