I have an app that I have deployed several times without issue. However, upon deploying my latest revision, I get an error stating that there was a problem extracting the bundle. It doesn't provide much more information than that so I am not really sure what could be wrong. The bundle does consist of a large number of files (about 36K) but the total size is only about 2 GB. I have searched but have been unable to find a solution for this.
Error: Unhandled Exception: child_task=1398133369 child_task_status=failed: Error downloading bundle: Error extracting Bundle: (CommandError(...), "Error running command 'tar xvf /opt/rstudio-lucid-legacy-builder/work/lucid-builder/bundles/bundlefxxvvu7x8414756/
In addition: There were 50 or more warnings (use warnings() to see the first 50)
Warning messages:
1: In writeBin(block, con) : problem writing to connection
2: In writeBin(block, con) : problem writing to connection
3: In writeBin(block, con) : problem writing to connection
4: In writeBin(block, con) : problem writing to connection
5: In writeBin(block, con) : problem writing to connection
Are you using the free or paid tiers of shinyapps.io. ?
How much data can I upload to shinyapps.io? – Posit Support
The bundle size that can be uploaded is limited to 1 GB for the Free and Starter plans, and up to 5 GB for the Basic, Standard, and Professional plans.
For the latter plans, note that the rsconnect package has a default bundle size limit of 3 GB, independent of your shinyapps.io plan limit. If you plan to deploy application bundles larger than 3 GB in size, you can override the rsconnect default by setting this option first: options(rsconnect.max.bundle.size=...) where the value is in bytes. Do not set the maximum size larger than the limit for your shinyapps.io plan, or application deployments will fail.
I have a paid (standard) subscription.
Hi. I've been looking into this issue from your support request.
The fundamental issue is that rsconnect uses utils::tar to generate a compressed tar file containing your application, but for some reason that file when generated by your system, which appears to be Windows, is not considered valid by our Linux servers.
rsconnect offers an option to control whether this file is generated natively with R code, or by an external program. I think a good next step would be to try again, forcing the method of compressed tar generation to be a method different from the one used on your prior attempts.
We can double check which method has been used on prior attempts if you access an R console with everything the same as when you ran rsconnect::deployApp(), and run
rsconnect:::getTarImplementation()
If this says internal, try running options(rsconnect.tar = "tar") and then trying the deploy again.
If this says anything other than internal, try running options(rsconnect.tar = "internal") and then trying the deploy again.
Let us know how it goes, and thanks for your patience!
Hi Mike,
Thanks for looking into this. You are correct that I am working on a Windows machine. I tried the steps you suggested. The option was previously set to "internal", so I tried setting it to "tar" and deploying again. It still fails, but with a new error message this time. I have attached the verbose log for that in case it is helpful. It seems to indicate that it created a file called manifest.json, but I am unable to find that file locally.
Fundamentally, there is nothing different about this deployment than previous ones that worked perfectly fine. The only thing I can think of is perhaps there are more files now (I can't remember how many there were before - it was still a lot though, at last 20,000). Could I be running into some limit on how many files can be zipped at once? I set max.bundle.files to 50,000.
Please let me know if there is anything else I can try.
(Attachment deploy_log_tar.txt is missing)
Attachment got rejected, trying again.
(Attachment deploy_log_1.txt is missing)
I cannot send an attachment (even a text document). Below is my original message (since I am not sure if that got though) with the log information appended to the bottom.
Yeah, nothing came through log-wise unfortunately.
Could you try pasting interesting looking parts of the log into your message directly? Lines near the manifest.json you mentioned would be one interesting part, the tar file is generated soon after that. You could also submit it as part of your support request if that's easier.
Bundling 36101 files: 000659322.rds, 001549504.rds, 010130427.rds, 010198331.rds, 010211494.rds, 010211501.rds, 010211534.rds, 010211783.rds, 010211797.rds, 010212435.rds, 010215227.rds, 010217211.rds, 010227195.rds, 010234189.rds, 010238552.rds, 010263628.rds, 010315414.rds, 010324279.rds, …,
[2024-03-29 10:49:27.897395] Bundling app dir
[2024-03-29 10:51:55.412747] Generate manifest.json
Capturing R dependencies with renv
Error in file(con, "w") : cannot open the connection
In addition: There were 50 or more warnings (use warnings() to see the first 50)
----- Deployment error -----
Error in file(con, "w") : cannot open the connection
Thanks!
My first suspicion as to why this might still not be working is that your machine doesn't have an external program called tar and so when we ask R to delegate to it instead of trying to do that work internally, it cannot. It could also be something related to the large number of files.
Can we prove or disprove my first suspicion quickly by checking if your machine has tar? We can do that by
- Open a command prompt by pressing WinKey+R, entering
cmd, and pressing Enter.
- In the command prompt window that opens, type
tar
On my Windows 10 machine with the standard R installation from CRAN, this outputs
tar: must specify one of -c, -r, -t, -u, -x
which looks like there is a tar program on this particular machine.
If you do have tar, it must be something to do with the large file count and we'll have to dig into what we can do about that.
I get the same output as you when I type 'tar' on the command line.
I tried reproducing this by
- Making an empty shiny project on a Windows machine
- Locating one of your attempts to deploy, and extracting as much of it as was readable into the new project. This resulted in 30,328 .rds files. I'm guessing that's fewer than you have since the tar file cannot be fully extracted (the root issue). I was unable to extract any non-.rds files...presumably if the file was fully readable there would have been R source files as well.
- Adding some more files to my project, just a basic shiny app. I was hoping that if there was some issue caused by sheer number of files, I would encounter it by going beyond the 30,328 file count.
- Deploying the app to shinyapps.io
But this worked for me, using both the internal and tar variants.
This seems to reduce the set of possible causes to things like
- A specific feature of one particular file you have is causing issues with packaging it into a
tar archive, and therefore when I try to reproduce it by reading the tar file you're sending us, I can't, because that file didn't make it into the archive.
- Different version of R and its built-in
utils package that performs the tar archiving...I'm using 4.3.3 and your logs indicated you're on 4.3.1, so there's some small chance they fixed an issue in those minor versions. Given that you've been deploying this application successfully in the past though, the first possibility seems more likely.
If you'd like, I'd be happy to start with exactly the project you are having difficulty deploying and try to reproduce errors deploying it. This would probably require you sending me the entire project data (using some archiving method other than tar, perhaps zip! :)).
How can I send you a large zip file?
I tried to deploy again with verbose logging and I got the following error stack. Is this helpful at all?
----- Error stack trace -----
10: rsconnect::deployApp(logLevel = "verbose")
9: bundleApp(appName = deployment$name, appDir = appDir, appFiles = appFiles,
appMetadata = appMetadata, quiet = quiet, verbose = verbose,
pythonConfig = pythonConfig, image = image, envManagement = envManagement,
...
8: createAppManifest(appDir = bundleDir, appMetadata = appMetadata,
users = users, pythonConfig = pythonConfig, retainPackratDirectory = TRUE,
image = image, envManagement = envManagement, envManagementR = envManagementR,
...
7: bundlePackages(bundleDir = appDir, extraPackages = extraPackages,
verbose = verbose, quiet = quiet)
6: computePackageDependencies(bundleDir, extraPackages, quiet = quiet,
verbose = verbose)
5: snapshotRenvDependencies(bundleDir, extraPackages, verbose = verbose)
4: recordExtraDependencies(bundleDir, extraPackages)
3: writeLines(paste0("library(", pkgs, ")\n"), depPath)
2: file(con, "w")
1: (function (e)
{
cat("----- Deployment error -----\n")
...
Regarding the most recent log message, that's interesting, it appears to be failing for a different reason there - unable to write a new file to your machine's disk that is written out before sending us the tar file.
How much free disk space does this machine have?
Regarding sending a large upload, you can use
https://posit-support.shinyapps.io/upload/
I've inquired though and it sounds like this tool unfortunately has a 1GB limit. Therefore you'd need to zip up about half of your files and send that, then send the other half as a separate upload. If you did that though I would be able to reconstruct the entire project.
I have ~1 TB of free space on the drive I am using, so I don't think it is a space issue.
I tried uploading some zip files for you, but after I attempt to upload the first one it just says "disconnected from server", so I don't think they are going through. I limited the file size to about 900 MB.
I haven't heard anything back for quite a while. Can I please get an update?
Hi there,
We suspect there is a problem with the tar file that your machine is generating. This could be a problem that is triggered by the contents or attributes of the actual files attempting to be uploaded. Would it be possible to upload the contents of the archive to us another way? Perhaps you can try uploading it to Google Drive and sharing the link.
Thanks
-Andy
I was able to get https://posit-support.shinyapps.io/upload/ to work just now. I uploaded four zip files. If you unzip all four into one directory, that is the entirety of what I am attempting to deploy. If you didn't get all the files, please provide me a private way to share a Google Drive link and I will do that.