Reference Internal R Package in DESCRIPTION That is Hosted on Azure DevOps via Remotes

Hello RStudio Community!

We have our internal R Packages hosted on Azure DevOps. This package has utility functions that we commonly use across projects. Naturally, we want to add our package to Imports in the DESCRIPTION file.

We have Azure Pipelines set up that will check the package, run testthat, and run code coverage. The pipeline (this is using Jim Hester's azure-pipelines btw) builds a fresh R environment and installs all the dependencies. Since our internal package is not on CRAN this step will throw an error.

I tried to explore how to use the Remotes field in the DESCRIPTION file but have not been very successful. I see many great examples for GitLab, Bitbucket, etc. I was wondering if anyone knows of a way to reference our repo on Azure DevOps using this method.

Or if there is another way to get around the install dependency error that would be great to! :slight_smile:

1 Like

I forgot to add the page I have been referencing!

https://cran.r-project.org/web/packages/devtools/vignettes/dependencies.html

Either of these should work, assuming your ADO repo uses git; just substitute your repo's URL. If your repo is private, you'll need to add authentication details: see this link for more info.

# Git
Remotes: git::git@bitbucket.org:djnavarro/lsr.git

# URL
Remotes: url::https://github.com/hadley/stringr/archive/master.zip
1 Like

I have tried a few different things with the exampels you provided but I am still not understanding something. Here is what i have in my DESCRIPTION file:

Remotes: git::https://myCompany@dev.azure.com/myCompany/myPackage/_git/myPackage

This is a private repo, however, this is strictly for Azure Pipelines where it is all contained in my companies ADO/organization. When I check the logs of the pipeline it fails on this line of code and shows the following output in the log.

pkg-install_dependencies.yml:

steps:
  - bash: |
      Rscript -e "install.packages(c('remotes', 'devtools', 'rcmdcheck', 'testthat'))"
      Rscript -e "remotes::install_deps(dependencies = TRUE)"
    displayName: 'Install package dependencies'

ADO log:

The downloaded source packages are in
	‘/tmp/RtmprbktV8/downloaded_packages’
NA         (NA -> NA    ) [Git]
dplyr      (NA -> 1.0.0 ) [CRAN]
furrr      (NA -> 0.1.0 ) [CRAN]
future     (NA -> 1.17.0) [CRAN]
lubridate  (NA -> 1.7.9 ) [CRAN]
tictoc     (NA -> 1.0   ) [CRAN]
tidyr      (NA -> 1.1.0 ) [CRAN]
globals    (NA -> 0.12.5) [CRAN]
listenv    (NA -> 0.8.0 ) [CRAN]
generics   (NA -> 0.0.2 ) [CRAN]
tidyselect (NA -> 1.1.0 ) [CRAN]
Skipping 1 packages not available: myPackage
Error: Failed to install 'unknown package' from Git:
  Error in 'git2r_remote_ls'
In addition: Warning message:
In git2r::remote_ls(remote$url, credentials = remote$credentials) :
  Error in 'git2r_remote_ls'

Execution halted

##[error]Bash exited with code '1'.
Finishing: Install package dependencies

Note; I changed the actual package name to myPackage above

I see that it does not understand that I want to install my internal package from my companies organization. Does it need credentials while it's still within my companies organization? Am I declaring the URL in the DESCRIPTION file correctly?

I appreciate the help, working with ADO has been an uphill battle :grimacing:

1 Like

Hi KoderKow,

The post is very detailed, and I'm facing the same error with downloading a package from my company's internal azure website.

Were you able to find a fix for this?

If yes, it would be great if you can share that.

Regards,
Neel

Yes. Add a PAT, as described here: Authenticate with personal access tokens - Azure DevOps | Microsoft Docs

Use it as described here: B GitHub Personal Access Tokens | Happy Git and GitHub for the useR (happygitwithr.com)

Still nothing to fix this issue ?

Circling back on this, as there's no answer and I'm recently finding myself in this situation and would like to share what's worked for me.

I'm (newly) working for a state agency that has a private Azure DevOps Server and I need to be able to coordinate across in-house R packages via standard developer approaches.

I have found that the following works from an active session to install a package from the ADOS instance as long as I'm on the network:

remotes::install_git(url = "https://<agency ADOS address>/<agency>/<project>/_git/<repo>", 
                     git = "external")

And it works to include that same url in the Remotes section of the DESCRIPTION file as

Remotes:
    git::https://<agency ADOS address>/<agency>/<project>/_git/<repo>

in another package if I run

remotes::install_deps(dependencies = TRUE)

from that repository, again on a machine that's on the network.

If I mis-type the url I get the same message @KoderKow listed above regarding skipping the not available package.

Given that, @KoderKow (if you're still even working on this at all), I'm wondering if the issue has anything to do with the url format. I'm very new to Azure DevOps and this is my first organization using it, so I'm not aware of the range of urls, but what works for me (and I included above) has the project name before the _git, rather than the package (or repo) name, which only comes after the _git.

Given organizational constraints, I have not yet gotten to building Azure pipelines, so I am not 100% certain that this approach will work as-is in a remote server workflow, but I imagine there might just need to be a credential set up for the runs to operate (as Hong explained and linked out).

Can you try using the devel version on pak, as documented at All about installing pak. — Installing pak • pak?

pak::pkg_install("git::https://...")

Using the devel version of pak, I can successfully run the following from the R console on a machine on network:

pak::pkg_install("git::https://<agency ADOS address>/<agency>/<project>/_git/<repo>")

So the handling of the credentials works seamlessly, thanks!

1 Like

@Gabor , I am having some really inconsistent and maddening errors coming through when I try to do development using the ADOS as described here withpak::pkg_install. Note that this is still using the devel version as the current release does not work at all for this.

The issue always arises when trying to use the repository URL, but not when using the local-on-machine location with pak::pkg_install. Further, remotes::install_git works just fine. I can build and check as such using devtools and can install using basic installation from local zip files, etc. So this really only appears when I'm using pakwith the remote.

If the local copy is up-to-date and I try to use the URL, I get a successful notification that No downloads are needed, so that tells me that pak can communicate with the repo and determine that the local version is up-to-date with the remote version.

However, when I update the remote version and then try to install it, I get the following error:

             
ℹ Packaging <package> <version>
Installing...
                                                                
✔ Packaged <package> <version> (300ms)
                                                                
ℹ Building <package> <version>
                  
✖ Failed to build <package> <version>
                                                                
Error: 
! error in pak subprocess
Caused by error in `stop_task_build(state, worker)`:
! Failed to build source package '<package>'
Full installation output:
* installing *source* package '<package>' ...
staged installation is only possible with locking
** using non-staged installation
** help
No man pages found in package  '<package>' 
*** installing help indices
** building package indices
** testing if installed package can be loaded
Warning: S3 method '<method>' was declared in NAMESPACE but not found
Error: package or namespace load failed for '<package>' in namespaceExport(ns, exports):
 undefined exports: <function names...>
Error: loading failed
Execution halted
ERROR: loading failed
* removing '<temp file location>'
Type .Last.error to see the more details.

which tells me the same set of errors regarding the "undefined exports" and lack of man pages. And to be clear, that method and those functions are defined and exported and listed in the NAMESPACE file, so that's not actually the problem.

If I try to run again I get a different error for every time thereafter:

Installing...
             
ℹ Building <package> <version>
Installing...
             
Error: 
! error in pak subprocess
Caused by error in `read.dcf(con)`:
! Line starting '<package>/DESCRIPTION ...' is malformed!
Type .Last.error to see the more details.

which tells me

Error: 
! error in pak subprocess
Caused by error in `read.dcf(con)`:
! Line starting '<package>/DESCRIPTION ...' is malformed!

but again, I know that it's not malformed when I make it locally, and I can run read.dcf on it in the local version without any issue.

I have had the same issue in multiple repositories, so it's not specific to one.

After a few hours digging, I have not been able to trace the error well given the structure of pak (esp the R6 aspects), nor have I been able to consistently recreate the error (sometimes pushing an update to the remotes fixes or causes the error, with seemingly no consistency, I have tried to strip the package down to bare bones but still have been hitting this issue).

Further making things a challenge, given that it's a private agency server, I can't share details and others won't be able to access. I also don't have a personal ADOS server (why would I ever?, I always use github) so I cna't recreate this outside of my workspace or create a reliable reproudcible example. Recognizing that, I realize that we have some serious limitations with what you can help with at this moment, but if you have any ideas, I'd be happy to try.

thanks so much!

I think you are probably running into this bug: Solving for tree order in git fetch response · Issue #339 · r-lib/pkgdepends · GitHub

1 Like

Ah, thanks for thank link out.

I'll take a look around there and see if I can track things down more.

@jlsimonis OK, the fix should be in the nightly devel build of pak now. The nightly build is not very smooth nowadays, so don't hesitate to let me know if it still does not work for you.

1 Like