Inclusion of a noctua conn object prevents Rmd document from rendering.

[dougfir]

I have a simple Rmd document that I would like to call via a cron job using rmarkdown::render()

Here is the example.Rmd file:

---
title: "Example blah"
output: html_document
---

blah blah blah

```{r}
library(dplyr)
library(dbplyr)
library(odbc)
library(DBI)
hive_conn <- DBI::dbConnect(odbc::odbc(), dsn = "Hive")
##s3_conn <- DBI::dbConnect(noctua::athena(), s3_staging_dir = "s3://glu-athena/results/") # for sending the actual data to S3

select s, 
       install_dt, 
       split(game_name, '_')[1] as platform,
       case when country = 'United States' then 1 else 0 end as usa
from device_metrics.game_install
where lower(game_name) rlike '^(?!.*QA).*' || 'fungame' || '.*' 
and year || '-' || month || '-' || day = '2020-01-01'

Check the object df exists

glimpse(df)

If I run this script by clicking knit, all works as expected. A page is rendered with the expected output.

If I uncomment out the line starting with s3_conn <- ... the script also runs successfully.

However, I would like to run this script as part of a cron job. I have another script called run_for_cron.R

library(rmarkdown)
rmarkdown::render("/home/myname/Sessions/example.Rmd")

This time the cron only completes as long as the s3_conn line is commented out. If I include the s3_conn line, I see an 'execution halted' message in the cron log. This does not happen if the s3_conn is commented out.

Not sure what this signifies, it's difficult to diagnose. How can I get the document to complete when being called in a cron?

[HanOostdijk]

Do you need some authorisation for accessing the noctua resource that is set in (RStudio) profile?

[dougfir]

Hi. Yes, actually I think I do.I have some aws acess credentials in my .Renviron file. Do I need to add those elsewhere?

[HanOostdijk]

So I understand that these are environment settings. Can't you use

Sys.setenv(...)

in an R-chunk (with echo=F of course)?

[dougfir]

Let me try that... will report back how it goes

[dougfir]

@HanOostdijk

Sys.setenv(...)

I tried this

Sys.setenv(AWS_ACCESS_KEY_ID="mykey", AWS_SECRET_ACCESS_KEY="my_secret_key", 
AWS_REGION="us-east-1")
test <- Sys.getenv("AWS_REGION")
print(test) # just verifying these set correctly as expected

I added this above the code block referencing noctua. However I hit the same issue where I could not render the document.

[HanOostdijk]

It should look then as

```{r echo=T} 
# change to echo=F when this is working
Sys.setenv(AWS_ACCESS_KEY_ID="mykey", 
           AWS_SECRET_ACCESS_KEY="my_secret_key", 
           AWS_REGION="us-east-1")
```

```{r}
library(dplyr)
library(dbplyr)
followed by other statements

but that is the case if I understand you correctly.
I have no further suggestions but you could have a look at the noctua page. In one of the examples all authorizations are specified in the dbConnect. You could try this to see if that works (not as a permanent solution).

[dougfir]

Thank you all the same

[larefly]

Hi @dougfir, sorry you are having difficulty with noctua. noctua can accept aws credentials in a few different ways.

1. Hard coded into the dbConnect function. This isn't advised as you will type the credentials in the script
2. environmental variables
3. .aws/config and .aws/credential files

I can see you're having difficulty with working with the environmental variables. My guess is that sys.setenv is only applying the credentials for that specific R session. Which will make it difficult to run through a cron job. A better option would to set a persistent environmental variable that will persist across different session (linux: How to permanently set environmental variables - Unix & Linux Stack Exchange, windows: windows - Set a persistent environment variable from cmd.exe - Stack Overflow) . Or you can use the .aws/config and .aws/credentials files.

To set up the .aws/config and .aws/credentials files, you can use aws cli Command Line Interface - AWS CLI - AWS. However if you are using a server you can get your Admins to pass the credentials to the EC2 (if you are working in AWS).

I hope this helps. If you are still having difficulty with this, please let me know

[dougfir]

Thanks for the suggestions. This may have been a separate issue unrelated to noctua. I was able to get things up and running with the env vars however my teammate was unable to. I suspect it's perhaps specific to their environment rather than to noctua. If I was able to delete this post I would.