Reading in NCDF files with tidync

tidyverse
1

A few months ago I had some code that did an awesome job of helping me read in NCDF files and flatten them out to a single tibble. This week, I needed some additional years, so I went back to the same script, but it no longer works. I have no idea what changed.

First I load the libraries

library(tidyverse)  
library(tidync)

Then I make a list of files from a folder

files <- list.files(path = "merra_hourly/mandan_merra", pattern = ".nc")

I reset the working directory and then try to use the tidync function on each file. This is where it stops working.

setwd("merra_hourly/mandan_merra")  
mandan<-purrr::map(files, tidync)

Error in nc_meta.character(...) : **
** failed to open 'x', value given was: "mandan0001.nc"

But, if it had worked like before, I just need three more lines for my tibble. So close.

names(mandan)<-gsub("\\.nc", "", files)

tidy_mandan<-purrr::map(mandan, ~hyper_tibble(.))%>%
bind_rows(.id = "day")

I hadn't updated the tidync package when I got the error the first time, but I have updated it since and gotten the same error. Also, it was run with the same version of R a few months ago and now (3.4.3), which I haven't updated (but I guess I should). When I try the original script with the original files, I get the same error.

What am I missing?

2

Hi @Ranae! To clarify, what result are you getting when you run your third code block? Because if you're getting an error when you run the second block (mapping tidync over the file list), mandan is probably NULL, and so the third block might not go anywhere either.

Unfortunately, some of the errors in tidync aren't super informative. Could you possibly:

  1. Try opening a single NetCDF file (without purrr), so that we can check whether tidync is still actually opening these files properly?
  2. Try downloading one of the backend packages, like ncdf4, and opening a file with that (eg. ncdf4::nc_open)? I've had one or two instances of files that would open with ncdf4 but not with tidync.
1 Like
3

Thanks! Really good things I should have tried before.

The list in the second block looks good, I think, just a long list of file names. I don't get an error until the third block, when I try reading in the files. I gave your questions a shot with the following errors:

  1. tidync("mandan0001.nc")
    Oops, connection to source failed.
    [1] "mandan0001.nc"

  2. ncdf4::nc_open("mandan0001.nc")
    Error in R_nc4_open: NetCDF: Unknown file format
    Error in ncdf4::nc_open("mandan0001.nc") :
    Error in nc_open trying to open file mandan0001.nc

I had originally thought maybe the files were bad, but I got the same errors trying to open files I had opened a few months ago.

1 Like
4

Alright! I've got this to work. I think there was something going on with the website where I get the files. The site wasn't working great yesterday, but I pulled files from it anyway. I am retrieving new files, and now the code works.

I must have been making a different mistake when I tried running everything on old files.

1 Like
5

Great news! Sounds like you just got a bad file :sweat_smile: