Filtering with Dates

stat_geek · September 11, 2019, 9:49pm

Trying to do a date comparison but keep getting an error that I don't understand.

Original data source - DATE1, type is POSIXct as noted.

claims1 <- claims_original %>%
                dplyr::filter(DATE1 >= as.POSIXct('2018-01-01'))

Error is:

**Error in filter_impl(.data, quo) :**
**   Cannot convert object to a function: [type=NULL; target=CLOSXP, SPECIALSXP, or BUILTINSXP].**

I have tried updating rlang package as I found noted on StackOverflow but that's not working either.

R version is 3.4.3 - cannot be updated due to company limits.

technocrat · September 11, 2019, 10:04pm

Change '2018-01-01" to "2018-01-01"

(I hate it when delimiter blindness strikes me.)

stat_geek · September 11, 2019, 10:10pm

Sorry, that's just a typo here. Our system is locked down so I can't copy/paste out of it. This means I had to type it in manually unfortunately. I'll update my question to remove that typo.

technocrat · September 11, 2019, 10:25pm

library(dplyr)
library(tibble)
claims_original
#> # A tibble: 3 x 1
#>   DATE1     
#>   <chr>     
#> 1 2017-12-01
#> 2 2018-01-01
#> 3 2018-01-02
claims1 <- claims_original %>%
    dplyr::filter(DATE1 >= as.POSIXct("2018-01-01"))
claims1
#> # A tibble: 2 x 1
#>   DATE1     
#>   <chr>     
#> 1 2018-01-01
#> 2 2018-01-02

^{Created on 2019-09-11 by the reprex package (v0.3.0)}

stat_geek · September 12, 2019, 12:20am

Why do you have DATE1 as a character variable? Shouldn't it also be POSIXct? or some date format?

stat_geek · September 12, 2019, 12:36am

The code should work in theory, I do understand that. However, it's not.

Only packages installed are tidyverse, readxl and rlang in an attempt to fix the issue already. Everything else is Base R so far.

technocrat · September 12, 2019, 1:02am

For lack of a reproducible example, called a reprex, I took a guess from your question that your source data was character. If you want to recast DATE1 as a date object, see lubridate

technocrat · September 12, 2019, 1:20am

I can't reproduce the error.

library(dplyr)
library(tibble)
demo <- enframe(c("2017-12-01", "2018-01-01", "2018-01-02"))
demo %>% filter(value >= as.POSIXct("2018-01-01"))
#> # A tibble: 2 x 2
#>    name value     
#>   <int> <chr>     
#> 1     2 2018-01-01
#> 2     3 2018-01-02
demo
#> # A tibble: 3 x 2
#>    name value     
#>   <int> <chr>     
#> 1     1 2017-12-01
#> 2     2 2018-01-01
#> 3     3 2018-01-02

^{Created on 2019-09-11 by the reprex package (v0.3.0)}

I'm not aware of anything that R 3.4.3 that would throw the message you're getting in connection with dplyr::filter, but it's possible that dplyr and rlang are not at compatible version levels. If the Gods of IT permit it, try updating those packages.

jcblum · September 12, 2019, 1:59am

How do your package versions compare to the ones used here?

library(dplyr)
#> 
#> Attaching package: 'dplyr'
#> The following objects are masked from 'package:stats':
#> 
#>     filter, lag
#> The following objects are masked from 'package:base':
#> 
#>     intersect, setdiff, setequal, union
library(tibble)
demo <- enframe(c("2017-12-01", "2018-01-01", "2018-01-02"))
demo %>% filter(value >= as.POSIXct("2018-01-01"))
#> # A tibble: 2 x 2
#>    name value     
#>   <int> <chr>     
#> 1     2 2018-01-01
#> 2     3 2018-01-02

sessionInfo()
#> R version 3.4.4 (2018-03-15)
#> Platform: x86_64-pc-linux-gnu (64-bit)
#> Running under: Ubuntu 16.04.6 LTS
#> 
#> Matrix products: default
#> BLAS: /usr/lib/atlas-base/atlas/libblas.so.3.0
#> LAPACK: /usr/lib/atlas-base/atlas/liblapack.so.3.0
#> 
#> locale:
#>  [1] LC_CTYPE=C.UTF-8       LC_NUMERIC=C           LC_TIME=C.UTF-8       
#>  [4] LC_COLLATE=C.UTF-8     LC_MONETARY=C.UTF-8    LC_MESSAGES=C.UTF-8   
#>  [7] LC_PAPER=C.UTF-8       LC_NAME=C              LC_ADDRESS=C          
#> [10] LC_TELEPHONE=C         LC_MEASUREMENT=C.UTF-8 LC_IDENTIFICATION=C   
#> 
#> attached base packages:
#> [1] stats     graphics  grDevices utils     datasets  methods   base     
#> 
#> other attached packages:
#> [1] bindrcpp_0.2.2 tibble_1.4.2   dplyr_0.7.5   
#> 
#> loaded via a namespace (and not attached):
#>  [1] Rcpp_0.12.17     knitr_1.20       bindr_0.1.1      magrittr_1.5    
#>  [5] tidyselect_0.2.4 R6_2.2.2         rlang_0.2.1      stringr_1.3.1   
#>  [9] tools_3.4.4      utf8_1.1.4       cli_1.0.0        htmltools_0.3.6 
#> [13] yaml_2.1.19      assertthat_0.2.0 rprojroot_1.3-2  digest_0.6.15   
#> [17] crayon_1.3.4     purrr_0.2.5      glue_1.2.0       evaluate_0.10.1 
#> [21] rmarkdown_1.10   stringi_1.2.3    compiler_3.4.4   pillar_1.2.3    
#> [25] backports_1.1.2  pkgconfig_2.0.1

Created on 2019-09-12 by the reprex package (v0.2.0).

stat_geek · September 12, 2019, 2:28am

About a version behind,
R is 3.4.3 and dplyr_0.7.4 it seems.

I tried reinstalling all the packages first before posting, that did not work. In addition, I tried the rlang portion as indicated when you google that error from StackOverflow. Worst part of this, exact same system was rolled out to two users on same OS with same permissions and only one of them works.

Giving up and going back to SQL now.

jcblum · September 12, 2019, 2:49am

I realize your patience for debugging this problem may have been exceeded, but do you have a link to the SO answer you have in mind? I can find a number of SO questions related to different Error(s) in filter_impl(.data, quo), but I'm not coming up with any related to your specific error.

stat_geek · September 12, 2019, 2:52am

I have a deadline to meet..which I'm already past and this isn't helping so not much of a choice.

If there's somewhere else to look up these errors or another process to debug I'd be happy to know about it. I've been programming for more than 25 years so I'm pretty decent most of the time...this however has not worked.

Edit:
Other approaches I've tried including reading date as character and then just substringng the date. Or even listing all dates but anything seems to generate these errors which is why I think it's package issues somewhere that isn't really solveable at this point without either reinstalling everything or some other major change - none of which I can do at this time due to administrative rights lockdown.

jcblum · September 12, 2019, 3:04am

Thanks for the link!

Dependency hell on a deadline is absolutely the worst. I'm sorry.

I agree, I think the most likely issue is mismatched package versions, and having to work within your locked down system constraints definitely makes it all harder. I'm still not clear on what package versions you have installed right now, other than dplyr 0.7.4. Any chance you could do a screenshot of the output of running library(dplyr) followed by sessionInfo()?

I'll admit that this is probably not going to help you right now, so I understand completely if you prefer to drop it.

stat_geek · September 12, 2019, 3:11am

I have to be up doing this anyways...and I'm tenacious

Here's the screenshot - I had to remove a path with ID but rest is the same.

Edit: I compared versions with the installation on my home system and it wasn't too different.

jcblum · September 12, 2019, 6:02am

Interesting! I suspect the problem here is that your rlang is too new for your dplyr. You seem to have wound up with a funny mishmash of tidyverse-related package versions. dplyr 0.7.4 was current back in late 2017, when rlang was at 0.1.4 and the then-current version of the tidyverse umbrella package was 1.2.0.

Looking at your installed package versions, some packages are consistent with the up-to-this-minute latest:

#> # A tibble: 6 x 3
#>   package   installed  latest    
#>   <chr>     <pckg_vrs> <pckg_vrs>
#> 1 rlang     0.4.0      0.4.0     
#> 2 ggplot2   3.2.1      3.2.1     
#> 3 tidyverse 1.2.1      1.2.1     
#> 4 lubridate 1.7.4      1.7.4     
#> 5 modelr    0.1.5      0.1.5     
#> 6 rvest     0.3.4      0.3.4

Other packages are stuck at the versions that were current when tidyverse 1.2.0 was released

#> # A tibble: 12 x 4
#>    package    installed  tidyverse_1_2_0 latest    
#>    <chr>      <pckg_vrs> <pckg_vrs>      <pckg_vrs>
#>  1 readxl     1.0.0      1.0.0           1.3.1     
#>  2 forcats    0.2.0      0.2.0           0.4.0     
#>  3 stringr    1.2.0      1.2.0           1.4.0     
#>  4 dplyr      0.7.4      0.7.4           0.8.3     
#>  5 purrr      0.2.4      0.2.4           0.3.2     
#>  6 readr      1.1.1      1.1.1           1.3.1     
#>  7 jsonlite   1.5        1.5             1.6       
#>  8 cli        1.0.0      1.0.0           1.1.0     
#>  9 rstudioapi 0.7        0.7             0.10      
#> 10 haven      1.1.0      1.1.0           2.1.1     
#> 11 xml2       1.1.1      1.1.1           1.2.2     
#> 12 httr       1.3.1      1.3.1           1.4.1

And then there’s a handful of packages that are at versions somewhere in between now and the tidyverse 1.2.0 era.

#> # A tibble: 4 x 4
#>   package installed  tidyverse_1_2_0 latest    
#>   <chr>   <pckg_vrs> <pckg_vrs>      <pckg_vrs>
#> 1 tidyr   0.8.3      0.7.2           1.0.0     
#> 2 tibble  1.4.1      1.3.4           2.1.3     
#> 3 hms     0.4.0      0.3             0.5.1     
#> 4 broom   0.4.3      0.4.2           0.5.2

I'm hypothesizing that your specific error above might go away if you were able to downgrade rlang back to the version that was current when dplyr was at 0.7.4 — but that would almost certainly break some of the other packages that are actually up to date (e.g., ggplot2).

rlang has been under rapid and momentous development over the last two years, and the tidyverse as a whole has been moving toward ever-greater integration with the newer rlang paradigms. So I think you'd ultimately be better off bringing all of your tidyverse packages up to date (or arguing to the powers-that-be for doing so!).

kenbutler · September 19, 2019, 2:28am

Observation: dates in yyyy-mm-dd format will sort correctly whether they are text or actual Dates, so you could avoid the as.POSIXct thing entirely and just keep everything as text (if that's how it was to begin with). If it is greater than 2018-01-01 as a date, it is also greater than "2018-01-01" as text, and conversely.

system · October 10, 2019, 2:29am

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.