Serious problems with filter and dplyr

[jfca283]

Hello,
I'm working with some data and I've faced "errors" that doesn't perform.
For example, if I filter something
data %>% filter(region=="Glasgow" & age==30)
I can see the data I desired. But when I try to do
data=data %>% filter(!(region=="Glasgow" & age==30)
In order to exclude a specific row, I noticed that R deletes no only the one I specified, even more, It deletes many more rows.
In order to avoid that, I think that
data=data %>% filter(!(region=="Glasgow" & age==30 & !is.na(age) & !is.na(region) )
The code above in Stata is just
data=data if region!="Glasgow" & age!=30
I've lost confidence doing that kind of task.
I've read about anti-join, but I'm not sure now.
What is the best way to deal with data and exclude rows with confidence?
Thanks for your time and interest.

[williaml]

Can you provide a reproducible example?

FAQ: What's a reproducible example (`reprex`) and how do I create one?

Why reprex? Getting unstuck is hard. Your first step here is usually to create a reprex, or reproducible example. The goal of a reprex is to package your code, and information about your problem so that others can run it and feel your pain. Then, hopefully, folks can more easily provide a solution. What's in a Reproducible Example? Parts of a reproducible example: background information - Describe what you are trying to do. What have you already done? complete set up - include any library() calls and data to reproduce your issue. data for a reprex: Here's a discussion on setting up data for a reprex make it run - include the minimal code required to reproduce your error on the data…

[Matthias]

This is true, interesting! It also removes the lines where Glasgow is NA, it must have to do something how the (negated) NA is handled.
If you don't need the NAs in the age you could also remove it beforehand.

data = data.frame(
  stringsAsFactors = FALSE,
  region = c("Glasgow","Edinburgh","Inverness","Aberdeen",
             "Glasgow","Edinburgh", "Inverness","Aberdeen",
             "Glasgow","Edinburgh", "Inverness","Aberdeen",
             "Glasgow", "Edinburgh","Inverness","Aberdeen",
             "Glasgow", "Edinburgh","Inverness","Aberdeen",
             "Glasgow", "Edinburgh","Inverness","Aberdeen",
             "Glasgow", "Edinburgh","Inverness","Aberdeen",
             "Glasgow", "Edinburgh","Inverness","Aberdeen"),
  age = c(30L,42L,44L,NA,30L,48L,32L,42L,28L,NA,
          48L,30L,42L,38L,30L,26L,46L,28L,43L,39L,20L,
          35L,33L,NA,37L,43L,30L,49L,NA,27L,41L,33L),
  line = 1:32
)

data %>% filter(region=="Glasgow" & age==30)
# this gives (correctly) 2 hits: line 1 and line 5

data %>% filter(!(region=="Glasgow" & age==30))
# this gives 29 - line 29 (Glasgow - NA) is missing

# using anti join
kick_out  = filter(data, region=="Glasgow" & age==30)

data %>% 
  anti_join(kick_out)
# this gives 30, correctly without line 1 and line 5

[jfca283]

Reading your code assures me that R is very tricking handling big databases.
Besides, I think It is tedious to check every time I declare a filter or !filter.
I will take the anti_join approach in the future. It seems the safest.