Improved regex statements with automatic variable selection and simplified coding

Slavek · January 28, 2020, 10:26am

Hi R Masters,
I have this challenge today.
I solved that but it looks awful and I'm sure it may be significantly simplified.

I have this simple dummy file with 3 character variables (with many spelling mistakes) and I want to recode them into specific categories under relevant recoded variables (with Rec prefix).

I had some issues with regex not picking up some words therefore my coding is long and complicated.
I also manually specified RecQ9a, RecQ9b and RecQ9c:

df <- data.frame(stringsAsFactors=FALSE,
                 URN = c("aaa", "bbb", "ccc", "ddd", "eee", "fff", "ggg", "hhh", "iii",
                         "jjj"),
                 Q9a = c("Satisfied", "Contentsatifiedstressfree",
                         "Happy satisfied please d", "Sattisfied",
                         "Veryeasytoarrangeappoinem", "Ease", "Reasonable", "Dissatisfied", "Happy",
                         "satisfying"),
                 Q9b = c(" satisfying", NA, NA, "Easy, good", "Reasonable", "Fabulous", " Profesionable",
                         "Unimpressed", " Reassured", "Safe"),
                 Q9c = c("Confident", NA, NA, "Better", "Professional", "Professional",
                         "Timing", "Disappointef", "Sattisfied", "Enjoyable")
)

df

library(stringr)

results <- df %>% mutate(
  RecQ9a = case_when(
    str_detect(Q9a, regex("Satisfied|sattisfied|satissfied.Satisfied?|.Sattisfied?|.Satissfied?|.satisfying|Satisfying|.Satisfying|Satisified|.Satisified|Satified|.Satified|Satisfaction|.Satisfaction", 
                          ignore_case = TRUE, multiline = TRUE)) 
    & !str_detect(Q9a, regex(".Disatisfied?|.Dissatisfied?|.Dissattisfied?|.unsatisfied?|.unssatisfied?|.unsattisfied?|.unsatisfactory?|.unsatifactory?", 
                             ignore_case = TRUE, multiline = TRUE)) ~ "Satisfied",
    str_detect(Q9a, regex("Professional|Proffesional|.Professional?|.Proffesional?|Profesionable|.Profesionable|Proffessional|.Proffessional|Profesional|.Profesional", ignore_case = TRUE, multiline = TRUE)) ~ "Professional",
    str_detect(Q9a, regex("Easy|.Easy|Ease|.Ease", ignore_case = TRUE, multiline = TRUE)) 
    & !str_detect(Q9a, regex("reasonable", ignore_case = TRUE, multiline = TRUE)) 
    & !str_detect(Q9a, regex("reassured|reassuring|reasure|reasured", ignore_case = TRUE, multiline = TRUE))
    & !str_detect(Q9a, regex("pleased", ignore_case = TRUE, multiline = TRUE)) ~ "Easy"),
  RecQ9b = case_when(
    str_detect(Q9b, regex("Satisfied|sattisfied|satissfied.Satisfied?|.Sattisfied?|.Satissfied?|.satisfying|Satisfying|.Satisfying|Satisified|.Satisified|Satified|.Satified|Satisfaction|.Satisfaction", 
                          ignore_case = TRUE, multiline = TRUE)) 
    & !str_detect(Q9b, regex(".Disatisfied?|.Dissatisfied?|.Dissattisfied?|.unsatisfied?|.unssatisfied?|.unsattisfied?|.unsatisfactory?|.unsatifactory?", 
                             ignore_case = TRUE, multiline = TRUE)) ~ "Satisfied",
    str_detect(Q9b, regex("Professional|Proffesional|.Professional?|.Proffesional?|Profesionable|.Profesionable|Proffessional|.Proffessional|Profesional|.Profesional", ignore_case = TRUE, multiline = TRUE)) ~ "Professional",
    str_detect(Q9b, regex("Easy|.Easy|Ease|.Ease", ignore_case = TRUE, multiline = TRUE)) 
    & !str_detect(Q9b, regex("reasonable", ignore_case = TRUE, multiline = TRUE)) 
    & !str_detect(Q9b, regex("reassured|reassuring|reasure|reasured", ignore_case = TRUE, multiline = TRUE))
    & !str_detect(Q9b, regex("pleased", ignore_case = TRUE, multiline = TRUE)) ~ "Easy"),
  RecQ9c = case_when(
    str_detect(Q9c, regex("Satisfied|sattisfied|satissfied.Satisfied?|.Sattisfied?|.Satissfied?|.satisfying|Satisfying|.Satisfying|Satisified|.Satisified|Satified|.Satified|Satisfaction|.Satisfaction", 
                          ignore_case = TRUE, multiline = TRUE)) 
    & !str_detect(Q9c, regex(".Disatisfied?|.Dissatisfied?|.Dissattisfied?|.unsatisfied?|.unssatisfied?|.unsattisfied?|.unsatisfactory?|.unsatifactory?", 
                             ignore_case = TRUE, multiline = TRUE)) ~ "Satisfied",
    str_detect(Q9c, regex("Professional|Proffesional|.Professional?|.Proffesional?|Profesionable|.Profesionable|Proffessional|.Proffessional|Profesional|.Profesional", ignore_case = TRUE, multiline = TRUE)) ~ "Professional",
    str_detect(Q9c, regex("Easy|.Easy|Ease|.Ease", ignore_case = TRUE, multiline = TRUE)) 
    & !str_detect(Q9c, regex("reasonable", ignore_case = TRUE, multiline = TRUE)) 
    & !str_detect(Q9c, regex("reassured|reassuring|reasure|reasured", ignore_case = TRUE, multiline = TRUE))
    & !str_detect(Q9c, regex("pleased", ignore_case = TRUE, multiline = TRUE)) ~ "Easy"))
  
  results

What I need is:

I have Q9a, Q9b and Q9c variables in this file but in the other they might be Q8a, Q8b, Q8c so I would like to use a code for any variables ending with a, b or c (to apply that to different data)
I would like to use one list of case_when rather then repeating the same thing for the first, the second and the third variable to be analysed (Q9a, Q9b and Q9c in this case)
I would like to simplify regex as I have a feeling that "satis", sattis" & exclude phrases starting with "dis" and "un" (like "dissatisfied", "unsattisfied" etc) would work better than stating all possible spelling options. I had similar issues with "Easy" where many irrelevant words (such as "reassured") were picked up therefore I had to make coding long and complicated.

Can you help?

Slavek · January 29, 2020, 11:41am

I have tried this:

library(stringr)
library(plyr)

results <- df %>% 
  select(URN, ends_with(".a|.b|.c")) %>% 
  mutate_if(~is.character(.),
            list(`Rec`= ~ case_when(
    str_detect(Q9a, regex("Satisfied|sattisfied|satissfied.Satisfied?|.Sattisfied?|.Satissfied?|.satisfying|Satisfying|.Satisfying|Satisified|.Satisified|Satified|.Satified|Satisfaction|.Satisfaction", 
                          ignore_case = TRUE, multiline = TRUE)) 
    & !str_detect(Q9a, regex(".Disatisfied?|.Dissatisfied?|.Dissattisfied?|.unsatisfied?|.unssatisfied?|.unsattisfied?|.unsatisfactory?|.unsatifactory?", 
                             ignore_case = TRUE, multiline = TRUE)) ~ "Satisfied",
    str_detect(Q9a, regex("Professional|Proffesional|.Professional?|.Proffesional?|Profesionable|.Profesionable|Proffessional|.Proffessional|Profesional|.Profesional", ignore_case = TRUE, multiline = TRUE)) ~ "Professional",
    str_detect(Q9a, regex("Easy|.Easy|Ease|.Ease", ignore_case = TRUE, multiline = TRUE)) 
    & !str_detect(Q9a, regex("reasonable", ignore_case = TRUE, multiline = TRUE)) 
    & !str_detect(Q9a, regex("reassured|reassuring|reasure|reasured", ignore_case = TRUE, multiline = TRUE))
    & !str_detect(Q9a, regex("pleased", ignore_case = TRUE, multiline = TRUE)) ~ "Easy")))

but I have this error:

Error in stri_detect_regex(string, pattern, negate = negate, opts_regex = opts(pattern)) : 
  object 'Q9a' not found

andresrcs · January 29, 2020, 12:00pm

ends_with() doesn't accept regular expressions as an argument. The only tidyselect helper that accepts regex is matches()

Slavek · January 29, 2020, 12:59pm

I'm trying but I still cannot find the best resolution

Slavek · January 31, 2020, 11:45am

I still believe that one of R masters can easily compress my code into a few lines as:

Combination of "select" and "list" may be used to replace 3 separate sets of codes by one
Most of negative contexts in my " & !str_detect" expressions might be probably replaced by some clever general statement as many English adjectives turn to negatives prefixes like "un", "diss", "not" added (I don't know why words containing "eas" were picked up initially, therefore I had to fix it by listing exclusions in last 3 lines)
My lists of regex phrases might be shortened with more clever expressions for example my multiple statements about Professional may be replaced by something clever as all these phrases with spelling errors contain either "profes" ot "proffes".

Can anyone help?

andresrcs · January 31, 2020, 12:38pm

You would have better luck if you try to do it your self and ask for help when you get stuck on specific coding problems, It is very unlikely that someone is going to do all the tedious work (like writing complex regular expressions) for you.

Slavek · January 31, 2020, 1:31pm

I have. I sorted many issues in the original, large file. This dummy file contains only 3 options on the list "Satisfied", "Professional" and "Easy".
I listed three things I had tried to improve myself but these are only ideas. My knowledge is too limited to know if my suggestions are even possible in R.
I have a strong feeling that my codding with errors could be fixed and replaced by a maximum 7-8 short (but clever) lines of code...

andresrcs · January 31, 2020, 3:14pm

Ok, this will help you with the R related part, I let you work on the regular expressions.

df <- data.frame(stringsAsFactors=FALSE,
                 URN = c("aaa", "bbb", "ccc", "ddd", "eee", "fff", "ggg", "hhh", "iii",
                         "jjj"),
                 Q9a = c("Satisfied", "Contentsatifiedstressfree",
                         "Happy satisfied please d", "Sattisfied",
                         "Veryeasytoarrangeappoinem", "Ease", "Reasonable", "Dissatisfied", "Happy",
                         "satisfying"),
                 Q9b = c(" satisfying", NA, NA, "Easy, good", "Reasonable", "Fabulous", " Profesionable",
                         "Unimpressed", " Reassured", "Safe"),
                 Q9c = c("Confident", NA, NA, "Better", "Professional", "Professional",
                         "Timing", "Disappointef", "Sattisfied", "Enjoyable")
)


library(tidyverse)

df %>% 
    mutate_at(vars(matches("Q\\d[a-c]")),
              .funs = list(Rec = ~ case_when(
                  str_detect(., regex("Satisfied|sattisfied|satissfied.Satisfied?|Sattisfied|.Satissfied?|.satisfying|Satisfying|.Satisfying|Satisified|.Satisified|Satified|.Satified|Satisfaction|.Satisfaction", 
                                        ignore_case = TRUE, multiline = TRUE)) 
                  & !str_detect(., regex(".Disatisfied?|.Dissatisfied?|.Dissattisfied?|.unsatisfied?|.unssatisfied?|.unsattisfied?|.unsatisfactory?|.unsatifactory?", 
                                           ignore_case = TRUE, multiline = TRUE)) ~ "Satisfied",
                  str_detect(., regex("Professional|Proffesional|.Professional?|.Proffesional?|Profesionable|.Profesionable|Proffessional|.Proffessional|Profesional|.Profesional", ignore_case = TRUE, multiline = TRUE)) ~ "Professional",
                  str_detect(., regex("Easy|.Easy|Ease|.Ease", ignore_case = TRUE, multiline = TRUE)) 
                  & !str_detect(., regex("reasonable", ignore_case = TRUE, multiline = TRUE)) 
                  & !str_detect(., regex("reassured|reassuring|reasure|reasured", ignore_case = TRUE, multiline = TRUE))
                  & !str_detect(., regex("pleased", ignore_case = TRUE, multiline = TRUE)) ~ "Easy"))
              )
#>    URN                       Q9a            Q9b          Q9c   Q9a_Rec
#> 1  aaa                 Satisfied     satisfying    Confident Satisfied
#> 2  bbb Contentsatifiedstressfree           <NA>         <NA> Satisfied
#> 3  ccc  Happy satisfied please d           <NA>         <NA> Satisfied
#> 4  ddd                Sattisfied     Easy, good       Better Satisfied
#> 5  eee Veryeasytoarrangeappoinem     Reasonable Professional      Easy
#> 6  fff                      Ease       Fabulous Professional      Easy
#> 7  ggg                Reasonable  Profesionable       Timing      <NA>
#> 8  hhh              Dissatisfied    Unimpressed Disappointef Satisfied
#> 9  iii                     Happy      Reassured   Sattisfied      <NA>
#> 10 jjj                satisfying           Safe    Enjoyable Satisfied
#>         Q9b_Rec      Q9c_Rec
#> 1     Satisfied         <NA>
#> 2          <NA>         <NA>
#> 3          <NA>         <NA>
#> 4          Easy         <NA>
#> 5          <NA> Professional
#> 6          <NA> Professional
#> 7  Professional         <NA>
#> 8          <NA>         <NA>
#> 9          <NA>    Satisfied
#> 10         <NA>         <NA>

Slavek · February 3, 2020, 12:31pm

Aaaah. Thank you very much!

system · February 10, 2020, 12:37pm

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.