scraping messages in forum using rvest

:partying_face::partying_face::partying_face::partying_face::partying_face::partying_face::partying_face::partying_face: Welcome to the RStudio Community forum @anacho :partying_face::partying_face::partying_face::partying_face::partying_face::partying_face:

The following code scrapes all the data you need. It is obvious you have some experience in web scraping so I will not spend too much time explaining what the code does at the moment, but feel free to ask me questions and I will be more than glad to provide you with more details. :slight_smile: However, regarding the messages in the threads, what I did is to scrape the links to each thread and use them to access the thread pages and further scrape the messages. The final results of this script is a tidy data frame with one list-column (since that are several messages in each thread.


# Scrape thread titles, thread links, authors and number of views

url <- ""
h <- read_html(url)

threads <- h %>%
  html_nodes("#threadslist .alt1 a") %>%

thread_links <- h %>%
  html_nodes("#threadslist .alt1 a") %>%
  html_attr(name = "href")

authors <- h %>%
  html_nodes("#threadslist .alt1 .smallfont") %>%
  html_text() %>%
  str_replace_all(pattern = "\t|\r|\n", replacement = "")

views <- h %>%
  html_nodes(".alt2:nth-child(6)") %>%
  html_text() %>%
  str_replace_all(pattern = ",", replacement = "") %>%

# Custom function to scrape messages in each thread

scrape_messages <- function(link){
  read_html(link) %>%
    html_nodes(css = ".smallfont~ hr+ div") %>%
    html_text() %>%
    str_replace_all(pattern = "\t|\r|\n", replacement = "") %>%

# Create master dataset (and scrape messages in each thread in process)

master_data <- 
  tibble(threads, authors, views, thread_links) %>%
  mutate(messages = map(thread_links, scrape_messages)) %>%
  select(threads:views, messages, thread_links)


  threads                            authors      views messages thread_links                                                            
  <chr>                              <chr>        <dbl> <list>   <chr>                                                                   
1 ADHD And Aspergers                 MyNameIsCra~  4973 <chr [3~
2 Adult Pants Pooping and Asperger'~ poopypants21  1680 <chr [4~
3 I did NOT spoil him!               mery          5939 <chr [7~
4 ASD Assessment as an adult, how?   Dragonfly W~  1243 <chr [2~
5 Sex and the single woman with AS   Madeofglass   7040 <chr [4~
6 I have aspergers and very severe ~ joe398        1445 <chr [2~

Hope this helps.

1 Like