Trying to scrape job descriptions on Indeed.com

tidyverse
,
1

I am trying to web scrape job descriptions from the webpage -https://www.indeed.com/cmp/HP/jobs
I am also new to web scrapping and im learning from scratch so I apologize if I am doing something stupid and do not realise it.
I have the following code

test_url <- read_html("https://www.indeed.com/cmp/HP/jobs")
  job_description <- test_url %>% 
    html_nodes(".cmp-JobDetailDescription") %>%
    html_text()

I used the CSS selector to find the node but for some reason I cannot get the description.

What do I have to change in my code to extract all the job descriptions in that url?
Someone please help Ive been breaking my head over this for the last few hours

Thank you in advance

2

Is this info that cannot be retrieved using Indeed's API?
https://opensource.indeedeng.io/api-documentation/

There are a few resources on web scraping best-practices/ethics that I link to in the thread here:

There are also a bunch of great web-scraping posts in hrbrmstr's blog here:

Including one on scraping when there is a free API available:

3

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.