Using the Linux Shell for Web Scraping

Let’s assume we want to scrape the “Most Popular in News” box from bbc.com. What we need first is a CSS selector to locate what we are interested in. In this case it is simply a div tag with the ID “mostPopular” and you can figure this out using the Developer Tools of your favorite web browser. And now we are going to apply a chain of command line tools – each feeding their output to the next tool (that is called piping btw) and in the end we have a layouted text representation of the box’ content:

~> echo "http://www.bbc.com" |
   wget -O- -i- | 
   hxnormalize -x | 
   hxselect -i "div.most_popular_content" |  
   lynx -stdin -dump > theMostPoupularInNews

So let’s see what is going on. First the echo pipes the URL to wget. I could have also provided the URL directly in the arguments but I chose to do it like this to make clear that the URL or a list of URLs itself might be the result of processing. wget fetches the HTML code from BBC, which is then normalized by hxnormalize to improve digestability by hxselect (both installed on Ubuntu by

sudo apt-get install html-xml-utils

), which then extracts the part of the code being identified by the CSS selector. Lynx finally turns the code into a layouted text that you would see in a browser.

And this is what we get in the end:

~> cat theMostPoupularInNews

[1]Shared

     * [2]1 Galileo satellites on wrong orbit
     * [3]2 UK imams condemn Isis in online film
     * [4]3 Boy held for 'killing pet dinosaur'
     * [5]4 Experts to review stroke clot-buster

[6]Read

     * [7]1 Merkel in Ukraine as crisis mounts
     * [8]2 Galileo satellites on wrong orbit
     * [9]3 UN call to 'prevent Iraq massacre'
     * [10]4 Many dead in Madrid plane crash
     * [11]5 Russian mother has 'giant' baby

[12]Watched/Listened

     * [13]1 What are options in fight against IS?
     * [14]2 Victoria Beckham does ice challenge
     * [15]3 SpaceX rocket explodes during testing
     * [16]4 Obama refuses ice bucket challenge
     * [17]5 'Wedding was saddest day of my life'

References

   1. file:///tmp/lynxXXXXnXNyNy/L3209-8165TMP.html
   2. http://www.bbc.co.uk/news/world-europe-28910662
   3. http://www.bbc.co.uk/news/uk-28270296
   4. http://www.bbc.co.uk/news/blogs-news-from-elsewhere-28897353
   5. http://www.bbc.co.uk/news/health-28900824
   6. file:///tmp/lynxXXXXnXNyNy/L3209-8165TMP.html
   7. http://www.bbc.co.uk/news/world-europe-28910215
   8. http://www.bbc.co.uk/news/world-europe-28910662
   9. http://www.bbc.co.uk/news/world-middle-east-28910674
  10. http://www.bbc.co.uk/2/hi/europe/7572643.stm
  11. http://www.bbc.co.uk/2/hi/europe/7015841.stm
  12. file:///tmp/lynxXXXXnXNyNy/L3209-8165TMP.html
  13. http://www.bbc.co.uk/news/uk-28902128
  14. http://www.bbc.co.uk/news/entertainment-arts-28896231
  15. http://www.bbc.co.uk/news/world-us-canada-28910812
  16. http://www.bbc.co.uk/news/world-us-canada-28892227
  17. http://www.bbc.co.uk/news/world-middle-east-28315346

I love simple solutions and this is as simple as it can be (I guess).

---

(original article published on www.joyofdata.de)