Web page scraping via Linux.

Picture of Web page scraping via Linux.
On of the most interesting things to do with linux is use the command line to do page scraping. You can hunt the web for the information you need without spending a a lot of time on-line. I also show you the pages where the data comes from. Here are three simple examples of how to scrape a web page from the command line. You do need to have some experience in using editors from the command line and bash to use this instructable.

Once you get more adept at this you can take the scraped data (i.e. sports stats) and enter them into database for later reference. That is a real incentive for using web page scraping.

You may need to add some software to your system such as lynx, elinks, and links2.

$ sudo apt-get update
$ sudo apt-get install lynx elinks links2 vim

Experiment, see what you can do to scrape data on your own. You could also extract data so it could be put into a database. but then that's another instructable. This is what search engines do, but at a more sophisticated level.

Note a file is included with all the old source code from my blog. Also if the web page you want to access are changed, then the scripts will have to be changed also. Web pages never stay the same so these scripts are not written in stone. They are examples to work from!
Remove these adsRemove these ads by Signing Up