How to Simplify Your Approach to Web Scraping

How to Simplify Your Approach to Web Scraping hackernoon.com4 years ago in #Dev Love54

Originally published by Octoparse on November 4th 2018 Web scraping is hard, as much as we want to claim it as simple click and fetch — this is not the whole truth. Well, think back in time, when we haven’t had visual web scrapers like Octoparse, Parsehub, or Mozenda, any persons lacking programming knowledge are held back from tech-intensive stuff like web scraping. Despite the time it takes to learn the software, we might come to appreciate more of what is offered by all these “intelligent” programs, that had made web scraping feasible for everyone. Why is web scraping hard? · Coding is not for everyone Learning to code is interesting, but only if you are interested. For those that lack the drive or time to learn, it could post a real obstacle to getting data from the web. · Not all websites are the same (apparently) Sites change all the time, and maintaining scrapers may get very time consuming and costly. While scraping ordinary html contents may not be that hard, but we know there are so much more than that. What about scraping from PDFs, CSVs, or Excels? · Web pages are designed to interact with users in many innovative ways Sites that are made of complicated Java Scripts and AJAX mechanisms (which happens to be most of the popular sites you know) are tricky to scrape. Also, sites that require login credentials to access the data or one that has data changed dynamically behind forms can create a serious headache for web scrapers. · Anti-scraping mechanisms With a growing awareness to web scraping, straight-forward scraping can be easily detected as a bot and get blocked. Captcha or limited access often occurs with frequent visits within a short time. Tactics such as rotating user agents, altering IP addresses, and switching proxies are used to defeat common anti-scraping schemes. Moreover, adding page download delays or adding any human-likes navigating actions may also give the impression that “you are not a bot”. · A “super” server is needed Scraping a few pages and scraping at a scale (like millions of pages) are totally different stories. Scraping at large scale will require a scalable system with I/O mechanism, distributed crawling, communication, task scheduling, checking for duplication, etc. Learn more about what is web scraping if you are interested. How does an “automatic” web scraper work? Most, if not all, of the automatic web scrapers, work by deciphering the HTML structure of the webpage. By “telling” the scraper what you need with “drag” and “click”, the program proceeds to “guess” what data you may be after using various algorithms, then eventually fetch the target text, HTML, or URL from the webpage. Should you consider using a web scraping tool? There’s not a perfect answer to this question. However, if you find yourself in any of the situations below you may want to check out what a scraping tool can do for you, 1) do not know how to code (and do not have the desire/time to dig deep) 2) comfortable using a computer program 3) have limited time/budget 4) looking to scrape from many websites (and the list changes) 5) wants to scrape on a consistent basis If you fit into one of the above,  » Read More

Like to keep reading?

This article first appeared on If you'd like to keep reading, follow the white rabbit.

View Full Article

Leave a Reply