Web Scraping: the Good, the Bad and the Inbetween

Web scraping isn’t a new concept. People have been collecting data from the internet since its invention. But, the technology used for web scraping has become more powerful in recent years. These tools work faster and can collect a lot more data than ever before.

When combined with other tools such as a residential proxy, web scrapers can even work unnoticed, avoiding detection and bans. This makes it difficult for websites to protect their public data from scrapers. Which raises the question of whether web scraping is a tool for good or evil?

In this article, we will introduce web scraping. We’ll also be looking at some of the recent uses of web scraping and their varying effects, both good and bad. If you’ve ever been interested in web scraping but are worried about the adverse effects it can have, this article will provide you with some valuable points to consider before getting started.

Introduction to Web Scraping

Web scraping collects public data from different websites and compiles it into a single source, such as a spreadsheet. Once all the data is in one central location, it can easily be evaluated and used in various ways. 

Many businesses can use WebScraping.AI for pricing intelligence, market research, brand monitoring, competitor analysis and more. Using data in this way can be very beneficial as it provides valuable insights to help owners make the best business decisions. You need a scraping tool such as Octoparse, Parsehub or Smart Scraper to start web scraping. These tools will automatically collect the data and then parse it so that it’s in a format that can be read. 

Finally, you also need residential proxies with your web scrapers. Residential proxies will keep your web scraping tasks safe by hiding your actual IP address and replacing it with one from their pool linked to an actual device with an ISP. Not only will residential proxies protect your identity while scraping, but they’ll also keep you from getting banned, as each request you make will be linked to a different IP address.

Is Web Scraping Good or Bad?

Many businesses have used web scraping to great benefit. With more data to rely on, companies can make better decisions. However, the tool can also be used negatively. So, it begs the question, is web scraping a source of good or bad?

The Good: Wayback Machine

There are already a lot of benefits for those who use web scraping responsibly. One of the best examples of web scraping used for the greater good is the Wayback Machine. The Wayback Machine works with the Internet’s Archive to bring universal access to all knowledge. To do this, web scraping is required to collect data. In this case, web scraping is also used on pages like Wikipedia to find out what books and websites are cited so that these sources can be digitized. Digitizing these sources makes it easier for those conducting research (students, journalists, etc.) to gain the right knowledge and accurate information.

The Bad: Clearview AI

This is a popular story that has made numerous headlines. Clearview AI used web scraping to collect photos and personal data from social media sites such as Facebook, Twitter, etc. The startup collected more than 3 billion images of people. This data was compiled into a database that worked with their AI-powered facial recognition software. The company stated that this database was provided to law enforcement to help with criminal investigations. Sounds good, right? 

Clearview AI collected this personal data secretly and without anyone’s consent. It was also later revealed that this data was not just used in law enforcement but also sold to other businesses and individuals across the world. This has massive security, privacy and even safety implications for the many individuals captured in this database. Imagine the implications this could have for refugees seeking asylum, or individuals who escaped an abusive relationship, or even individuals in witness protection?

The Inbetween: Ryanair

Ryanair sued the travel price aggregator, Expedia, for scraping their data to be used for price comparisons. Expedia collects price and travel data to provide customers with accurate comparisons so that they can book the best deals. One of the reasons they could’ve done this was because of the backlash they were receiving after changing their rules regarding carry-on luggage. 

Ryanair started implementing a rule that passengers who weren’t priority passengers would only be allowed one small carry-on and would have to pay for anything bigger than a backpack or handbag. This rule is not the norm, and therefore doesn’t show in comparisons such as those by Expedia, meaning many passengers were in for a big surprise when booking their flights.

Final Thoughts

Web scraping is a powerful tool that many businesses and individuals can benefit from. However, how the tool is used depends on the user. As such, the user is the one who determines if the tools should be used for good or bad. And while most agree that the benefits of web scraping outweigh the risks, there is more legislature and governance required to ensure that web scraping is not used for malicious or harmful reasons.