Introduction
The internet is an absolutely massive source of data — data that we can access using web scraping and Python!
In fact, web scraping is often the only way we can access data. There is a lot of information out there that isn’t available in convenient CSV exports or easy-to-connect APIs. And websites themselves are often valuable sources of data — consider, for example, the kinds of analysis you could do if you could download every post on a web forum.
To access those sorts of on-page datasets, we’ll have to use web scraping.
In today’s competitive world everybody is looking for ways to innovate and make use of new technologies. Web scraping (also called web data extraction or data scraping) provides a solution for those who want to get access to structured web data in an automated fashion. Web scraping is useful if the public website you want to get data from doesn’t have an API, or it does but provides only limited access to the data.
In this article, we are going to shed some light on web scraping, here’s what you will learn:
- What is web scraping?
- The basics of web scraping
- What is the web scraping process?
- What is web scraping used for?
- The best resources to learn more about web scraping
What is web scraping?
Web scraping is the process of collecting structured web data in an automated fashion. It’s also called web data extraction. Some of the main use cases of web scraping include price monitoring, price prediction, news monitoring, price generation, and market research among many others.
In general, web data extraction is used by people and businesses who want to make use of the vast amount of publicly available web data to make smarter decisions.
If you’ve ever copy and pasted information from a website, you’ve performed the same function as any web scraper, only on a microscopic, manual scale. Unlike the mundane, mind-numbing process of manually extracting data, web scraping uses intelligent automation to retrieve hundreds, millions, or even billions of data points from the internet’s seemingly endless frontier.
The basics of web scraping
It’s extremely simple, in truth, and works by way of two parts: a web crawler and a web scraper. The web crawler is the horse, and the scraper is the chariot. The crawler leads the scraper, as if by hand, through the internet, where it extracts the data requested.
The web scraping process
If you do it yourself
This is what a general web scraping process looks like:
- Identify the target website
- Collect URLs of the pages where you want to extract data from
- Make a request to these URLs to get the HTML of the page
- Use locators to find the data in the HTML
- Save the data in a JSON or CSV file or some other structured format
Web Scrapping Example
- Find the URL that you want to scrape
In this blog we are going to scrape Flipkart website for product info.
We will scrap data from above URL.
2. Inspecting the Page and find data that you want
Right click on website and click on inspect.
Find tag of data that you want to scrape.
write the code
Output:
Thank You…
