iWeb Scraping
Despite Amazon’s dominance in e-commerce marketplaces, eBay still has a significant presence in the online retail business. To obtain a competitive advantage, brands selling online should keep an eye on prices on eBay.
Scraping product data from eBay on a large scale regularly is a difficult task for data scientists to solve. Here’s an example of how to use Python to scrape eBay for mobile phone costs.
Consider a scenario in which you need to keep track of the price of a product, such as an eBay phone. You’ll also want to see the range of price options accessible on the cell phone you’re tracking. Furthermore, you may want to examine the prices of different mobile phones that you are considering.
In this blog, we’ll scrape eBay for phone costs and compare them to what’s available on the eBay website.
Scraping eBay Product Information
1. Selecting the Necessary Information
Identifying the target web page is the first step in web scraping services. It is the web page from which you must extract all of the necessary data.
We’ll be scraping eBay for product listings, so all we have to do is go to the eBay website and enter our product into the search area. All you have to do now is copy the URL from the browser once the website is loaded with most of the product listings for that product. Our desired URL will be this one.
The “nkw” (new keyword) and “pgn” (page number) parameters in this URL should be noted. The search query is defined by these arguments in the URL. If we alter the “pgn” variable to 2, it will enter the new page of product descriptions for the Galaxy Note 8 phone, and if we change the “nkw” parameter to iPhone X, eBay will search for iPhone X and show you the results.
2. Confirming the Tags for Extraction
We’ll need to comprehend the HTML layout of the intended web page once we’ve completed it to scrape the results. This is the most fundamental and crucial phase in web scraping, and it requires a basic understanding of HTML.
When on the target web page, press CTRL+SHIFT+I to open the developer tools window or do “inspect element” to open the developer tools window. The source code of the target web page will appear in a new window. Because all of the products are mentioned as list components in our situation, we must collect all of these lists.
An identifier must be associated with an HTML element to grab it. It could be that element’s id, a class name, and any other HTML attribute. As an identifier, we’re utilizing the class name. The class name for all of the lists is s-item.
Further investigation yielded the class names “s-item title” and “s-item price” for the product name and price, respectively. We’ve completed step 2 with this information!
3. Placing the Scraped Information in a Structured Manner
We only need to extract particular bits of the HTML content after we’ve created our extractors/identifiers. After that, we must organize the data in a logical, structured fashion. We’re going to make a table along with all the product names in one column and their prices in the other.
4. Visualizing the Results
We’ll also visualize the results because we’re comparing price offers on two different mobile phones. This isn’t a required step for web scraping, but it is a step in the process of converting your acquired data into useful information. Boxplots will be used to determine the distribution of pricing ranges on both the Galaxy Note 8 and the iPhone 8.
Necessary Libraries and Installations
You’ll need python, pip (python package installer), and the BeautifulSoup library in python to implement web scraping for this use case. To put the acquired data into a structured format, you’ll also need the pandas and numpy libraries.
1. Installing Python and Pip
You can set up Python and Pip in your system by following this blog link, depending on your operating system.
2. Installing BeautifulSoup Library
apt-get install python-bs4 pip install beautifulsoup4
3. Installing Pandas and numpy
pip install pandas pip install numpy
We’ve completed the setup of our environment and can now move on to the scraping implementation in Python. The steps outlined in the previous section make up the implementation.
Python Implementation for Scraping eBay
We’ll do two scraping operations: one for iPhone 8 and another for the Samsung Galaxy Note 8. For easy comprehension, the implementation was replicated for the two mobile phones. Two independent scrapping activities could be consolidated into one in a more streamlined version, but this isn’t necessary right now.
Scraping eBay for Galaxy Note 8 Products
item_name = [] prices = []
for i in range(1,10): ebayUrl = "https://www.ebay.com/sch/i.html?_from=R40&_nkw=note+8&_sacat=0&_pgn="+str(i)
r= requests.get(ebayUrl) data=r.text soup=BeautifulSoup(data) listings = soup.find_all('li', attrs={'class': 's-item'})
for listing in listings: prod_name=" " prod_price = " " for name in listing.find_all('h3', attrs={'class':"s-item__title"}):
if(str(name.find(text=True, recursive=False))!="None"): prod_name=str(name.find(text=True, recursive=False)) item_name.append(prod_name) if(prod_name!=" "):
price = listing.find('span', attrs={'class':"s-item__price"})
prod_price = str(price.find(text=True, recursive=False))
prod_price = int(sub(",","",prod_price.split("INR")[1].split(".")[0]))
prices.append(prod_price)
from scipy import stats
import numpy as np data_note_8 = pd.DataFrame({"Name":item_name, "Prices": prices})
data_note_8 = data_note_8.iloc[np.abs(stats.zscore(data_note_8["Prices"]))< 3,]Scraping eBay for iPhone 8
item_name = [] prices = [] for i in range(1,10):
ebayUrl = "https://www.ebay.com/sch/i.html?_from=R40&_nkw=iphone+8_sacat=0_pgn="+str(i)
r= requests.get(ebayUrl) data=r.text soup=BeautifulSoup(data)
listings = soup.find_all('li', attrs={'class': 's-item'})
for listing in listings: prod_name=" "
prod_price = " " for name in listing.find_all('h3', attrs={'class':"s-item__title"}):
if(str(name.find(text=True, recursive=False))!="None"):
prod_name=str(name.find(text=True, recursive=False))
item_name.append(prod_name) if(prod_name!=" "):
price = listing.find('span', attrs={'class':"s-item__price"})
prod_price = str(price.find(text=True, recursive=False))
prod_price = int(sub(",","",prod_price.split("INR")[1].split(".")[0]))
prices.append(prod_price) from scipy import stats
import numpy as np data_note_8 = pd.DataFrame({"Name":item_name, "Prices": prices})
data_note_8 = data_note_8.iloc[np.abs(stats.zscore(data_note_8["Prices"])) < 3,]Visualizing the Price of the Products
It’s time to put the collected data into perspective. The distribution of mobile phone prices will be visualized using boxplots.
We can use a box plot to show a numerical trend. The average of the collected price information is represented by the green line. The box encompasses the data from the first through third quartiles, with a line connecting them at the median (Q2). To represent the range of the data, the whiskers extend from the box’s edges.
The most of iPhone 8 phones are priced between INR 25k and 35k, whereas the majority of Galaxy Note 8 phones are priced between INR 25k and 30k.
However, the iPhone 8 has a considerably wider pricing range than the Galaxy Note 8. On eBay, the iPhone 8 costs a minimum of INR 15k, while the Samsung Galaxy Note 8 costs a minimum of INR 15,000.
iWeb Scraping as a Scraping Partner
There are numerous programs available to assist you in scraping data. iWeb Scraping can aid you if you require professional support with little technical knowledge. We have a very well and transparent approach for pulling data from the internet in real-time and delivering it in the manner that you require. We’ve worked with companies in a variety of industries. iWeb Scraping has developed comprehensive solutions for the majority of these use-cases, ranging from support to the recruitment sector python/ to retail solutions.
Before it’s too late, you should join the data-scraping bandwagon in your operations. It will assist you in improving your company’s performance. It will also assist you in gaining insights that you’ll never be aware of right now. This will allow you to make more informed decisions in your business processes.
Final Thoughts
We have successfully used Python to scrape eBay for various products and their pricing. We have also compared various prices for galaxy note 8 and iPhone 8 for a better purchase decision.
Want to avail more services for scraping eBay product data using Python?
Contact iWeb Scraping, today!!!
