Retail Price Monitoring

Success Story -

Client’s Challenge

On regular basis, the client required for 250 million SKUs as competition statistics. They expected 250 million SKUs within a year and planned to employ this information to inform their pricing policies and monitor retail prices.

Product part number, quantity, the price associated with the quantity, stock levels, and lead times were the data variables that they wanted to scrape. They needed quick access to data, so instead of continuing with their internal web scraping infrastructure, they decided to partner with, a managed service provider.

They needed clean, ready-to-use data that could be directly uploaded to their database so that they could conduct retail price monitoring for their company.

iWeb Scraping Solution

Client Requirements: The client gave us a list of competitor websites to crawl and collect data from. The crawl frequency was set to once a month, which meant that new data sets would be extracted once a month.

Custom Crawler Setup: Within two days, the iWeb Scraping team had set up the crawlers for the business. Because the source websites in the database had varied architecture and layout, this was a use case for our site crawl product.

Data Access: The data obtained had to be in CSV format and submitted to the client's Dropbox account. We automated the crawler so that it could supply the data at the needed frequency.

Data Template: A data template was created based on the client's schema, and data was structured according to the template.

Data Delivery: After the crawler was set up, we began delivering information to the customer in the format and delivery manner that they preferred.

Maintenance Required: We put up continuous monitoring for the origin sites to identify source site modifications, so there was no need for any more manual intervention on the client's part.