A shopper visits a price comparison website to buy a set of earphones. The site shows a list of online retailers where he can buy the product, displaying the price and availability of the earphones at each retailer. The shopper then clicks on the retailer that has the earphones in stock at the lowest price, and proceeds to the check-out on the retailer’s website.
At check-out however, he will often discover that the product is either not in stock, or that it’s sold for a higher price than the one displayed on the comparison site.
How does this happen? When retailers provide their pricing and availability to the brand, they may falsify the price, or claim the product is in stock when it’s not. They do that to entice the shopper to enter their site, because once the shopper has already landed on the retailer’s website, he is likely to further engage, even if he realized the data has been falsified. The user will spend more money than expected, or purchase a similar product in stock, to the advantage of the falsifying retailer.
To avoid this falsification, price comparison sites retain the service of companies that anonymously scrape the prices from the retailer’s sites using custom software, and routing their requests through proxies in data centers. However, the targets often know that they are being scrapped (since the proxies are located in data centers and thus identified as scrapers), and once again, the targets will falsify the data.
To provide accurate data, these companies must rely on two technologies:
1) First, they must use scraping technologies that behave like human users, so that the scraping activity will not be identified as a robot and thus the pricing won’t be falsified.
2) Second, they must rely on a real anonymity network that provides residential IPs (not data center IPs), so that the target sites won’t detect the IP of the scraper as such.
To face the phenomenon, some of the companies offering these solutions are gathering accurate data from retailers by using its internal scraping technologies, and relying on the Hola Luminati network, a global P2P proxy network that routes data through residential IPs, so that the prices reflect those that are given to real users.
A cheaper option would be to use a traditional proxy network, whose IPs are in data centers. However, the price comparison websites may choose the premium path of residential IPs to maintain their confidence in the results provided to their brands.
Thanks to the use of the Luminati network, the shopper will be able to purchase the set of earphones with no surprises at the check-out.