Don’t worry, we’re not talking about questionable tax returns or businesses that sell ‘adult products’ to people who would have to be contortionists or weirdos to use them! Sometimes, if you’re running a business and you want to find out what your competitors are doing (and stop them finding out what you’re doing!) then it’s a good idea to use all that internet technology has to offer in keeping your activities under the radar.
Let’s look at a couple of possible scenarios
Imagine you’re running an ecommerce business selling collectible books. It’s a competitive market and pricing models aren’t set as they are for regular retail products. For example, if you’re selling car spares, then certain products are priced across the market at a price range that probably only differs by 10 or 15 % from the highest to the lowest.
It fits your car, or it doesn’t – there’s no difference in the item whether you buy it from Smith Ltd or Jones & Co. But if you’ve got a famous first edition book signed by the author, there might only be two of them up for sale, in differing conditions, across the whole world at any given time. Then the price structure for your wares becomes very much more focused on what the competition has to offer.
The quickest and easiest way to ascertain what your competitors are selling, where and for what price, is to use a technique called ‘web scraping’. Web scraping is not criminally illegal, but if you’re caught doing it, you could end up on the wrong end of some expensive civil litigation. That’s because web scraping is a method of using automated ‘bots’ to explore e-commerce websites and record all the content of pricing, availability, shipping costs, returns policies and the like.
Doing business on your own terms
Almost every e-commerce site will have terms and conditions (T&Cs) that specifically prevent web scraping by competitors, even though the information being ‘scraped’ is completely in the public domain. For example, you can choose any well-known e-commerce site at random, say, booking.com – their T&Cs state:
“You’re not allowed to monitor, copy, scrape/crawl, download, reproduce or otherwise use anything on our Platform for any commercial purpose without written permission of Booking.com or its licensors.”
So whilst you’re not breaking the law, Booking.com have very likely got some hardcase lawyers who will ensure that you’re properly bankrupted for scraping their site to try to offer better prices than them!
So how to keep an eye on the competition without them knowing? Simple – use a pool of rotating residential proxies – they are effectively servers that are normally provided by internet service providers to residential and domestic internet protocol (IP) addresses. In short, most businesses tend to use data center servers because they guarantee greater uptimes and can handle much more data, faster, than residential servers. But by using a residential proxy server, you can still enjoy all the advantages of a commercially provided server, but ‘cloaked’ behind an IP address that appears to be residential.
This way, if you’re data scraping a competitor’s website, their security scans will just pick up a very busy domestic online shopper performing price comparisons. That’s because website scraping detectors usually scan for data center servers and ignore Joe Public’s IP addresses.
A dip in the pool
Earlier we mentioned a pool of proxy servers. The simple reason for a business to use more than one residential proxy server is that the activity on each server is less traceable as a business’s online activity skips between several proxies. It also stands to reason that the more servers available in your pool, the less likely any competitor or potentially malicious hacker will be able to identify your business online. This is especially important when using collaborative design tools like Adobe Creative Cloud or Figma. The more you’re out there, the more visible you are.
Let’s not forget that some disgruntled customers can be very tech savvy, their revenge for a poor service or a bad deal might range from a stinking review on TrustPilot to a full on hack of all your organization’s machines and installation of ransomware across your company network! But if your online presence appears to be constantly shifting across an indeterminate number of residential IP addresses, across one or more countries, hackers are going to have a hard time tracking your business down. After all, if Microsoft can get hacked, so can you.
Quit sneaking around…
Finally, using residential proxies in several locations also enables the technique of effective dynamic pricing and SEO (search engine optimization) techniques. Many companies might want to have their products found for different terminologies and / or price-points, depending upon where those products are located for sale, and what they are called. That’s why regional variations and colloquialisms for naming and pricing goods is a common e-commerce technique in SEO advertising campaigns.
For example, in the U.K., gym shoes are known as “trainers” – whereas in most of the USA they’re called “Tennis Shoes” – but in the NE states of America they’re referred to as “sneakers”. If you want to test out your SEO strategies in these disparate locations, it’s essential to use a residential proxy local to where you want those localized nomenclatures to appear in online search results.
In the final analysis, if you’re running a business of any size, it’s often better to be unseen online as an interested shopper than being the highly visible provider of a busy e-commerce site.
