Feed Management

What is website scraping?

By August 19, 2015 No Comments

FusePump’s website-scraping software is used to produce databases of product information for retailers and brands. Our proprietary technology and tools allow us to automate the extraction of information from any consumer-facing website. But what does that actually mean?

How our technology works

In the same way as search engines (like Google) ‘crawl’ websites to find out what they are about, our little crawlers go to websites to see what’s contained on the pages.

Imagine a custom-built robot pretending to be a user, going through a retailer’s website in the same way a shopper would. The robot reads all the relevant information (including AJAX and JavaScript), extracts it (takes a copy of all the text, links, images, and so on) and turns this information into a database.

This product data can then be used for a range of marketing purposes.

How do we use scraped data?

1. Our Feed Management team works with retailers who sell products online. We use our web-scraping technology to access all the information about all the products on their site, from product name to price to availability to full details and product features. (We can also add data from other sources, perhaps a spreadsheet with more detailed information, when creating the master feed.) This is turned into individual feeds for your different marketing channel partners, such as Google, affiliate networks, price comparison sites and display/retargeting campaigns. It can also be used for APIs and in the creation of front-end tools and widgets.

2. Our BuyNow team works with businesses who sell products through retail partners online. The technology is the same, but for this we scrape details on brand products from their various retail partners to let brands deploy the BuyNow solution. We can also detect position of offers on the page, and find competitor offers and product reviews, to help brands understand their online ‘share of shelf’.

In other words, anything you might want to quickly gather from one or more websites to use for marketing purposes… can be collected through our technology.

Why website scraping?

Often, a retailer’s website is the only place where all the correct and latest information about products, perhaps fed from different sources, comes together.

It’s also the information that consumers will see when they click through from an advert or listing, so the details need to match.

We also import APIs and accept feeds that have been created on the client’s side, for the same purposes.

Does the client need to do anything?

Not really. We have some tips for building a ‘friendly’ website but our solution works without any technical involvement from the owners of the websites we scrape.

Is website scraping legal?

Yes. It’s absolutely not against the law to gather information from public websites.