Q and A with Chris Conn (CTO) on using product data feeds and API's for online channel marketing
- How do FusePump provide the product data feed to Affiliate Networks, Comparison Shopping Engines (CSE’s), Digital Agencies etc. What’s the difference between an XML file and doing this by URL and what are the advantages of FTP transfer?
We extract data from our e-commerce clients’ websites using bespoke web agents. Two of the main formats for distributing product feeds are XML and delimiter-separated values (including CSV and pipe-delimited formats). XML feeds use a hierarchical structure to describe products whereas delimited files are similar to spreadsheets in that each row is a product entry, with each column containing an attribute value. The main advantage of XML is that you can “nest” multiple attributes, which allows for the representation of much richer product data in the feed. The main advantages of delimited text files are their smaller file size and the fact they are easy to open and view in spreadsheet applications.
In most cases, we provide a URL from which partners download the feed, as this is simple to set up and use. We can also provide an FTP location for the partner to log in to. In this case we still host the feed on our servers, but they need to use a different method, or “protocol”, to download it. This has the benefit of providing them with a browsable repository of files, which is particularly useful if they have multiple feeds to download.
Alternatively the partner can provide their own FTP details, in which case we will upload the feed to their server each day.
FTP includes username/password authentication but is insecure because the password and file are not encrypted during transfer. There are two main secure alternatives that we use: SFTP and FTPS.
- How do FusePump deliver product feeds for search? And how do our search partners use the feed?
One of the benefits of our data feeds is the fact we can include numerous relevant keywords for each product. We can do this because we already have rich product attributes extracted from the site. These can include colour, size, specifications, model number etc. Often we can do additional attribute mapping from our side, for example to map model numbers out of titles. This has advantages for long tail search matching.
We can also implement bespoke logic within the feed. We’ve done this recently for a client with a Kenshoo feed, to include per-category and per-brand rules to generate model number variations e.g. “AB-123L”, “AB123L”, “AB 123” This can be loaded into meta-descriptions. Someone searching using only part of a product code can therefore be directed to a relevant page.
Our search partners will use the data feed to help create the long-tail search terms as well as to create the advert copy dynamically. We make this easy for them by providing them with custom feeds down to the category and sub-category levels if required.
- How do we work with data feed API’s and what are these commonly used for?
APIs are used at FusePump in two main contexts:
Firstly, we can perform data extraction from an API, where we use a client’s existing API rather than the website to generate the product data. The main benefit of this is that data extraction from an API is typically much faster than “scraping” (typically 10-100 x faster). This is particularly useful for travel, where we always have to limit the number of dates/destinations etc. By using an API we can extract many more products without putting excessive load on the client’s website. Another benefit is that good APIs, such as those conforming to the OpenTravel Alliance standards, very rarely change in a way that would break our data extraction code. With some clients, we will supplement an API data extraction with a site scrape, to include rich static content that might not be exposed by the API (such as hotel descriptions and images).
Secondly, we can distribute data via an API, where we use the product data we have obtained by other means (e.g. scraping) to create an API for the client. The API can be queried on-demand for any number of purposes, including retrieving product data to use in display ads. Benefits for partners and affiliates include the fact they don’t have to download an entire feed – instead they can request just the data they need. They can retrieve the data programmatically, rather than logging in to a FeedCreator and manually configuring feeds. Also, an affiliate could create their own banners or widgets without any need for a database, because they can request data from the API on demand.
- Surely once you’ve delivered the feed to Google (or any other consumer of feeds) there’s very little to do to “optimise” this as you’re simply working to a specification? What sort of optimisation can you do, and why would you do this?
A feed integration isn’t over once we’ve extracted the client’s data and fitted it to a specification. Each feed consumer has their own unique requirements and recommendations that we can accommodate. Google Product Search is a good example. There are numerous ways in which we optimise our Google feeds to elevate product rankings. Firstly, we try to ensure that we include data in all Google’s mandatory attributes and in as many optional attributes as possible. A few examples are brand name, EAN and MPN, at least two of which are required for all products, with a few exceptions such as custom-made goods and clothing.
We also leverage keywords, in cases where the product titles don’t convey enough information about the product. One of the most common changes we make is to include part of the categorisation in the title, for example “Chinos” becomes “Men’s Chinos”.
Google’s feed specification is ever-changing. Recent updates include new attributes for product variants (such size and colour variations), and a requirement to use Google’s own category taxonomy for certain product types. Failing to comply with these changing requirements can result in account suspension and a subsequent loss of revenue, so it’s crucial to stay up to date.
Useful Links:
FusePump’s tips on integrating into Google Product Search can be found at: http://www.fusepump.com/fusepumps-5-top-tips-successful-google-shopping-channel-marketing-how-avoid-disaster
Summarized download of Google Shopping changes: http://www.fusepump.com/our-services/google-shopping