Server-side product indexing and search

Summary

Server-side product indexing and search is the 2nd of the 4 sub-tasks of the Project:Personalized_Search funded by the NGI0 Discovery Fund managed by NlNet.

This page documents the progress made in Q2, Q3 and Q4 2020.

Overview

Diagram source: https://vecta.io/app/edit/-M2XyVv8ZoaLNrW-zQoT

1. The Open Food Facts mobile app (and 3rd party apps) make generic search requests that do not contain user preferences
2. The server returns a big number of generic results
3. The app uses the user preferences stored locally to personalize the search results

Initial research and specifications (completed in May and June 2020)

At the start of the project, we evaluated the different options for server-side product indexing and search:

• Server-side product indexing and search - Functional Specs
• Server-side product indexing and search - Technical Specs

New Search API (completed in August and September 2020)

The existing Open Food Facts search API is outdated and hacky (it was built on top of the OFF web site search form and is unnecessarily convoluted) and does not support some of the requirements for the Personal Search project (in particular being able to retrieve a given set of products using their barcodes).

We have create a new a new Open Food Facts Search API Version 2 that is simpler but also more powerful.

Key features of the new Search API:

• Simplified parameter specification
• Tag parameters (e.g. categories, labels, ingredients) can be searched in any language
• Support for AND, NOT and OR queries for tags fields (e.g. product with all those labels, none of those labels, or one of those labels)
• Allow to sort results by popularity of products (most scanned products)
• New /api/v2/search (JSON) and /search (OFF web site) endpoints that accept the same parameters

New Product Attributes for all search features (completed from August to October 2020)

We have created new Product Attributes that allow clients (like apps but also the OFF web site) to easily filter and rank search results according to the user preference, and to explain to users how well the products match their preferences.

Key features of product attributes:

• All search criteria / features (e.g. nutritional quality, if a product is vegan, contains a specific allergen etc.) are computed individually on the server side, and then made available to clients in the same normalized format
  • Clients can implement local filtering and ranking with a very simple algorithm. All the complexity is of the different search features is handled on the server side.
  • Clients can get localized names, descriptions etc. for each search feature in the language they choose.
  • New search features (e.g. a new diet, a new environmental or social score) can be made available to clients without any code change on the clients.

Next milestone: Client-side libraries for personalized product filtering and ranking

To make it even easier to use the new Search API and Product Attributes, we are developing Client-side libraries for personalized product filtering and ranking in Javascript and DART (for Flutter apps).