Events system

Introduction

This is a brainstorm page, to find a good solution to monitor events in Open Food Facts.

• *We need an event and action tracking and management system for the Platform for Producers: to monitor imports, verify new users etc.*
• We can also consider other use cases to see if we can create a solution that could work for them too

What kind of events do we want to track

Info events

Something happened.

• User actions
  • e.g. an user registered, a product was added or edited...
  • could be useful to have recent changes filterable by facets
• Processes: some process started, failed, or completed
  • -> this is the most pressing need, for which we don't have a good solution today
  • Regular maintenance tasks (e.g. daily processing of something)
  • Adhoc maintenance tasks (e.g. re-processing of products to apply a new taxonomy)
  • User requested tasks
    • Imports and exports on the Producers Platform
  • Prospective data check reveals problem.
• Data Import
  • eg USDA Branded foods API is checked and reveals new info
• Errors
  • Software errors
    • MongoDB down
  • Product quality issues

Events that require an action

Something needs to be done.

• User request that needs to be manually validated
  • exporting products from the Producers Platform to the public database (at least for the first time)

Desirable features

• Event log: being able to see individual events
• Filtering by event type, country, product, organization, user etc.
• Integration with OFF
  • It's not just about generating events, but also being able to display events directly into OFF
    • e.g. show status of user generated tasks
• Aggregated metrics, graphs etc. of events
  • Nice to have, but not the main focus.
• Event status for events that require an action (e.g. open / close)
• Notifications (by mail, Slack channel etc.) for specific types of events
• Public events and private events

Possible options

External events system

• Sentry
  • Perl client: https://docs.sentry.io/clients/perl/
• Matomo
• Minion
  • We could turn maintenance tasks into Minion jobs
• Nagios - nagios itself is having problems, but a nagios-like tool which:
  • has script returning GOOD, WARN, BAD
  • has a display

OFF events system

• Events collection in MongoDB
  • Makes it easy to have lots of OFF specific facets for events

Notes

• We probably do not want to turn the events system in an analytics system
  • Keep things (e.g. the events database) relatively small