Events system

From Open Food Facts wiki
Revision as of 18:10, 8 September 2023 by Rkiddy (talk | contribs) (→‎Info events: ideas)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)

Introduction

This is a brainstorm page, to find a good solution to monitor events in Open Food Facts.

  • *We need an event and action tracking and management system for the Platform for Producers: to monitor imports, verify new users etc.*
  • We can also consider other use cases to see if we can create a solution that could work for them too

What kind of events do we want to track

Info events

Something happened.

  • User actions
    • e.g. an user registered, a product was added or edited...
    • could be useful to have recent changes filterable by facets
  • Processes: some process started, failed, or completed
    • -> this is the most pressing need, for which we don't have a good solution today
    • Regular maintenance tasks (e.g. daily processing of something)
    • Adhoc maintenance tasks (e.g. re-processing of products to apply a new taxonomy)
    • User requested tasks
      • Imports and exports on the Producers Platform
    • Prospective data check reveals problem.
  • Data Import
    • eg USDA Branded foods API is checked and reveals new info
  • Errors
    • Software errors
      • MongoDB down
    • Product quality issues

Events that require an action

Something needs to be done.

  • User request that needs to be manually validated
    • exporting products from the Producers Platform to the public database (at least for the first time)


Desirable features

  • Event log: being able to see individual events
  • Filtering by event type, country, product, organization, user etc.
  • Integration with OFF
    • It's not just about generating events, but also being able to display events directly into OFF
      • e.g. show status of user generated tasks
  • Aggregated metrics, graphs etc. of events
    • Nice to have, but not the main focus.
  • Event status for events that require an action (e.g. open / close)
  • Notifications (by mail, Slack channel etc.) for specific types of events
  • Public events and private events

Possible options

External events system

  • Sentry
  • Matomo
  • Minion
    • We could turn maintenance tasks into Minion jobs
  • Nagios - nagios itself is having problems, but a nagios-like tool which:
    • has script returning GOOD, WARN, BAD
    • has a display

OFF events system

  • Events collection in MongoDB
    • Makes it easy to have lots of OFF specific facets for events

Notes

  • We probably do not want to turn the events system in an analytics system
    • Keep things (e.g. the events database) relatively small