Events system: Difference between revisions

From Open Food Facts wiki
(Created page with "= Introduction = This is a brainstorm page, to find a good solution to monitor events in Open Food Facts. = What kind of events do we want to track = == Info events == Som...")
 
 
(3 intermediate revisions by one other user not shown)
Line 2: Line 2:


This is a brainstorm page, to find a good solution to monitor events in Open Food Facts.
This is a brainstorm page, to find a good solution to monitor events in Open Food Facts.
* *We need an event and action tracking and management system for the Platform for Producers: to monitor imports, verify new users etc.*
* We can also consider other use cases to see if we can create a solution that could work for them too


= What kind of events do we want to track =
= What kind of events do we want to track =
Line 12: Line 15:
** e.g. an user registered, a product was added or edited...
** e.g. an user registered, a product was added or edited...
** could be useful to have recent changes filterable by facets
** could be useful to have recent changes filterable by facets
* *Processes: some process started, failed, or completed*
* '''Processes: some process started, failed, or completed'''
** -> this is the most pressing need, for which we don't have a good solution today
** -> this is the most pressing need, for which we don't have a good solution today
** Regular maintenance tasks (e.g. daily processing of something)
** Regular maintenance tasks (e.g. daily processing of something)
Line 18: Line 21:
** User requested tasks
** User requested tasks
*** Imports and exports on the Producers Platform
*** Imports and exports on the Producers Platform
** Prospective data check reveals problem.
* Data Import
** eg USDA Branded foods API is checked and reveals new info
* Errors
** Software errors
*** MongoDB down
** Product quality issues


== Events that require an action ==
== Events that require an action ==
Line 45: Line 55:


* Sentry
* Sentry
** Perl client: https://docs.sentry.io/clients/perl/
* Matomo
* Matomo
* Minion
* Minion
** We could turn maintenance tasks into Minion jobs
** We could turn maintenance tasks into Minion jobs
* Nagios - nagios itself is having problems, but a nagios-like tool which:
** has script returning GOOD, WARN, BAD
** has a display


== OFF events system ==
== OFF events system ==

Latest revision as of 18:10, 8 September 2023

Introduction

This is a brainstorm page, to find a good solution to monitor events in Open Food Facts.

  • *We need an event and action tracking and management system for the Platform for Producers: to monitor imports, verify new users etc.*
  • We can also consider other use cases to see if we can create a solution that could work for them too

What kind of events do we want to track

Info events

Something happened.

  • User actions
    • e.g. an user registered, a product was added or edited...
    • could be useful to have recent changes filterable by facets
  • Processes: some process started, failed, or completed
    • -> this is the most pressing need, for which we don't have a good solution today
    • Regular maintenance tasks (e.g. daily processing of something)
    • Adhoc maintenance tasks (e.g. re-processing of products to apply a new taxonomy)
    • User requested tasks
      • Imports and exports on the Producers Platform
    • Prospective data check reveals problem.
  • Data Import
    • eg USDA Branded foods API is checked and reveals new info
  • Errors
    • Software errors
      • MongoDB down
    • Product quality issues

Events that require an action

Something needs to be done.

  • User request that needs to be manually validated
    • exporting products from the Producers Platform to the public database (at least for the first time)


Desirable features

  • Event log: being able to see individual events
  • Filtering by event type, country, product, organization, user etc.
  • Integration with OFF
    • It's not just about generating events, but also being able to display events directly into OFF
      • e.g. show status of user generated tasks
  • Aggregated metrics, graphs etc. of events
    • Nice to have, but not the main focus.
  • Event status for events that require an action (e.g. open / close)
  • Notifications (by mail, Slack channel etc.) for specific types of events
  • Public events and private events

Possible options

External events system

  • Sentry
  • Matomo
  • Minion
    • We could turn maintenance tasks into Minion jobs
  • Nagios - nagios itself is having problems, but a nagios-like tool which:
    • has script returning GOOD, WARN, BAD
    • has a display

OFF events system

  • Events collection in MongoDB
    • Makes it easy to have lots of OFF specific facets for events

Notes

  • We probably do not want to turn the events system in an analytics system
    • Keep things (e.g. the events database) relatively small