Workflow

hermes-server

• 1. loads its local cache
• 2. checks if its dataschema has changed since last run, and emits the resulting removed events (if any), and the new dataschema
• 3. fetches all data required by its datamodel from datasource(s)
  • 3.1. enforces merge constraints
  • 3.2. merges data
  • 3.3. replaces inconsistencies and merge conflict by cached values
  • 3.4. enforce integrity constraints
• 4. generate a diff between its cache and the fetched remote data
• 5. loop over each diff type: added, modified, removed
  • 5.1. for each diff type, loop over each data type in their declaration order in the datamodel, except for removed diff type, for which it is the reverse declaration order
    • 5.1.1. loop over each diff item of current data type
      • 5.1.1.1. generate the corresponding event
      • 5.1.1.2. emit the event on message bus
      • 5.1.1.3. if event was successfully emitted:
        • 5.1.1.3.1. run datamodel commit_one action if any
        • 5.1.1.3.2. update the cache to reflect the new value of the item affected by event
• 6. once all events have been emitted
  • 6.1. run datamodel commit_all action if any
  • 6.2. save cache on disk
• 7. wait for updateInterval and restart from step 3. if app has not been requested to stop

If any exception is raised in step 2., this step is restarted until it succeeds.

If any exception is raised in steps 3. to 7., the cache is saved on disk, and the server restart from step 3..

Integrity constraints

Hermes-server can handle several data types, with link (foreign keys) between them, and to enforce integrity constraints.

Let’s use a typical Users / Groups / GroupsMember use case to illustrate this.

classDiagram
    direction BT
    GroupsMembers <-- Users
    GroupsMembers <-- Groups
    class Users{
      user_id
      ...
    }
    class Groups{
      group_id
      ...
    }
    class GroupsMembers{
      user_id
      group_id
      integrity() _SELF.user_id in Users_pkeys and _SELF.group_id in Groups_pkeys
    }

In this scenario, entries in GroupsMembers that have a user_id that doesn’t exist in Users, or a group_id that doesn’t exist in Groups will be silently ignored.

For more details, please see integrity_constraints in hermes-server configuration.

Multi source data aggregation

In a multi-source scenario, Hermes can aggregate entries providing from multiple sources as if they were providing from one, and optionally enforce aggregation constraints to ensure data consistency.

Let’s take a use case, with a university data set where Hermes should manage user accounts. Employees and students data are stored on two separate data sources. Hermes will be able to merge the two datasources in one virtual Users, but must ensure that primary keys doesn’t collide.

Here we got two distinct data sources for a same data type.

classDiagram
    direction BT
    Users <|-- Users_employee
    Users <|-- Users_students
    class Users{
      user_id
      login
      mail
      merge_constraints() s.user_id mustNotExist in e.user_id
    }
    class Users_students{
      s.user_id
      s.login
      s.mail
    }
    class Users_employee{
      e.user_id
      e.login
      e.mail
    }

In this scenario, entries in Users_students that have a user_id that exist in Users_employee will be silently ignored.
But entries in Users_employee that have a user_id that exist in Users_students will still be processed.

For more details, please see pkey_merge_constraint and merge_constraints in hermes-server configuration.

Multi source data merging

In a multi-source scenario, Hermes can recompose entries providing from multiple sources by merging their data, and optionally setting merge constraints to ensure data consistency.

Let’s take a use case, where Hermes should manage user accounts. Main data and wifi profile name are stored on two separate data sources. Hermes will be able to aggregate the two datasources in one virtual Users, but must ensure that primary keys of second exists in first.

Here we got two distinct data sources for a same entry.

classDiagram
    direction BT
    Users <|-- Users_main
    Users <|-- Users_auxiliary
    class Users{
      user_id
      login
      mail
      wifi_profile
      merge_constraints() a.user_id mustAlreadyExist in m.user_id
    }
    class Users_auxiliary{
      a.user_id
      a.wifi_profile
    }
    class Users_main{
      m.user_id
      m.login
      m.mail
    }

In this scenario, entries in Users_auxiliary that have a user_id that doesn’t exist in Users_main will be silently ignored.
But entries in Users_main that have a user_id that doesn’t exist in Users_auxiliary will be processed, and therefore the resulting Users entry won’t have a wifi_profile value.

For more details, please see pkey_merge_constraint and merge_constraints in hermes-server configuration.

Events emitted

Event categories

An event always belongs to one of those categories:

• base: standard event, can be of type:

  • dataschema: propagate the new dataschema to clients, after a server datamodel update
  • added: a new entry has been added to specified data type, with specified attributes and values
  • removed: entry of specified pkey has been removed from specified data type
  • modified: entry of specified pkey has been modified. Contains only added, modified, and removed attributes with their new values

• initsync: indicate that the event is a part of an initsync sequence, can be of type:

  • init-start: beginning of an initsync sequence, also contains the current dataschema
  • added: a new entry has been added to specified data type, with specified attributes and values. As the server sends the contents of its cache to initialize clients, entries can only be added
  • init-stop: end of an initsync sequence

Cache files

_hermes-server.json

Contains state of the server:

• lastUpdate: datetime.datetime | None

  Datetime of latest update.

• errors: dict[str, dict[str, dict[str, Any]]]

  Dictionary containing current errors, to be able to notify of any changes.

• exception: str | None

  String containing latest exception trace.

_dataschema.json

Contains the Dataschema, built upon the Datamodel. This cache file permit to server to process step 2. from Workflow.

DataType.json

There’s one file per data type declared in Datamodel, containing the data cache of this data type, as a list of dict. Each dict from the list is an entry.