Data Access Patterns

Below is a list of operations that dictmap does for accessing data. RED = Cassandra operations BLUE = Object storage operations

  • Refreshing user root index

    • Frequency: very often – nearly every time a user logs in or a mail is delivered
    • Lookup object names from user_index_objects for the given username.
    • Lookup object names from user_index_diff_objects for the given username.

  • Refreshing folder index

    • Frequency: somewhat often
      • This lookup is usually done only when saving the index to local cache for the first time. Afterwards it’s not done unless the user’s cache becomes invalidated (user modified by another backend or the folder cache deleted).
    • Lookup object names from user_mailbox_index_objects for the given (username, folder GUID).
    • Lookup object names from user_mailbox_index_diff_objects for the given (username, folder GUID).
    • Lookup max_bucket from user_mailbox_buckets for the given (username, folder GUID)
    • Lookup object names from user_mailbox_objects for the given (username, folder GUID, 0..max_bucket). This is necessary for finding any newly delivered emails since the last folder index upload.

  • Writing user root self/diff index

    • Frequency: not often
      • The first time this backend modifies the user’s mailbox in any way.
      • Whenever user’s folders are created/renamed/deleted.
    • Insert to user_index_diff_objects (which overwrites the existing row)

  • Writing user root base index

    • Frequency: rarely * Normally self/diff index is updated; only after there have been a lot of changes a new base index is created.
    • Insert to user_index_objects
    • Delete from user_index_objects by (user, object name)

  • Writing folder diff/self index

    • Frequency: often
      • Every 10th mail delivery
      • Every 5 minutes after IMAP client has changed the folder (flag changes, deletions)
    • Insert to user_mailbox_index_diff_objects (which overwrites the existing row)

  • Writing folder base index

    • Frequency: not very often * Normally self/diff index is updated; only after there have been a lot of changes a new base index is created.
    • Insert to user_mailbox_index_objects
    • Delete from user_mailbox_index_objects by (user, folder GUID, object name)

  • Delivering a new email via LMTP, or saving a new email via IMAP APPEND

    • Frequency: Write folder index (as described above) for every 10th mail delivery (default)
    • Insert to user_mailbox_objects

  • Reading email

    • Usually no lookup, because object ID is stored also in Dovecot indexes

  • Deleting email

    • Lookup object ID from user_mailbox_objects_reverse to get list of (user, folder, object name). Dovecot only cares if the result is empty or non-empty.
    • Delete from user_mailbox_objects (user, folder, object name) and user_mailbox_objects_reverse (object ID)
    • Folder index is written lazily within the next 5 minutes

  • Copying email

    • Lookup object ID from user_mailbox_objects_reverse to get list of (user, folder, object name). Dovecot only cares if the result is empty or non-empty.
    • Write folder index (as described above) for every 10th mail delivery (default)

  • Moving email

    • This is identical to a combination of copying and then deleting the email.

  • Running “doveadm force-resync”

    • Frequency: Rarely, and always a manual operation
    • Refresh user & folder indexes as described above.
    • Lookup folder GUIDs from user_mailbox_index_diff_objects for the specified user to find any missing folders. With Cassandra this returns several duplicates (one per each index object in folder), which are de-duplicated internally.