Amazon S3

mail_location = obox:%8{md5;format=hex:user}/%u:INDEX=~/:CONTROL=~/

We’ll use the first 8 characters of the hex representation of the MD5 hash of the username at the beginning of each object path. This is S3’s dispersion prefix which identifies which internal shard the data is stored in.

In AWS, by default, the sharding prefix is ignored for a bucket and it can be enabled per request to AWS support.

Note

The AWS sharding prefix is limited to hex characters [0-9a-f] only.

When a S3 bucket is created, AWS creates a single shared partition for the bucket with a default limit of 3,500 TPS for PUTs/DELETEs/POSTs § and 5500 GET requests per second (source).

This 3,500 TPS limit is generally too small and quickly surpassed by Dovecot which results in a spike of 503: Slow Down log events. It is strongly recommended to contact AWS to request they manually set up 1 layer of hex partitioning (0-9a-f), to create16 dedicated partitions for your bucket.

1 hex layer of partitioning thus means a theoretical capacity of 56,000 PUT/POST/DELETE and 88,000 GETs per second.

Per AWS, you can go pretty deep in the number of layers, but most customers do not need more than 2 layers of partitioning, (2 layers = 16x16 = 256 partitions = this would theoretically provide you up to: ~896,000 PUT/POST/DELETE TPS and 1,408,000 GET TPS if requests are distributed evenly across the partitions).

plugin {
  obox_fs = fscache:1G:/var/cache/mails:compress:gz:6:s3: https://ACCESSKEY:SECRET@BUCKETNAME.s3.amazonaws.com/
  obox_index_fs = compress:gz:6:s3: https://ACCESSKEY:SECRET@BUCKETNAME.s3.amazonaws.com/
}

Get ACCESSKEY and SECRET from www.aws.amazon.com -> My account -> Security credentials -> Access credentials. Create the BUCKETNAME from AWS Management Console -> S3 -> Create Bucket.

If the ACCESSKEY or SECRET contains any special characters, they can be %hex-encoded.

Note

dovecot.conf handles %variable expansion internally as well, so % needs to be escaped as %% and ‘:’ needs to be escaped as %%3A.

S3 driver uses the AWS2 signing method by default, but AWS4 can be used by adding the bucket region parameter to the S3 URL:

plugin {
  obox_index_fs = compress:gz:6:s3: https://ACCESSKEY:SECRET@host/?bucket=BUCKETNAME&region=eu-central-1
}