IMAP overview

IMAP lets mail clients such as Thunderbird, Outlook and mobile apps read your mail. The server in server/imap is built on Gluon, Proton's IMAP library, through the enjoys-in/gluon fork, which adds QUOTA, SORT and THREAD. Mail is stored on disk in Pebble, a key-value database modelled on RocksDB.

How mail gets into IMAP

Internet ──25──▶ Node mail server ──▶ checks (SPF, DKIM, DMARC, reverse DNS, sender)
                     │                   └─ suspicious? add "X-AE-Spam-Flag: YES"
                     └──LMTP :24──▶ IMAP server ──▶ INBOX, or Junk when flagged
                                        ├─▶ Pebble (on disk, /data/mail)
                                        └─▶ Gluon update ──▶ every open mail client, at once

1. The Node server accepts the message on port 25 and runs its checks.
2. It hands the message to the IMAP server over LMTP (RFC 2033), the standard way a mail server delivers into mailboxes. Set IMAP_LMTP_HOST=imap in server/.env to turn this on.
3. The IMAP server stores it in Pebble, then pushes it to Gluon through the connector's update stream (GetUpdates). Clients that are already connected see the new message straight away, without reconnecting.

LMTP answers once per recipient, so each recipient gets its own result. The Node server turns those results into the SMTP reply:

When the sender retries after a 450, mailboxes that already have the message skip it, so nobody gets two copies. The IMAP server records each delivered Message-ID per user in a small SQLite file, dedupe.db, in the Gluon data directory (/data/gluon). The record:

• is unique: the table has one row per user and Message-ID, and SQLite refuses a second one. Delivery claims that row before it stores the message. If the same message arrives twice at the same moment, one copy is stored and the other is turned away with a temporary error (451), so its sender retries and then gets the stored copy. A claim left behind by a crash is released after 5 minutes;
• survives a restart, because it is on the data volume;
• is deleted when the user deletes the message (expunge, or deleting its folder), so a message deleted on purpose can arrive again. Moving or copying a message does not release it;
• expires after 24 hours otherwise;
• is never used for a message with no Message-ID header, which is always stored.

If the file cannot be read, the message is stored anyway: a duplicate is better than lost mail.

The LMTP port is only for the Docker network. It accepts connections only from the networks in <lmtp trusted>, and the compose file never publishes it.

Live updates

Delivery does not wait for a client to ask. After the message is stored, the connector puts a "message created" update on its GetUpdates stream and waits until Gluon has applied it. Gluon then announces the new message (* N EXISTS) to every session of that user that is waiting in IDLE or has the mailbox open. In the Docker tests, two open sessions of one user saw a new message 10 to 40 ms after delivery started.

By default Gluon holds these notices for 500 ms and merges them. This server turns that off, so each message is announced at once:

<idle bulk="0s"/>

A value such as bulk="500ms" sends fewer, merged updates when many messages arrive together. Changing it needs a container restart.

Mail clients that use IDLE (Thunderbird, Apple Mail and most phone apps) show new mail immediately. A client that does not use IDLE learns about it on its next command, which is how IMAP works.

Spam and the Junk folder

The work is split the same way as in Postfix, Rspamd and Dovecot setups:

• The mail server decides. Its checks (SPF, DKIM, DMARC, reverse DNS and others) can tag a message. A tagged message gets X-AE-Spam-Flag: YES. See Receiving mail for which checks tag.
• Delivery files it. When any header in spam-headers starts with YES, the message goes to the spam mailbox instead of INBOX. That mailbox is created on first use and carries the standard \Junk attribute, so mail clients show it as their spam folder.

<delivery spam-mailbox="Junk" spam-headers="X-AE-Spam-Flag,X-Spam-Flag"/>

X-Spam-Flag is the header SpamAssassin and Rspamd add, so a content scanner placed in front later works without changes here. spam-mailbox="INBOX" keeps spam in the inbox. The rules apply on reload.

Folders

Every IMAP user gets these folders, created at login and again after every reload:

The attribute tells mail clients which folder plays which role, so Thunderbird, Apple Mail and phone apps use them without setup.

One folder per role. Spam is the same as Junk, Bin and Deleted Items the same as Trash, and Sent Items or Sent Mail the same as Sent. A user who already has one of these keeps it, and no second folder is created. Spam is then filed into that folder.

Starred holds every flagged message, like Gmail's Starred. Flagging a message anywhere adds it to Starred, and the message stays in its own folder too. Unflagging removes it. Copying or moving a message into Starred flags it, and deleting it from Starred unflags it. A message that is only in Starred moves to Archive when it is unflagged, so it is never lost.

Your own folders. List them in imap.xml, nested with /:

<folders defaults="Sent,Drafts,Outbox,Archive,Junk,Trash,Starred,Receipts,Clients/Acme"/>

defaults="" creates no default folders. Users can create, rename and delete their own folders in their mail client as usual.

Connect a mail client

Add an account in Thunderbird, Outlook, Apple Mail or a phone app and enter the settings by hand:

Port 143 is published on 127.0.0.1 only, so without a certificate the server can be used from the same machine and nothing is sent in the clear over a network. Thunderbird warns that the connection is not encrypted. Confirm it for a local test.

Reading mail needs no outgoing (SMTP) server. If the client insists on one, enter any value and ignore errors about it. To send, see Sending mail.

• Folders. A client caches its folder list. If a default folder does not appear, refresh the account or tick it in the client's Subscribe dialog.
• Quota. Thunderbird shows it under the folder's Properties, on the Quota tab.
• Live updates. Clients that use IDLE show new mail at once. See Live updates.

IMAP commands

The server speaks IMAP4rev1 (RFC 3501) and these extensions:

UIDPLUS adds APPENDUID and COPYUID to the replies, so clients know the UID of a saved or copied message without searching for it.

SUBSCRIBE and UNSUBSCRIBE answer OK when the folder is already in that state. Every new folder starts subscribed, and Thunderbird subscribes right after it creates one.

Quota. Each user has one quota root, "", holding their storage and message limits from imap.xml or users.xml:

C: a1 GETQUOTAROOT INBOX
S: * QUOTAROOT INBOX ""
S: * QUOTA "" (STORAGE 1530 1048576 MESSAGE 42 50000)

STORAGE is in units of 1024 bytes and MESSAGE is the number of messages. A limit that is not set is left out. Saving a message that would go over the limit is refused with NO [OVERQUOTA], and Thunderbird shows the quota under the folder's properties.

Not supported

None of these stops Thunderbird, Outlook, Apple Mail or phone apps from working.

Updating Gluon

server/imap/go.mod replaces github.com/ProtonMail/gluon with the dev branch of enjoys-in/gluon. To take in new upstream fixes, sync the fork with ProtonMail/gluon on GitHub, then update the pin and rebuild:

cd server/imap
go mod edit -replace github.com/ProtonMail/gluon=github.com/enjoys-in/gluon@dev
GOPROXY=direct go mod tidy
docker compose -f deploy/docker-compose.yml --profile imap up -d --build imap

go mod tidy turns @dev into the exact commit, so every build uses the same code until you update again. Use GOPROXY=direct because Go's public proxy caches branch lookups, so right after a merge it can still resolve dev to the previous commit and leave the pin unchanged. Check that the replace line in go.mod shows the new commit.

Who can use IMAP

Every user in the directory is an SMTP user. IMAP works for a user only when IMAP is enabled for them. Users without it cannot log in over IMAP, and LMTP refuses mail for them (550 5.1.1).

At start-up and after every reload, the server loads the IMAP-enabled users into Gluon and unloads everyone else.

How it is built

Each part is a separate package with a small interface, so you can replace any of them.

Configuration

Two files, both gitignored. Start from the examples in server/imap/conf/:

cp server/imap/conf/imap.example.xml server/imap/conf/imap.xml
cp server/imap/conf/users.example.xml server/imap/conf/users.xml

imap.xml says how the server is put together:

<imap>
  <settings>
    <listen address="0.0.0.0:143"/>
    <data dir="/data/gluon"/>
    <tls cert="/etc/ssl/imap/fullchain.pem" key="/etc/ssl/imap/privkey.pem" listen="0.0.0.0:993"/>
    <lmtp listen="0.0.0.0:24" trusted="127.0.0.0/8,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16"/>
    <idle bulk="0s"/>
  </settings>
  <folders defaults="Sent,Drafts,Outbox,Archive,Junk,Trash,Starred"/>
  <directory kind="file" source="/etc/imap/users.xml"/>
  <storage engine="pebble" path="/data/mail"/>
  <quota storage="1GB" messages="50000"/>
  <delivery spam-mailbox="Junk" spam-headers="X-AE-Spam-Flag,X-Spam-Flag"/>
</imap>

users.xml lists the users for the file kind:

<users>
  <domain name="example.com" imap="true">
    <user id="user1" password="change-me" quota-storage="2GB"/>
    <user id="noreply" password="change-me" imap="false"/>
  </domain>
</users>

The login name is user@domain. imap on a domain is the default for its users, and imap on a user overrides it.

Reload, like caddy reload

docker compose -f deploy/docker-compose.yml exec imap imapctl reload

It works like caddy reload: the new files are validated first and swapped in atomically, with no restart and no dropped connections. If a file is invalid, its old version stays live and imapctl exits with code 1 and says why. A reload re-reads imap.xml, the TLS certificate and users.xml, loads or unloads users in Gluon, and creates any default folders that users are missing.

imapctl reloadxml and docker kill -s HUP <container> do the same. The one difference from Caddy: changing a listen address (143, 993, LMTP), the data dir, the IDLE bulk time, the user kind or the storage engine needs a container restart, and the reload output says so.

Admin commands

All of them run in the container with imapctl:

Use docker compose exec -T when you pipe a file into imapctl mail deliver. If docker compose complains about a missing env file, run imapctl in the container directly, for example docker exec node-mail-server-imap-1 imapctl status. See Docker Compose.

Writing users.xml keeps the previous file as users.xml.bak. The file is rewritten from the user list, so comments in it are not kept.

TLS and port 993

Put fullchain.pem and privkey.pem in server/imap/tls/ (gitignored, mounted read-only at /etc/ssl/imap) and enable the <tls> line. The server then listens on 993 with implicit TLS and offers STARTTLS on 143, with TLS 1.2 as the minimum. After a renewal, run imapctl reload: new connections get the new certificate.

STARTTLS is advertised only on a connection that is not encrypted yet, so it is missing from the capabilities on 993 and after a client has upgraded. Without a certificate, STARTTLS is not offered, and a client that asks for it anyway gets NO TLS is unavailable and can carry on unencrypted. That is why a client set to "STARTTLS" fails on a server with no certificate, while one set to "None" works.

Storage

The pebble engine keeps every account's mailboxes, messages, flags and usage on disk under /data/mail. Every write is synced before it is acknowledged. Nothing is kept in memory: Gluon keeps its own cache (under /data/gluon) for fast reads.

It is built on internal/storage/kv:

To put another database under the same engine, implement kv.Store and register an engine that calls storage.NewKVEngine(yourStore).

Docker volumes

The container runs as uid 1000. On a Linux host: sudo chown -R 1000:1000 server/imap/conf.

Settings

Add your own backend

User kinds and storage engines are registered by name. Embed the Unimplemented type (every method returns "not implemented"), override what you support, and register it in an init function. Then select it in imap.xml, for example <storage engine="s3" bucket="mail"/>.

Not done yet

• Content spam scanning. A scanner such as Rspamd would sit in the Node server before the LMTP hand-off and add X-Spam-Flag. Filing already honours it.
• Telling the sender about refused recipients. When some recipients have no mailbox but others do, the message is accepted and the refusals are only logged. A bounce to the sender is not sent.
• Hashed passwords. A user kind that reads your SMTP user store should compare hashes.