One of those a-ha moments.

The majority of my time on Meldware these days seems to be spent on our IMAP implementation. IMAP is a funny spec, every time I think I have a handle on it I discover another interesting quirk. Recently I have been trying to understand how to handle multi-accessed mailboxes. I.e. mailboxes that can be access by multiple clients at the same time. The behaviour in this situation is defined in a different rfc to the main IMAP4 spec.

The trickiest part of this is how to handle mailbox expunges. An IMAP server is supposed send updates of changes to a mailbox to any client that has that mailbox selected, except expunges. Expunges must not be sent unsolicited nor when a FETCH, STORE or SEARCH is executing. RFC 2180 specifies 4 different approaches to supporting expunge. Of the 4 approaches only 2 are worth considering.

Initially I intended implementing the first option. In order to do this I needed a locking mechanism that supported shared locks and upgrading of locks from shared to exclusive. Quite easy with the new Java concurrency API, well that is until we want to support clustering. Two quite obvious approaches jump out. Firstly do something in the database, which is okay until you need to deal with nodes that may die without cleanly removing their locks. We can work around this by applying timeouts to locks. Another option would be use JGroups and do implement some form of clustered locking manager. I rejected this, after trying to bend my mind round all of the possible race conditions that could occur in such an implementation.

After considering this for some time, finally the penny dropped. What I should do is implement the second behaviour using a "version" number to determine which messages are currently visible. In the latest implementation, the Folder (our entity representing an IMAP mailbox) holds an expunge version number. We also maintain an expunge version value on the FolderEntry. A FolderEntry represents a many-many relationship between Messages and Folders. Unlike other mail servers, we don't duplicate emails when sending to multiple receivers on the same server. On initial delivery the FolderEntry's expunge version is set to Long.MAX_VALUE (2^63 - 1). When a client selects a mailbox it uses the expunge version to determine which messages are visible SELECT * FROM folderentry where expungeversion > :folder_expunge_version. When a client decides to expunge a folder it increments the Folder's expunge version and sets the expunge version of all of the deleted FolderEntries to that value. Other clients will not see the messages that have been deleted, until they decide to refresh their instance of the Folder entity.

The only side effect of this solution is that we can't immediately delete the actual rows from the database. When a message is expunged we set an expunge date on the FolderEntry. With both IMAP and POP we set a timeout value for the connection, thereby ensuring that POP and IMAP clients are never more that about 30 minutes out of date with the current mailbox state. We will need to have some periodic mechanism for clearing down expunged messages. Probably after they have been expunged for more than 24 hours.

Technorati Tags: imap4 pop3