Revision 5 as of 2008-01-13 17:44:25

Clear message

Stable URLs

There are many reasons to want a message to be addressed by a stable URL, i.e. one that does not change if the message is edited (for the most part), moved to a different archive, or referred to by different access methods. Further, this stable URL should be calculable with just minimal information, and with access to a non-list copy of the message. Some use cases for a message's stable URL include:

  • Archives URLs that survive regeneration, even if messages are deleted from the archive or edited by a list administrator.
  • The ability to pre-calculate the stable URL for inclusion in a message footer, without having to talk to the archiver (which may be a remote system).
  • The ability for programmatic access to a message in a message store when all you have is the off-list copy of the message.
  • The ability of a 3rd party archiver to implement a "mail me this message" function.

Stephen Turnbull starts the conversation off with this thread from the mailman-developers mailing list. I suggest you read the entire thread, but below is my counter proposal.

Message-IDs

RFC 2822 describes the Message-ID header. Everyone assumes that Message-ID is globally unique, so why can't it just be used as the stable URL?

Well, maybe it can, but not in its raw form. It's worth noting that RFC 2822 does not require the header, instead specifying that the header SHOULD be included. The header also SHOULD be globally unique but of course, because this is supplied by the client, it may not actually be unique. The Message-ID header certainly isn't very user-friendly and it is not url-friendly because it may contain characters that would have to be url-encoded.

Jeff Breidenbach of The Mail Archive did some analysis of their very large corpus of messages and makes a convincing argument that Message-ID is unique enough to rely on in the real world. It still suffers from lack of user- and url-friendliness. The following proposal uses Message-ID while supporting these other constraints.

RFC 5064

RFC 5064 (draft) is a specification for the Archived-At header. This is a very interesting proposal which should be honored by Mailman. It specifies where the message url should be included in the list copy, though we'll probably also include it in the message footer (the same url will be used in both places). Section 3.2 describes implementation considerations which mirror the same goals we're trying to achieve here, but the draft suffers from the same problems of user- and url-friendliness we've described here.

Thus, you can consider the following specification as a replacement for section 3.2 in the draft RFC 5064.

Specification

Here then is an informal specification for stable URL generation, such that could be used in a web service to provide messages on demand, or included in message footers without requiring communication with the archive.

Headers

We will use the RFC 5064 Archived-At header to contain the full url to the archived message. We'll also introduce a new header called X-Message-ID-Hash which will contain a user- and url-friendly token calculated from the Message-ID and provided as the last component of the Archived-At header. The X-Message-ID-Hash header is provided as a convenience only and is not required for this algorithm to work.

  • X-Message-ID-Hash is calculated from the Base 32 encoded SHA 1 hash of the complete Message-ID header (including angle bracket delimiters) of the original message. If the incoming message is missing its Message-ID or Date header, the mailing list manager SHOULD add its own version of the header, with the understanding that the non-list copy of the message will not contain this header. If the incoming message has more than one such header, the mailing list manager MUST use the first header found and MAY delete subsequent such headers. A mail server feeding the mailing list manager MAY reject messages with duplicate or missing Message-ID headers.

  • } is calculated using the following template: <baseurl>/<listname>/<midhash>.  So for example, a message for the Mailman Developers mailing list at {{{mail.python.org}}} with the {{{Message-ID}}} value of {{{<87myycy5eh.fsf@uwakimon.sk.tsukuba.ac.jp>}}} might have an

Stable URL calculation

The use of X-Message-ID-Hash, List-Archive, and Archived-At headers provides a unique, stable, easily calculated location for the message. Here's a more complete example of a message as posted through the mailing list.

Subject: An important message
Date: Wed, 04 Jul 2007 16:49:58 +0900
Message-ID: <87myycy5eh.fsf@uwakimon.sk.tsukuba.ac.jp>
X-Message-ID-Hash: AGDWSNXXKCWEILKKNYTBOHRDQGOX3Y35
List-Archive: http://mail.python.org/archives/mailman-developers
Archived-At: http://mail.python.org/archives/mailman-developers/AGDWSNXXKCWEILKKNYTBOHRDQGOX3Y35

Off-list copy

What if you receive an off-list copy of the message? How could you locate this message in the archive? Let's assume you know that the list's base archive url is at http://mail.python.org/archives/mailman-developers, and let's assume that the off-list copy was well-formed, with one unique Message-ID header. You could calculate the X-Message-ID-Hash easily, for example with the following bit of Python:

>>> from hashlib import sha1
>>> from base64 import b32encode
>>> base_url = 'http://mail.python.org/archives/mailman-developers'
>>> token = b32encode(sha1(msg['message-id']).digest())
>>> archive_url = '%s/%s' % (base_url, token)

Why Base 32?

Base 32 was chosen because of its limited alphabet and because it consists of only ASCII numbers and letters. This makes it easy (if slightly verbose) for humans to read and pronounce. For example, the numbers 0 and 1 are omitted from the alphabet to reduce confusion between them and the letters O and I in some fonts. Also, base 32 contains upper case letters only, however an archive MAY treat urls as case insensitive (accepting any combination of upper and lower case letters). An archive MAY also accept 0 for O and 1 for I in the X-Message-ID-Hash part only. The base32 hash is also completely url-safe, requiring no encoding in web applications.

Base 64 was rejected because, while providing minimal space savings, the expanded alphabet and case sensitivity was deemed to be less adaptable to human mistake (i.e. "be liberal in what you accept").

Open issues

Mailman's message scrubber also uses the archiver in a way not handled by the current specification. When a message reaches Mailman containing attachments, these attachments can be stripped and stored in the archive. The list copy of the message is then altered to contain just a url to the attachment as it appears in the archive. This is called scrubbing the message. How can we handle scrubbed attachments?


Comments

Barry Warsaw

An alternative for the Mail Archive is described here: http://www.mail-archive.com/faq.html#listserver

Brad Knowles

If you're going to generate a hash, it should be generated across all the required RFC-2822 headers (in addition to the others being discussed), including From:, To:, Subject:, and Date:.   This would help to guarantee the uniqueness of the hash, even if the message-id were to collide.

Second, although you want to use a hashing algorithm that is considered reasonably secure today (e.g., SHA-256), you also want to explicitly include up-front an extension/alternative mechanism, so that in the future when SHA-256 gets thrown onto the same scrapheap as MD5 and SHA-1, you can easily do so.