Ken Sheppardson Subscribe to RSS Feed archives home

Simple Microblogging Transport Protocol

07/05/2008

If you’re reading this post, then you’re probably familiar with Twitter, FriendFeed, Plurk, Identi.ca, Pownce, Jaiku, Kwippy, and perhaps three or four other “microblogging platforms”. I apply the term somewhat loosely here. One reason for that is I don’t exactly know what it’s supposed to mean. Another is that many would argue these services can’t all be lumped into the same category because Twitter is in no way like FriendFeed, or that one of the others doesn’t deserve to be on the list, or that Jaiku is dead, or that we’re all on crack. I’m not here to argue those points.

Generally, I’d describe a “microblogging platform” as a system that allows someone to share short messages with one or more of their online contacts or broadcast to the public. These messages, when grouped by theme or topic, make up “threads” or conversations. They can also be searched, forwarded, replied to, etc. They’re either little nuggets of our witty and insightful thinking that we want to share with others, or they’re information like “I’m out of beans.”

Twitter

I am mad at Twitter. (To the person I know who works for Twitter, I’m not mad at you. Honest.) I think I’ll leave it at that.

The Answer to Your Prayers

In response to Twitter’s recent issues, many have suggested we need some sort of open, decentralized Twitter-like microblogging network. Dave Winer, for one, has written extensively on the subject . Steve Gillmor, Marc Canter, et al have also discussed some sort of " Plan B ". That conversation has fragmented out over the web and generated some very interesting technical discussions .

Unfortunately, I don’t really enjoy trying to track down and wade through other people’s posts, technical specs, and standards documents. For example, I’d hate to have to read both the OpenMicrobloggingProtocol specification and the Microblogging over XMPP specification and try to figure out how they each work, let alone which is better. So rather than try to address anyone else’s specific points or aggregate all that wisdom and all those differing opinions into some consensus, I thought it would be best to propose a new standard that should pretty much cover everything.

Before I continue, I’ll give you a chance to go check FriendFeed, Google Reader, Twitter, and your email since this isn’t a quick little 30 second comment wrapped around someone else’s article. You might have to set aside a solid five minutes for this. Alright, ready to settle in? Here we go…

Simple Microblogging Transport Protocol (SMTP)

[Note: I’d originally considered calling this the “Sheppardson Microblogging Transport Protocol” but that seemed a bit too… well… long. Plus I expect lots of folks would misspell my name.]

Unique User Address

We’re going to be building a global distributed system here, so each user will need some sort of SMTP ID that’s good across all the nodes that make up the SMTP network. This is really an address, of sorts, that points to our microblogging “home” node on the web. Our home node is where we usually post and read our microblogging content. If someone wants to direct a message to us, they can send it to the place specified by our SMTP ID. A reader can tell who posted a particular microblog message by examining the author’s SMTP ID.

I propose we use the convention of username (e.g. kshep) followed by the address of the SMTP system on which that users lives (in my case, kshep.net). Let’s string those together, perhaps separated by a special character of some sort, to form a unique address for each user. I’d suggest either a # or a %, e.g. “kshep#kshep.net” or perhaps “kshep%kshep.net”.

Message Content Type&Size

Users will probably want to publish and consume microblog content on a wide variety of devices, so let’s limit content to text. If a user wants to reference a picture or a mixed media document of some sort, I’m sure someone else can come up with some sort of standard for a short (tiny, even) text string that we could insert into a message that will tell us where that document is located on the network. Many users seem to enjoy microblogging to and from their cell phones so let’s say the messages can be no more than 140 characters long, which is apparently the longest message cell phones can display unless it’s 160 characters.

User Interface

It’s likely that users will want to both generate and consume SMTP messages on a variety of devices, so it’s important that we support a variety of use cases and UI layouts. We’re more concerned with the development of the SMTP protocol at this point, so it’s probably not necessary to delve into the UI design too much. However it’s worth noting that we’ll want to support posting/reading in the web browser, cell phone, and perhaps even desktop clients of some sort.

Message Storage, Archiving&Search

Much of the criticism of existing microblogging architectures has been directed at their reliance on relational database systems, and some critics have proposed a file system based approach for the next generation of services. For example, a directory could be created for each user on an SMTP server and microblog messages could be stored as text in this directory. This would facilitate not just indexing and data mining techniques, but rendering this stream of microblog messages in the UI wouldn’t require expensive DB access. Luckily we’re dealing with very small messages here, relative to all the streaming media usually carried across the internet, so the bandwidth and storage requirements probably aren’t that taxing.

Let the Games Begin

As you can see, I haven’t thought this out completely yet. It’s probably 60% baked at this point. I’d encourage you to leave comments, ask questions, and raise issues. For example, it’s not quite clear to me how you’d send an SMTP message to everyone on a particular server, or what you’d do with a message intended for someone on a remote server if that server is down, or how you’d make sure the person sending you a message is who she says she is.

Also, in order to get Steve Gillmor on board (and I do honestly hope he comes on board. Seriously.) we need to figure out how to get track to work. This means each SMTP server will probably need to examine and filter messages based on their content and route those messages accordingly. Again, this is an important area for further discussion and investigation.

I do have one request: Please limit your conversation on the new SMTP specification to the comments here on this post, as I want everyone to realize this was my idea. Thanks in advance.

blog comments powered by Disqus