An Open Twitter-like Network, Powered by Email.

There’s been a few articles lately about what an open twitter-like network would be. A while back I was thinking about the same problem and how it could be implemented, so I wanted to throw my 2 cents in on the subject. My solution is super simple, does not require any new technology to be deployed, all it would take is a clear message standard, and clients to be created for a few popular platforms.

So here’s my rambling thoughts on what I’ll codename OpenTweet. I wanted to put these down on paper so they stop rattling about in my head, and maybe someone would like to point out some issues with what I’m proposing too or add their own thoughts. Feedback invited.

Firstly though, let me just say, what Twitter have created, the idea of a short, timely, but ultimately throw-away, 1-to-many unidirectional message is a powerful invention. The problem is that invention is held entirely by them and with the closing of API’s lately it’s becoming apparent that they intend to tightly control the access to it. That makes me uneasy.

Never Assume

The first big assumption is this: a tweet is just a no-subject-line, 140 character email (with some special headers). From that assumption we know the infrastructure we already have, capable of sending billions of messages a day around the internet, already exists. It’s already capable (arguably more so than twitter themselves) of dealing with spam. It’s not ‘owned’ by any one person or company.

My proposal is a layer on top of email and particularly mailing lists. This way people are free to run their own ‘twitter server’, or use a hosted one (basically just a POP/IMAP/SMTP server – every host in the world currently offers this – and obviously there would be completely hosted options like hotmail/gmail/yahoo.com too. API support is available in every modern programming language, they can all connect to POP/IMAP and SMTP servers to send and receive emails.

Let’s say I make the email address twitter@aschroder.com. This is my OpenTweet address, see how it’s just an email address? The only difference is that instead of Sparrow or Mailplane to check and send emails with this account, I’ll use my OpenTweet client. The OpenTweet client will know how to respond to various special messages sent and received from the mail account – but otherwise, it’s just email. Obviously it’ll render the emails like twitter cleints render tweets though – you can still have favorites, and ‘unread’ messages, as you do now.

The Basics

So let’s imagine there’s twitter@aschroder.com and it would be a mailing list.

If you subscribe to that list, you receive emails sent to the list, if you unsubscribe, you do not. The only person allowed to send emails onto the list, is the owner (me in this case). See how that’s basically what a twitter feed is?

How do lists work? easy. In OpenTweet, lists are just mailing lists too. For example. nzwebdevelopment@nzherald.co.nz. This would be the NZ Herald’s list of NZ web developers.

Someone who controls the list would then subscribe that list to say, my mailing list and Robert’s and Kristof’s. Then anyone can in turn subscribe to the OpenTweet list nzwebdevelopment@nzherald.co.nz and receive tweets from any of those subscribed feeds.

How about a private twitter feed? Simple, private mailing list, only add an email to the list when it’s approved by the owner – and don’t publish the mailing lists publicly.

Mentions, direct messages? These are just specialized messages that are interpreted by the client. I could equally send a special message to your address to find out who has mentioned you lately.

Discovery

Just as there is a robots.txt and sitemap.xml for instructing web robots, I’d suggest an opentweet.txt file in the root of any website, which tells clients what my OpenTweet address is. It’s also where I’d publish any lists I want to share. The file would contain the information on all OpenTweet accounts hosted at the domain. So www.gmail.com/opentweet.txt would likely be a big file. That’s OK though, sitemap.xml files can be big too, and we have really well-tested, proven methods for splitting them, zipping them and scaling to sites with millions of pages.

OpenTweet:aschroder.com
aschroder:twitter@aschroder.com
Ashley Schroder:twitter@aschroder.com
archive:aschroder:/aschroder.twitter.html
List:MagentoFolks:magentofolks@aschroder.com
Avatar: http://www.aschroder.com/me.png
Description: Magento developer from New Zealand...
Web:aschroder.com
Keywords: ashley, Schroder, Magento, New Zealand...

Search engines would begin indexing these files and then when someone wants to search for an OpenTweet user, or list – they can offer up results based on the files. This file also contains meta-information such as profile picture/avatar, description and website.

You can put it on your business card, just the same as you do now. There could be a particular header in a tweet message, and if a regular email (sans special header) is sent to the email address it auto-replies to let the person know it’s an OpenTweet address and they should follow it with their OpenTweet client – it may even suggest a few suitable clients.

Update: Christian has noted there are already standards that could handle this part of the network.

Information exchange

If I want to know how many followers someone has, and who they are – my client sends a special message to a particular users’ OpenTweet email address likewise for checking who someone follows.

One problem with decentralization though, they could be lying. However the data is verifiable, in that I can send a message to each party checking who they follow, the two would need to match up – if they don’t well, the final say sits with the ‘follower’ in that if their client ignores messages received from an account they do not record as ‘following’, they’re not following.

Client Consistency

How do I make sure my iPhone client and my desktop client share the same lists, followers and followed users? There’s an app for that too. The same way we currently share a list of contacts between email clients. Mac folks might use iCloud – Google has Google Drive. It’d be up to a given client, and may even spawn services specializing in just that.

Search

Ok, so there’s a glaring problem here – search. With all of this decentralization, there’s no obvious way for me to search for say ‘olympics’ and see what’s going on right now with the olympics. All the mailing lists are islands, no one list has all the tweets about ‘the olympics’.

What I suggest is that each twitter mailing list is simply published as web pages, albeit frequently updated web pages. And then just as with web sites, these mailing lists will be indexed and made searchable by various search engines.

I’m sure the results pages will be laden with ads and ‘who to follow’ featured spots etc. That’s all totally fine, that’s how the internet works, search engines will simply get really good at rapidly reading and indexing millions of pages (like they do now) only the pages will contain short messages sent to mailing lists.

I’d expect all the major search engines to be quite capable of this already, and a few small players might even come up with better ways to do it too. Then it’s just a matter of point in your OpenTweet client at you favorite OpenTweet search engine.

Other

Ownership is different too. Because the owner of the server publishes their tweets, they’d be free to assert copyright on them too – you own your own OpenTweet content. But just as you are responsible for backing up your blog posts, you’d need to backup and archive your own tweets too (or your provider would, if you don’t host your own).

Speed & Synchronicity. I realize that because the message passing is by way of asynchronous emails, not synchronous JSON or XML API’s there’s likely to be latency and delays. But if you’ve ever been on the phone to someone and said ‘just sending this through now’ and 1 second later they say ‘yep got it’ you’d know that in the most part, email is pretty fast and reliable. It’s reliable enough for the purposes of OpenTweet, where we’re not offering 100% reliability, I’m not expecting anyone to use OpenTweet to send nuclear passcodes when we go DEFCON 5.

Spam. This could be handled both by the server and by client specific rules. For example I’d be happy to filter out mentions from users who have sent mentions for 90% of their last say 100 messages and who have URLs in >50% of their tweets – as an example.

Getting it started and Transitioning

The good thing about this setup is it can be layered onto of Twitter itself too, messages sent into my mailing list, could be re-broadcast to my Twitter feed too – and messages my twitter account currently receives, could be fed into my own mailing list, for my OpenTweet client to receive. That way those that don’t wish to use OpenTweet can still benefit from sending and receiving tweets to/from OpenTweet users.

All of this is purely hypothetical of course. It does seem like a possibility, with a clear standard, and a few clients to get started. Anyone care to pick holes, did I miss anything?