This is a very clever move from AOL. They decided that the value lies in their users base and in the network they provide. They thus want to grow the community usage of their platform. Revenue is based on advertising. You can filter ads in your client but you can also decide to build an ads supported software and benefit from AOL advertising network. You will thus be able to share advertissement revenues with AOL. I bet that MSN and Yahoo! will now think about the large set of opportunity that this model opens.

What does this move mean for the Instant Messaging community ? Simply, it means that building a client for AIM is now officially possible. Building an Instant Messaging aggregator is possible regarding the AIM / ICQ protocol aggregation.

For ProcessOne, it also means that AOL and ICQ gateway software can be published officially. Maybe it is time to dream about the most scalable AIM / ICQ gateway ever published (hint, hint !).

AOL has now opened the AIM / ICQ platform and protocol. Now, there is no major reason to not fully adopt the XMPP protocol. See my previous blog post on that topic for more information: The AOL XMPP scalability challenge. The missing part is still allowing to chat with a contact without requiring an AIM or ICQ account. In the XMPP world, this is called federation. I am not convinced that they will do this move if they managed to extend advertisements and revenue sharing to this larger community.

For now, let's celebrate !

Experience in Erlang and/or XMPP protocol is a big plus, but is not mandatory.

The jobs are preferably based in France, but our company being international and our team distributed across the world, remote work is possible.

If you are interested, please drop me a mail (mremond at this website domain) with a summary of your technical achievements and a description of why you would love working at Process-One.

I think this is an interesting read.

The article has been written by Jérôme Sautret and presents basic ejabberd architecture and concepts.

Proof is given with the following sequence:

Memphis-2:~ mremond$ telnet chat.live.yahoo.com 5222
Trying 68.180.220.125...
Connected to chat.live.yahoo10.akadns.net.
Escape character is '^]'.
<?xml version='1.0' encoding='UTF-8'?>
<stream:stream xmlns:stream='http://etherx.jabber.org/streams' version='1.0'
 xmlns='jabber:client' to='chat.live.yahoo.com' >

<?xml version='1.0' encoding='UTF-8'?><stream:stream
 xmlns:stream="http://etherx.jabber.org/streams" xmlns="jabber:client"
 from="chat.live.yahoo.com"
 id="5d46000b" xml:lang="en" version="1.0"><stream:features>
<mechanisms xmlns="urn:ietf:params:xml:ns:xmpp-sasl"><mechanism>PLAIN</mechanism>
<mechanism>JIVE-SHAREDSECRET</mechanism></mechanisms><compression
 xmlns="http://jabber.org/features/compress"><method>zlib</method></compression>
<auth xmlns="http://jabber.org/features/iq-auth"/></stream:features>

This is a bit technical, but what what this command shows, is that Yahoo! server is replying on the standard XMPP port. When sending the stream opening XML, it replies with a server stream, meaning that there is this is really an XMPP server.

Note that we have no reason to believe yet that those experimentations will ever come to Yahoo! Instant Messaging service. However, as they gather XMPP knowledge, they are more and more likely to progressively add XMPP support to their main instant messaging service.

Update 1: I have checked and server-to-server connections are not open: It means you cannot join a Yahoo! Live chat room with your usual XMPP account and client.

Introduction

Since I designed the first version of the pluggable pubsub module for ejabberd in early 2007, I had in mind to turn it into a powerful application server. I have already blogged about the power of the new API of ejabberd 2.0 pubsub engine. However, this single article does not do any justice to this idea and how it can change the face of the web.

Customizable services based on pubsub is XMPP (eXtensible Messaging and Presence Protocol) at its full speed. The XMPP protocol has been designed since the beginning to be a near real time routing / distribution engine. A piece in the puzzle was missing however. This is where I think our new plugin based pubsub API fills the gap and turn ejabberd into the first XMPP application server.

As a proof of concept and as a way to robustify our API, we have written the Personal Eventing via Pubsub (PEP) as a plugin of our pubsub engine. It is a big specification and should clearly show the potential of the framework.

The XMPP application server: The Twitter case

The PEP implementation was still not enough to demonstrate that ejabberd is a fault-tolerant, highly scalable application development platform for the Web 2.0 era. I decided to start a serie of articles demonstrating how to build a distributed Twitter-like microblogging platform based on ejabberd 2.0.

Step 1

The goals of the first development step are the following:

• Write a new ejabberd service as a pubsub extension: this is a XMPP Twitter-like application code itself.
• Show that you can integrate with other pieces of the XMPP framework and not only the event distribution: The code uses the service browser as a user interface, building a fully dynamic hierarchy.
• Write the near realtime publishing / distribution code. To show how powerfull it is, I wrote the code in a way that node creation and subscriptions are not necessary. You only have to publish content and it is distributed to anyone that can see your presence. The end result, like PEP, is closely integrated to the user roster.

The architecture is distributed across as many user base (domain) as you want: A user can get events from users having their account in a large number of different servers.

Writing a new plugin to define custom node hierarchy

I wanted to build a special service hierarchy so that the application integrates nicely in XMPP service discovery. I wanted a dynamic discovery tree whose content was depending on who was doing the query. This would make the application more user friendly and the example much more impressive.

To write a custom node hierarchy with our API, you simply have to write a nodetree plugin. I have written the nodetree_twitter module to handle our special hierachy looking like the following:

• Root node: "twitter"
• Special node: MyStream. This is a collection displaying your latest published posts.
• Special node: FriendsStream. It contains the latest posts of my friends.
• Special node: MyFriends. It contains the list of your friends and allow you to see the latest posts of a given contact.

Friends are taken from the instant messaging contacts list: I defined this friends list as "the people that already share their presence with me". This way users do not have to manage the cumbersome subscription process specially for this application: Their friends from their contact list are cleverly reused.

Note that the node hierarchy is not defined in the Pubsub specification (XEP-0060), but is totally needed to make the application flexible. This pubsub node hierarchy does not violate the Pubsub specification, so we are fine with it.

This is the result as shown by Psi's service discovery::

The contact Blaine (thank you for your comments, by the way) is coming directly from my roster.

Posting entries

Posting an event is simply sending a small piece of XML code in ejabberd. As Psi does not support publishing events to our special virtual nodes (yet ), we will send it from Psi XML console.

For example, I can send the following piece of code in the XML console. I publish the post on my special pubsub node MyStream:

<iq type='set'
    to='pubsub.localhost'
    id='publish1'>
   <pubsub xmlns='http://jabber.org/protocol/pubsub'>
     <publish node='/twitter/MyStream'>
      <item>Demoing my Twitter-like code</item>
    </publish>
  </pubsub>
</iq>

The message is received instantly by all the contacts that have subscribed to my presence. No polling mechanism is needed. This is the beauty of using ejabberd XMPP App Server.

The received message is the following:

<message from="pubsub.localhost" to="test1@localhost" >
  <event xmlns="http://jabber.org/protocol/pubsub#event">
    <items node="/twitter/FriendsStream" >
      <item id="4B2C8277256F" >Demoing my Twitter-like code</item>
    </items>
  </event>
</message>

Note that the dynamic hierarchy is very powerful: We publish on our special node. The message seems to come from another special node on the other end, for the sake of consistency.

Persistence

I did nothing special for persistence of the published items. It is handled as a default by the ejabberd pubsub framework.

We can browse our new service to check that the blog post has been actually published:

I did not yet write the part of the code to browse friends posts. This will be part of a second article.

Conclusions

The end result is the basis of a Twitter-like application. It is fully distributed at the web scale. You can run this application on different servers and get events from people with account on different servers. It is compliant with ejabberd clustering and high scalability features. Most of all, thanks to our Pubsub API, I wrote this first code in 3 hours. It tooks something like 200 to 300 lines of code to write this extremely powerful infrastructure. Yes, that's amazing, I agree, as I was surprised myself by how easy and fast it was to write !

After a bit of clean-up, the code will be added to ProcessOne contributions Subversion repository. If you are impatient to play with the code, please drop me a mail

In a next article, I will present the second stage of this application. I will add better and complete browsing of the items and publish directly for the service browser thank to adhoc commands supported in Psi.

In the meantime, I hope to get this experiment running with client developers: It would be very nice see the messages received properly displayed in OneTeam and in Psi. Stay tuned !

Let's try some guess work imagining what it could mean in the Instant Messaging space. To keep the reasoning, we will concentrate on the biggest IM services.

A bigger IM community ?

I see three possible options for the Microsoft and Yahoo! public Instant Messaging and Presence service:

• Microsoft keeps both networks: I do not think it will happen, as Microsoft has always tried to increase its MSN IM community. The advantage in having one big community is much larger than probably they keep both networks, but interconnect them (already being done in MS / Yahoo! interop programme)
• MSN Instant Messaging user base is merged inside Yahoo! service: I do not think this will happen. MSN community is bigger than Yahoo!. MSN is a very strong leader in several parts of the world. It make more sense to do the other way around.
• Yahoo! IM user base will be merged in MSN. This sounds currently the most likely to happen.

Forces on the public IM battlefield

Merging the two Instant Messaging networks is not enough for MSN to become the biggest Instant Messaging network.

The best figures I found are the following: in 2006, according to Neilsen/Netratings, the market share in the public IM space was as follow:

• AOL: 53 million users
• MSN: 27 million users
• Yahoo: 22 million users
• Google: 866,000 users

The figures are a bit outdated, but the hierarchy is probably still the same.

As said earlier, we can expect that in a distant future MSN and Yahoo services will be at least fully and totally interoperable, if not completly merged.

Recently, it has been found that AOL was experimenting XMPP access to their service. XMPP stands for eXtensible Messaging and Presence Protocol and is becoming very hot. It is the same protocol that Google is using. This protocol is also used by a large federation of IM servers around the world.

It means that the battle lines are shaping up, splitting the public IM world in two or three parts, depending on these hypothesis:

• AOL: 53 million users
• MSN + Yahoo!: 49 million users
• AOL: Google + all federated XMPP networks: estimated 100 millions (pessimistic figure if we trust Wikipedia article, probably more)

AOL needs absolutely to join the XMPP front if they want to keep their leadership in Public Instant Messaging service.

Conclusions

1. What it could mean is that AOL is constrained to accelerate the move to XMPP to merge forces with XMPP players. This move seems very good for XMPP: by possibly reducing the number of proprietary protocols in the Instant Messaging world, it possibly strengthen the open standard one.
2. It also makes more fragile companies whose business model rely on Instant Messaging aggregation. The less different protocols, the fewer accounts you need, the less you need aggregation tools.

I bet that the future of XMPP and XMPP companies is even brighter today.

]]> Jabber / XMPP, ejabberd 2008-02-01T18:25:00+01:00