Introducing the XMPP application server: The Twitter example
Posted by Mickaël Rémond on February 03, 2008ejabberd is probably the first XMPP application server ever released. This article introduces this new concept and explains how to build a distributed event-based infrastructure for social networking. The example application that illustrate this article is a distributed Twitter-like microblogging platform.
Introduction
Since I designed the first version of the pluggable pubsub module for ejabberd in early 2007, I had in mind to turn it into a powerful application server. I have already blogged about the power of the new API of ejabberd 2.0 pubsub engine. However, this single article does not do any justice to this idea and how it can change the face of the web.
Customizable services based on pubsub is XMPP (eXtensible Messaging and Presence Protocol) at its full speed. The XMPP protocol has been designed since the beginning to be a near real time routing / distribution engine. A piece in the puzzle was missing however. This is where I think our new plugin based pubsub API fills the gap and turn ejabberd into the first XMPP application server.
As a proof of concept and as a way to robustify our API, we have written the Personal Eventing via Pubsub (PEP) as a plugin of our pubsub engine. It is a big specification and should clearly show the potential of the framework.
The XMPP application server: The Twitter case
The PEP implementation was still not enough to demonstrate that ejabberd is a fault-tolerant, highly scalable application development platform for the Web 2.0 era. I decided to start a serie of articles demonstrating how to build a distributed Twitter-like microblogging platform based on ejabberd 2.0.
Step 1
The goals of the first development step are the following:
- Write a new ejabberd service as a pubsub extension: this is a XMPP Twitter-like application code itself.
- Show that you can integrate with other pieces of the XMPP framework and not only the event distribution: The code uses the service browser as a user interface, building a fully dynamic hierarchy.
- Write the near realtime publishing / distribution code. To show how powerfull it is, I wrote the code in a way that node creation and subscriptions are not necessary. You only have to publish content and it is distributed to anyone that can see your presence. The end result, like PEP, is closely integrated to the user roster.
The architecture is distributed across as many user base (domain) as you want: A user can get events from users having their account in a large number of different servers.
Writing a new plugin to define custom node hierarchy
I wanted to build a special service hierarchy so that the application integrates nicely in XMPP service discovery. I wanted a dynamic discovery tree whose content was depending on who was doing the query. This would make the application more user friendly and the example much more impressive.
To write a custom node hierarchy with our API, you simply have to write a nodetree plugin. I have written the nodetree_twitter module to handle our special hierachy looking like the following:
- Root node: "twitter"
- Special node: MyStream. This is a collection displaying your latest published posts.
- Special node: FriendsStream. It contains the latest posts of my friends.
- Special node: MyFriends. It contains the list of your friends and allow you to see the latest posts of a given contact.
Friends are taken from the instant messaging contacts list: I defined this friends list as "the people that already share their presence with me". This way users do not have to manage the cumbersome subscription process specially for this application: Their friends from their contact list are cleverly reused.
Note that the node hierarchy is not defined in the Pubsub specification (XEP-0060), but is totally needed to make the application flexible. This pubsub node hierarchy does not violate the Pubsub specification, so we are fine with it.
This is the result as shown by Psi's service discovery::
The contact Blaine (thank you for your comments, by the way) is coming directly from my roster.
Posting entries
Posting an event is simply sending a small piece of XML code in ejabberd. As Psi does not support publishing events to our special virtual nodes (yet ;)), we will send it from Psi XML console.
For example, I can send the following piece of code in the XML console. I publish the post on my special pubsub node MyStream:
<iq type='set'
to='pubsub.localhost'
id='publish1'>
<pubsub xmlns='http://jabber.org/protocol/pubsub'>
<publish node='/twitter/MyStream'>
<item>Demoing my Twitter-like code</item>
</publish>
</pubsub>
</iq>
The message is received instantly by all the contacts that have subscribed to my presence. No polling mechanism is needed. This is the beauty of using ejabberd XMPP App Server.
The received message is the following:
<message from="pubsub.localhost" to="test1@localhost" >
<event xmlns="http://jabber.org/protocol/pubsub#event">
<items node="/twitter/FriendsStream" >
<item id="4B2C8277256F" >Demoing my Twitter-like code</item>
</items>
</event>
</message>
Note that the dynamic hierarchy is very powerful: We publish on our special node. The message seems to come from another special node on the other end, for the sake of consistency.
Persistence
I did nothing special for persistence of the published items. It is handled as a default by the ejabberd pubsub framework.
We can browse our new service to check that the blog post has been actually published:
I did not yet write the part of the code to browse friends posts. This will be part of a second article.
Conclusions
The end result is the basis of a Twitter-like application. It is fully distributed at the web scale. You can run this application on different servers and get events from people with account on different servers. It is compliant with ejabberd clustering and high scalability features. Most of all, thanks to our Pubsub API, I wrote this first code in 3 hours. It tooks something like 200 to 300 lines of code to write this extremely powerful infrastructure. Yes, that's amazing, I agree, as I was surprised myself by how easy and fast it was to write !
After a bit of clean-up, the code will be added to ProcessOne contributions Subversion repository. If you are impatient to play with the code, please drop me a mail :)
In a next article, I will present the second stage of this application. I will add better and complete browsing of the items and publish directly for the service browser thank to adhoc commands supported in Psi.
In the meantime, I hope to get this experiment running with client developers: It would be very nice see the messages received properly displayed in OneTeam and in Psi. Stay tuned !
Categories: Jabber / XMPP ejabberd Erlang
Comments
Excellent article, thanks for writing it!
I keep seeing XMPP as an application server myself. It often seems to me that all of the current social networking sites capabilities could easily (famous last words ;-) ) be wrapped up with pub/sub and or a couple mods/components. The project that I keep percolating in the back of my head is some kind of simple photo gallery that could be integrated into a XMPP client/server, once you have a photo gallery and a micro-blogging app there is no other corner stones that IMHO that the other social networks offer - XMPP FTW! :-D
Posted by Ryan Parrish on 04 Feb 2008 at 14:50
Such a project exists already for years. It is called xDash (http://xdash.jabberstudio.org/) and uses Perl/PostgreSQL. xDash is a framework for application integration (EAI). It is using jabber/XMPP as the communication protocol and a set of agents. All the needed stuff is automatically generated from a CSV file, which can be comfortably edited in a spreadsheet editor. There is a tutorial and live demo for xDash available as a VMware image at . It can be used with the free VMware Player.
Posted by Jerzy Wachowiak on 05 Feb 2008 at 08:19
Hello Jerzy,
Thank you for your feedback.
I know the EAI (Enterprise Application Integration) and ESB (Enterprise Service Bus) topics very well and I will try to explain why an XMPP application server platform and an EAI are a different things.
I have designed J-EAI, an XMPP-based ESB (or EAI). The project itself was launched in 2004 and its origin comes from my previous work on IDX-Buster XML bus when I was working at IDEALX.
Most of the assets of J-EAI have now been integrated into ejabberd 2.0. xDash would be in some sense comparable to what J-EAI was doing.
However, the field covered by EAI is only a very small part of what you need to build an XMPP application server.
An XMPP application server is more featured and complete for the following reasons:
- The XMPP server does manage not only the message distribution and reliability and monitoring, but has an API to write modules, components, plugins that are run and controled by the application server itself. In other terms, the application server controls the life cycle of the components. xDash rely on external agents (like J-EAI). An XMPP application server provides facilities to develop software in the application server framework itself. You can do much more things with this approach as you have access to everything including internal API.
- An XMPP application server provides API and facilities that cover a much larger scope. ejabberd XMPP application server provides API to handle presence events in your component. It provides API to add features to the XMPP server itself (new service such as contact list management, avatar management). It also provide feature to handle typical synchronous call (IQ packets in XMPP terms), whereas EAI focus mainly on asynchronous messages calls.
- As the example shows, we not only control the message flow, but we also developed a custom user interface directly on the client side, using discovery feature (and next article will show how to handle adhoc command to add interactive forms).
As a conclusion, an EAI platform is mainly designed to manage inter-applications communications. An XMPP application server is designed to handle richer inter-applications workflows, but also to mix humans and applications into the same workflow.
Posted by Mickaël Rémond on 05 Feb 2008 at 11:17
Angie looks great. Will it be possible to run it with Epeios (and another XMPP server) ?
Posted by kael on 06 Feb 2008 at 20:26
Hello Kael,
To benefit from Angie scalability, robustness or feature set, you need to run ejabberd.
Epeios can be used to run standard ejabberd module, but you will miss a big part of what makes ejabberd 2.0 great.
Hi Mickaël,
@ I read you explantion above and understand your idea of XMPP application server as a sort of platform for realisation of systems sticking to the event-driven architecture (http://en.wikipedia.org/wiki/Event_Driven_Architecture).
@ Maybe I am coming from the wrong corner (system integration with SAP, MQ, JMS, enterprise applications) but don’t see the practical use of XMPP application server. Could you provide a real-live example, where the use of a XMPP server would be an advanatage over the state-of-the art technology?
Cheers,
Jerzy
Any source code release yet for this proof of concept?
Posted by Patrice Girard on 19 Feb 2008 at 04:22
To keep you updated, I am preparing the second article in this serie and will publish the code at the same time.
I think I will publish it this week-end.
Any updates on the next version/the code? Looking forward to it.
Posted by Itay on 02 Mar 2008 at 19:12
Me too, I am really itching to have a closer look at this and would love to see some code!
Posted by Fredrik on 04 Mar 2008 at 18:31
On attend avec impatience la suite (et le code) Mickaël !
Very good read :)
hi Mickael,
can u please tell me which web client suits for ejabbed server
regards
rams
Does the viral GPL license mean that ejabberd cannot be redistributed (sold) with closed-source value-added plugins ?
It seems that any ‘application server’ should be licensed in a way that does not virally infect all application components that may be developed on top of it, perhaps something like LGPL with linking exclusions for the plugin APIs (but I am not a lawyer).
Mik
Posted by mikhailfranco on 20 Mar 2008 at 09:42
P.S. A good analogy would be JBoss J2EE ‘application server’
(which is also in the Gnu-ish camp) and ships with an LGPL license.
Mik
Posted by mikhailfranco on 20 Mar 2008 at 09:49
@Mik: It depends on how your application works and is packaged, but you can write closed source plugins.
Posted by Mickaël Rémond on 23 Mar 2008 at 15:36
Sorry for the delay, guys. We have received unusually large numbers of customers requests and projects on ejabberd and I will follow up on this as soon as I find a day to work on this.
Posted by Mickaël Rémond on 23 Mar 2008 at 15:38
Thank you for your answer, Mickaël. Is there any documentation about writing a pubsub plugin so far ?
Posted by Florian on 31 Mar 2008 at 16:27
Hej Mickaël,
implementing my own pubsub node hierachy sounds like I always need to implemnt it using plugins. Is there a way to have some “normal” pubsub nodes right below /, like “monitoring/events” on the same level as “home/<server>” ?
Thanks
Posted by Christoph Görn on 01 Apr 2008 at 20:39
Christoph,
From looking at the ejabberd codebase, you would need to modify node_default.erl to do so. The reason is that it specifically checks that you are creating a node in the “/home/server/user” subtree, otherwise it says you are not authorized to continue.
You could fairly easily create a node_christoph, delegating nearly everything except the permission checking to node_default.
Does that make sense? Feel free to email me at my firstname ‘at’ lastname .net if you have any questions.
Itay
Posted by Itay Neeman on 22 Apr 2008 at 06:02
I’ve just written about (and linked this post) a Jabber implementation of Distributed Twitter, and whether or not it’s overkill for the timeliness requirement of microblogging. I’d be interested in your thoughts.
Posted by Joe Cascio on 06 May 2008 at 19:56Add comment
Stay Informed
Subscribe to our RSS feed or follow us on Twitter to receive alerts when we post new news stories and blogs.
Subscribe to ProcessOne RSS feed
Follow Mickaël on Twitter
Search our blogs
To make it easier for you to find blogs on topics that you are interested in, we have grouped them into categories depending on the different themes addressed in each blog. Categories include:
ProcessOne
Jabber / XMPP
ejabberd
Erlang
CEAN
Tsung
French
Employment opportunities
IMtrends
All categories
Become a ProcessOne partner
Find out about the benefits of joining ProcessOne’s partnership programme.
Click here
Our products and services
We offer packaged solutions comprising all of the server-side and client-side technology necessary to create valuable new instant messaging applications. In addition, we offer a range of services for delivering customised solutions.
Click here for full details