Neat project! I notice in your slides that you say it will work on a cluster of “several machines”. How well do you think this would scale up to a cluster of thousands of machines?

Thanks,
Steve

Hello Steve,

The cluster we are talking about is an “Erlang cluster”. This is a cluster to run Erlang applications on, with the distributed Erlang mode.
Erlang distributions mode has not been designed for thousands of machines, but for running an hosted distributed application in a cluster for fault-tolerance and scalability. The Erlang distribution protocol does not target peer to peer application.

So to answer your question, no, the purpose is not to scale to thousands of machines.

Cheers,

--
Mickaël Rémond

Thanks for answering.

If distributed erlang can’t scale to thousands of machines, how would somebody launch an ejabberd cluster that could handle, say, Google Talk’s traffic? Since I work at Google, this question is not academic to me. There must be tricks people use to shard traffic so as to split up their clusters into smaller, more manageable pieces?

Thanks,
Steve

hehe,

Not academic, but difficult to answer in a few words.
You have to use architectural tricks to reach this level of scalability. The scope of the tricks you can use depends on the contraints of the application you are building. Do you control the clients that are connecting to your site or not to hide some internals or do you want to allow any client to connect, etc ?

When you have defined precisely your requirements, you can start building your architecture. Ultimately, if you contrains are very tight, you will actually end up splitting your cluster into several manageable parts with communication between the separate parts only when needed (and not using the Erlang distributed protocol but a custom TCP protocol).

I hope this helps (at least a bit)

You can contact me privately if you have more precise questions in mind,

Cheers,

--
Mickaël Rémond