Challenges in Building Real Time Applications

Real time is everywhere. Users are now expecting that applications can update and display pieces of information in real time.

Whether you are building a chat application, a website, a mobile app, or a business application, users want to be notified, receive pushes (properly targeted), be able to react instantly and have the user interface always up to date.

However, the real time domain is quite broad. A chat application is very different from a business dashboard. Far too often, I have seen “real time” web applications that were just doing refresh queries every 5 seconds to keep the data up to date. This is a naive way to build real time applications, but the real problem is that most of the time, it does not scale. In the case at hand, each request was doing a whole set of database queries. It worked fine with ten users but collapsed under load with just a hundred users.

What you need is to stop pulling data and switch to a server push approach. However, it needs to be designed that way from the beginning, as you need to think your system in terms of subscription scope, consistent set of data (channels). You need to design how real time information will propagate and update your system.

This is just a simple example coming from a real case. I can find many other examples and I am sure you can too.

» Do you want to enhance your business with real time systems? «
We help companies achieve goals through real time streaming architectures.
Contact us

So, depending on the type of applications you need to write, you must consider the toolset you have available and possibly mix them. You need to design your data flow, your events, your integration points, etc. real time tools are often seen as a commodity, but not the skills needed to build such systems. This expertise is very rare. Real time is not something you learn at school. It is a very advanced topic in information system design.

In practice, many developers and software architects struggle to introduce real time in their applications or information systems. You not only need to introduce a new set of skills, but they also need to learn about the available tools to implement real time systems.

I have been very aware of the issues developers were having. I have helped tens of companies build real time messaging systems since 2002. However, this is only in the last years that I realized that I needed to put in place a holistic approach for real time software design. Real time is much more than building chat systems. It needs to be placed at the heart of your application. So, that’s why we expanded our core skills to new tools, protocols, and type of applications. We have a unified vision of real time that often mix XMPP, MQTT, Kafka, among other protocols, to connect people, things and applications together. It helps us use the right tool for the real time job at hand. It broadens the vision and helps us help developers.

We can thus help developers and software architects better understand what is the type of real time data they will manipulate, how critical they are, what are the best architectural design and what are the best tools for the jobs and why.

However, to get further, we need to understand more broadly your struggles. What are you facing as challenges in developing and building your system? Which tools have you selected? How is it working for you?

You want to build a better development team, that think about architecture and integrates real time from the beginning? What is restraining you to do so? Are you fighting with server platforms? Software architecture design? Client developments? Skills? Which tools are you using today? What do you plan to use tomorrow?

I am preparing contents to help you get better at overcoming the challenges of real time systems, so do not hesitate to comment or mail me (mickael.remond@process-one.net). Please, share your feedback and issues to help me help you.

Thanks!