Security Advisory: eMudhra Intermediate CA Compromise and ejabberd

The recent eMudhra Intermediate CA Compromise is a security threat for ejabberd if you use s2s, modules that make outgoing HTTPS requests, or TLS connections to MQTT brokers (mod_mqtt_bridge).

To check if your ejabberd is vulnerable, you can use the following code snippet (works on ejabberd 19.08 and later).

Open the debug console with:

ejabberdctl debug

Paste the following code into the debug console (the indented version of this code is available at the end of this post):

(fun() -> Host = "dv-g3b-emudhra.we-have-a-problem.com", Fail = fun(Err) -> {"unable to connect to test server", Err} end, case gen_tcp:connect(Host, 443, [binary, {active, false}]) of {ok, Sock} -> case fast_tls:tcp_to_tls(Sock, [connect, {cafile, ejabberd_option:ca_file()}]) of {ok, TSock} -> _ = fast_tls:send(TSock, list_to_binary("HEAD / HTTP/1.0\r\nHost: " ++ Host ++ "\r\n\r\n")), case fast_tls:recv(TSock, 0, 10000) of {ok, _} -> case fast_tls:get_peer_certificate(TSock) of {ok, Cert} -> FT = case fast_tls:get_cert_verify_string(fast_tls:get_verify_result(TSock), Cert) of <<"ok">> -> "vulnerable"; _ -> "ok" end, HT = case list_to_integer(erlang:system_info(otp_release)) >= 26 of true -> case httpc:request("https://" ++ Host) of {ok, _} -> "vulnerable"; _ -> "ok" end; false -> "not verified (OTP < 26)" end, "fast_tls status: " ++ FT ++ ", ssl status: " ++ HT; E4 -> Fail(E4) end; E3 -> Fail(E3) end; E2 -> Fail(E2) end; E1 -> Fail(E1) end end)().

Exit the console with Ctrl-C twice.

It returns "fast_tls status: ok, ssl status: ok" if you are not vulnerable. If either status is vulnerable, please read on.

• If ssl status is not verified (OTP < 26): on Erlang/OTP < 26, outgoing TLS connections made with OTP's ssl (HTTP requests, MQTT bridge) don't verify server certificates at all; upgrading to OTP 26+ is recommended.
• If it returns {"unable to connect to test server", ...}, the check couldn't run (no outbound access, DNS, firewall…); this is not a "not vulnerable" result.

If you are running ejabberd with the official Docker image, you can update to 26.09.1 to fix the problem. It just removes the problematic certificates from the OS; it still runs ejabberd 26.09. Pull ghcr.io/processone/ejabberd:26.09.1 and recreate your container.

To fix it manually without restarting ejabberd, you can follow this procedure (requires Erlang/OTP 25+; on older versions, restart ejabberd).

First, update the CA bundle on your OS by upgrading the ca-certificates package:

# Debian / Ubuntu
apt update && apt install --only-upgrade ca-certificates
# RHEL / Rocky / Alma / Fedora
dnf update ca-certificates
# Alpine
apk upgrade ca-certificates

If your distribution has not shipped an updated package yet, you can distrust the certificates manually: see update-ca-certificates(8) on Debian/Ubuntu, or Using shared system certificates on RHEL-based systems.

Once the OS bundle is updated, paste the following into the debug console to reload it:

public_key:cacerts_clear(), public_key:cacerts_load(), ejabberd_admin:reload_config().

Exit the console with Ctrl-C twice.

Finally, re-run the check above to verify that you are not vulnerable anymore.

If you are still vulnerable, your ejabberd may be using CA certs from a specific location. You can use the following functions in the debug console to check where to update your certs (ca_file shows the CA file actually in use; if s2s_cafile is undefined, s2s uses ca_file).

ejabberd_option:ca_file().
ejabberd_option:s2s_cafile().

If you are using some s2s CA cert per vhost, pass it as a parameter, for example:

ejabberd_option:s2s_cafile(<<"example.com">>).

Here is the check above, indented for readability:

(fun() ->
    Host = "dv-g3b-emudhra.we-have-a-problem.com",
    Fail = fun(Err) -> {"unable to connect to test server", Err} end,
    case gen_tcp:connect(Host, 443, [binary, {active, false}]) of
        {ok, Sock} ->
            case fast_tls:tcp_to_tls(Sock, [connect, {cafile, ejabberd_option:ca_file()}]) of
                {ok, TSock} ->
                    _ = fast_tls:send(TSock, list_to_binary("HEAD / HTTP/1.0\r\nHost: " ++ Host ++ "\r\n\r\n")),
                    case fast_tls:recv(TSock, 0, 10000) of
                        {ok, _} ->
                            case fast_tls:get_peer_certificate(TSock) of
                                {ok, Cert} ->
                                    FT = case fast_tls:get_cert_verify_string(fast_tls:get_verify_result(TSock), Cert) of
                                        <<"ok">> -> "vulnerable";
                                        _ -> "ok"
                                    end,
                                    HT = case list_to_integer(erlang:system_info(otp_release)) >= 26 of
                                        true ->
                                            case httpc:request("https://" ++ Host) of
                                                {ok, _} -> "vulnerable";
                                                _ -> "ok"
                                            end;
                                        false ->
                                            "not verified (OTP < 26)"
                                    end,
                                    "fast_tls status: " ++ FT ++ ", ssl status: " ++ HT;
                                E4 -> Fail(E4)
                            end;
                        E3 -> Fail(E3)
                    end;
                E2 -> Fail(E2)
            end;
        E1 -> Fail(E1)
    end
end)().