Most teams hit their first Elixir scaling wall not because the BEAM can’t handle the load — it can — but because they bring thread-based intuitions into an actor-based world. They reach for connection pools when they should be spawning processes, add mutexes where supervision trees belong, and horizontally scale before they understand what’s happening inside a single node.
This is a field guide to what actually worked when we took a real-time payment processing platform from 40,000 requests per second to over 1 million, across a 12-node cluster, with sub-10ms P99 latency and zero planned downtime during the scale-up.
Why the BEAM Changes the Rules
Before getting into architecture, it’s worth understanding why Elixir is different at a fundamental level — not because the documentation says so, but because it changes every decision you make.
Traditional runtimes (JVM, Node, Go) model concurrency on top of OS primitives. A goroutine is still scheduled by the Go runtime and ultimately mapped to OS threads. A JVM virtual thread still allocates a stack. These abstractions reduce cost, but the underlying model remains: shared memory, locks, and the constant risk of one slow operation blocking others.
The BEAM is different. Each process is a fully isolated unit with its own heap, its own garbage collector, and its own message queue. When a process crashes, it crashes alone. When the GC runs, it runs on one process’s heap without stopping the world. At startup, a BEAM process costs around 2–3 KB — so spinning up 100,000 of them is not a resource crisis, it’s just normal operation.
This isn’t theoretical. At 1M RPS sustained, we had roughly 800,000 live processes at peak. On a JVM stack, that number would be absurd. On the BEAM, it was a Tuesday.
Modeling the Domain as Actors
The first architectural shift is conceptual: stop thinking in terms of request handlers and start thinking in terms of domain entities with their own lifecycle.
In our payment platform, each account has its own stateful process. Routing a transaction means finding or spawning that account’s process and sending it a message. The account process serializes operations naturally — no database-level locking required for balance checks, because the process is the serialization boundary.
defmodule Payments.AccountProcess do
use GenServer
def start_link(account_id) do
GenServer.start_link(__MODULE__, account_id,
name: {:via, Registry, {Payments.Registry, account_id}}
)
end
def init(account_id) do
state = Payments.Store.load_account(account_id)
{:ok, state}
end
def handle_call({:authorize, txn}, _from, state) do
case authorize_transaction(txn, state) do
{:ok, new_state} ->
{:reply, {:ok, txn.id}, new_state}
{:error, reason} ->
{:reply, {:error, reason}, state}
end
end
defp authorize_transaction(%{amount: amount}, %{balance: balance})
when amount > balance,
do: {:error, :insufficient_funds}
defp authorize_transaction(txn, state) do
new_state = %{state |
balance: state.balance - txn.amount,
pending: [txn.id | state.pending]
}
{:ok, new_state}
end
endThe key line is {:via, Registry, {Payments.Registry, account_id}}. This registers the process under the account ID in a local registry, so any caller can route to it without knowing the PID. The registry lookup is an O(1) ETS operation — fast enough to do on every request.
Building Supervision Trees You Can Trust
The second shift is operational: your supervision tree is your reliability contract. It’s not a curiosity you configure once and forget.
At 1M RPS, you will have process crashes. A malformed transaction, a network timeout to a downstream service, an unexpected nil — processes will die. The question isn’t whether crashes happen but whether you’ve designed the restart semantics correctly.
We structured supervision at three layers:
defmodule Payments.Application do
use Application
def start(_type, _args) do
children = [
# Infrastructure — start first, never restart siblings on failure
{Registry, keys: :unique, name: Payments.Registry},
Payments.Store,
Payments.EventBus,
# Account processes — isolated, restart individually
{DynamicSupervisor,
name: Payments.AccountSupervisor,
strategy: :one_for_one,
max_restarts: 10,
max_seconds: 1},
# Ingestion pipeline — if this dies, restart everything in order
{Supervisor,
children: [Payments.Router, Payments.RateLimiter],
strategy: :rest_for_one}
]
Supervisor.start_link(children, strategy: :one_for_one)
end
end:one_for_one on the top level means infrastructure failures stay isolated. :rest_for_one on the ingestion pipeline means if the Router crashes, the RateLimiter (which depends on it) also restarts — in order. These aren’t arbitrary choices; they encode the dependency graph of your system.
One lesson learned the hard way: set max_restarts and max_seconds deliberately. The default allows 3 restarts in 5 seconds. Under a thundering-herd scenario, a process tied to a broken downstream can exhaust that budget instantly, escalate to its supervisor, and cascade up the tree. We tightened this to 10 restarts per second at the leaf level and added circuit breakers at the boundary.
Horizontal Clustering with libcluster
A single BEAM node handled around 90k RPS in our benchmarks. To reach 1M, we needed 12 nodes with proper distribution. Elixir’s built-in distribution makes this surprisingly straightforward using libcluster.
# config/releases.exs
config :libcluster,
topologies: [
payments: [
strategy: Cluster.Strategy.Kubernetes.DNS,
config: [
service: "payments-headless",
application_name: "payments",
polling_interval: 5_000
]
]
]With nodes connected, :pg (Process Groups) gives you cluster-wide process discovery without a central coordinator:
defmodule Payments.Router do
def route(txn) do
node = least_loaded_node(txn.account_id)
:erpc.call(node, Payments.AccountProcess, :authorize, [txn])
end
defp least_loaded_node(account_id) do
# Consistent hashing keeps the same account on the same node
# unless that node is unavailable — avoids split-brain on balances
nodes = [Node.self() | Node.list()]
Enum.at(nodes, :erlang.phash2(account_id, length(nodes)))
end
endConsistent hashing here is critical. If account abc123 always routes to node-3, that node’s AccountProcess has a warm in-memory state. If we round-robin instead, every request cold-starts from the database. At 1M RPS, that difference is the gap between a working system and a database outage.
Backpressure with Broadway
Accepting 1M RPS is one problem. Processing them reliably without overloading downstream systems is another. This is where Broadway — Elixir’s demand-driven data processing library — earns its place.
defmodule Payments.IngestPipeline do
use Broadway
def start_link(_opts) do
Broadway.start_link(__MODULE__,
name: __MODULE__,
producer: [
module: {BroadwayKafka.Producer, [
hosts: [kafka: 9092],
group_id: "payments-ingest",
topics: ["raw-transactions"]
]},
concurrency: 4
],
processors: [
default: [concurrency: 50]
],
batchers: [
authorized: [concurrency: 10, batch_size: 500, batch_timeout: 50],
rejected: [concurrency: 4, batch_size: 200, batch_timeout: 100]
]
)
end
def handle_message(_, %Broadway.Message{data: txn} = msg, _) do
case Payments.Router.route(txn) do
{:ok, _} -> Broadway.Message.put_batcher(msg, :authorized)
{:error, _} -> Broadway.Message.put_batcher(msg, :rejected)
end
end
def handle_batch(:authorized, messages, _, _) do
ids = Enum.map(messages, & &1.data.id)
Payments.Store.bulk_confirm(ids)
messages
end
endBroadway’s demand-driven model means producers only emit messages as fast as processors can consume them. There’s no unbounded buffer filling up during a traffic spike — back-pressure propagates upstream to Kafka, which applies the pressure to upstream producers. The system self-regulates.
The batchers are particularly valuable for database writes. Instead of 500 individual INSERT calls, we batch them into single bulk operations every 50ms. At our throughput, this reduced database write load by roughly 80%.
What the Numbers Actually Looked Like
After full deployment across 12 nodes:
- Sustained throughput: 1.2M RPS
- P50 latency: 2ms
- P99 latency: 8ms
- P99.9 latency: 31ms
- Process count at peak: ~800,000 live processes
- Memory per node: 4.2 GB average (32 GB available)
- Deployment strategy: rolling restarts, zero traffic drop
The rolling restart number is the one we’re most proud of. Because each node maintains consistent-hash ownership of its account processes, we could drain a node, restart it with new code, and reconnect it — while traffic shifted to the remaining 11 nodes — without a single transaction loss. The BEAM’s hot code loading made this possible; the supervision tree design made it safe.
Lessons Worth Keeping
Model domains as processes, not records. If an entity has state that changes over time, it should probably be a process. The serialization you’d otherwise implement in SQL or Redis comes for free.
Supervision trees encode your failure model. Before you write a single use GenServer, draw the dependency graph and decide which restarts cascade and which don’t. The wrong strategy at 40k RPS is a bug. At 1M RPS, it’s an outage.
Consistent hashing beats round-robin at scale. Locality matters. Warm process state is free. Cache misses at a million requests per second are not.
Broadway for any I/O-bound pipeline. If you’re consuming from Kafka, SQS, or any external source and writing to a database, Broadway’s demand-driven model will save you from reinventing back-pressure badly.
Measure before scaling horizontally. We spent two weeks optimizing a single node before adding the second. Every bottleneck we found there — a blocking database call in an initialization path, a supervision tree misconfiguration, a Registry contention spike — would have multiplied across the cluster.
The BEAM was built for this. The question is whether your architecture is.
Tagged in
