How to deploy a Prometheus Alertmanager stack with Terraform
An alerting stack that lives in the same state as the infrastructure it watches deserves the same discipline as everything else in that state, and that changes decisions you'd otherwise skip: persistence, configuration ownership, and who's allowed to change a routing rule without a review.
Most teams encounter Alertmanager once Prometheus is already scraping metrics. A rule fires, and somebody realizes there's no sane way to get that alert routed to the right person without paging the entire on-call rotation. Alertmanager solves that specific problem.
However, when it comes to deployment, a production-grade Alertmanager isn't just a binary with a config file; it's a stateful service with its own storage requirements, its own high-availability story, and a configuration file that changes often enough to need real review, not a kubectl edit against a live pod.
This article treats that deployment as a Terraform concern from the first resource block, not an afterthought bolted onto a Helm chart after the fact.
You'll get an explanation of what Alertmanager is first, and precisely where its responsibilities start and stop relative to Prometheus, then a full build: provisioning Prometheus, configuring Alertmanager's routing and grouping, persisting its state, wiring in exporters, and pointing Grafana at the result.
We also share a webhook example, as that is the integration point most teams need. Best practices and an honest look at the alternatives close things out.
If you're already running Terraform's CLI daily and treating a plan before every apply as non-negotiable, you'll learn how Alertmanager's config file gets templated, what its persistent volume actually needs to survive, and how a webhook receiver gets wired up end to end rather than just described in the abstract.
What is Prometheus Alertmanager
Prometheus Alertmanager is the component that takes firing alerts sent by client applications, usually one or more Prometheus servers, and turns them into deduplicated, grouped, routed notifications through its own grouping and routing logic.
Prometheus only does half the job: it evaluates alerting rules against the metrics it has scraped and, when a rule's condition holds true for the configured duration, it starts sending that alert to Alertmanager over HTTP.
After that handoff, Alertmanager decides everything: which incoming alerts belong together, which ones should suppress others, and which notification channel each one uses.
A Prometheus server with no Alertmanager attached can still evaluate rules and expose firing alerts through its own web interface, but nothing will actually notify anyone. Conversely, an Alertmanager instance with nothing pointed at it just sits idle.
The two are separate binaries, separately scaled, communicating over a defined API, and treating them as a single unit obscures the decisions that actually have an impact once you're running in production: how many Alertmanager replicas you need, and how your Prometheus server's alerting rules are structured relative to Alertmanager's own routing rules.
How Alertmanager fits into the Prometheus architecture
Every Prometheus server that should generate notifications is configured with one or more Alertmanager targets under its own alerting block.
When a rule transitions to firing, Prometheus pushes that alert (labels, annotations, the works) to every configured Alertmanager target. It's then the Alertmanager side, not Prometheus, that decides whether that specific alert gets grouped with others, delayed, silenced, or dropped entirely by an inhibition rule.
The Alertmanager instance also stops being singular at this point, at least in any serious deployment.
Alertmanager supports a high-availability mode where multiple instances gossip over their own protocol to agree on what's already been notified, meaning that pointing several Prometheus servers (or several replicas of the same one) at a cluster of Alertmanager instances doesn't produce three copies of the same page.
Getting that clustering right, not just running a lone Alertmanager pod, is what helps your alertmanager implementation survive a node eviction.
Key features for production
If you already know the Prometheus ecosystem, these are the features worth naming, i.e., the ones that decide whether alerting rules translate into useful pages or into noise:
- Grouping. Alertmanager bundles related alerts that share a set of label names into a single notification instead of firing one message per alert, controlled by the
group_bylist on each route. - Routing. A tree of routes, starting from a root route, matches incoming alerts against label conditions and sends them down the corresponding child route (a sub-route can nest further, inheriting or overriding grouping and timing from its parent).
- Inhibition. An inhibition rule suppresses a whole class of alerts once a different, related alert is already firing, making it useful if a node going down fires alerts for everything running on it.
- Silencing. A time-boxed mute for alerts matching a label set, managed through the web interface or the API, distinct from an inhibition rule because a silence is manually created rather than inferred from another alert.
Deploying the stack with Terraform
Most teams running this in production will have Kubernetes as the deployment target, so the example below provisions Prometheus and Alertmanager through the prometheus-community/prometheus Helm chart, managed entirely through Terraform's helm_release resource rather than a manually applied values.yaml.
If your platform is a set of standalone VMs instead of Kubernetes, the shape changes (you'd provision compute, write the config file to disk through a provisioner or a templated user-data script, and manage the systemd unit directly) but the underlying decisions in each section below still apply.
Start with a namespace, as every other resource needs it to exist first:
Provisioning Prometheus for monitoring
The Prometheus server comes from the chart's server values block. The scrape configuration can be found here too, since it's what tells the Prometheus server which targets to actually pull metrics from:
Two things to acknowledge:
- The chart's own scrape config defaults are usually enough to get the Prometheus server discovering in-cluster targets via Kubernetes service discovery, so there's rarely a reason to override
server.extraScrapeConfigsunless you're targeting something outside the cluster. extraArgsis where command line flags that don't have a first-class values key end up,storage.tsdb.retention.timebeing the obvious one for controlling how long Prometheus itself holds onto raw samples.
Configuring Alertmanager for alerting
The alertmanager.config key above is where the actual Alertmanager configuration file exists, rendered through templatefile() rather than pasted inline, so a change to routing rules, notification policies, or a receiver's webhook URL is a one-line diff to a real file, not a multi-hundred-line YAML block buried in a Terraform values map.
Alertmanager itself reloads a new configuration at runtime without a restart, whether that reload is triggered by a SIGHUP or a POST to its own /-/reload endpoint. However, a Terraform apply is still the far more auditable way to get there.
That file follows Alertmanager's own configuration schema directly:
The root route's group_by covers alertname and namespace, which keeps related alerts firing from the same failure grouped into a single notification instead of paging once per pod.
The child route for severity: critical shortens group_wait and repeat_interval so critical alerts arrive faster and repeat more often than the default, while continue: true lets a critical alert still fall through to any routes defined after it rather than stopping at the first match.
The inhibit_rules entry suppresses every other alert sharing the same node label once a NodeDown alert for that node is already firing, which is what separates one useful page from twenty duplicate ones when a node actually goes down.
As this file is templated through Terraform rather than edited on a running pod, a routing change goes through plan and review exactly like a change to the underlying Terraform CI/CD pipeline that ships everything else.
If that review happens by running the plan in the pull request itself rather than after the fact, a bad routing change gets caught before it ever reaches a cluster, useful once more than one person is allowed to touch alerting policy.
Setting up data persistence
Alertmanager keeps two important things on disk:
- Its notification log, which alerts have already been sent, and when, so a restart doesn't re-notify everything.
- Its silences, which alerts are currently muted, and until when.
Observation
Skip persistence, and both reset to empty on every pod restart, which means every silence your team created gets undone and every dedup window resets at the worst possible moment: right as the pod that just crashed comes back up needing to page someone.
The chart's alertmanager.persistence block, set inside the same values map above, provisions a real PersistentVolumeClaim for the Alertmanager StatefulSet rather than falling back to an emptyDir:
Two gigabytes is generous for a single Alertmanager instance under low alert volume; the notification log and silence store are both small relative to the metrics data Prometheus itself accumulates.
The setting worth getting right isn't the size; it's the storage class: pick one your cluster's provisioner actually supports (gp3 on EKS, pd-balanced on GKE, or managed-csi on AKS), because an unresolvable storageClassName leaves the claim stuck pending forever, and a pending claim means Alertmanager never starts at all.
It's the same instinct that encourages you to audit your Terraform state rather than assume it matches reality: a stateful service is only as trustworthy as the storage backing it, and Alertmanager's silences and dedup history are state in exactly that sense, just smaller and easier to overlook than a .tfstate file.
Setting up exporters
Prometheus has nothing to alert on unless something is exposing metrics for it to scrape, and for infrastructure-level signals (node CPU, memory, disk pressure) the standard answer is node_exporter, which the same chart bundles as a subchart, toggled the same way as the server and alertmanager keys shown earlier:
Application-specific exporters are a separate concern, and Terraform manages them the same way it manages any other workload. A postgres_exporter sidecar deployment, for example, is just another resource in the same graph as the database it's watching:
Pointing Prometheus at it requires you to add a scrape target, either through the chart's server.extraScrapeConfigs or, more durably, a Kubernetes Service with the right annotations if the deployment relies on Prometheus's own service discovery rather than a static target list.
Integrating Grafana dashboards
Assuming Grafana is already running (its own deployment is outside the scope of an Alertmanager article), then pointing it at the Prometheus server you just provisioned, alongside any other data sources it already has configured, is a single grafana_data_source resource:
The url here uses the in-cluster service DNS name rather than any externally exposed address. As both Grafana and Prometheus exist inside the cluster, there's no reason to route dashboard queries out through an ingress and back in.
If you ship this data source through the same Terraform apply as the rest of the stack, a fresh environment gets a working Grafana connection on the first run, not a manual "add data source" click you may forget to document.
An example of a Prometheus Alertmanager webhook
A webhook receiver is the integration point most teams will look to, whether the destination is an internal incident tool, a custom Slack app, or something in between, because it's the one receiver type that doesn't assume you're using a specific vendor.
The Alertmanager side is the webhook_configs block already shown in the routing example above.
Alertmanager POSTs a JSON payload to the configured url for every notification, and the shape of that payload is fixed regardless of what's on the other end:
A minimal receiving service just needs to accept that POST and do something useful with the alerts array, since a single notification can (and usually does) bundle more than one alert once grouping kicks in:
Point the receiver's public URL at var.webhook_receiver_url in the Terraform example we provided in the previous section, and the loop closes: Prometheus fires a rule, Alertmanager groups and routes it, and the receiver decides what happens next.
We've left that last step (turning the raw payload into a page, a ticket, or a chat message) generic deliberately, since it's the one part of this pipeline that's genuinely specific to whatever incident management tools a given team already runs.
Implementation Detail
send_resolved: true on the receiver means Alertmanager will also POST when the alert stops firing, with status set to resolved. A receiver that only handles firing and silently drops everything else ends up looking like alerts never actually close, which instigates alert fatigue.
Prometheus Alertmanager best practices
Most of the operational pain with Alertmanager traces back to a handful of repeated mistakes, all of them easy to avoid once you know to look for them.
Group by more than just the alert name
A group_by of ['alertname'] alone means every instance of the same alert across every service lands in one notification, which sounds efficient until an incident spans twenty pods and the resulting single notification is too dense to act on.
Include a dimension that actually separates unrelated occurrences (namespace, job, or cluster, depending on your topology) so that grouping reduces noise without hiding scope.
Write inhibition rules for the failures that cascade
When a node goes down, it doesn't just trigger a NodeDown alert; it takes down every workload alert for pods that were scheduled there.
Without an inhibition rule matching on the shared node label, that's a dozen alerts for one root cause. The inverse mistake, an inhibition rule that's too broad, can suppress alerts that turn out to be unrelated once someone actually investigates, so scope the equal labels tightly to what genuinely correlates.
Don't route everything through a single receiver
It's tempting to point every route at one webhook and let the receiving service sort things out, but Alertmanager has a routing tree precisely so that certain alerts (critical and/or customer-facing) can get a different repeat interval and destination than everything else.
A root route with no child routes at all is a signal the routing tree hasn't been designed yet, just defaulted.
Run Alertmanager in high availability mode in production
Having one Alertmanager instance creates a single point of failure for every notification your Prometheus servers are trying to send, a risk that compounds for Enterprise teams running dozens of services against one alerting pipeline.
Clustering multiple instances (via the --cluster.peer flag pointing them at each other) means the gossip protocol handles deduplication across replicas, so a pod eviction doesn't also mean a missed page.
Treat the config file with the same review discipline as everything it relies on
Here is where a Terraform-native approach has its benefits.
A stack like this (namespace, secrets, a persistent volume, a stateful Alertmanager pod, and a Grafana data source pointed at all of it) is a genuinely interdependent resource graph, and Stategraph's dependency-aware execution model, which stores that dependency graph and plans only the affected subgraph on a change, is built for exactly this shape of challenge rather than the flat, unordered plans a lot of teams still run against stacks like this one.
Alternatives to a self-hosted Prometheus Alertmanager stack
There are other ways to run alerting. Below are a few alternatives.
Cloud provider options: GKE, AKS, and standalone VMs
On GKE, Autopilot removes node management entirely, which means one less thing for the Alertmanager StatefulSet's persistent volume to worry about at the infrastructure layer, at the cost of less control over exactly how pods get scheduled relative to the nodes they're monitoring.
AKS sits in a similar place: managed control plane, still your responsibility for node pool sizing and upgrade cadence, and generally the path of least resistance if the rest of your estate is already on Azure.
For a small number of services with a stable footprint, you can run Prometheus and Alertmanager as systemd units on a couple of standalone VMs, provisioned and configured through the same Terraform.
The shape becomes aws_instance or google_compute_instance resources, one of the project's precompiled binaries pulled down through a provisioner, and a templated config in place of Helm values. This option creates less operational surface area than standing up a Kubernetes cluster just to host it.
Alternative monitoring tools like Datadog and New Relic
Datadog and New Relic both trade the operational burden of running this stack yourself (upgrades, storage sizing, and HA clustering) for a subscription and less control over retention and alerting semantics.
It's worth considering if your team doesn't want to own another stateful service. Otherwise, someone has to be on call for Alertmanager itself, not just the systems it watches.
However, choose self-hosted Prometheus and Alertmanager if you plan to really scale, as a managed platform's per-host or per-metric pricing keeps compounding as a fleet grows, in a way a fixed self-hosted footprint doesn't.
Self-hosted Prometheus and Alertmanager also makes sense if you want full control over the alerting graph and routing rules that reflect your actual escalation policy rather than whatever a vendor's UI happens to support.
Conclusion
Alertmanager is straightforward to understand and easy to under-provision: run it stateless with no persistence, edit its configuration file by hand against a live pod, and skip high availability mode, and it'll work fine right up until the moment a restart wipes every silence or a lone replica goes down mid-incident.
A Terraform-native deployment closes those gaps.
By provisioning Prometheus, Alertmanager, its persistent storage, its exporters, and its Grafana integration through the same state you already manage everything else, you ensure that none of those gaps get forgotten: they're decisions made once and reviewed like everything else.
Try Stategraph free and see how graph-aware execution keeps the Terraform behind your alerting stack fast, reviewable, and safe.
Prometheus Alertmanager FAQs
Does Alertmanager need its own database?
No. Alertmanager keeps its silences and notification log either in memory or, if a persistent volume is configured, in a local file on disk; there's no external database to provision or manage.
How many Alertmanager replicas should run in production?
Three is the common baseline, enough for the gossip-based high availability mode to tolerate one instance being unavailable without losing the ability to deduplicate notifications across the remaining two. Running a single replica works for a test environment, but it reintroduces the single point of failure the clustering exists to remove.
Can Alertmanager send the same alert to multiple receivers?
Yes, through the continue: true field on a route. Setting it means a match on that route doesn't stop the alert from also being evaluated against subsequent routes, so a critical alert can page an on-call receiver and still land in a general-purpose webhook or chat channel through a separate route further down the tree.
What happens to silences when Alertmanager restarts without persistent storage?
They're gone. A silence lives in Alertmanager's local state, and without a persistent volume backing that state, a pod restart returns to an empty silence store, meaning every alert that was intentionally muted starts firing again, often at the worst possible moment relative to whatever caused the restart in the first place.