← Back to Blog RSS

Terraform best practices are bullshit

Terraform State Management Infrastructure Platform Engineering

Split your state. Wrap it in Terragrunt. Wire the pieces back together with remote state. Keep root modules small, serialize applies, plan everything on every pull request. Every team that runs Terraform at scale converges on this list and teaches it to the next team as best practice. Most of it is not best practice. It is a decade of workarounds for a single defect, and calling them best practices is what kept the defect from being fixed.

The two kinds of best practice

  1. Practices that are true about the world, independent of any tool. Write tests. Review changes before they ship. Keep secrets out of version control. These hold no matter what you use. This is engineering.
  2. Practices that are true about your tool, specifically about where it is broken. They tell you how to work around a defect, and they only have value while the defect exists.

A workaround gets promoted to best practice the same way every time. Someone hits the wall. A single global lock on the whole state file means one plan blocks the entire team. They work around it, it works, and they write it up. Others adopt it. It gets a conference talk, then a chapter, then a certification question. The defect drops out of the story, and the workaround is left standing as the way you are supposed to work.

So the test for a real best practice is simple. A real best practice survives the tool being fixed. A workaround does not, because once the tool is fixed there is nothing left to work around.

Every practice traces to the same defect

Terraform stores state as a flat JSON file behind a single global lock. Its execution engine reasons about that state as a dependency graph; the storage layer does not. The lock covers the entire file even when a change touches one resource. That mismatch is the defect. We have covered why and how it works elsewhere. The point here is narrower. Every practice on the list works around this one lock.

Every one traces to the same defect. Match the scope of the lock to the scope of the change and there is nothing for any of them to work around. They do not get better. They stop being necessary. These are not separate best practices. They are the same defect under different names.

Seniority is knowing the workarounds

The industry turned this into a career ladder. Terraform maturity, in practice, means fluency in the workarounds. The senior engineer is the one who knows how finely to split state, which directory layout keeps the orchestrator happy, how to structure remote state reads to contain the blast radius. That is real, hard-won knowledge. It is also knowledge of a problem that should not exist.

An ecosystem now depends on the problem staying unsolved. Gruntwork built Terragrunt because Terraform's one-state-per-root model broke at scale, and in 2016 that was the right call. Atmos, Spacelift, env0, books, consultancies. All of it competent, all of it solving a real problem, and all of it built on the assumption that the problem is permanent. We were part of this too. We built Terrateam to run Terraform through pull requests, and it worked, but we were automating around the same defect instead of removing it, which is what pushed us to rebuild the state layer underneath. When the workaround is the product, fixing the state layer is not an upgrade. It is a threat to the product.

The cost is organizational, not only technical. A file lock turns your pipeline into a queue. Teams batch changes, apply off-hours, and route around each other, then call the coordination discipline. It is not discipline. It is the overhead of a shared mutable file with one lock.

When knowing the workarounds is how you demonstrate seniority, the workarounds stop looking like a problem, and nobody has an incentive to remove the thing their expertise is measured against.

Terragrunt is the worst offender

Its entire job is to make state splitting bearable, which means its entire job is to keep the defect in place instead of fixing it. A decade of orchestration glue sold as best practice, papering over a problem the state layer should never have had, while every team that adopts it sinks deeper into the workaround and the rest of the ecosystem follows Gruntwork down with it. Terragrunt is dead, and it should be.

What holds up when you fix the state layer

Fix the defect. Model state as a graph and lock at the resource instead of the file, which is what we build at Stategraph. Then apply the test to the whole list and see what remains.

Version control, code review, testing, and least-privilege credentials are unaffected. Drift detection gets easier, because state you can query is state you can audit. We covered the DevOps practices that hold up separately. None of them depended on the state file being broken, so none of them change. That is what a real best practice looks like.

The workarounds are a different story. Splitting state to shrink a lock is pointless when the lock is already at the resource. Orchestrating the split pieces is pointless when the graph orders itself. Reading a whole remote state for one output is pointless when you can read the field directly. Serializing applies is pointless when non-overlapping changes never contended. Replanning everything is pointless when the tool already knows which resources a change affects.

None of these practices got better. They became unnecessary. That is the difference between fixing a problem and getting good at working around it.

This is not the fault of the engineers who adopted them. Splitting state was the rational choice, it worked, and it kept production running. The failure is in the label. We called a set of workarounds best practices, and the label stopped anyone from asking whether they still needed to exist.

Drop the label and look at what is left. Most of it is not architecture. It is a set of workarounds for a defect that has gone unfixed for a decade.