Terraform best practices are bullshit
Split your state. Wrap it in Terragrunt. Wire the pieces back together with remote state. Keep root modules small, serialize applies, plan everything on every pull request. Every team that runs Terraform at scale converges on this list and teaches it to the next team as best practice. Most of it is not best practice. It is a decade of workarounds for a single defect, and calling them best practices is what kept the defect from being fixed.
The two kinds of best practice
- Practices that are true about the world, independent of any tool. Write tests. Review changes before they ship. Keep secrets out of version control. These hold no matter what you use. This is engineering.
- Practices that are true about your tool, specifically about where it is broken. They tell you how to work around a defect, and they only have value while the defect exists.
A workaround gets promoted to best practice the same way every time. Someone hits the wall. A single global lock on the whole state file means one plan blocks the entire team. They work around it, it works, and they write it up. Others adopt it. It gets a conference talk, then a chapter, then a certification question. The defect drops out of the story, and the workaround is left standing as the way you are supposed to work.
So the test for a real best practice is simple. A real best practice survives the tool being fixed. A workaround does not, because once the tool is fixed there is nothing left to work around.
Every practice traces to the same defect
Terraform stores state as a flat JSON file behind a single global lock. Its execution engine reasons about that state as a dependency graph; the storage layer does not. The lock covers the entire file even when a change touches one resource. That mismatch is the defect. We have covered why and how it works elsewhere. The point here is narrower. Every practice on the list works around this one lock.
- Split your state. The lock covers the whole file, so you make the file smaller. This spreads the contention across many state files. It does not remove it.
- Keep root modules small. The same lock, one level up. The root module is the unit that gets locked, so you keep it small to reduce how often two people collide on it.
- Adopt Terragrunt or Atmos. Splitting the state leaves many pieces to run in order. A wrapper orchestrates them from the outside, because the state model cannot.
- Wire it back with terraform_remote_state. Splitting severs dependencies between resources. You reconnect them by reading an entire remote state to get one output value.
- Serialize applies through a CI queue. The lock cannot distinguish two changes that conflict from two that are merely concurrent, so you serialize everything, including changes that never touch the same resource.
- One workspace per environment. The backend gives you one coarse unit of isolation, so you map environments onto workspaces even though workspaces were built for something else.
- Run plan on everything, every PR. The unit of work is the whole state, so the tool cannot cheaply tell you what a change affects. You replan everything on every pull request to compensate.
Every one traces to the same defect. Match the scope of the lock to the scope of the change and there is nothing for any of them to work around. They do not get better. They stop being necessary. These are not separate best practices. They are the same defect under different names.
Seniority is knowing the workarounds
The industry turned this into a career ladder. Terraform maturity, in practice, means fluency in the workarounds. The senior engineer is the one who knows how finely to split state, which directory layout keeps the orchestrator happy, how to structure remote state reads to contain the blast radius. That is real, hard-won knowledge. It is also knowledge of a problem that should not exist.
An ecosystem now depends on the problem staying unsolved. Gruntwork built Terragrunt because Terraform's one-state-per-root model broke at scale, and in 2016 that was the right call. Atmos, Spacelift, env0, books, consultancies. All of it competent, all of it solving a real problem, and all of it built on the assumption that the problem is permanent. We were part of this too. We built Terrateam to run Terraform through pull requests, and it worked, but we were automating around the same defect instead of removing it, which is what pushed us to rebuild the state layer underneath. When the workaround is the product, fixing the state layer is not an upgrade. It is a threat to the product.
The cost is organizational, not only technical. A file lock turns your pipeline into a queue. Teams batch changes, apply off-hours, and route around each other, then call the coordination discipline. It is not discipline. It is the overhead of a shared mutable file with one lock.
When knowing the workarounds is how you demonstrate seniority, the workarounds stop looking like a problem, and nobody has an incentive to remove the thing their expertise is measured against.
Terragrunt is the worst offender
Its entire job is to make state splitting bearable, which means its entire job is to keep the defect in place instead of fixing it. A decade of orchestration glue sold as best practice, papering over a problem the state layer should never have had, while every team that adopts it sinks deeper into the workaround and the rest of the ecosystem follows Gruntwork down with it. Terragrunt is dead, and it should be.
What holds up when you fix the state layer
Fix the defect. Model state as a graph and lock at the resource instead of the file, which is what we build at Stategraph. Then apply the test to the whole list and see what remains.
Version control, code review, testing, and least-privilege credentials are unaffected. Drift detection gets easier, because state you can query is state you can audit. We covered the DevOps practices that hold up separately. None of them depended on the state file being broken, so none of them change. That is what a real best practice looks like.
The workarounds are a different story. Splitting state to shrink a lock is pointless when the lock is already at the resource. Orchestrating the split pieces is pointless when the graph orders itself. Reading a whole remote state for one output is pointless when you can read the field directly. Serializing applies is pointless when non-overlapping changes never contended. Replanning everything is pointless when the tool already knows which resources a change affects.
None of these practices got better. They became unnecessary. That is the difference between fixing a problem and getting good at working around it.
This is not the fault of the engineers who adopted them. Splitting state was the rational choice, it worked, and it kept production running. The failure is in the label. We called a set of workarounds best practices, and the label stopped anyone from asking whether they still needed to exist.
Drop the label and look at what is left. Most of it is not architecture. It is a set of workarounds for a defect that has gone unfixed for a decade.