Terraform backends explained: Types, config, and limits
Terraform backends are the foundation of any collaborative infrastructure workflow, but the standard file-based model was designed for smaller, more isolated workflows. If you understand what a Terraform backend actually does, and where the default model stops scaling, you make better decisions long before state becomes a production problem.
Every terraform plan and terraform apply depends on terraform state, and the backend is what decides where that state data lives, who can read it, and how concurrent operations are coordinated.
For a single engineer using the default local backend on a laptop, this can feel almost invisible. For multiple team members, CI runners, and multiple workspaces spread across environments, it becomes one of the most important backend settings in the stack.
This article covers backend types, backend configuration, local and remote state, partial configuration, how teams handle multiple backends in practice, how to remove or migrate a backend safely, and where standard terraform backends hit limits that are easy to ignore early and expensive to ignore later.
What is a Terraform backend?
A Terraform backend defines where Terraform stores its persistent state data and how it coordinates access to that state during operations that can change it. Every Terraform configuration has exactly one backend. If you don't configure one, Terraform uses the default backend called local, which stores a terraform.tfstate file on disk in the working directory.
You declare the backend inside the top-level terraform {} block with a nested backend "<type>" block. The backend does not change what Terraform does to cloud resources. It changes how Terraform reads state, writes updated state, and, when supported, locks state so concurrent operations do not corrupt it.
A minimal s3 backend block looks like this.
If you leave backend configuration unspecified, the state file stays on the local filesystem. That's acceptable for a throwaway experiment, but it leaves the state invisible to teammates and CI systems, and any locking that exists is only local to the machine holding the file rather than shared across a team.
What Terraform backends actually do
In practice, a Terraform backend is responsible for state storage, state locking, and backend configuration that can be partially supplied at initialization time.
State storage
The backend is Terraform's system of record. At the start of a run, Terraform reads existing state from the backend to understand which infrastructure resources already exist and how they map to configuration. At the end of a successful run, Terraform writes updated state back to that backend.
Without consistent state storage, Terraform loses its memory and starts treating existing infrastructure as if it needs to be created again.
With the local backend, Terraform stores state as a local JSON file on disk. That's fine for learning, demos, and isolated personal work, but it breaks down quickly once you need shared access, access control, or durable remote state. Remote backends store state files in shared systems such as an s3 bucket, google cloud storage, or azure blob storage container.
State files should not live in version control. They change often, they create merge conflicts, and they can contain sensitive data in plain text. Terraform state can contain sensitive data that should be stored remotely, encrypted, with versioning enabled, and least-privilege access applied.
State locking
State locking exists to prevent two writers from modifying the same state at the same time. If a backend supports locking, Terraform acquires a lock before operations that can write state and refuses to continue if it cannot get that lock. This mechanism stands between ordinary concurrency and corrupted state.
Not all backends support locking.
- The local backend does support local system-level locking, but the file itself lives on one machine so there's no shared team coordination.
azurermsupports state locking with Azure Blob Storage native capabilities.gcssupports state locking.- The
s3 backendsupports locking when you setuse_lockfile = true, while older S3 setups often still use a DynamoDB table, i.e., the common pattern before native lockfiles were documented.
Observation
Standard Terraform backends lock the whole state file. Terraform may only be changing one security group, one IAM role, or one route table, but the locking primitive is attached to the entire state object. That reality sounds small until a team has enough concurrent operations for it to become a queue.
Partial configuration
Backend blocks are evaluated before Terraform can resolve input variables, locals, or data sources, which means you cannot use normal Terraform named values inside a backend block. Terraform has to know where state lives before it can do almost anything else.
You can handle that limitation without hardcoding everything using partial configuration. You leave some backend arguments blank in the configuration file, then provide the remaining values at terraform init time with -backend-config command line flags or a backend configuration file such as config.s3.tfbackend.
Use environment variables for credentials and other sensitive data because values supplied directly in backend config can be written in plain text into the .terraform directory and saved plan files.
When it comes to secrets and backend access, a common pattern is to keep stable values such as bucket name, key path, region, or storage account details in the backend block, while injecting credentials through environment variables or CI secrets during terraform initialization.
Local and remote Terraform state backends
Now that the backend's responsibilities are visible, the local versus remote choice becomes clearer. It's the dividing line between single-user state storage and shared infrastructure operations.
The local backend stores the terraform state file in the working directory on the local filesystem. That means no shared remote state and no collaboration model beyond whoever has a copy of the directory. It's appropriate for learning Terraform, for isolated personal projects, and for short-lived experiments. It's not appropriate for anything a second person, a second laptop, or a CI job will touch.
Remote state is the mechanism that lets multiple people access the same state and work together on the same infrastructure.
Remote Terraform state backends move state to a shared service and are the default choice for team environments. The remote backend stores state in systems such as Amazon S3, Google Cloud Storage, Azure Blob Storage, or HCP Terraform, where you can apply access control, durability guarantees, and backend settings that multiple actors can use consistently.
The practical failure mode of using a local backend with CI/CD is straightforward. A fresh pipeline runner starts without the existing local file, Terraform sees no prior state, and the run can attempt to create infrastructure that already exists. Remote state becomes a requirement the moment automation enters the picture, not an optional cleanup step.
Types of Terraform backends
With the local versus remote decision made, the types of backend become easier to evaluate. Most teams are not choosing between abstract categories. They are deciding which backend configuration best fits their cloud, their authentication model, and how much backend infrastructure they want to manage themselves.
Local
The default backend is local, and Terraform uses it whenever no other backend block is present. The local backend stores state on disk, and you can optionally set the path yourself. Setup is low friction, which is why the default local backend is useful for development, but the moment you need shared state, auditing, or reliable team workflows, it stops being a serious production option.
S3
The s3 backend is still the most common remote backend in AWS environments. It stores the state as an object in a specified bucket and key, and the current documentation supports native locking with use_lockfile = true. You should enable Bucket Versioning on the S3 bucket for recovery from accidental deletion or overwrite, while S3 buckets encrypt new object uploads by default.
The below requires Terraform 1.10 or later. For older versions, use a DynamoDB table for locking.
For older setups, you'll still see DynamoDB-based locking alongside an S3 bucket. Although it's older than the current lockfile model, it's still a viable option. In new backend configuration, the bigger question is less about syntax and more about operational hygiene. Use versioning, keep credentials out of the file terraform configuration, and prefer environment variables or workload identity over static access key values.
AzureRM
The AzureRM backend stores state as a blob in an Azure storage account and supports state locking with Azure Blob Storage native capabilities. There are OpenID Connect and Microsoft Entra ID authentication options directly in backend settings, which avoid baking long-lived credentials into configuration files.
If your estate already lives in Azure, this is usually the most direct remote backend because it keeps state storage, locking, authentication, and access control inside the same cloud boundary as the rest of the platform.
GCS
The GCS backend stores state as an object in a pre-existing Google Cloud Storage bucket and supports state locking. It's a good idea to enable Object Versioning on the bucket for state recovery, which matters for the same reason versioning matters in S3. State is a live operational artifact, and recovery paths should be cheap before you need them.
One reason many teams like it is that, compared with S3, the configuration is minimal. Authentication usually comes from Application Default Credentials, service account impersonation, or the GOOGLE_APPLICATION_CREDENTIALS environment variable rather than values hardcoded into the backend block.
HCP Terraform
HCP Terraform sits in a slightly different category. HashiCorp now recommends the cloud block over the older remote backend when connecting Terraform CLI workflows to HCP Terraform workspaces, and HCP Terraform acts as the remote backend for state while also supporting remote execution and broader workspace controls.
HCP Terraform is attractive for teams that want managed state, remote runs, policy controls, and less self-managed backend infrastructure.
Be aware that this is not only a storage choice. It can shift where plan and apply run, which is a bigger workflow decision than pointing state at object storage. Beyond these common backend types, Terraform still ships others such as consul, http, and pg, but most teams end up in one of the options above.
Using multiple backends in Terraform
Many teams discover that Terraform's backend model is narrower than they expected. One root module gets one backend, one set of backend settings, and one state file. If you declare more than one backend block, Terraform errors.
Workspaces are the lightest-weight answer. A single backend configuration can hold multiple named workspaces, each with its own state. This works well when dev, staging, and prod all use roughly the same Terraform configuration and differ mostly in variable values. It doesn't work as well once environments diverge materially or you need strong per-environment access control and change isolation.
State splitting across multiple root modules is the more common pattern at scale. Each service, environment, or platform layer gets its own backend block and its own state key, and dependencies between states are handled through terraform_remote_state data sources or provider-native lookups.
terraform_remote_state is a way to retrieve root module outputs from another state backend, which is why it shows up everywhere in layered Terraform codebases.
Pattern Recognition
This setup is still a workaround for the same backend model. You trade one giant state file for many smaller ones, then pay for it with more root modules, more orchestration, more backend migration work, and more configuration sprawl.
How to remove a Terraform backend
Backend migration is one of those operations that feels scary because it should. The state file is Terraform's memory, so changing where that memory lives isn't something to do without due consideration.
If you're switching from one remote backend to another, such as s3 to gcs, update the backend block in your Terraform configuration files and run terraform init -migrate-state. Terraform detects the backend change, prompts for confirmation, and copies the existing state to the new backend. After that, verify the state exists in the new location before cleaning up the old one.
If you're removing a remote backend and returning to local state, remove the backend block entirely and run terraform init -migrate-state. Terraform then copies the existing remote state into a local terraform.tfstate file. In both cases, terraform init is doing real migration work, not just initializing provider plugins or updating the dependency lock file.
There's also terraform init -reconfigure, which forces Terraform to forget the previous backend configuration and reinitialize against the current settings without migrating state automatically. terraform init -reconfigure is useful in narrow cases, but it's not the best solution when you want to preserve state.
The thing you shouldn't do is manually copy JSON around, manually push a state file, or delete backend objects by hand and hope the local view catches up. Backend changes should be handled through Terraform's migration flow.
Where standard Terraform backends fall short
This is the part that matters once infrastructure grows past a few operators and a few isolated runs. The limits of standard terraform backends aren't really about missing flags or imperfect backend configuration, they come from the shape of the storage model itself.
File-level locking is the core constraint. Standard backends such as S3, GCS, and AzureRM all protect state by locking the whole state file during writes. But the actual operation usually touches only a small subset of resources. One engineer updates an IAM policy, another updates a load balancer listener, a pipeline changes a security group rule, and all of them queue behind the same lock if those resources live in the same state. The scope of the lock is the entire file. The scope of the change is not.
That is why teams keep splitting state, not because everyone independently discovered a love of many tiny root modules, but because coarse locking turns concurrency into coordination overhead. Pipelines wait, engineers wait, and the answer becomes operational partitioning rather than fixing the primitive that is causing the queue.
Design Principle
There's a deeper mismatch underneath that behavior. Terraform itself builds and walks a dependency graph, graph walking that can process independent nodes in parallel. However, the state that standard backends persist is still treated as a file object. The execution engine reasons in graphs while the persistence layer reasons in blobs. That mismatch is why file-level locking keeps reappearing. The storage model does not preserve the natural shape of the system Terraform is actually operating on.
The visibility model is weak for the same reason. If your infrastructure is spread across many state files, answering simple questions about what exists means running terraform state list, terraform show, or custom JSON parsing against each state in turn. Terraform exposes inspection commands, but it doesn't turn all of your state data into a single queryable system out of the box.
Stategraph takes a different approach. Stategraph replaces the flat state file with a database-backed graph, stores state in PostgreSQL with full fidelity and transaction history, parses and indexes resources, exposes queryable infrastructure data, and provides resource-level conflict detection so non-overlapping changes can run in parallel.
It works with existing Terraform and OpenTofu configurations, modules, and providers, and any PostgreSQL 14 or newer instance is enough for thousands of resources.
Migration is simple. You're not rewriting Terraform code to get out of the flat-file bottleneck, just changing the backend model so the storage layer matches the dependency structure Terraform already knows exists.
Conclusion
Terraform backends decide where state lives, how state is locked, and how safely multiple people and systems can work against the same infrastructure. Once you understand the difference between a local backend, remote backends, partial configuration, state splitting, and backend migration, the standard model becomes easier to use well and easier to critique honestly.
Standard backends eventually inherit the limits of a flat file with coarse locking. If your team has already hit that wall, or you want to avoid building toward it, Stategraph is worth evaluating as a backend that stores state in the form Terraform already reasons about. Our Stategraph docs are the right place to start.
Terraform backend FAQs
Can Terraform have multiple backends?
Not in one root module. Terraform allows exactly one backend block per configuration, and the backend initially has one workspace with one state associated with that configuration. If you need multiple states, you usually need to use multiple workspaces with the same backend or split infrastructure across multiple root modules.
What is the difference between a local and remote Terraform backend?
A local backend stores state on the local machine, while a remote backend stores state in a shared service such as S3, GCS, Azure Blob Storage, or HCP Terraform. Remote backends provide collaboration, access control, and better durability for team workflows.
What is partial configuration in Terraform backends?
Partial configuration means leaving some backend arguments empty in the backend block and supplying the rest at terraform init time through -backend-config flags or a .tfbackend file.
Backend blocks cannot use input variables, and sensitive data should come from environment variables rather than committed configuration files.
What backend should I use with AWS?
For most AWS-native workflows, the default choice is the S3 backend. Teams that want a fully managed control plane rather than self-managed object storage may also choose HCP Terraform.
Why is my Terraform backend causing a lock error?
A lock error usually means another operation is already holding the state lock, the previous run crashed and left stale lock metadata behind, or the backend does not have the permissions needed to create or release the lock object. On standard backends, the lock scope is usually the whole state file, so unrelated changes can still block each other if they share the same state.