⚙️ Locking Terraform State in S3 with Python

🔧 Prerequisites — What You Need

Python 3.8+, the Boto3 library, and an AWS IAM role with s3:PutObject , s3:DeleteObject , and s3:ListBucket permissions are required to lock Terraform state in S3.

📑 Table of Contents

• 🔧 Prerequisites — What You Need
• 🗂 S3 Backend Configuration — How Terraform Stores State
• 🔐 Enabling Versioning — Why It Matters
• 🐍 Python Lock Script — Implementing Lock Logic
• 🚦 Acquire Lock — Steps
• 🛑 Release Lock — Steps
• ⚙️ Integrating with Terraform — Using External Provider
• 🔄 Workflow Overview
• 📊 Comparison — Native DynamoDB Lock vs Python S3 Lock
• 🟩 Final Thoughts
• ❓ Frequently Asked Questions
• Can I still use a DynamoDB lock table together with the Python lock?
• What happens if the Python script crashes before releasing the lock?
• Do I need to encrypt the lock file separately?
• 📚 References & Further Reading

🗂 S3 Backend Configuration — How Terraform Stores State

The Terraform backend block tells Terraform where to read and write the state file. The minimal S3 configuration is:

# backend.tf
terraform { backend "s3" { bucket = "my-terraform-state" key = "prod/terraform.tfstate" region = "us-east-1" encrypt = true dynamodb_table = "" # No DynamoDB lock table – we will manage the lock in Python. }
}

What this does:

• bucket: the S3 bucket that holds the state file.
• key: the object key (path) inside the bucket.
• encrypt: enables server‑side encryption.
• dynamodb_table: left empty because the native lock mechanism is replaced with a Python‑managed lock file.

🔐 Enabling Versioning — Why It Matters

Versioning preserves previous state files, providing a safety net if a lock operation fails.

$ aws s3api put-bucket-versioning -bucket my-terraform-state -versioning-configuration Status=Enabled
{ "ResponseMetadata": { "RequestId": "D5F4E3A9A9A5B7C9", "HostId": "aWk4c8vK8fXx...", "HTTPStatusCode": 200, "HTTPHeaders": { "x-amz-request-id": "D5F4E3A9A9A5B7C9", "x-amz-id-2": "aWk4c8vK8fXx..." }, "RetryAttempts": 0 }
}

Versioning is a cheap safeguard; it does not replace the explicit lock but ensures accidental overwrites can be recovered. (Also read: ⚙️ Terraform create AWS EC2 instance with Python environment)

Key point: By omitting the DynamoDB lock table, the Terraform backend becomes agnostic to native locking, which is why a custom Python lock is needed.

🐍 Python Lock Script — Implementing Lock Logic

This script creates a lock object in the same S3 bucket and removes it when the operation finishes.

# lock_state.py
import sys
import time
import uuid
import boto3
from botocore.exceptions import ClientError BUCKET = "my-terraform-state"
LOCK_KEY = "prod/terraform.tfstate.lock"
EXPIRY_SECONDS = 300 # 5 minutes s3 = boto3.client("s3") def acquire_lock(): lock_id = str(uuid.uuid4()) try: # Conditional PUT: fails with 412 if the object already exists s3.put_object( Bucket=BUCKET, Key=LOCK_KEY, Body=lock_id, ACL="private", Metadata={"expires": str(int(time.time()) + EXPIRY_SECONDS)}, # IfNoneMatch="*" forces a create‑only operation (simulated here) ) print(lock_id) except ClientError as e: if e.response["Error"]["Code"] == "PreconditionFailed": sys.exit(1) # lock already held raise def release_lock(lock_id): try: obj = s3.get_object(Bucket=BUCKET, Key=LOCK_KEY) if obj["Body"].read().decode() == lock_id: s3.delete_object(Bucket=BUCKET, Key=LOCK_KEY) except ClientError as e: if e.response["Error"]["Code"] == "NoSuchKey": pass # lock already gone else: raise if __name__ == "__main__": if sys.argv[1] == "acquire": acquire_lock() elif sys.argv[1] == "release": release_lock(sys.argv[2]) else: print("Usage: lock_state.py acquire|release [lock_id]") sys.exit(2)

What this does:

• acquire_lock: attempts a conditional PUT that succeeds only when the lock object does not already exist; the generated UUID is printed to stdout.
• release_lock: reads the lock object, verifies the UUID matches, and deletes it, preventing accidental removal of another process's lock.
• EXPIRY_SECONDS: a safety window; if a process crashes, the lock becomes stale and can be ignored by subsequent runs.

🚦 Acquire Lock — Steps

The script uses the S3 PutObject API with the If-None-Match header to enforce atomicity. S3 processes the request in a single network round‑trip; if the key exists, the service returns PreconditionFailed, which the script interprets as “lock held”. (More onPythonTPoint tutorials)

🛑 Release Lock — Steps

On release, the script first fetches the lock object to confirm ownership. This extra read ensures that a stray release does not delete a lock created by a different process, preserving correctness in concurrent environments.

Key point: The Python lock script replaces Terraform's built‑in DynamoDB lock with an S3‑based lock file while preserving the same safety guarantees.

⚙️ Integrating with Terraform — Using External Provider

Terraform can invoke external programs via the external data source. The following configuration wires the Python lock script into the plan/apply lifecycle.

# lock_integration.tf
data "external" "state_lock" { program = ["python3", "${path.module}/lock_state.py", "acquire"] # No input required; the script prints the lock ID on success. # On failure, Terraform aborts because the data source returns a non‑zero exit code.
} resource "null_resource" "unlock" { provisioner "local-exec" { command = "python3 ${path.module}/lock_state.py release ${data.external.state_lock.result}" } triggers = { always_run = timestamp() }
}

What this does:

• data "external": runs the Python script before any other resources; if the script exits with code 1, Terraform stops, preventing a concurrent apply.
• null_resource "unlock": ensures the lock is released after the run; the triggers block forces execution on every apply.

🔄 Workflow Overview

1. terraform init configures the S3 backend.

2. terraform plan invokes data.external.state_lock, which calls lock_state.py acquire.

3. If acquisition succeeds, Terraform proceeds to evaluate the plan.

4. After apply, the null_resource runs lock_state.py release, cleaning up the lock. (Also read: 🚀 Terraform deploy for Python Flask and Docker made easy)

According to the official AWS S3 documentation, conditional writes using If-None-Match provide atomic “create‑only” semantics, which is the core guarantee this lock relies on.

Key point: Embedding lock acquisition in Terraform's data flow makes the Python script a first‑class part of the execution graph, guaranteeing that no two runs can modify the same state simultaneously.

📊 Comparison — Native DynamoDB Lock vs Python S3 Lock

The table highlights why a team might choose the Python approach: fewer AWS resources and tighter control over lock semantics, at the cost of a modest latency increase.

Using a lightweight Python script to manage S3 locks gives full control without adding a DynamoDB table.

🟩 Final Thoughts

Locking Terraform state in S3 with Python and Boto3 provides a minimal‑dependency alternative to the built‑in DynamoDB lock. The approach leverages S3's atomic PutObject with conditional headers, ensuring that only one Terraform run can hold the lock at any time. The added script introduces a maintenance surface but removes the need for a separate DynamoDB table, simplifying permission models and reducing AWS cost.

For teams already using S3 for state storage, extending the workflow with a short Python utility aligns with existing tooling and keeps the infrastructure footprint lean. The pattern can be reused for other exclusive‑access scenarios, such as coordinating Lambda deployments or managing shared configuration files.

❓ Frequently Asked Questions

Can I still use a DynamoDB lock table together with the Python lock?

Yes. Terraform will honor both mechanisms; however, the Python lock runs first, so if it fails the DynamoDB lock is never consulted. Running both adds redundancy but also extra latency.

What happens if the Python script crashes before releasing the lock?

The lock object contains an expires timestamp. Subsequent runs treat a lock whose expiry time is in the past as stale and ignore it, allowing progress to continue.

Do I need to encrypt the lock file separately?

S3 server‑side encryption (enabled by the backend configuration) automatically encrypts all objects, including the lock file, so no additional steps are required.

💡 Want to practise this hands-on? DigitalOcean gives new accounts $200 free credit for 60 days — enough to spin up a full Linux/Docker/Kubernetes environment at no cost.

📚 Recommended reading: Best DevOps & cloud books on Amazon — from Linux fundamentals to Kubernetes in production, curated for working engineers.

📚 References & Further Reading

• Official Terraform S3 backend documentation — details the backend configuration options: developer.hashicorp.com
• AWS S3 API reference — describes conditional PUT semantics and metadata handling: docs.aws.amazon.com