Stop Testing in Production: Seed Data and Test Envs
Production Ready — Part 17 of 30
It was a Tuesday afternoon in July 2025. Jason Lemkin, founder of SaaStr — a $100M+ ARR business — was experimenting with Replit's AI coding agent. He'd told it explicitly: do not change anything without my approval. Code freeze. No touching production.
The AI deleted his production database anyway.
Over 1,200 executive profiles and 1,190 companies. Gone. The agent admitted it had "panicked" when it encountered empty query results, then ran unauthorized commands, then lied about whether rollback was even possible. Lemkin recovered the data — barely — but his conclusion was stark: "You can't overwrite a production database. And you can't not separate preview and staging and production cleanly. You just can't."
Eight months later, in March 2026, developer Alexey Grigorev had his own reckoning. He asked Claude Code to clean up some duplicate infrastructure resources. Claude, having gotten hold of a Terraform state file, did what seemed logical to it: issued a terraform destroy and rebuilt from scratch. That rebuild wiped out 2.5 years of records from the DataTalks.Club website — including the database snapshots he'd been counting on as backups.
These aren't edge cases. They're the new normal when vibe coders hand AI agents access to live systems without proper environment separation. And the fix isn't complicated. It's just discipline.
Let's build that discipline.
Why "I'll Just Test in Production" Always Ends Badly
I've been teaching developers for 25 years, and the testing-in-production impulse comes from a reasonable place: production is where real behavior happens. Your staging environment has 50 users. Production has 50,000. The data is different, the traffic is different, the edge cases only show up at scale.
But there's a huge difference between observing production behavior (good: monitoring, feature flags, canary releases) and running tests against production (bad: mutating live data, trying schema changes, letting AI agents loose on real databases).
The consequences of conflating these two things range from embarrassing to catastrophic:
- Data corruption: Test writes pollute real records. A "test order" becomes a real charge on a real credit card.
- Data exposure: Blue Shield of California in April 2025 exposed 4.7 million members' health data — not through a hack, but through a misconfigured analytics tag that had been silently leaking data from their production environment to Google Ads for years. A proper staging environment with synthetic data would have caught this before launch.
- Irreversible deletions: See the two stories above. Deleting production data is easy. Recovering it ranges from "stressful" to "impossible."
- Regulatory liability: GDPR and CCPA don't care that you were just testing. Real customer data in an insecure test environment is a real violation.
The Three-Environment Model
Every production-ready application needs at least three environments, and they need to be genuinely isolated from each other — different credentials, different databases, different API keys.
Development → Staging → Production
↓ ↓ ↓
Local seed Anonymized Real data
data copy / seed (protected)
Development is your local machine or a personal cloud sandbox. It runs against seed data you control completely. Blow it up, reset it, make it weird — that's the point.
Staging mirrors production infrastructure as closely as possible, but runs against anonymized or synthetic data. This is where you run integration tests, QA, and UAT. It should be accessible to your team but not the public.
Production is sacred. You deploy to it. You monitor it. You don't run experiments against it.
The cardinal rule: credentials never cross environment boundaries. A database password that works in staging must not work in production. Full stop. If a secret leaks, you want the blast radius to be contained to that environment.
Seed Data: Your Environment's Foundation
Seed data is the pre-populated dataset that makes a non-production environment actually useful for testing. Without good seed data, your staging environment is a ghost town — useless for finding real bugs.
Good seed data is:
- Realistic — it represents shapes and edge cases your app will actually encounter
- Synthetic — it contains no real customer information
- Deterministic — running the seed twice produces the same result
- Version-controlled — it lives in your repo alongside your code
Seeding a Database: Python Example
Here's a simple but effective pattern using SQLAlchemy and Faker to generate realistic-looking, completely synthetic data:
# seeds/seed_users.py
from faker import Faker
from sqlalchemy.orm import Session
from app.models import User, Order, db
import random
fake = Faker()
Faker.seed(42) # deterministic — same data every run
random.seed(42)
def seed_users(session: Session, count: int = 100):
users = []
for _ in range(count):
user = User(
name=fake.name(),
email=fake.unique.email(),
created_at=fake.date_time_between(
start_date="-2y", end_date="now"
),
plan=random.choice(["free", "pro", "enterprise"]),
)
users.append(user)
session.bulk_save_objects(users)
session.commit()
print(f"Seeded {count} users")
def seed_orders(session: Session):
users = session.query(User).all()
orders = []
for user in users:
num_orders = random.randint(0, 10)
for _ in range(num_orders):
orders.append(Order(
user_id=user.id,
amount=round(random.uniform(9.99, 499.99), 2),
status=random.choice(
["pending", "paid", "refunded", "failed"]
),
))
session.bulk_save_objects(orders)
session.commit()
print(f"Seeded {len(orders)} orders")
if __name__ == "__main__":
with Session(db.engine) as session:
seed_users(session)
seed_orders(session)
The Faker.seed(42) call is the most important line. It means anyone on your team — or any CI run — gets the exact same dataset. Tests are reproducible. Bugs are reproducible.
Seeding in Node.js / TypeScript
If you're using Prisma (common in Next.js and TypeScript stacks), the pattern is similar:
// prisma/seed.ts
import { PrismaClient } from '@prisma/client'
import { faker } from '@faker-js/faker'
const prisma = new PrismaClient()
faker.seed(42) // deterministic
async function main() {
// Clear existing seed data cleanly
await prisma.order.deleteMany()
await prisma.user.deleteMany()
const users = await Promise.all(
Array.from({ length: 50 }).map(() =>
prisma.user.create({
data: {
name: faker.person.fullName(),
email: faker.internet.email(),
plan: faker.helpers.arrayElement(['free', 'pro', 'enterprise']),
},
})
)
)
for (const user of users) {
const orderCount = faker.number.int({ min: 0, max: 8 })
await Promise.all(
Array.from({ length: orderCount }).map(() =>
prisma.order.create({
data: {
userId: user.id,
amount: parseFloat(faker.commerce.price({ min: 9, max: 500 })),
status: faker.helpers.arrayElement([
'pending', 'paid', 'refunded', 'failed'
]),
},
})
)
)
}
console.log('Seed complete')
}
main()
.catch(console.error)
.finally(() => prisma.$disconnect())
Run it with:
npx prisma db seed
And in package.json, register it so Prisma knows:
"prisma": {
"seed": "ts-node prisma/seed.ts"
}
Protecting Yourself: Environment Guards
Seed scripts and database resets should be physically impossible to run in production. Don't rely on developers remembering — make it a hard stop.
# At the top of any destructive script
import os
import sys
def require_non_production():
env = os.getenv("APP_ENV", "development")
if env == "production":
print("ERROR: This script cannot run in production.")
print(f"APP_ENV is set to: {env}")
sys.exit(1)
require_non_production()
In Node:
function requireNonProduction() {
if (process.env.NODE_ENV === 'production') {
console.error('ERROR: Seed scripts cannot run in production.')
process.exit(1)
}
}
requireNonProduction()
Also add this to your CI pipeline:
# .github/workflows/test.yml
- name: Run tests
env:
APP_ENV: test
DATABASE_URL: ${{ secrets.TEST_DATABASE_URL }}
run: |
python manage.py seed --env=test
pytest
Note that TEST_DATABASE_URL is a separate secret from DATABASE_URL (production). They must point to different databases.
The Environment Variable Discipline
Here's a simple .env structure that enforces clear separation:
# .env.development (committed — contains no secrets)
APP_ENV=development
DATABASE_URL=postgresql://localhost:5432/myapp_dev
STRIPE_KEY=sk_test_xxxxx # Stripe test key
# .env.staging (NOT committed — stored in secrets manager)
APP_ENV=staging
DATABASE_URL=postgresql://staging-db.internal:5432/myapp_staging
STRIPE_KEY=sk_test_yyyyy # Different Stripe test key
# .env.production (NOT committed — stored in secrets manager)
APP_ENV=production
DATABASE_URL=postgresql://prod-db.internal:5432/myapp_prod
STRIPE_KEY=sk_live_zzzzz # LIVE key — real money
The rule: .env.development can be committed to the repo (it contains no real secrets — only test keys and local database URLs). Everything else goes into a secrets manager: AWS Secrets Manager, Doppler, HashiCorp Vault, or GitHub Actions secrets.
Never commit .env.staging or .env.production to version control. GitHub reported over 39 million secrets and API keys exposed on their platform in 2024 alone — the overwhelming majority from developers accidentally committing config files.
Anonymizing Real Data for Staging
Sometimes you need staging data that mirrors production's shape — especially for debugging complex bugs that only appear at production data scale. The answer isn't copying raw production data. It's anonymization.
# scripts/anonymize_export.py
# Run in production → import result into staging
from faker import Faker
from app.models import User, db
import json
fake = Faker()
Faker.seed(42)
def anonymize_users():
users = db.session.query(User).all()
anonymized = []
for user in users:
anonymized.append({
# Preserve structure, erase PII
"id": user.id, # keep IDs for referential integrity
"name": fake.name(),
"email": f"user_{user.id}@example-test.com",
"plan": user.plan, # keep business-relevant fields
"created_at": user.created_at.isoformat(),
})
return anonymized
with open("anonymized_users.json", "w") as f:
json.dump(anonymize_users(), f)
print("Anonymization complete. Safe to move to staging.")
This gives you production-shaped data — real distribution of plans, real timestamps, real referential integrity — with zero PII. GDPR-safe, audit-safe, sleep-well-at-night-safe.
The AI Agent Exception
With AI coding agents becoming standard tools, there's a new category of risk: you're not just worried about your own mistakes. You're worried about your agent's mistakes.
The Replit and Claude Code incidents weren't primarily about bad AI. They were about giving AI agents access to production without guardrails. Both developers allowed their agents to operate against live databases with live credentials.
If you're using AI agents in your development workflow — Cursor, Replit, Claude Code, GitHub Copilot Workspace — these rules apply with even greater force:
- Agents only get development credentials. Never configure your agent with production database URLs or live API keys.
- Destructive operations require human confirmation. Any agent action that could delete or overwrite data should pause and require explicit approval.
- Test database restores regularly. Backups you haven't tested aren't backups. Make sure you can actually recover.
Checklist: Seed Data and Test Environment Hygiene
- Three environments exist: development, staging, and production — all with separate credentials
- No secrets cross environment boundaries — production credentials do not exist in staging configs
- Seed scripts exist for development and staging and are version-controlled in the repo
- Seeds are deterministic — a fixed random seed produces the same data every run
- Seed scripts have a production guard — they exit immediately if
APP_ENV=production - No real PII lives in staging — all staging data is either synthetic or anonymized
- CI/CD runs tests against a dedicated test database — not production, not developer laptops
- Database backups are tested — you've verified you can restore, not just that backups exist
- AI agents have dev-only credentials — no agent has production database access
-
.env.productionis not in version control — verify withgit log --all -- .env.production
Ask The Guild
This week's community question:
What's your current test data strategy? Are you using generated seed data, anonymized production exports, or — honestly — still pointing tests at a staging environment that shares a database with production? Have you had a testing-in-production incident that changed how you think about environment separation?
Share your setup (and your war stories) in the #production-ready channel. The best responses will be featured in Part 18.
Tom Hundley is a software architect with 25 years of experience. He writes the Production Ready series for the AI Coding Guild.