We Wiped Our Production Database in 90 Minutes: A Prisma Postgres Post-Mortem

At 3:51 PM on a Friday afternoon, we deployed a new analytics feature to Shilpiworks, our AI-powered sticker shop. By 5:00 PM, every product in the database was gone. 590 stickers, all agent run history, all orders — wiped. The site showed an empty catalog.

This is the full post-mortem: what happened, why it happened, how we recovered, and the rules we now follow to prevent it from ever happening again.

The Setup: Prisma Postgres Is Not What You Think

Shilpiworks uses Prisma Postgres — a managed database service from Prisma that gives you a PostgreSQL-compatible connection string at db.prisma.io. From your application's perspective, it looks and feels like a normal Postgres database. Your Prisma schema points to it, your ORM queries work, your data is there.

But Prisma Postgres is actually a proxy layer. When you run ORM queries like prisma.product.findMany(), the proxy intercepts them and routes them to the real underlying database. This is what enables features like connection pooling and Prisma Accelerate.

The critical detail we missed: raw SQL bypasses the proxy's routing logic and executes directly against the managed database. ORM queries and raw SQL do not hit the same target.

What Went Wrong

We wanted to add a simple analytics dashboard — track page views, search queries, and product interactions. The feature needed a new analytics_events table. Here's the sequence of events:

1. 13:51 PM — Deployed analytics feature. Used $executeRawUnsafe(CREATE TABLE analytics_events...) to create the table through the Prisma client.
2. 23:53 PM — Realized the table was created, but in the proxy's own database layer, not alongside the Product table. ORM queries still worked, but raw SQL was hitting a different target.
3. 34:01 PM — Ran prisma migrate to try to fix the schema. This created a baseline migration with CREATE TABLE "Product" — which conflicted with the existing data.
4. 44:15 PM — Products API started returning 500 errors. The Prisma client was confused about which database layer contained the tables.
5. 54:30 PM — In a debugging attempt, ran DROP TABLE commands to "clean up" the tables we'd accidentally created. This wiped the _prisma_migrations table and analytics_events.
6. 64:45 PM — Discovered the Product table was also gone. The entire database was empty. Zero tables.

The Moment of Realization

We ran a simple diagnostic and got the worst possible output:

const tables = await client.query(
  "SELECT tablename FROM pg_tables WHERE schemaname='public'"
);
console.log('tables:', tables.rows);
// tables: []

An empty array. No Product table. No Order table. No AgentRun table. No Tag table. 590 products, months of agent run history, all customer orders — gone.

The Recovery

The one thing that saved us: Prisma Postgres has automatic daily backups. We found them in the Prisma Console (console.prisma.io) under the Backups tab. Four daily snapshots going back to February 25th, each about 10 MB.

But the restore process had two surprises:

1. 1Restore creates a new database. It doesn't overwrite the existing one. You get a brand new database with a new connection string. This is actually smart — it prevents a bad restore from compounding the disaster.
2. 2Vercel env vars are locked. Prisma's Vercel integration creates managed environment variables (DATABASE_URL, POSTGRES_URL) that you cannot edit or remove from the Vercel dashboard. They're tied to the old database.

The workaround was pragmatic. We changed the Prisma schema to read from a new environment variable:

// Before (locked to old, empty database)
datasource db {
  provider = "postgresql"
  url      = env("DATABASE_URL")
}

// After (points to restored database)
datasource db {
  provider = "postgresql"
  url      = env("RESTORED_DATABASE_URL")
}

We added RESTORED_DATABASE_URL as a new environment variable in the Vercel dashboard (no locking issues with custom vars), pointed it to the restored database's connection string, and deployed. Within minutes, the site was back:

curl -s "https://www.shilpiworks.com/api/products" | node -e "
  let d='';
  process.stdin.on('data',c=>d+=c);
  process.stdin.on('end',()=>console.log('Products:',JSON.parse(d).length))
"
# Products: 585

585 products restored. We lost about 5 products that were created between the last backup and the incident — a small price for a full recovery.

The Code Reset

In addition to the database restore, we did a hard git reset to the last known working commit — the one right before the analytics feature was added. This removed all 21 commits from the failed analytics implementation:

git reset --hard ae673b0  # "feat: add welcome article to /learn"
git push --force-with-lease origin main

The analytics feature can be re-implemented later — but this time, without any raw SQL touching the Prisma Postgres proxy.

Prevention Rules

We've added these rules to our project's deployment lessons document. They're non-negotiable:

• Never use $executeRawUnsafe() or $executeRaw() with Prisma Postgres. Raw SQL executes against the proxy's managed layer, not the real database. The results are unpredictable.
• Never run prisma migrate against a Prisma Postgres database. Migrations use raw SQL internally and will create conflicting state.
• Never run DROP TABLE on production without verifying backups exist first. This sounds obvious. It wasn't obvious at 4:30 PM on a Friday.
• Use only Prisma ORM methods. findMany, create, update, delete — these are properly proxied and safe.
• For raw SQL needs, use a separate direct connection. If you need to create tables or run DDL, use the pg npm package with a direct database URL, not through the Prisma client.

The Broader Lesson

Managed database services abstract away complexity — and that's usually a good thing. But abstraction creates a gap between your mental model and reality. We thought db.prisma.io was "our database." It's actually a proxy that routes ORM queries one way and raw SQL another way. We didn't understand the abstraction boundary, and we paid for it.

The site is fully recovered. The sticker agents are running again. And we have a new rule painted on the wall: No raw SQL through Prisma Postgres. Ever. Browse the restored shop at shilpiworks.com →