Troubleshooting Failed Postgres Migrations

Context

When upgrading a self-hosted LangSmith platform, Postgres schema migrations run automatically as a Kubernetes Job (Helm post-upgrade hook). In some cases this job may fail silently or get cleaned up before completing.

Symptoms:

- Errors in Postgres pod logs such as:

  ERROR: column o.invites_enabled does not exist (SQLSTATE 42703)

- ingest-queue pods failing to connect to Postgres

- No migration-related jobs visible in kubectl get jobs

Solution

Manually run the Postgres migration from a backend pod:

kubectl exec -it $(kubectl get pod -l app.kubernetes.io/component=langsmith-backend -o jsonpath='{.items[0].metadata.name}') -- alembic upgrade head

This executes the same Alembic migration that the Helm hook would have run, bringing the Postgres schema up to date with the deployed version.