Database migrations in the pre-deploy script without downtime

The pre-deploy script is where database migrations run on JetDeploy: after the build, before the new version receives traffic. That order protects you from one failure, a new version starting against an old schema. It does not protect you from the opposite one: the old version keeps serving while the migration runs, and while the new version rolls out. A migration that the old code cannot live with causes errors during every deploy, however short.

How the pre-deploy script runs

• It runs in the freshly built image, with the App's environment variables, in a container of its own.
• It runs under /bin/sh -c with set -e in front, so a multi-line script stops at the first failing command. The image needs /bin/sh.
• It has 510 seconds, restarts included. The script and the start of the new version together have 600.
• A script that fails is not run again. If it fails or times out, the deploy fails and the previous version keeps serving.
• It is started again in a new container only if the platform interrupts it, for example during a node drain. Every attempt begins with a Pre-deploy attempt N line in the deploy logs.

A typical script per framework:

python manage.py migrate --noinput     # Django
alembic upgrade head                   # SQLAlchemy / FastAPI
bin/rails db:migrate                   # Rails
php artisan migrate --force            # Laravel
npx prisma migrate deploy              # Prisma

All of these skip migrations that are already applied, so running them twice is safe. Anything else you add to the script must be safe to run twice as well.

Which changes are safe in one deploy

The test is simple: can the version currently running work with the new schema?

Everything marked "no" can still be done without errors. It takes more than one deploy.

Expand, migrate, contract

Say a users.username column has to become users.handle. Instead of one rename, ship four deploys:

1. Deploy 1
   users

   usernameuntouched

   handlenullable

   Expand. Add handle as a nullable column. The new code writes both columns and reads handle, falling back to username when it is empty. The old code keeps working, since username is untouched.

2. Deploy 2
   users

   usernamestill written

   handleNOT NULL

   Migrate the data. Copy username into handle for existing rows, then make handle NOT NULL. The code reads handle only, and still writes both columns, since the version from step 1 is still serving during this deploy.

3. Deploy 3
   users

   usernamenullable, unused

   handleNOT NULL

   Stop using it. Make username nullable and remove it from the code, including the model, so the ORM stops selecting it. The column stays in the database, since the version from step 2 still reads and writes it during this deploy. In Django, that is a SeparateDatabaseAndState: AlterField on the database, RemoveField on the state.

4. Deploy 4
   users

   usernamedropped

   handleNOT NULL

   Contract. Drop the username column. By now no running version reads or writes it.

In Django, step 2's data copy is a RunPython migration:

from django.db import migrations, models


def copy_handles(apps, schema_editor):
    User = apps.get_model("accounts", "User")
    User.objects.filter(handle__isnull=True).update(handle=models.F("username"))


class Migration(migrations.Migration):
    dependencies = [("accounts", "0014_user_handle")]
    operations = [migrations.RunPython(copy_handles, migrations.RunPython.noop)]

The filter(handle__isnull=True) makes it safe to run twice.

Large tables and long migrations

On PostgreSQL, build indexes without locking writes:

from django.contrib.postgres.operations import AddIndexConcurrently
from django.db import migrations, models


class Migration(migrations.Migration):
    atomic = False
    dependencies = [("orders", "0031_previous")]
    operations = [
        AddIndexConcurrently("order", models.Index(fields=["created_at"], name="order_created_idx")),
    ]

When a step will take longer than the 510 seconds of the pre-deploy script, run it outside:

1. 1. Remove the step from the pre-deploy script and deploy. The new code must already work without it.
2. 2. Run the step once in the new version, from the shell on the App page or through the API:
   bashCopy

   curl -sS -N -X POST -H "Authorization: Bearer $JD_TOKEN" \
     -H 'Content-Type: application/json' \
     -d '{"command": ["python", "manage.py", "migrate", "--noinput"], "timeout": 600}' \
     https://jetdeploy.com/api/v1/apps/42/pods/7/exec

   A command through the API runs for at most 600 seconds. The shell on the App page runs longer, but it closes after 30 minutes without typing and after 4 hours in any case, and with the browser tab. Keep it active, or split the step into shorter parts.

3. 3. Put the step back in the pre-deploy script for the next deploys.

MariaDB: no transactional DDL

On MariaDB, keep each migration small, with one schema change per migration where you can, so an interrupted run leaves a state you can resume from.

A checklist before pushing a migration