Dmitry Dvornitsky

Set the variables before you create the volume

Postgres reads its password once, when the data directory is empty. The order of operations is configuration.

1 hour lost

The problem

Production on Railway, about 15 hours after the first pull request. A Postgres service, a volume, the variables. Every migration (dbmate up) died with 28P01: password authentication failed.

What happened

  • On Railway, creating a volume deploys the service within the second.
  • The postgres image applies POSTGRES_PASSWORD only while it initialises an empty data directory.
  • We created the volume first and set the variables after.

The database was born with the password the service had at that second, which was staging’s. Changing the variable later touched nothing: the cluster already existed.

The fix

On an empty database: delete the volume and create it again, with the variables already set. The deploy log proves the cluster is fresh: PG_VERSION=missing.

Two more things learned on the way:

  • A deleted volume lingers as isPendingDeletion, and environment { volumeInstances } keeps returning it. project { volumes } shows which one is real.
  • Neither environment had a repository connected, so the dashboard’s Deploy button had nothing to build. Every release was railway up from a working copy. That became the first line of “deploy on merge”.

Takeaway

Any image that initialises on first start (Postgres, MySQL, MongoDB) reads its credentials once. Variables first, storage second. Write the order down: it is the step nobody re-reads.

All decisions