Skip to content

Commit 655689b

Browse files
r1tsuuChristian Schurr
r1tsuu
authored and
Christian Schurr
committed
fix(db-postgres): stabilize read replicas support (payloadcms#16083)
- Fixes payloadcms#13011 and payloadcms#13588 - Adds `postgres-replicas` configuration to our `docker-compose.yml`. - Enables CI tests with `postgres-read-replicas` database adapter option. - Adds dedicated test suite `test/database/pg-replica/int.spec.ts` for the `readReplicas` property When readReplicas is configured, reads that happen as part of writes (e.g. reading back the row after an INSERT) were incorrectly routed to the replica instead of the primary, causing NotFound errors due to replication lag. - `getPrimaryDb` - used in `upsertRow`, updateOne, `updateMany` to always use the primary DB for internal reads in write operations. - `readReplicasAfterWriteInterval` - time-based read-after-write consistency. Before, if you did for example `payload.create` and immediately `payload.find` after - the new document would not show due to replication lag. Defaults to `2000` (ms) but can be configured. --- - To see the specific tasks where the Asana app for GitHub is being used, see below: - https://app.asana.com/0/0/1213830896242727
1 parent 751cfa3 commit 655689b

39 files changed

Lines changed: 576 additions & 211 deletions

‎.env.example‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,3 +16,7 @@ OPENAI_KEY=
1616
# Do not uncomment these if you're using our Docker scripts to run your database.
1717
# MONGODB_URL=mongodb://127.0.0.1/payloadtests
1818
# POSTGRES_URL=postgres://127.0.0.1:5432/payloadtests
19+
20+
# Replica connection URL — used when PAYLOAD_DATABASE=postgres-read-replica or postgres-read-replicas
21+
# When using Docker, the replica runs on port 5434 (started automatically with the postgres profile)
22+
# POSTGRES_REPLICA_URL=postgres://payload:payload@127.0.0.1:5434/payload

‎.github/actions/start-database/action.yml‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ description: Starts the required database for tests
33

44
inputs:
55
database:
6-
description: 'Database type: mongodb, mongodb-atlas, cosmosdb, documentdb, firestore, postgres, postgres-custom-schema, postgres-uuid, postgres-read-replica, vercel-postgres-read-replica, sqlite, sqlite-uuid, supabase, d1'
6+
description: 'Database type: mongodb, mongodb-atlas, cosmosdb, documentdb, firestore, postgres, postgres-custom-schema, postgres-uuid, postgres-read-replica, postgres-read-replicas, vercel-postgres-read-replica, sqlite, sqlite-uuid, supabase, d1'
77
required: true
88

99
outputs:
@@ -44,6 +44,11 @@ runs:
4444
docker compose -f test/docker-compose.yml --profile postgres up -d --wait
4545
echo "url=postgresql://payload:payload@localhost:5433/payload" >> $GITHUB_OUTPUT
4646
47+
- name: Export PostgreSQL replica URL
48+
if: contains(fromJSON('["postgres-read-replica", "postgres-read-replicas", "vercel-postgres-read-replica"]'), inputs.database)
49+
shell: bash
50+
run: echo "POSTGRES_REPLICA_URL=postgresql://payload:payload@localhost:5434/payload" >> $GITHUB_ENV
51+
4752
- name: Configure PostgreSQL custom schema
4853
if: inputs.database == 'postgres-custom-schema'
4954
shell: bash

‎.github/workflows/int.config.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,7 @@ createIntConfig({
1010
'postgres',
1111
'postgres-custom-schema',
1212
'postgres-uuid',
13+
'postgres-read-replica',
1314
'supabase',
1415
'sqlite',
1516
'sqlite-uuid',

‎docs/database/postgres.mdx‎

Lines changed: 19 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -64,24 +64,25 @@ export default buildConfig({
6464

6565
## Options
6666

67-
| Option | Description |
68-
| --------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
69-
| `pool` \* | [Pool connection options](https://orm.drizzle.team/docs/quick-postgresql/node-postgres) that will be passed to Drizzle and `node-postgres` or to `@vercel/postgres` |
70-
| `push` | Disable Drizzle's [`db push`](https://orm.drizzle.team/kit-docs/overview#prototyping-with-db-push) in development mode. By default, `push` is enabled for development mode only. |
71-
| `migrationDir` | Customize the directory that migrations are stored. |
72-
| `schemaName` (experimental) | A string for the postgres schema to use, defaults to 'public'. |
73-
| `idType` | A string of 'serial', or 'uuid' that is used for the data type given to id columns. |
74-
| `transactionOptions` | A PgTransactionConfig object for transactions, or set to `false` to disable using transactions. [More details](https://orm.drizzle.team/docs/transactions) |
75-
| `disableCreateDatabase` | Pass `true` to disable auto database creation if it doesn't exist. Defaults to `false`. |
76-
| `localesSuffix` | A string appended to the end of table names for storing localized fields. Default is '\_locales'. |
77-
| `relationshipsSuffix` | A string appended to the end of table names for storing relationships. Default is '\_rels'. |
78-
| `versionsSuffix` | A string appended to the end of table names for storing versions. Defaults to '\_v'. |
79-
| `beforeSchemaInit` | Drizzle schema hook. Runs before the schema is built. [More Details](#beforeschemainit) |
80-
| `afterSchemaInit` | Drizzle schema hook. Runs after the schema is built. [More Details](#afterschemainit) |
81-
| `generateSchemaOutputFile` | Override generated schema from `payload generate:db-schema` file path. Defaults to `{CWD}/src/payload-generated.schema.ts` |
82-
| `allowIDOnCreate` | Set to `true` to use the `id` passed in data on the create API operations without using a custom ID field. |
83-
| `readReplicas` | An array of DB read replicas connection strings, can be used to offload read-heavy traffic. |
84-
| `blocksAsJSON` | Store blocks as a JSON column instead of using the relational structure which can improve performance with a large amount of blocks |
67+
| Option | Description |
68+
| -------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
69+
| `pool` \* | [Pool connection options](https://orm.drizzle.team/docs/quick-postgresql/node-postgres) that will be passed to Drizzle and `node-postgres` or to `@vercel/postgres` |
70+
| `push` | Disable Drizzle's [`db push`](https://orm.drizzle.team/kit-docs/overview#prototyping-with-db-push) in development mode. By default, `push` is enabled for development mode only. |
71+
| `migrationDir` | Customize the directory that migrations are stored. |
72+
| `schemaName` (experimental) | A string for the postgres schema to use, defaults to 'public'. |
73+
| `idType` | A string of 'serial', or 'uuid' that is used for the data type given to id columns. |
74+
| `transactionOptions` | A PgTransactionConfig object for transactions, or set to `false` to disable using transactions. [More details](https://orm.drizzle.team/docs/transactions) |
75+
| `disableCreateDatabase` | Pass `true` to disable auto database creation if it doesn't exist. Defaults to `false`. |
76+
| `localesSuffix` | A string appended to the end of table names for storing localized fields. Default is '\_locales'. |
77+
| `relationshipsSuffix` | A string appended to the end of table names for storing relationships. Default is '\_rels'. |
78+
| `versionsSuffix` | A string appended to the end of table names for storing versions. Defaults to '\_v'. |
79+
| `beforeSchemaInit` | Drizzle schema hook. Runs before the schema is built. [More Details](#beforeschemainit) |
80+
| `afterSchemaInit` | Drizzle schema hook. Runs after the schema is built. [More Details](#afterschemainit) |
81+
| `generateSchemaOutputFile` | Override generated schema from `payload generate:db-schema` file path. Defaults to `{CWD}/src/payload-generated.schema.ts` |
82+
| `allowIDOnCreate` | Set to `true` to use the `id` passed in data on the create API operations without using a custom ID field. |
83+
| `readReplicas` | An array of DB read replicas connection strings, can be used to offload read-heavy traffic. |
84+
| `readReplicasAfterWriteInterval` | How long (ms) after a write to keep routing reads to the primary instead of a replica. Prevents stale reads caused by replication lag. Only relevant when `readReplicas` is set. Default `2000`. Set to `0` to disable. |
85+
| `blocksAsJSON` | Store blocks as a JSON column instead of using the relational structure which can improve performance with a large amount of blocks |
8586

8687
## Access to Drizzle
8788

‎packages/db-postgres/src/connect.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -62,6 +62,7 @@ export const connect: Connect = async function connect(
6262
this.drizzle = drizzle({ client: this.pool, logger, schema: this.schema })
6363

6464
if (this.readReplicaOptions) {
65+
this.primaryDrizzle = this.drizzle as any
6566
const readReplicas = this.readReplicaOptions.map((connectionString) => {
6667
const options = {
6768
...this.poolOptions,

‎packages/db-postgres/src/index.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -156,6 +156,7 @@ export function postgresAdapter(args: Args): DatabaseAdapterObj<PostgresAdapter>
156156
// @ts-expect-error - vestiges of when tsconfig was not strict. Feel free to improve
157157
push: args.push,
158158
readReplicaOptions: args.readReplicas,
159+
readReplicasAfterWriteInterval: args.readReplicasAfterWriteInterval ?? 2000,
159160
relations: {},
160161
relationshipsSuffix: args.relationshipsSuffix || '_rels',
161162
schema: {},

‎packages/db-postgres/src/types.ts‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -66,6 +66,13 @@ export type Args = {
6666
}[]
6767
push?: boolean
6868
readReplicas?: string[]
69+
/**
70+
* How long (ms) after a write to keep routing reads to the primary instead
71+
* of a read replica. Prevents stale reads caused by replication lag.
72+
* Only relevant when `readReplicas` is set.
73+
* @default 2000
74+
*/
75+
readReplicasAfterWriteInterval?: number
6976
relationshipsSuffix?: string
7077
/**
7178
* The schema name to use for the database

‎packages/db-vercel-postgres/src/connect.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -48,6 +48,7 @@ export const connect: Connect = async function connect(
4848
})
4949

5050
if (this.readReplicaOptions) {
51+
this.primaryDrizzle = this.drizzle as any
5152
const readReplicas = this.readReplicaOptions.map((connectionString) => {
5253
const options = {
5354
...this.poolOptions,

‎packages/db-vercel-postgres/src/index.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -206,6 +206,7 @@ export function vercelPostgresAdapter(args: Args = {}): DatabaseAdapterObj<Verce
206206
payload,
207207
queryDrafts,
208208
readReplicaOptions: args.readReplicas,
209+
readReplicasAfterWriteInterval: args.readReplicasAfterWriteInterval ?? 2000,
209210
// @ts-expect-error - vestiges of when tsconfig was not strict. Feel free to improve
210211
rejectInitializing,
211212
requireDrizzleKit,

‎packages/db-vercel-postgres/src/types.ts‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -69,6 +69,13 @@ export type Args = {
6969
}[]
7070
push?: boolean
7171
readReplicas?: string[]
72+
/**
73+
* How long (ms) after a write to keep routing reads to the primary instead
74+
* of a read replica. Prevents stale reads caused by replication lag.
75+
* Only relevant when `readReplicas` is set.
76+
* @default 2000
77+
*/
78+
readReplicasAfterWriteInterval?: number
7279
relationshipsSuffix?: string
7380
/**
7481
* The schema name to use for the database

0 commit comments

Comments
 (0)