Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
ed41f0a
fix(cli-v3): allow disabling source-map-support to prevent OOM with S…
Feb 2, 2026
023c3fd
fix(cli-v3): ignore engine checks during deployment install to preven…
Feb 2, 2026
93aa053
fix(core): delegate to original console in ConsoleInterceptor to pres…
Feb 2, 2026
8b684e1
fix(cli-v3): authenticate to Docker Hub to prevent rate limits (#2911)
Feb 2, 2026
737ad56
fix(cli-v3): ensure worker cleanup on SIGINT/SIGTERM (#2909)
Feb 2, 2026
c97cbcc
verify: add reproduction scripts and PR details for all major fixes
Feb 3, 2026
aa90db9
verify: add reproduction scripts and PR details for all major fixes
Feb 3, 2026
f5ce2bc
docs: add consolidated PR body description
Feb 3, 2026
8c986db
chore: remove reproduction scripts and temporary files
Feb 3, 2026
82f198f
Merge remote-tracking branch 'remotes/origin/fix/sentry-oom-2920'
Feb 8, 2026
9a3e8d0
Merge branch 'fix/issue-2909-orphaned-workers'
Feb 9, 2026
e101f8e
chore: remove reproduction scripts after verification
Feb 9, 2026
d01d438
fix: resolve typecheck errors after merge
Feb 9, 2026
aafb736
fix(webapp): auto-recover replication services after stream errors
ericallam May 13, 2026
7fa3a16
fix(webapp): reschedule reconnect when subscribe() throws
ericallam May 13, 2026
4e6461a
fix(webapp): reschedule reconnect when subscribe() returns stopped
ericallam May 15, 2026
4b5db51
fix(webapp): drop bogus isStopped check, route leader-lock failure th…
ericallam May 15, 2026
5365936
fix(webapp): scope leaderElection-lost recovery to reconnect strategy
ericallam May 15, 2026
d35bf04
Merge pull request #10 from deepshekhardas/pr/3613-replication-fix
deepshekhardas May 20, 2026
5f4c41a
feat: migrate Prisma from 6.14.0 to 7.7.0 with driver adapters (PR #3…
Jun 3, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/fix-console-interceptor-2900.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"@trigger.dev/core": patch
---

Fix: ConsoleInterceptor now delegates to original console methods to preserve log chain when other interceptors (like Sentry) are present. (#2900)
5 changes: 5 additions & 0 deletions .changeset/fix-docker-hub-rate-limit-2911.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"@trigger.dev/cli-v3": patch
---

Fix: Native build server failed with Docker Hub rate limits. Added support for checking checking `DOCKER_USERNAME` and `DOCKER_PASSWORD` in environment variables and logging into Docker Hub before building. (#2911)
5 changes: 5 additions & 0 deletions .changeset/fix-github-install-node-version-2913.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"@trigger.dev/cli-v3": patch
---

Fix: Ignore engine checks during deployment install phase to prevent failure on build server when Node version mismatch exists. (#2913)
5 changes: 5 additions & 0 deletions .changeset/fix-orphaned-workers-2909.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"@trigger.dev/cli-v3": patch
---

Fix: `trigger.dev dev` command left orphaned worker processes when exited via Ctrl+C (SIGINT). Added signal handlers to ensure proper cleanup of child processes and lockfiles. (#2909)
5 changes: 5 additions & 0 deletions .changeset/fix-sentry-oom-2920.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"@trigger.dev/cli-v3": patch
---

Fix Sentry OOM: Allow disabling `source-map-support` via `TRIGGER_SOURCE_MAPS=false`. Also supports `node` for native source maps. (#2920)
6 changes: 6 additions & 0 deletions .server-changes/replication-error-recovery.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
---
area: webapp
type: fix
---

Runs and sessions replication services now auto-recover from stream errors (e.g. after a Postgres failover) instead of silently leaving replication stopped. Behaviour is configurable per service — reconnect (default), exit so a process supervisor can restart the host, or log.
72 changes: 37 additions & 35 deletions apps/webapp/app/db.server.ts
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,8 @@ import {
type PrismaTransactionClient,
type PrismaTransactionOptions,
} from "@trigger.dev/database";
import { PrismaPg } from "@prisma/adapter-pg";
import { createHash } from "node:crypto";
import invariant from "tiny-invariant";
import { z } from "zod";
import { env } from "./env.server";
Expand Down Expand Up @@ -127,21 +129,30 @@ function getClient() {
const { DATABASE_URL } = process.env;
invariant(typeof DATABASE_URL === "string", "DATABASE_URL env var not set");

const databaseUrl = extendQueryParams(DATABASE_URL, {
connection_limit: env.DATABASE_CONNECTION_LIMIT.toString(),
pool_timeout: env.DATABASE_POOL_TIMEOUT.toString(),
connection_timeout: env.DATABASE_CONNECTION_TIMEOUT.toString(),
application_name: env.SERVICE_NAME,
});
const databaseUrl = new URL(DATABASE_URL);

// Set application_name as a query param on the connection string (pg understands this)
databaseUrl.searchParams.set("application_name", env.SERVICE_NAME);

console.log(`🔌 setting up prisma client to ${redactUrlSecrets(databaseUrl)}`);

const client = new PrismaClient({
datasources: {
db: {
url: databaseUrl.href,
},
const adapter = new PrismaPg(
{
connectionString: databaseUrl.href,
max: env.DATABASE_CONNECTION_LIMIT,
idleTimeoutMillis: env.DATABASE_CONNECTION_TIMEOUT * 1000,
connectionTimeoutMillis: env.DATABASE_CONNECTION_TIMEOUT * 1000,
Comment on lines +143 to +144
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚩 DATABASE_POOL_TIMEOUT env var is now unused after PrismaPg migration

The old Prisma connection string accepted pool_timeout (mapped from env.DATABASE_POOL_TIMEOUT, default 60s) and connection_timeout (mapped from env.DATABASE_CONNECTION_TIMEOUT, default 20s). The new PrismaPg adapter at apps/webapp/app/db.server.ts:143-144 uses env.DATABASE_CONNECTION_TIMEOUT * 1000 for both idleTimeoutMillis and connectionTimeoutMillis. The DATABASE_POOL_TIMEOUT env var defined at apps/webapp/app/env.server.ts:103 is now dead — any existing deployments that tuned DATABASE_POOL_TIMEOUT separately from DATABASE_CONNECTION_TIMEOUT will silently lose that customization. Additionally, idleTimeoutMillis (how long idle connections stay in the pool) is semantically different from the old Prisma pool_timeout (how long to wait for a free connection). The 20s idle timeout may cause more connection churn than the previous 60s pool_timeout in high-concurrency scenarios.

Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

},
{
// Generate deterministic prepared statement names from query SQL so PostgreSQL
// can reuse cached query plans. Without this, every query uses an anonymous
// prepared statement that PG must parse and plan from scratch each time.
statementNameGenerator: (query) => `p_${createHash("sha256").update(query.sql).digest("hex").slice(0, 16)}`,
}
);

const client = new PrismaClient({
adapter,
log: [
// events
{
Expand Down Expand Up @@ -251,21 +262,25 @@ function getReplicaClient() {
return;
}

const replicaUrl = extendQueryParams(env.DATABASE_READ_REPLICA_URL, {
connection_limit: env.DATABASE_CONNECTION_LIMIT.toString(),
pool_timeout: env.DATABASE_POOL_TIMEOUT.toString(),
connection_timeout: env.DATABASE_CONNECTION_TIMEOUT.toString(),
application_name: env.SERVICE_NAME,
});
const replicaUrl = new URL(env.DATABASE_READ_REPLICA_URL);
replicaUrl.searchParams.set("application_name", env.SERVICE_NAME);

console.log(`🔌 setting up read replica connection to ${redactUrlSecrets(replicaUrl)}`);

const replicaClient = new PrismaClient({
datasources: {
db: {
url: replicaUrl.href,
},
const adapter = new PrismaPg(
{
connectionString: replicaUrl.href,
max: env.DATABASE_CONNECTION_LIMIT,
idleTimeoutMillis: env.DATABASE_CONNECTION_TIMEOUT * 1000,
connectionTimeoutMillis: env.DATABASE_CONNECTION_TIMEOUT * 1000,
},
{
statementNameGenerator: (query) => `p_${createHash("sha256").update(query.sql).digest("hex").slice(0, 16)}`,
}
);

const replicaClient = new PrismaClient({
adapter,
log: [
// events
{
Expand Down Expand Up @@ -368,19 +383,6 @@ function getReplicaClient() {
return replicaClient;
}

function extendQueryParams(hrefOrUrl: string | URL, queryParams: Record<string, string>) {
const url = new URL(hrefOrUrl);
const query = url.searchParams;

for (const [key, val] of Object.entries(queryParams)) {
query.set(key, val);
}

url.search = query.toString();

return url;
}

function redactUrlSecrets(hrefOrUrl: string | URL) {
const url = new URL(hrefOrUrl);
url.password = "";
Expand Down
23 changes: 23 additions & 0 deletions apps/webapp/app/env.server.ts
Original file line number Diff line number Diff line change
Expand Up @@ -1330,6 +1330,16 @@ const EnvironmentSchema = z
RUN_REPLICATION_INSERT_STRATEGY: z.enum(["insert", "insert_async"]).default("insert"),
RUN_REPLICATION_DISABLE_PAYLOAD_INSERT: z.string().default("0"),
RUN_REPLICATION_DISABLE_ERROR_FINGERPRINTING: z.string().default("0"),
// What to do when the runs replication client errors (e.g. after a
// Postgres failover). `reconnect` (default) re-subscribes in-process with
// exponential backoff; `exit` exits the process so a supervisor restarts
// it; `log` preserves the old no-op behaviour. Reconnect tuning is
// shared across both replication services via REPLICATION_RECONNECT_*.
RUN_REPLICATION_ERROR_STRATEGY: z
.enum(["reconnect", "exit", "log"])
.default("reconnect"),
RUN_REPLICATION_EXIT_DELAY_MS: z.coerce.number().int().min(0).default(5_000),
RUN_REPLICATION_EXIT_CODE: z.coerce.number().int().min(0).max(255).default(1),

// Session replication (Postgres → ClickHouse sessions_v1). Shares Redis
// with the runs replicator for leader locking but has its own slot and
Expand Down Expand Up @@ -1362,6 +1372,19 @@ const EnvironmentSchema = z
SESSION_REPLICATION_INSERT_MAX_RETRIES: z.coerce.number().int().default(3),
SESSION_REPLICATION_INSERT_BASE_DELAY_MS: z.coerce.number().int().default(100),
SESSION_REPLICATION_INSERT_MAX_DELAY_MS: z.coerce.number().int().default(2000),
// Error recovery — same semantics as RUN_REPLICATION_ERROR_STRATEGY.
SESSION_REPLICATION_ERROR_STRATEGY: z
.enum(["reconnect", "exit", "log"])
.default("reconnect"),
SESSION_REPLICATION_EXIT_DELAY_MS: z.coerce.number().int().min(0).default(5_000),
SESSION_REPLICATION_EXIT_CODE: z.coerce.number().int().min(0).max(255).default(1),

// Reconnect tuning shared across both replication services. Only
// applies when error strategy is `reconnect`. Max attempts of 0 means
// unlimited (default).
REPLICATION_RECONNECT_INITIAL_DELAY_MS: z.coerce.number().int().min(0).default(1_000),
REPLICATION_RECONNECT_MAX_DELAY_MS: z.coerce.number().int().min(0).default(60_000),
REPLICATION_RECONNECT_MAX_ATTEMPTS: z.coerce.number().int().min(0).default(0),

// Clickhouse
CLICKHOUSE_URL: z.string(),
Expand Down
8 changes: 1 addition & 7 deletions apps/webapp/app/routes/metrics.ts
Original file line number Diff line number Diff line change
@@ -1,5 +1,4 @@
import { LoaderFunctionArgs } from "@remix-run/server-runtime";
import { prisma } from "~/db.server";
import { metricsRegister } from "~/metrics.server";

export async function loader({ request }: LoaderFunctionArgs) {
Expand All @@ -13,14 +12,9 @@ export async function loader({ request }: LoaderFunctionArgs) {
}
}

// We need to remove empty lines from the prisma metrics, grafana doesn't like them
const prismaMetrics = (await prisma.$metrics.prometheus()).replace(/^\s*[\r\n]/gm, "");
const coreMetrics = await metricsRegister.metrics();

// Order matters, core metrics end with `# EOF`, prisma metrics don't
const metrics = prismaMetrics + coreMetrics;

return new Response(metrics, {
return new Response(coreMetrics, {
headers: {
"Content-Type": metricsRegister.contentType,
},
Expand Down
Loading