incremental-sync.ts and api/cw/sync.ts imported getBoss() from workert.ts.
When workert.ts (the entry point) dynamically imported incremental-sync.ts,
it triggered a circular module re-evaluation that hung indefinitely.
Extract the PgBoss singleton and getBoss() factory to a new boss-instance.ts
module that neither has top-level async side-effects nor imports from
workert.ts. All consumers (workert.ts, index.ts, incremental-sync.ts,
cw/sync.ts) now import from boss-instance.ts instead.
- Add statement_timeout=30000ms to PgBoss connection URL to prevent
SQL queries from hanging indefinitely
- Add connectionTimeoutMillis=15s to PgBoss config for connection timeout
- Wrap boss.start() in 30s Promise.race timeout with process.exit(1)
on failure to ensure container restarts instead of hanging silently
- Add debug logging around PgBoss startup to diagnose connection issues
User rows have a FK constraint to CwMember (User_cwMemberId_fkey). Syncing
Users first caused all 140 User upserts to fail since the CwMember table was
empty. This cascade failure then caused all Opportunity upserts to fail because
Opportunity.primarySalesRepId is FK-constrained to User.cwIdentifier.
Fix: reorder steps so CW Members syncs first, then Users.
- Add ownerLevelRecId -> locationId mapping to opportunity translation
- Include soOppStatus in opportunity query and derive closedFlag from
status.closedFlag (with fallback to legacy oldCloseFlag field)
- Add locationId sanitization guard in both sync.ts and sync-by-table.ts
Note: departmentId is not available in CW SO_Opportunity table and
remains null for synced records.
When no User accounts have cwMemberId linked, the context map was empty
and all opportunities got primarySalesRepId = null. Now also populate
the map from CwMember rows directly (User-linked entries take precedence),
so rep identifiers resolve correctly regardless of user account linkage.
Prisma MSSQL adapter keeps connections open after the sync finishes,
preventing the process from exiting naturally. The k8s job was staying
in Running state indefinitely. Call process.exit(0) on success so the
job completes and the GH workflow step passes.
Full initial sync has 500k+ rows across all tables and exceeded the
30-minute activeDeadlineSeconds. Bump both the k8s job deadline and
the kubectl wait timeout to 7200s (2 hours).
envFrom was loading api-env-secret but CW_DATABASE_URL was absent from the
deployed secret, causing sync.ts to fall back to DATABASE_URL (Postgres) as
the MSSQL connection string -> 'Invalid port number: //optima'.
- Replaced envFrom with explicit CW_DATABASE_URL and API_DATABASE_URL env
entries so the mapping is unambiguous
- Patched api-env-secret in cluster to add CW_DATABASE_URL
The CW MSSQL and API Postgres addresses are internal to the cluster and
unreachable from GitHub-hosted runners, so the sync must run inside k8s.
- Add dalpuri-sync Docker stage to api/Dockerfile: installs deps,
generates both Prisma clients, and runs dalpuri/src/sync.ts
- Add dalpuri/kubernetes/sync-job.yaml: mounts api-env-secret (which
already contains CW_DATABASE_URL) and maps DATABASE_URL -> API_DATABASE_URL
- build-api job now also pushes optima-dalpuri-sync:TAG image
- sync-cw-to-api CI job replaced with kubectl apply/wait pattern,
needs [build-api, build-worker], blocks deploy-api and deploy-worker
The socket retrieved from ensureManagerSocketReady() was never passed to
enqueueDalpuriFullSync(), so inside createWorkerJob the socket.emit('requestId')
call crashed with 'TypeError: undefined is not an object (evaluating A.emit)'.
This caused every full sync job to fail immediately, leaving the DB empty.
The 5s incremental sync interval then flooded the queue with 4700+ jobs that
all failed too since there was no data.
Also manually cleared the backlog of 4720 failed/pending incremental jobs and
2 failed full sync jobs from the production queue.
Two bugs in the catch-up migration that only manifest with real production data:
1. Company (4520 rows): uid was added as TEXT NOT NULL DEFAULT '' causing
all existing rows to get uid='' which makes the PRIMARY KEY constraint
fail with 'could not create unique index, Key (uid)=() is duplicated'.
Fix: add uid as nullable, UPDATE uid = id (copies the existing CUID text
PK into uid), then SET NOT NULL, then swap PK. Also populate the new
integer id column from cw_CompanyId (which is fully populated in prod).
2. UnifiSite (180 rows): old approach just dropped the text companyId and
added a null integer column, destroying all company relationships.
Fix: add companyId_int, UPDATE via JOIN on Company.uid (= old Company.id
text), drop old text column, rename integer column.
Also fix the P3009 handler in migrate-entrypoint.sh: Prisma may emit ANSI
color codes even without a TTY, wrapping backticks in escape sequences and
breaking the regex match. Fix: strip ANSI codes with sed before extracting
the migration name. Also simplify the regex from a rigid format match to a
simpler backtick-content grep.
Production DB manually unblocked (migrate resolve --rolled-back) so the
next deploy will cleanly apply the corrected migration.
POSIX sh exits a script on the assignment line when command substitution
exits non-zero under set -e -- before the subsequent echo ever runs.
DEPLOY_OUTPUT=$(cmd 2>&1) # <- script exits here if cmd fails
EXIT_CODE=$?
echo "$DEPLOY_OUTPUT" # <- never reached
Fix: use the || idiom, which puts the LHS in a compound-command context
where set -e does not apply, and still captures the real exit code:
EXIT_CODE=0
DEPLOY_OUTPUT=$(cmd 2>&1) || EXIT_CODE=$?
echo "$DEPLOY_OUTPUT" # <- always runs
Applied the same fix to the resolve call.
All schema changes that were applied via 'prisma db push' over the past
several months were never captured in migration files. When the postgres
pod restarted just before the migration job ran, the database was rebuilt
from the 15 existing migrations -- creating an old schema that was missing
~20 tables and significant structural changes to User, Opportunity,
CatalogItem, and Company.
This migration bridges the gap idempotently:
- New enums: PhoneType, FaxType, BillingMethod, BillingType, GenderType,
USState, Country, OpportunityInterest
- User: add firstName/lastName/title/active/hidden/cwMemberId/updatedBy;
drop emailVerified/name; make userId nullable
- CatalogItem: TEXT id → INTEGER id + TEXT uid PK; restructure FK columns
- Company: TEXT id → INTEGER id + TEXT uid PK; drop old CW columns; add
dateDeleted/deleteFlag/phone/taxExempt/taxId/website/enteredById
- Opportunity: TEXT id → INTEGER id + TEXT uid PK; drop ~25 flat CW
columns; add typeId/statusId/contactId/siteId/locationId/departmentId/
closedById/primarySalesRepId/secondarySalesRepId/eneteredBy/updatedBy/
oppNarrative/taxCodeId/interest; drop cwDateEntered
- UnifiSite: companyId TEXT → INTEGER
- 20+ new tables: CorporateLocation, InternalDepartment, CompanyAddress,
Contact, CatalogItemType, CatalogCategory, CatalogSubcategory,
CatalogManufacturer, Warehouse, WarehouseBin, ProductInventory,
MinimumStockByWarehouse, ProductData, ServiceTicket, ServiceTicketNote,
ServiceTicketType, ServiceTicketBoard, ServiceTicketLocation,
ServiceTicketSource, ServiceTicketImpact, ServiceTicketPriority,
ServiceTicketServerity, ServiceTicketFinalData, OpportunityType,
OpportunityStatus, ScheduleStatus, ScheduleType, ScheduleSpan,
Schedule, TaxCode
Verified: all 16 migrations apply cleanly on a fresh DB and produce zero
schema drift (prisma migrate diff outputs '-- This is an empty migration.')
Fixes P2022 ColumnNotFound errors on login and all model queries.
HoloPanio
Jackson