HorizonStackSoftware/optima
1
0
Fork 0
Code Issues Actions Packages Projects Releases Wiki Activity

fix: remove nested .git folders, re-add as normal directories

Browse Source
This commit is contained in:
Jackson Roberts
2026-03-22 17:50:47 -05:00
parent f55c7e47c9
commit 6b7eec67b8
1870 changed files with 4170168 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
api/CACHING.md
+399
View File
@@ -0,0 +1,399 @@
# Caching Architecture
This document describes the caching layer used in the Optima API, covering the Redis-backed opportunity cache, TTL algorithms, background refresh mechanics, retry logic, and debugging tools.
---
## Overview
The API caches expensive ConnectWise (CW) API responses in **Redis** to reduce latency and avoid CW rate limits. The primary cache layer is the **opportunity cache** (`src/modules/cache/opportunityCache.ts`), which is proactively warmed by workers on a background interval.
The API also maintains a Redis-backed **sales member metrics cache** (`src/modules/cache/salesOpportunityMetricsCache.ts`) refreshed every 5 minutes. It precomputes per-member dashboard/reporting figures (pipeline revenue, won/lost counts, win rate, avg days to close, and related metrics) for fast reads from `/v1/sales/opportunities/metrics`.
### Key design principles
- **Adaptive TTLs** — cache durations are computed dynamically based on how "hot" an opportunity is (recently updated = shorter TTL = fresher data).
- **Background refresh** — a 20-minute interval runs a unified opportunity refresh pass (collector-first full sync + cache warm across active and archived opportunities).
- **Bounded concurrency** — CW API calls are throttled via thunk-based batching to prevent overwhelming the upstream API.
- **Graceful degradation** — transient CW errors (timeouts, network failures) are caught, logged, and retried on the next cycle rather than crashing the process.
- **Priority ordering** — most recently updated opportunities are refreshed first so active deals get fresh data before stale ones.
---
## What is cached
Each opportunity can have up to 7 cached payloads in Redis:
| Cache Key Pattern | Data | Source |
| ----------------------------------- | ------------------------------------ | --------------------------------------------------------------------- |
| `opp:cw-data:{cwOpportunityId}` | Raw CW opportunity response | `GET /sales/opportunities/:id` |
| `opp:activities:{cwOpportunityId}` | CW activities array | `GET /sales/activities?conditions=opportunity/id=:id` |
| `opp:notes:{cwOpportunityId}` | CW notes array | `GET /sales/opportunities/:id/notes` |
| `opp:contacts:{cwOpportunityId}` | CW contacts array | `GET /sales/opportunities/:id/contacts` |
| `opp:products:{cwOpportunityId}` | Forecast + procurement products blob | `GET /sales/opportunities/:id/forecast` + `GET /procurement/products` |
| `opp:company-cw:{cw_CompanyId}` | Hydrated company + contacts blob | `GET /company/companies/:id` + contacts endpoints |
| `opp:site:{cwCompanyId}:{cwSiteId}` | Company site data | `GET /company/companies/:id/sites/:siteId` |
Inventory-adjustment-driven catalog sync adds a targeted product cache:
| Cache Key Pattern | Data | Source |
| ------------------------ | ---------------------------------------------------------- | -------------------------------------------------------------------------------------------- |
| `catalog:item:cw:{cwId}` | Full CW catalog item + computed `onHand` + DB row snapshot | `GET /procurement/adjustments` + `GET /procurement/catalog/:id` + catalog inventory endpoint |
Sales opportunity metrics caching adds member-focused keys:
| Cache Key Pattern | Data | Source |
| ------------------------------------- | -------------------------------------- | ------------------------------------------------------------------------------------- |
| `sales:metrics:members:all` | Envelope of all active-member metrics | Precomputed from active CW members + assigned opportunities + products cache/CW fetch |
| `sales:metrics:member:{cwIdentifier}` | One member's computed metrics snapshot | Same as above |
| `sales:metrics:oppRevenue:{cwOppId}` | Per-opportunity computed revenue blob | Metrics refresh lookups (products cache-first, then manager/controller fallback) |
---
## TTL Algorithms
Three algorithms compute cache TTLs. All share the same input signals:
- `closedFlag` — whether the opportunity is closed
- `closedDate` — when it was closed
- `expectedCloseDate` — projected close date (forward-looking signal)
- `lastUpdated` — last CW modification date (backward-looking signal)
### Primary TTL (`computeCacheTTL`)
**File:** `src/modules/algorithms/computeCacheTTL.ts`
Used for: opportunity CW data, activities, company CW data.
| # | Condition | TTL | Human |
| --- | ------------------------------------------------------- | ---------- | ------------ |
| 1a | Closed > 30 days ago | `null` | Do not cache |
| 1b | Closed within 30 days | 900,000 ms | 15 minutes |
| 2 | `expectedCloseDate` or `lastUpdated` within **5 days** | 30,000 ms | 30 seconds |
| 3 | `expectedCloseDate` or `lastUpdated` within **14 days** | 60,000 ms | 60 seconds |
| 4 | Everything else | 900,000 ms | 15 minutes |
Rules are evaluated top-to-bottom; first match wins.
### Sub-Resource TTL (`computeSubResourceCacheTTL`)
**File:** `src/modules/algorithms/computeSubResourceCacheTTL.ts`
Used for: notes, contacts.
| # | Condition | TTL | Human |
| --- | --------------------- | ---------- | ------------ |
| 1a | Closed > 30 days ago | `null` | Do not cache |
| 1b | Closed within 30 days | 300,000 ms | 5 minutes |
| 2 | Within **5 days** | 60,000 ms | 60 seconds |
| 3 | Within **14 days** | 120,000 ms | 2 minutes |
| 4 | Everything else | 300,000 ms | 5 minutes |
### Products TTL (`computeProductsCacheTTL`)
**File:** `src/modules/algorithms/computeProductsCacheTTL.ts`
Used for: forecast + procurement products.
| # | Condition | TTL | Human |
| --- | ------------------------------------------- | ------------ | ---------- |
| 1 | Status is Won/Lost/Pending Won/Pending Lost | `null` | No cache |
| 2 | Main cache TTL is `null` | `null` | No cache |
| 3 | `lastUpdated` within **3 days** | 15,000 ms | 15 seconds |
| 4 | Everything else | 1,200,000 ms | 20 minutes |
Products on terminal-status opportunities are never proactively cached. Non-hot products use a **lazy on-demand** cache — they're fetched when requested and cached for 20 minutes.
### Site TTL
Sites use a fixed TTL of **20 minutes** (1,200,000 ms). Site/address data rarely changes. Sites are **not** proactively warmed by the background refresh — they are populated lazily on the first detail-view request.
---
## Background Refresh
**Worker path:** `enqueueActiveOpportunityRefreshJob()` in `src/index.ts` → `src/workert.ts` → `refreshActiveOpportunitiesWorker()` in `src/modules/workers/cache/refreshActiveOpportunities.ts`
**Interval:** Every 20 minutes, triggered from `src/index.ts`.
### Refresh cycle
1. **Full sync first** — worker runs `refreshOpportunities()` with collector-first strategy (`fetchOpportunities` collector call, CW fallback) to keep DB opportunity records current.
2. **Collector cache seeding** — when collector data is used, Redis keys for opportunity CW data, activities, notes, contacts, and products are seeded directly from collector payload before cache warm planning.
3. **Query DB** — fetch all opportunities ordered by `cwLastUpdated DESC`.
4. **Compute TTL per opportunity** — adaptive TTL for active/recent opportunities, and `TTL_ARCHIVED_MS` (24h) for archived opportunities where adaptive TTL resolves to `null`.
5. **Batch EXISTS check** — use a single Redis pipeline to check which cache keys already exist (5 EXISTS commands per opportunity: oppCwData, activities, notes, contacts, products).
6. **Build thunk list** — for each opportunity with missing keys, push a **thunk** (lazy function) into the task list. No HTTP requests fire at this point.
7. **Execute with bounded concurrency** — process thunks through a worker pool (`ACTIVE_REFRESH_CONCURRENCY`, default **12**) and progress logging (`ACTIVE_REFRESH_PROGRESS_EVERY`, default **50**).
8. **Emit events** — `cache:opportunities:refresh:started` and `cache:opportunities:refresh:completed` events are emitted for the event debugger.
### Inventory-adjustment listener cycle
**Function:** `listenInventoryAdjustments()` in `src/modules/cw-utils/procurement/listenInventoryAdjustments.ts`
**Interval:** Every 60 seconds, triggered from `src/index.ts`.
1. Fetch `GET /procurement/adjustments?pageSize=1000`.
2. Build a normalized snapshot of tracked inventory rows (`cwCatalogId`, `onHand`, `inventory`) per adjustment.
3. Compare to previous snapshot; extract only changed product IDs.
4. For each changed product ID, fetch fresh CW catalog item + current on-hand.
5. Upsert `CatalogItem` in Postgres and write Redis key `catalog:item:cw:{cwId}` with a 20-minute TTL.
Guardrails to prevent request storms:
- Diffing is computed at **product state** level (grouped by `cwCatalogId`), not raw adjustment-row churn.
- Per-cycle syncs are capped (`CW_ADJUSTMENT_SYNC_MAX_PER_CYCLE`, default `50`).
- Product resync cooldown is enforced (`CW_ADJUSTMENT_SYNC_COOLDOWN_MS`, default `600000` ms / 10 min).
This avoids full-catalog sweeps for small inventory movements and updates only the products implicated by adjustments.
### Full procurement catalog refresh
**Function:** `refreshCatalog()` in `src/modules/cw-utils/procurement/refreshCatalog.ts`
**Interval:** Every 30 minutes, triggered from `src/index.ts`.
The full catalog cache/DB sync uses the same slow-parallel thunk strategy as opportunity cache refreshes:
- Build arrays of thunk tasks (`() => Promise<void>`) for CW item fetches, inventory fetches, and DB upserts.
- Execute with bounded concurrency (`CONCURRENCY=6`).
- Pause between batches (`BATCH_DELAY_MS=250`) to avoid CW burst pressure.
- Log task failures and retry naturally on the next cycle.
This keeps full-catalog refresh conservative while inventory-adjustment listener handles near-real-time targeted updates.
### Full inventory sweep fallback
`refreshInventory()` remains as a safety net but is intentionally infrequent:
- Runs every **6 hours** from `src/index.ts` (no startup-time full sweep).
- Uses the same slow-parallel pattern (`CONCURRENCY=6`, `BATCH_DELAY_MS=250`) to avoid burst traffic.
Most on-hand freshness now comes from the 60-second adjustment listener plus 30-minute full catalog refresh.
### Concurrency control
The thunk pattern is critical. Previously, tasks were pushed as already-executing promises (`refreshTasks.push(fetchAndCache(...))`), which meant all HTTP requests fired simultaneously regardless of the batching loop. The fix was changing the array type from `Promise<void>[]` to `(() => Promise<void>)[]` so requests only start when explicitly invoked: `batch.map((fn) => fn())`.
### Current tuning
| Parameter | Value | Effect |
| ------------------------------- | ---------- | ------------------------------------------ |
| `ACTIVE_REFRESH_CONCURRENCY` | 12 | Max simultaneous CW API requests |
| `ACTIVE_REFRESH_PROGRESS_EVERY` | 50 | Task completion cadence for progress logs |
| Refresh interval | 20 minutes | How often the unified sweep runs |
At these settings, a full sweep of ~500 expired keys completes in ~1-2 minutes with zero CW errors and ~230ms median latency.
### Archived opportunities in unified refresh
Archived opportunities are those returned `null` by `computeCacheTTL` — specifically opportunities with `closedFlag = true` AND `closedDate` older than 30 days (or `closedDate = null`).
In the unified 20-minute refresh pass, these rows are no longer skipped. Instead, cache writes use `TTL_ARCHIVED_MS = 86,400,000 ms` (**24 hours**) while still using the same missing-key EXISTS strategy as active opportunities.
### Sales metrics refresh job
**Function:** `refreshSalesOpportunityMetricsCache()` in `src/modules/cache/salesOpportunityMetricsCache.ts`
**Interval:** Every 5 minutes, triggered from `src/index.ts`.
**Startup behavior:** On app startup, the refresh is invoked once with `forceColdLoad=true`, which clears metrics-owned Redis keys and bypasses metrics/product cache reuse for that initial rebuild. Subsequent interval runs use the normal warm path.
Refresh flow:
1. Fetch all active CW members (`inactiveFlag=false`).
Source: local `CwMember` table (kept in sync by the existing members refresh job).
2. Query DB opportunities assigned to those members (primary or secondary rep), scoped to open opportunities plus YTD-closed opportunities.
3. For each opportunity, compute revenue cache-first from `sales:metrics:oppRevenue:{cwOppId}` then `opp:products:{cwOpportunityId}`, and fallback through the manager/controller path (`opportunities.fetchRecord(...).fetchProducts()`) on miss.
4. Aggregate member metrics (pipeline revenue, won/lost MTD+YTD counts, avg days to close, weighted pipeline, win/loss rates, and related KPIs).
5. Write per-opportunity revenue blobs plus all-member and per-member snapshots to Redis with a 10-minute TTL.
Safety controls:
- **Single-flight lock** prevents overlapping refresh runs if a prior run is still in progress.
- **Per-opportunity timeout guard** ensures slow CW product lookups degrade to zero-revenue fallback instead of stalling the full refresh.
- **Force-cold-load mode** clears `sales:metrics:*` runtime state owned by the metrics cache before rebuilding startup data.
This cache-first model prioritizes metrics-owned opportunity revenue keys first, then opportunity product cache entries, and only reaches CW when needed.
---
## Retry Logic (`withCwRetry`)
**File:** `src/modules/cw-utils/withCwRetry.ts`
Wraps CW API calls with exponential backoff retry on transient errors.
### Retryable errors
- `ECONNABORTED` (timeout)
- `ECONNRESET`
- `ETIMEDOUT`
- `ECONNREFUSED`
- `ERR_NETWORK`
- `ENETUNREACH`
- HTTP 5xx server errors
### Default configuration
| Parameter | Default | Description |
| ------------- | ------- | ----------------------------------------------------------- |
| `maxAttempts` | 3 | Total attempts including the first |
| `baseDelayMs` | 1,000 | Delay before first retry (doubles each retry: 1s → 2s → 4s) |
| `label` | — | Optional tag for log messages |
### Usage
```ts
import { withCwRetry } from "./withCwRetry";
const response = await withCwRetry(
() => connectWiseApi.get(`/company/companies/${id}`),
{ label: `fetchCompany#${id}`, maxAttempts: 3, baseDelayMs: 1_500 },
);
```
Non-transient errors (404, 400, etc.) are re-thrown immediately without retry.
---
## CW API Logger
**File:** `src/modules/cw-utils/cwApiLogger.ts`
Axios interceptor that logs every CW API call to a JSONL file. Logging is **opt-in** — set the `LOG_CW_API` environment variable to enable it. Each process start creates a new timestamped file in the `cw-api-logs/` directory (e.g., `cw-api-logs/2026-03-02T14-30-05.123Z.jsonl`).
### Enabling logging
```bash
# Via the dev:log shorthand script
bun run dev:log
# Or manually with any command
LOG_CW_API=1 bun run dev
```
### Log entry fields
| Field | Type | Description |
| ------------ | ----------------- | ----------------------------------- |
| `timestamp` | string (ISO-8601) | When the request completed |
| `method` | string | HTTP method |
| `url` | string | Request URL (relative or absolute) |
| `baseURL` | string | Axios baseURL |
| `status` | number \| null | HTTP status (null on network error) |
| `durationMs` | number | Wall-clock time in milliseconds |
| `error` | string \| null | Error code + message, if any |
| `timeout` | number | Configured timeout in ms |
### Analysis
Run the analyzer script to analyze the most recent log file:
```bash
bun run utils:analyze_cw
```
Or specify a particular file:
```bash
python3 debug-scripts/analyze-cw-calls.py cw-api-logs/2026-03-02T14-30-05.123Z.jsonl
```
This executes `debug-scripts/analyze-cw-calls.py` which produces:
- Overview (total calls, error rate, time span)
- Duration statistics (min, max, mean, p50, p90, p95, p99, distribution histogram)
- Error breakdown by type and endpoint
- Top 20 slowest calls
- Per-endpoint stats (count, errors, mean, p50, p95, max, total time)
- Timeline (per-minute throughput and errors)
- Concurrency hotspot detection
- Summary with recommendations
To clear all logs:
```bash
rm -rf cw-api-logs/
```
---
## Cache Invalidation
Mutation endpoints invalidate the relevant cache keys so the next read fetches fresh data from CW:
| Mutation | Cache invalidated |
| ------------------------------ | ---------------------------------------------------------------- |
| Create/update/delete note | `opp:notes:{cwOpportunityId}` via `invalidateNotesCache()` |
| Create/update/delete contact | `opp:contacts:{cwOpportunityId}` via `invalidateContactsCache()` |
| Add/update/resequence products | `opp:products:{cwOpportunityId}` via `invalidateProductsCache()` |
| Refresh opportunity | All keys for that opportunity (via re-fetch) |
---
## ConnectWise API Configuration
The shared Axios instance (`connectWiseApi`) is configured in `src/constants.ts`:
| Setting | Value | Purpose |
| --------- | ---------------------------------------------------- | ------------------------------ |
| `baseURL` | `https://ttscw.totaltech.net/v4_6_release/apis/3.0/` | CW API base |
| `timeout` | 30,000 ms (30s) | Per-request timeout |
| Logger | `attachCwApiLogger()` | Writes to `cw-api-calls.jsonl` |
---
## Architecture diagram
```
src/index.ts
│
├─ setInterval(enqueueActiveOpportunityRefreshJob, 20m)
│
└─► src/workert.ts (REFRESH_ACTIVE_OPPORTUNITIES worker)
│
├─ refreshOpportunities() // collector-first full DB sync
│
└─► refreshActiveOpportunitiesWorker()
│
├─ prisma.opportunity.findMany(orderBy: cwLastUpdated DESC)
├─ redis.pipeline().exists(...) ← batch key check
│
├─ Build thunk list (lazy functions)
│
└─ Execute thunks with ACTIVE_REFRESH_CONCURRENCY
│
├─► fetchAndCacheOppCwData() ─► opportunityCw.fetch()
├─► fetchAndCacheActivities() ─► activityCw.fetchByOpportunityDirect()
├─► fetchAndCacheNotes() ─► opportunityCw.fetchNotes()
├─► fetchAndCacheContacts() ─► opportunityCw.fetchContacts()
├─► fetchAndCacheProducts() ─► opportunityCw.fetchProducts() + fetchProcurementProducts()
├─► fetchAndCacheCompanyCwData() ─► fetchCwCompanyById() + contacts
└─► fetchAndCacheSite() ─► fetchCompanySite() (lazy only)
│
└─► connectWiseApi.get(...) ← withCwRetry + cwApiLogger interceptors
│
└─► Redis SET with computed TTL
```
---
## File reference
| File | Purpose |
| ---------------------------------------------------------------- | ------------------------------------------------------------- |
| `src/modules/cache/opportunityCache.ts` | Cache read/write helpers and key utilities |
| `src/modules/workers/cache/refreshActiveOpportunities.ts` | Unified opportunity cache refresh worker (active + archived) |
| `src/workert.ts` | Queue wiring and collector-first full refresh orchestration |
| `src/modules/algorithms/computeCacheTTL.ts` | Primary adaptive TTL algorithm |
| `src/modules/algorithms/computeSubResourceCacheTTL.ts` | Sub-resource (notes, contacts) TTL algorithm |
| `src/modules/algorithms/computeProductsCacheTTL.ts` | Products TTL algorithm |
| `src/modules/cw-utils/withCwRetry.ts` | Retry wrapper with exponential backoff |
| `src/modules/cw-utils/cwApiLogger.ts` | Axios interceptor for JSONL call logging |
| `src/modules/cw-utils/fetchCompany.ts` | Company fetch with retry |
| `src/modules/cw-utils/procurement/listenInventoryAdjustments.ts` | Adjustment listener for targeted catalog-item cache + DB sync |
| `src/modules/cache/salesOpportunityMetricsCache.ts` | 5-minute active-member opportunity metrics cache |
| `src/constants.ts` | CW Axios instance config (timeout, logger) |
| `src/index.ts` | Refresh interval registration and worker job enqueueing |
| `debug-scripts/analyze-cw-calls.py` | CW API call analysis script |