feat(water-log): Smartsheet sync integration
Deploy to route.crispygoat.com / deploy (push) Successful in 5m34s
Deploy to route.crispygoat.com / deploy (push) Successful in 5m34s
Per-brand Smartsheet integration that pushes new water_log_entries to a
configured sheet. Config UI lives at /admin/water-log/settings.
Database (0093_water_smartsheet_sync.sql):
- water_smartsheet_config: one row per brand; AES-256-GCM encrypted API
token (BYTEA), sheet id, column mapping JSONB, sync_frequency
(realtime | every_15_minutes | hourly), enable flag, last-sync
metadata. RLS via existing current_brand_id() pattern.
- water_smartsheet_sync_queue: one row per entry awaiting sync;
tracks status / attempts / exponential backoff. UNIQUE(brand_id,
entry_id) for idempotent enqueue.
- water_smartsheet_sync_log: append-only audit of every sync attempt
(success or failure); backs the Recent Activity panel in the UI.
Server:
- src/lib/crypto.ts: AES-256-GCM encrypt/decrypt + token mask helper.
Reads SMARTSHEET_TOKEN_ENC_KEY. Throws on missing/wrong-size key.
- src/lib/smartsheet.ts: typed REST wrapper (no SDK; the official
smartsheet npm has Node 22 compat issues). getSheetMeta, addRow,
findRowByColumn + typed SmartsheetApiError.
- src/services/smartsheet-sync.ts: syncEntryToSmartsheet (one entry),
drainSyncQueue (batch with backoff), runScheduledSync (cron entry).
Sequential processing stays well under Smartsheet's 300 req/min.
Max 5 attempts, then row stays 'failed' permanently.
- src/actions/water-log/smartsheet.ts: 6 server actions. Token never
returned in plaintext — UI gets maskedToken only. Permission check
matches the rest of the water-log module (can_manage_water_log).
- src/app/api/water-log/smartsheet-sync/route.ts: POST cron route,
bearer auth via SMARTSHEET_CRON_SECRET (falls back to CRON_SECRET).
Optional {brandId} body for manual retry.
- vercel.json: added */15 * * * * cron entry. Single entry covers
both 15-min and hourly frequencies (route filters per-brand).
UI:
- src/components/admin/water-log/SmartsheetIntegrationCard.tsx:
self-contained client component with useReducer. Sections: enable
toggle, sheet URL/ID + API token + Test Connection, sync frequency
radio group, column mapping dropdowns (populated after Test),
Recent Activity panel.
- src/app/admin/water-log/settings/page.tsx: mounted the card as new
'§ 05 — Integrations' section after the existing admin-PIN settings.
Card takes brandId as a prop (CLAUDE.md 'Brand ID Threading').
Hooks:
- src/actions/water-log/field.ts::submitWaterEntry: calls
triggerSyncForEntry(brandId, entryId) in a fire-and-forget void
after the entry insert. Sync failures NEVER bubble to the field
worker.
Env vars (.env.example):
- SMARTSHEET_TOKEN_ENC_KEY (REQUIRED, 32 random bytes base64)
- SMARTSHEET_CRON_SECRET (optional bearer)
- SMARTSHEET_SYNC_TIMEOUT_MS (optional, default 8000)
Deploy:
1. openssl rand -base64 32 → SMARTSHEET_TOKEN_ENC_KEY in .env.local
AND Vercel dashboard
2. npm run migrate:one 93 (pushes 0093_water_smartsheet_sync.sql)
3. Push triggers .gitea/workflows/deploy.yml
Rollback: remove cron entry, DROP the 3 tables, revert field.ts +
settings page hooks, git revert the merge commit. See MEMORY.md
for full rollback story + sharp edges (no key rotation, slug URLs).
This commit is contained in:
@@ -0,0 +1,188 @@
|
||||
-- ============================================================================
|
||||
-- 0093_water_smartsheet_sync.sql
|
||||
--
|
||||
-- Adds per-brand Smartsheet integration for the Water Log module.
|
||||
--
|
||||
-- Three tables:
|
||||
-- 1. water_smartsheet_config — one row per brand; encrypted token,
|
||||
-- sheet id, column mapping, frequency,
|
||||
-- enable flag, last-sync metadata
|
||||
-- 2. water_smartsheet_sync_queue — one row per water_log_entry that
|
||||
-- needs syncing; tracks attempts + status
|
||||
-- for retry / observability
|
||||
-- 3. water_smartsheet_sync_log — append-only log of every sync attempt
|
||||
-- (success OR failure) — backs the
|
||||
-- "Recent Activity" panel in the UI
|
||||
--
|
||||
-- Security:
|
||||
-- - API token is AES-256-GCM encrypted at rest using a key from
|
||||
-- SMARTSHEET_TOKEN_ENC_KEY. The token NEVER leaves the server in
|
||||
-- plaintext via any server action / API response.
|
||||
-- - All tables brand-scoped via the existing `app.current_brand_id`
|
||||
-- GUC + `current_brand_id()` function (see 0001_init.sql).
|
||||
-- ============================================================================
|
||||
|
||||
-- ── 1. Config table ────────────────────────────────────────────────────────
|
||||
|
||||
CREATE TABLE IF NOT EXISTS water_smartsheet_config (
|
||||
brand_id UUID PRIMARY KEY REFERENCES brands(id) ON DELETE CASCADE,
|
||||
|
||||
-- The numeric Smartsheet sheet ID (not the share URL).
|
||||
-- UI accepts either form and parses to numeric here.
|
||||
sheet_id TEXT NOT NULL,
|
||||
|
||||
-- AES-256-GCM encrypted API token (stored as BYTEA so we can store
|
||||
-- raw bytes, not base64). Use the helpers in src/lib/crypto.ts.
|
||||
encrypted_api_token BYTEA NOT NULL,
|
||||
token_iv BYTEA NOT NULL, -- 12 random bytes per encrypt
|
||||
token_auth_tag BYTEA NOT NULL, -- 16-byte GCM auth tag
|
||||
|
||||
-- Maps our internal water-log fields → Smartsheet column IDs.
|
||||
-- Shape: { entry_id: string, logged_at: string, headgate: string,
|
||||
-- measurement: string, unit: string, irrigator: string,
|
||||
-- notes: string | null }
|
||||
-- `entry_id` and `logged_at` are required (used for dedup).
|
||||
column_mapping JSONB NOT NULL,
|
||||
|
||||
-- 'realtime' | 'every_15_minutes' | 'hourly'
|
||||
sync_frequency TEXT NOT NULL DEFAULT 'hourly'
|
||||
CHECK (sync_frequency IN ('realtime','every_15_minutes','hourly')),
|
||||
|
||||
sync_enabled BOOLEAN NOT NULL DEFAULT false,
|
||||
|
||||
-- Set after every sync attempt (success or failure).
|
||||
last_sync_at TIMESTAMPTZ,
|
||||
last_sync_error TEXT,
|
||||
|
||||
-- User IDs from `neon_auth.user`; stored as text so we don't
|
||||
-- require a FK to a specific auth backend.
|
||||
created_by TEXT,
|
||||
updated_by TEXT,
|
||||
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
|
||||
);
|
||||
|
||||
-- Auto-bump updated_at on row UPDATE.
|
||||
DROP TRIGGER IF EXISTS water_smartsheet_config_set_updated_at ON water_smartsheet_config;
|
||||
CREATE TRIGGER water_smartsheet_config_set_updated_at
|
||||
BEFORE UPDATE ON water_smartsheet_config
|
||||
FOR EACH ROW EXECUTE FUNCTION set_updated_at();
|
||||
|
||||
-- RLS: brand-scoped.
|
||||
ALTER TABLE water_smartsheet_config ENABLE ROW LEVEL SECURITY;
|
||||
|
||||
DROP POLICY IF EXISTS tenant_isolation ON water_smartsheet_config;
|
||||
CREATE POLICY tenant_isolation ON water_smartsheet_config FOR ALL
|
||||
USING (brand_id = current_brand_id() OR is_platform_admin())
|
||||
WITH CHECK (brand_id = current_brand_id() OR is_platform_admin());
|
||||
|
||||
-- ── 2. Sync queue (one row per water_log_entry awaiting sync) ──────────────
|
||||
|
||||
CREATE TABLE IF NOT EXISTS water_smartsheet_sync_queue (
|
||||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
brand_id UUID NOT NULL REFERENCES brands(id) ON DELETE CASCADE,
|
||||
|
||||
-- FK to the entry. ON DELETE CASCADE means deleting an entry also
|
||||
-- removes its queue row, preventing orphaned sync attempts.
|
||||
entry_id UUID NOT NULL REFERENCES water_log_entries(id) ON DELETE CASCADE,
|
||||
|
||||
-- The Smartsheet row ID returned from a successful sync.
|
||||
-- NULL = not yet synced.
|
||||
smartsheet_row_id TEXT,
|
||||
|
||||
-- 'pending' | 'syncing' | 'synced' | 'failed'
|
||||
status TEXT NOT NULL DEFAULT 'pending'
|
||||
CHECK (status IN ('pending','syncing','synced','failed')),
|
||||
|
||||
-- Bumped on every attempt (success or failure). Caps at 5 (after
|
||||
-- which the row stays 'failed' permanently; admin must re-enable).
|
||||
attempts INT NOT NULL DEFAULT 0,
|
||||
|
||||
-- Last failure reason (sanitized — token never appears here).
|
||||
last_error TEXT,
|
||||
|
||||
-- When the next retry is allowed. Set to NOW() initially; pushed
|
||||
-- out by exponential backoff on failure (2^attempts minutes, cap 1h).
|
||||
next_attempt_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
synced_at TIMESTAMPTZ,
|
||||
|
||||
-- One queue row per entry per brand. CASCADE handles the case where
|
||||
-- the entry is deleted (rare; only via admin override).
|
||||
CONSTRAINT water_smartsheet_queue_entry_unique UNIQUE (brand_id, entry_id)
|
||||
);
|
||||
|
||||
-- Hot path: drain query scans `(brand_id, status, next_attempt_at)`.
|
||||
CREATE INDEX IF NOT EXISTS water_smartsheet_queue_brand_status_idx
|
||||
ON water_smartsheet_sync_queue (brand_id, status, next_attempt_at);
|
||||
|
||||
-- Recent-attempts view.
|
||||
CREATE INDEX IF NOT EXISTS water_smartsheet_queue_brand_created_idx
|
||||
ON water_smartsheet_sync_queue (brand_id, created_at DESC);
|
||||
|
||||
ALTER TABLE water_smartsheet_sync_queue ENABLE ROW LEVEL SECURITY;
|
||||
|
||||
DROP POLICY IF EXISTS tenant_isolation ON water_smartsheet_sync_queue;
|
||||
CREATE POLICY tenant_isolation ON water_smartsheet_sync_queue FOR ALL
|
||||
USING (brand_id = current_brand_id() OR is_platform_admin())
|
||||
WITH CHECK (brand_id = current_brand_id() OR is_platform_admin());
|
||||
|
||||
-- ── 3. Sync log (append-only, backs Recent Activity UI) ────────────────────
|
||||
|
||||
CREATE TABLE IF NOT EXISTS water_smartsheet_sync_log (
|
||||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
brand_id UUID NOT NULL REFERENCES brands(id) ON DELETE CASCADE,
|
||||
|
||||
entry_id UUID REFERENCES water_log_entries(id) ON DELETE SET NULL,
|
||||
smartsheet_row_id TEXT,
|
||||
|
||||
-- 'sync' (initial push) | 'retry' (after a failure) | 'skip'
|
||||
-- (dedup hit — entry already had smartsheet_row_id) | 'test'
|
||||
-- (the "Test Connection" button does NOT log here; that's a config
|
||||
-- event recorded via water_audit_log instead).
|
||||
action TEXT NOT NULL
|
||||
CHECK (action IN ('sync','retry','skip','queue')),
|
||||
|
||||
success BOOLEAN NOT NULL,
|
||||
error TEXT,
|
||||
duration_ms INT,
|
||||
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS water_smartsheet_log_brand_recent_idx
|
||||
ON water_smartsheet_sync_log (brand_id, created_at DESC);
|
||||
|
||||
ALTER TABLE water_smartsheet_sync_log ENABLE ROW LEVEL SECURITY;
|
||||
|
||||
DROP POLICY IF EXISTS tenant_isolation ON water_smartsheet_sync_log;
|
||||
CREATE POLICY tenant_isolation ON water_smartsheet_sync_log FOR ALL
|
||||
USING (brand_id = current_brand_id() OR is_platform_admin())
|
||||
WITH CHECK (brand_id = current_brand_id() OR is_platform_admin());
|
||||
|
||||
-- ── 4. Helper view: latest sync per entry (for dedup decisions) ────────────
|
||||
|
||||
-- One row per entry with the most recent sync attempt. Used by the
|
||||
-- sync service to decide "has this entry been pushed already?".
|
||||
CREATE OR REPLACE VIEW water_smartsheet_latest_sync AS
|
||||
SELECT DISTINCT ON (q.brand_id, q.entry_id)
|
||||
q.brand_id,
|
||||
q.entry_id,
|
||||
q.smartsheet_row_id,
|
||||
q.status,
|
||||
q.attempts,
|
||||
q.last_error,
|
||||
q.next_attempt_at,
|
||||
q.created_at AS queued_at,
|
||||
q.synced_at
|
||||
FROM water_smartsheet_sync_queue q
|
||||
ORDER BY q.brand_id, q.entry_id, q.created_at DESC;
|
||||
|
||||
COMMENT ON TABLE water_smartsheet_config IS
|
||||
'Per-brand Smartsheet integration config. Token is AES-256-GCM encrypted.';
|
||||
COMMENT ON TABLE water_smartsheet_sync_queue IS
|
||||
'One row per water_log_entry awaiting / completed sync to Smartsheet.';
|
||||
COMMENT ON TABLE water_smartsheet_sync_log IS
|
||||
'Append-only audit of Smartsheet sync attempts (success and failure).';
|
||||
+183
-1
@@ -24,11 +24,19 @@ import {
|
||||
index,
|
||||
uniqueIndex,
|
||||
integer,
|
||||
check,
|
||||
customType,
|
||||
} from "drizzle-orm/pg-core";
|
||||
import { brands } from "./brands";
|
||||
import { adminUsers } from "./brands";
|
||||
|
||||
// Drizzle's stock `bytea` import is not exported in every version, so
|
||||
// register a small custom type that maps to the existing `bytea` PG type.
|
||||
const bytea = customType<{ data: Buffer; default: false }>({
|
||||
dataType() {
|
||||
return "bytea";
|
||||
},
|
||||
});
|
||||
|
||||
export const waterHeadgates = pgTable(
|
||||
"water_headgates",
|
||||
{
|
||||
@@ -255,3 +263,177 @@ export type WaterAlertLog = typeof waterAlertLog.$inferSelect;
|
||||
export type WaterAdminSettings = typeof waterAdminSettings.$inferSelect;
|
||||
export type WaterAdminSession = typeof waterAdminSessions.$inferSelect;
|
||||
export type WaterAuditLog = typeof waterAuditLog.$inferSelect;
|
||||
|
||||
// ── Smartsheet integration (added in 0093_water_smartsheet_sync.sql) ────────
|
||||
|
||||
/**
|
||||
* Allowed sync frequencies. Stored as TEXT to match the convention
|
||||
* in CLAUDE.md ("Status enums stored as TEXT — no PostgreSQL ENUM type").
|
||||
*/
|
||||
export const SMARTSHEET_FREQUENCIES = [
|
||||
"realtime",
|
||||
"every_15_minutes",
|
||||
"hourly",
|
||||
] as const;
|
||||
export type SmartsheetFrequency = (typeof SMARTSHEET_FREQUENCIES)[number];
|
||||
|
||||
/**
|
||||
* The Water-log fields a brand can map to their Smartsheet columns.
|
||||
* `entry_id` and `logged_at` are required (used for dedup).
|
||||
*/
|
||||
export type SmartsheetColumnKey =
|
||||
| "entry_id"
|
||||
| "logged_at"
|
||||
| "headgate"
|
||||
| "measurement"
|
||||
| "unit"
|
||||
| "irrigator"
|
||||
| "notes";
|
||||
export const SMARTSHEET_COLUMN_KEYS: readonly SmartsheetColumnKey[] = [
|
||||
"entry_id",
|
||||
"logged_at",
|
||||
"headgate",
|
||||
"measurement",
|
||||
"unit",
|
||||
"irrigator",
|
||||
"notes",
|
||||
] as const;
|
||||
|
||||
/**
|
||||
* Shape stored in `water_smartsheet_config.column_mapping`. The values
|
||||
* are Smartsheet column IDs (numeric strings, e.g. "8309876543210123").
|
||||
*
|
||||
* Only `entry_id` and `logged_at` are required (used for dedup). All
|
||||
* other fields are nullable — `null` means "not mapped, don't write
|
||||
* this column". The UI uses an empty string in the dropdown to mean
|
||||
* the same thing and we coerce on the server.
|
||||
*/
|
||||
export type SmartsheetColumnMapping = {
|
||||
entry_id: string;
|
||||
logged_at: string;
|
||||
headgate: string | null;
|
||||
measurement: string | null;
|
||||
unit: string | null;
|
||||
irrigator: string | null;
|
||||
notes: string | null;
|
||||
};
|
||||
|
||||
export const waterSmartsheetConfig = pgTable("water_smartsheet_config", {
|
||||
brandId: uuid("brand_id")
|
||||
.primaryKey()
|
||||
.references(() => brands.id, { onDelete: "cascade" }),
|
||||
sheetId: text("sheet_id").notNull(),
|
||||
encryptedApiToken: bytea("encrypted_api_token").notNull(),
|
||||
tokenIv: bytea("token_iv").notNull(),
|
||||
tokenAuthTag: bytea("token_auth_tag").notNull(),
|
||||
columnMapping: jsonb("column_mapping")
|
||||
.$type<SmartsheetColumnMapping>()
|
||||
.notNull(),
|
||||
syncFrequency: text("sync_frequency")
|
||||
.$type<SmartsheetFrequency>()
|
||||
.notNull()
|
||||
.default("hourly"),
|
||||
syncEnabled: boolean("sync_enabled").notNull().default(false),
|
||||
lastSyncAt: timestamp("last_sync_at", { withTimezone: true }),
|
||||
lastSyncError: text("last_sync_error"),
|
||||
createdBy: text("created_by"),
|
||||
updatedBy: text("updated_by"),
|
||||
createdAt: timestamp("created_at", { withTimezone: true })
|
||||
.notNull()
|
||||
.defaultNow(),
|
||||
updatedAt: timestamp("updated_at", { withTimezone: true })
|
||||
.notNull()
|
||||
.defaultNow(),
|
||||
});
|
||||
|
||||
export type WaterSmartsheetConfig = typeof waterSmartsheetConfig.$inferSelect;
|
||||
export type WaterSmartsheetConfigInsert = typeof waterSmartsheetConfig.$inferInsert;
|
||||
|
||||
/**
|
||||
* Sync queue status. Mirrors the SQL CHECK constraint.
|
||||
*/
|
||||
export const SMARTSHEET_QUEUE_STATUSES = [
|
||||
"pending",
|
||||
"syncing",
|
||||
"synced",
|
||||
"failed",
|
||||
] as const;
|
||||
export type SmartsheetQueueStatus = (typeof SMARTSHEET_QUEUE_STATUSES)[number];
|
||||
|
||||
export const waterSmartsheetSyncQueue = pgTable(
|
||||
"water_smartsheet_sync_queue",
|
||||
{
|
||||
id: uuid("id").primaryKey().defaultRandom(),
|
||||
brandId: uuid("brand_id")
|
||||
.notNull()
|
||||
.references(() => brands.id, { onDelete: "cascade" }),
|
||||
entryId: uuid("entry_id")
|
||||
.notNull()
|
||||
.references(() => waterLogEntries.id, { onDelete: "cascade" }),
|
||||
smartsheetRowId: text("smartsheet_row_id"),
|
||||
status: text("status")
|
||||
.$type<SmartsheetQueueStatus>()
|
||||
.notNull()
|
||||
.default("pending"),
|
||||
attempts: integer("attempts").notNull().default(0),
|
||||
lastError: text("last_error"),
|
||||
nextAttemptAt: timestamp("next_attempt_at", { withTimezone: true })
|
||||
.notNull()
|
||||
.defaultNow(),
|
||||
createdAt: timestamp("created_at", { withTimezone: true })
|
||||
.notNull()
|
||||
.defaultNow(),
|
||||
syncedAt: timestamp("synced_at", { withTimezone: true }),
|
||||
},
|
||||
(t) => ({
|
||||
entryUnique: uniqueIndex("water_smartsheet_queue_entry_unique").on(
|
||||
t.brandId,
|
||||
t.entryId,
|
||||
),
|
||||
brandStatusIdx: index("water_smartsheet_queue_brand_status_idx").on(
|
||||
t.brandId,
|
||||
t.status,
|
||||
t.nextAttemptAt,
|
||||
),
|
||||
brandRecentIdx: index("water_smartsheet_queue_brand_created_idx").on(
|
||||
t.brandId,
|
||||
t.createdAt,
|
||||
),
|
||||
}),
|
||||
);
|
||||
|
||||
export type WaterSmartsheetSyncQueue = typeof waterSmartsheetSyncQueue.$inferSelect;
|
||||
export type WaterSmartsheetSyncQueueInsert = typeof waterSmartsheetSyncQueue.$inferInsert;
|
||||
|
||||
export const SMARTSHEET_LOG_ACTIONS = ["sync", "retry", "skip", "queue"] as const;
|
||||
export type SmartsheetLogAction = (typeof SMARTSHEET_LOG_ACTIONS)[number];
|
||||
|
||||
export const waterSmartsheetSyncLog = pgTable(
|
||||
"water_smartsheet_sync_log",
|
||||
{
|
||||
id: uuid("id").primaryKey().defaultRandom(),
|
||||
brandId: uuid("brand_id")
|
||||
.notNull()
|
||||
.references(() => brands.id, { onDelete: "cascade" }),
|
||||
entryId: uuid("entry_id").references(() => waterLogEntries.id, {
|
||||
onDelete: "set null",
|
||||
}),
|
||||
smartsheetRowId: text("smartsheet_row_id"),
|
||||
action: text("action").$type<SmartsheetLogAction>().notNull(),
|
||||
success: boolean("success").notNull(),
|
||||
error: text("error"),
|
||||
durationMs: integer("duration_ms"),
|
||||
createdAt: timestamp("created_at", { withTimezone: true })
|
||||
.notNull()
|
||||
.defaultNow(),
|
||||
},
|
||||
(t) => ({
|
||||
brandRecentIdx: index("water_smartsheet_log_brand_recent_idx").on(
|
||||
t.brandId,
|
||||
t.createdAt,
|
||||
),
|
||||
}),
|
||||
);
|
||||
|
||||
export type WaterSmartsheetSyncLog = typeof waterSmartsheetSyncLog.$inferSelect;
|
||||
export type WaterSmartsheetSyncLogInsert = typeof waterSmartsheetSyncLog.$inferInsert;
|
||||
|
||||
Reference in New Issue
Block a user