metrics: stand up the weekly funnel job (FRE-202)

`marketing/metrics/weekly.md` has been a placeholder since the marketing tree
landed — it promised a funnel and the one thing that moved, and nothing
produced it. This adds the producer: one command reads the live databases and
rewrites the file.

`scripts/weekly-metrics.ts` writes `marketing/metrics/weekly.md` from
`VisitorAnalytics` (visits, downloads, campaigns) and the-nook's `trials`,
`licenses`, `activations` (the paid funnel). It needs four env vars —
TURSO_DB_URL/TOKEN and NOOK_DB_URL/TOKEN — and imports no server env, so the
job runs from a checkout that has no Stripe or Apple secrets.
`src/server/weekly-metrics.ts` holds the queries and the renderer, taking both
connections as arguments; the tests run them against an in-memory SQLite shaped
like production, the same way the backfill suite does.

The campaign breakdown groups exactly as `getAnalyticsSummary.topCampaigns`
does (top 10, untagged excluded), so the weekly read and /analytics cannot
drift into different label sets.

The first real report says more about the instrumentation than about demand,
and the renderer computes that rather than asserting it:

- every addressed row the table has ever held (12,035 of 12,048) is loopback,
  so "unique visitors" is one host and per-channel reach cannot be read;
- the landing door and the file route both write `/api/downloads/thenook`, so
  "download click" and "DMG served" are one label — one served click writes two
  rows, a failed click one, a Sparkle update one;
- `POST /api/the-nook/checkout` records nothing, so checkout start is only a
  page-view proxy;
- `VisitorAnalytics` has no host dimension, so the Nook's `/` is freno.me's `/`.

Those three gaps are reported beside the numbers, not papered over. Week 0
(2026-10-22 → 2026-10-29): visit 61, download 156, trial 2, checkout 0,
purchase 0, activation 0 — and the one thing that moved is the download stage,
0 → 156, because the label was introduced that week.

Tests: `bun test src/server/weekly-metrics.test.ts` — 12 pass, 0 fail. The full
suite's 15 failures are pre-existing: this checkout's .env lacks
NOOK_LICENSE_PRIVATE_KEY, NOOK_STRIPE_SK/WEBHOOK/PRICE and LINEAGE_JWT_SECRET,
and every failure is that env validation.

Scripts/ is gitignored, so `scripts/weekly-metrics.ts` was added with `-f`, the
way the existing tracked scripts are.

FRE-202
This commit is contained in:
Paperclip Agent
2026-10-29 15:58:51 +00:00
parent d44133fcb3
commit 89f56fc78a
6 changed files with 1184 additions and 2 deletions

View File

@@ -0,0 +1,591 @@
/**
* The weekly funnel reading: `marketing/metrics/weekly.md`'s data half.
*
* The funnel the marketing cluster agreed on is
*
* visit → download click → DMG served → trial started → checkout started
* → purchase → activation
*
* and this module reads it from the two live databases, plus the campaign
* dimension the /analytics page already exposes through
* `getAnalyticsSummary.topCampaigns`.
*
* Two properties matter more than the numbers:
*
* - It takes both connections as arguments. `~/server/analytics` imports the
* full server env (Stripe keys, Apple secret, licence key …); the weekly
* job only needs the two databases, and must be runnable from a checkout
* whose `.env` carries nothing else. The SQL here is the same shape as the
* analytics summaries, so the weekly read and the dashboard cannot drift
* into different label sets.
* - Every stage carries what it actually measured. Two of the seven agreed
* stages share one label in production today and a third is only a page-view
* proxy; the report states that next to the number instead of inventing a
* figure (see `renderWeeklyMarkdown`).
*
* No `~/env`, no `~/server/database`: the caller owns the connections, the
* tests own an in-memory SQLite, and the script owns the libsql clients.
*/
import { DOWNLOAD_PATH_PREFIX } from "~/lib/download-asset";
/** The one asset key the Nook lands under — `downloadPath("thenook")`. */
export const NOOK_DOWNLOAD_PATH = `${DOWNLOAD_PATH_PREFIX}thenook`;
/** An `execute`-shaped connection: the libsql client and the test adapter both fit. */
export interface MetricsConn {
execute(query: {
sql: string;
args?: any[];
}): Promise<{ rows: unknown[]; rowsAffected?: number }>;
}
export interface Connections {
/** freno.me Turso DB — VisitorAnalytics. */
analytics: MetricsConn;
/** the-nook Turso DB — trials, licenses, activations. */
nook: MetricsConn;
}
export type StageKey =
| "visit"
| "download"
| "trial"
| "checkout"
| "purchase"
| "activation";
export interface StageDef {
key: StageKey;
label: string;
/** True when the number is a proxy or a merged label, not a clean stage. */
approximate: boolean;
}
/**
* The seven agreed stages. `download` deliberately carries two of them: the
* door and the file route both write `/api/downloads/thenook`, so a click and
* the DMG it fetched are one label in the data (proved in the report's caveats).
*/
export const STAGES: StageDef[] = [
{ key: "visit", label: "visit (page views)", approximate: false },
{
key: "download",
label: "download click → DMG served (one label)",
approximate: true
},
{ key: "trial", label: "trial started", approximate: false },
{
key: "checkout",
label: "checkout page visit",
approximate: true
},
{ key: "purchase", label: "purchase (licence issued)", approximate: false },
{ key: "activation", label: "activation", approximate: false }
];
export interface FunnelCounts {
visit: number;
download: number;
trial: number;
checkout: number;
purchase: number;
activation: number;
}
export interface CampaignRow {
source: string | null;
medium: string | null;
campaign: string | null;
count: number;
}
/** Traffic-shape facts the report has to state next to the funnel. */
export interface WindowTraffic {
analyticsRows: number;
loopbackRows: number;
/** All-time row counts, so "every row ever" is a checked claim, not a guess. */
allTimeRows: number;
allTimeAddressedRows: number;
allTimeLoopbackRows: number;
downloadRows: number;
downloadLoopbackRows: number;
downloadNonBrowserRows: number;
}
export interface Mover {
key: StageKey;
label: string;
now: number;
before: number;
delta: number;
}
export interface WeeklyReport {
window: { start: string; end: string; days: number };
previous: { start: string; end: string };
counts: FunnelCounts;
previousCounts: FunnelCounts;
campaigns: CampaignRow[];
taggedRows: number;
traffic: WindowTraffic;
mover: Mover | null;
generatedAt: string;
}
/** Loopback addresses — every row the live table has ever held. */
const LOOPBACK = ["127.0.0.1", "::1", "::ffff:127.0.0.1"];
/** SQLite's `datetime()` string, UTC: `YYYY-MM-DD HH:MM:SS`. */
export function sqlTime(date: Date): string {
return date.toISOString().slice(0, 19).replace("T", " ");
}
export interface Windows {
current: { start: string; end: string };
previous: { start: string; end: string };
}
/**
* The current window is `[now - days, now)` and the previous one sits directly
* before it, so a weekly read always compares like with like. Boundaries are
* passed as SQL parameters rather than `datetime('now', …)` so the file a run
* writes is reproducible and the two queries cannot straddle a second.
*/
export function windowsFor(now: Date, days: number): Windows {
const span = days * 24 * 60 * 60 * 1000;
const end = now;
const start = new Date(end.getTime() - span);
const previousEnd = start;
const previousStart = new Date(previousEnd.getTime() - span);
return {
current: { start: sqlTime(start), end: sqlTime(end) },
previous: { start: sqlTime(previousStart), end: sqlTime(previousEnd) }
};
}
function toNumber(value: unknown): number {
if (typeof value === "number") return value;
if (typeof value === "bigint") return Number(value);
if (typeof value === "string") return Number(value) || 0;
return 0;
}
async function scalar(
conn: MetricsConn,
sql: string,
args: any[]
): Promise<number> {
const result = await conn.execute({ sql, args });
const row = result.rows[0] as Record<string, unknown> | undefined;
if (!row) return 0;
return toNumber(Object.values(row)[0]);
}
async function countInWindow(
conn: MetricsConn,
sql: string,
start: string,
end: string
): Promise<number> {
return scalar(conn, sql, [start, end]);
}
/** One window's funnel. Two databases, one call. */
export async function readFunnel(
conns: Connections,
window: { start: string; end: string }
): Promise<FunnelCounts> {
const { analytics, nook } = conns;
const { start, end } = window;
const visit = await countInWindow(
analytics,
`SELECT COUNT(*) AS n FROM VisitorAnalytics
WHERE created_at >= ? AND created_at < ?
AND path NOT LIKE '/api/%'`,
start,
end
);
const download = await scalar(
analytics,
`SELECT COUNT(*) AS n FROM VisitorAnalytics
WHERE created_at >= ? AND created_at < ?
AND path = ?`,
[start, end, NOOK_DOWNLOAD_PATH]
);
const checkout = await countInWindow(
analytics,
`SELECT COUNT(*) AS n FROM VisitorAnalytics
WHERE created_at >= ? AND created_at < ?
AND (path LIKE '/checkout%' OR path LIKE '/nook/checkout%')`,
start,
end
);
// The licence DB stores ISO-8601 (`2026-10-28T20:28:46.806Z`); the analytics
// DB stores `YYYY-MM-DD HH:MM:SS`. `datetime()` normalises both sides so a
// boundary seconds apart cannot include or drop a row on string comparison.
const trial = await scalar(
nook,
`SELECT COUNT(*) AS n FROM trials
WHERE datetime(started_at) >= datetime(?) AND datetime(started_at) < datetime(?)`,
[start, end]
);
const purchase = await scalar(
nook,
`SELECT COUNT(*) AS n FROM licenses
WHERE datetime(created_at) >= datetime(?) AND datetime(created_at) < datetime(?)`,
[start, end]
);
const activation = await scalar(
nook,
`SELECT COUNT(*) AS n FROM activations
WHERE datetime(activated_at) >= datetime(?) AND datetime(activated_at) < datetime(?)`,
[start, end]
);
return {
visit,
download,
trial,
checkout,
purchase,
activation
};
}
/**
* Traffic shape for one window: how much of it came from this host, and how
* much of the download label was automation. Without these two facts the
* numbers read as demand, which they are not.
*/
export async function readTraffic(
conns: Connections,
window: { start: string; end: string }
): Promise<WindowTraffic> {
const { analytics } = conns;
const { start, end } = window;
const placeholders = LOOPBACK.map(() => "?").join(", ");
const allTimeRows = await scalar(
analytics,
`SELECT COUNT(*) AS n FROM VisitorAnalytics`,
[]
);
const allTimeAddressedRows = await scalar(
analytics,
`SELECT COUNT(*) AS n FROM VisitorAnalytics WHERE ip_address IS NOT NULL`,
[]
);
const allTimeLoopbackRows = await scalar(
analytics,
`SELECT COUNT(*) AS n FROM VisitorAnalytics
WHERE ip_address IN (${placeholders})`,
[...LOOPBACK]
);
const analyticsRows = await countInWindow(
analytics,
`SELECT COUNT(*) AS n FROM VisitorAnalytics
WHERE created_at >= ? AND created_at < ?`,
start,
end
);
const loopbackRows = await scalar(
analytics,
`SELECT COUNT(*) AS n FROM VisitorAnalytics
WHERE created_at >= ? AND created_at < ?
AND ip_address IN (${placeholders})`,
[start, end, ...LOOPBACK]
);
const downloadRows = await scalar(
analytics,
`SELECT COUNT(*) AS n FROM VisitorAnalytics
WHERE created_at >= ? AND created_at < ? AND path = ?`,
[start, end, NOOK_DOWNLOAD_PATH]
);
const downloadLoopbackRows = await scalar(
analytics,
`SELECT COUNT(*) AS n FROM VisitorAnalytics
WHERE created_at >= ? AND created_at < ? AND path = ?
AND ip_address IN (${placeholders})`,
[start, end, NOOK_DOWNLOAD_PATH, ...LOOPBACK]
);
// Anything whose UA is not a browser engine: curl, Sparkle's updater,
// link-preview crawlers. A real install click arrives as `Mozilla/5.0 …`.
const downloadNonBrowserRows = await scalar(
analytics,
`SELECT COUNT(*) AS n FROM VisitorAnalytics
WHERE created_at >= ? AND created_at < ? AND path = ?
AND (user_agent IS NULL OR user_agent NOT LIKE 'Mozilla/%')`,
[start, end, NOOK_DOWNLOAD_PATH]
);
return {
analyticsRows,
loopbackRows,
allTimeRows,
allTimeAddressedRows,
allTimeLoopbackRows,
downloadRows,
downloadLoopbackRows,
downloadNonBrowserRows
};
}
/**
* The campaign dimension, grouped exactly as `getAnalyticsSummary.topCampaigns`
* groups it: top 10 by volume, untagged rows excluded (the untagged bucket is
* a different question and would otherwise swallow the list).
*/
export async function readCampaigns(
analytics: MetricsConn,
window: { start: string; end: string }
): Promise<CampaignRow[]> {
const result = await analytics.execute({
sql: `SELECT utm_source, utm_medium, utm_campaign, COUNT(*) AS count
FROM VisitorAnalytics
WHERE created_at >= ? AND created_at < ?
AND (utm_source IS NOT NULL OR utm_medium IS NOT NULL OR utm_campaign IS NOT NULL)
GROUP BY utm_source, utm_medium, utm_campaign
ORDER BY count DESC
LIMIT 10`,
args: [window.start, window.end]
});
return result.rows.map((raw) => {
const row = raw as Record<string, unknown>;
return {
source: (row.utm_source as string | null) ?? null,
medium: (row.utm_medium as string | null) ?? null,
campaign: (row.utm_campaign as string | null) ?? null,
count: toNumber(row.count)
};
});
}
/**
* "The one thing that moved": the stage with the largest absolute change
* against the previous window. Ties go to the earlier stage, so a week when
* nothing moved returns null rather than a made-up winner.
*/
export function pickMover(
current: FunnelCounts,
previous: FunnelCounts
): Mover | null {
let best: Mover | null = null;
for (const stage of STAGES) {
const delta = current[stage.key] - previous[stage.key];
if (delta === 0) continue;
if (!best || Math.abs(delta) > Math.abs(best.delta)) {
best = {
key: stage.key,
label: stage.label,
now: current[stage.key],
before: previous[stage.key],
delta
};
}
}
return best;
}
export interface ReportOptions {
/** End of the current window; defaults to the wall clock. */
now?: Date;
/** Window length in days; defaults to 7. */
days?: number;
}
/** Read everything `.md` needs, in one call. */
export async function getWeeklyReport(
conns: Connections,
options: ReportOptions = {}
): Promise<WeeklyReport> {
const days = options.days ?? 7;
if (!Number.isFinite(days) || days <= 0) {
throw new Error(`days must be a positive number, got ${days}`);
}
const now = options.now ?? new Date();
const windows = windowsFor(now, days);
const [counts, previousCounts, traffic, campaigns] = await Promise.all([
readFunnel(conns, windows.current),
readFunnel(conns, windows.previous),
readTraffic(conns, windows.current),
readCampaigns(conns.analytics, windows.current)
]);
return {
window: { ...windows.current, days },
previous: windows.previous,
counts,
previousCounts,
campaigns,
taggedRows: campaigns.reduce((sum, row) => sum + row.count, 0),
traffic,
mover: pickMover(counts, previousCounts),
generatedAt: now.toISOString().replace(/\.\d{3}Z$/, "Z")
};
}
function cell(value: string | null): string {
return value === null || value === "" ? "—" : value;
}
function signed(value: number): string {
if (value === 0) return "0";
return value > 0 ? `+${value}` : `−${Math.abs(value)}`;
}
/**
* Render the weekly file. Interpretation lives in the CMO's read; this renderer
* only states what the data is, what moved, and what the numbers cannot say.
* The Gaps section is not boilerplate: each bullet is computed from the window
* it describes, so it empties itself if the underlying hole is ever fixed.
*/
export function renderWeeklyMarkdown(report: WeeklyReport): string {
const { counts, previousCounts, traffic } = report;
const lines: string[] = [];
lines.push("# Weekly funnel snapshot");
lines.push("");
lines.push(
`Window: **${report.window.start} → ${report.window.end} UTC** (${report.window.days} days). ` +
`Previous: ${report.previous.start} → ${report.previous.end} UTC.`
);
lines.push("");
lines.push(
"Generated by `bun --env-file=.env scripts/weekly-metrics.ts`; regenerated " +
"weekly, never hand-edited. The analysis is the CMO's."
);
lines.push("");
lines.push("## Funnel");
lines.push("");
lines.push("| Stage | This week | Last week | Change |");
lines.push("|---|---:|---:|---:|");
for (const stage of STAGES) {
const now = counts[stage.key];
const before = previousCounts[stage.key];
const label = stage.approximate ? `${stage.label} †` : stage.label;
lines.push(`| ${label} | ${now} | ${before} | ${signed(now - before)} |`);
}
lines.push("");
lines.push("## Campaigns");
lines.push("");
if (report.campaigns.length === 0) {
lines.push("_No tagged traffic in the window._");
} else {
lines.push(
"Top 10 by volume, untagged excluded — the same grouping " +
"`getAnalyticsSummary.topCampaigns` returns."
);
lines.push("");
lines.push("| Source | Medium | Campaign | Rows |");
lines.push("|---|---|---|---:|");
for (const row of report.campaigns) {
lines.push(
`| ${cell(row.source)} | ${cell(row.medium)} | ${cell(row.campaign)} | ${row.count} |`
);
}
}
lines.push("");
lines.push(
`${report.taggedRows} of ${traffic.analyticsRows} rows in the window carry ` +
`campaign tags; the rest are untagged and therefore unattributed.`
);
lines.push("");
lines.push("## The one thing that moved");
lines.push("");
if (!report.mover) {
lines.push("_Nothing moved: every stage held its previous-week value._");
} else {
const mover = report.mover;
lines.push(
`**${mover.label}: ${mover.before} → ${mover.now} (${signed(mover.delta)}).**`
);
}
lines.push("");
lines.push("## Before acting on these numbers");
lines.push("");
if (traffic.analyticsRows > 0 && traffic.loopbackRows > 0) {
lines.push(
`- **${traffic.loopbackRows} of ${traffic.analyticsRows} rows in the window came from a loopback address** ` +
`(127.0.0.1 / ::1)` +
(traffic.allTimeAddressedRows > 0 &&
traffic.allTimeAddressedRows === traffic.allTimeLoopbackRows
? `; every one of the ${traffic.allTimeAddressedRows} addressed rows the table ` +
`has ever held is loopback (${traffic.allTimeRows - traffic.allTimeAddressedRows} carry no address)`
: `; across the table's history, ${traffic.allTimeLoopbackRows} of ` +
`${traffic.allTimeAddressedRows} addressed rows are loopback`) +
`. Either this host is the only audience, or the app sits behind a local proxy ` +
`and \`getRequestIP\` records the proxy, not the visitor. Either way the address ` +
`identifies nobody — "unique visitors" collapses to one host and per-channel ` +
`reach cannot be read from it.`
);
}
if (traffic.downloadRows > 0) {
lines.push(
`- **The download stage is one label, not two.** The landing-door route and the file ` +
`route that serves the DMG both write \`${NOOK_DOWNLOAD_PATH}\`, so ` +
`"download click → DMG served" cannot be split: one served click writes two rows, ` +
`a click that fails to resolve writes one, and a Sparkle update writes one. ` +
`Of this window's ${traffic.downloadRows} download rows, ` +
`${traffic.downloadLoopbackRows} are loopback and ` +
`${traffic.downloadNonBrowserRows} are not browser user agents (curl, Sparkle, ` +
`link-preview crawlers).`
);
}
lines.push(
`- **Checkout start is not recorded.** \`POST /api/the-nook/checkout\` — the call that ` +
`actually creates the Stripe session — writes nothing to analytics; the only trace is ` +
`the /checkout page view reported above, which is a page, not a started checkout.`
);
lines.push(
`- **There is no host dimension.** \`VisitorAnalytics\` records a path, not the site, so ` +
`the Nook's landing page (\`/\` on nook.freno.me) is the same row as freno.me's root. ` +
`"visit" above is site-wide page views, not Nook visits.`
);
lines.push(
`- **No conversion rate before the cohort matures.** Trial→paid belongs to the mature ` +
`Day 28–35 read (FRE-197); a ratio from a partial cohort is not a metric.`
);
lines.push("");
lines.push(
"† Proxy or merged label — see *Before acting on these numbers* above."
);
lines.push("");
lines.push("## Sources");
lines.push("");
lines.push("- `VisitorAnalytics` (freno.me Turso DB) — visits, downloads, campaigns.");
lines.push("- `trials`, `licenses`, `activations` (the-nook Turso DB) — the paid funnel.");
lines.push(`- Generated ${report.generatedAt}.`);
lines.push("");
return lines.join("\n");
}