agent-security-engineer/freno-dev
1
0
Fork 0
forked from Mike/freno-dev
Code Pull Requests Activity

analytics: make each weekly funnel stage separately measurable (FRE-366)

Browse Source 
Gap 1 — click and served share one label: trackDownload now takes a
stage ("click" | "served") carried on a new VisitorAnalytics.stage
column; the Nook door and the tRPC proxy record clicks, the DMG file
route records served. summariseDownloads() and the /analytics downloads
block are untouched — the stage is a separate column, so the path
grouping cannot change.

Gap 2 — checkout start records nothing: the checkout route records a
created Stripe session through the shared trackRequest writer (same
referrer, campaign, country dimensions as every door) under its own
path, and STAGES.checkout reads it instead of the /checkout page proxy.

Gap 3 — loopback addresses, no host dimension: every writer now
resolves the client address through resolveClientIp, which trusts
x-forwarded-for only when the direct peer is the loopback production
proxy (trust boundary stated in request-client.ts), and records the
host on each row through the same site-context extractor routing uses.
stage and host reach the table through VISITOR_ANALYTICS_ADDED_COLUMNS;
the weekly job applies the same bootstrap before reading so it renders
seven stages against a database no deploy has migrated yet.

The weekly read splits downloadClick/dmgServed, scopes visits to
nook.freno.me, and computes its caveats from the window: pre-instrumentation
rows are stated and excluded, and the loopback warning now fires only
when rows recorded since the instrumentation still carry it.
This commit is contained in:
Paperclip Agent
2026-10-29 17:02:39 +00:00
parent 89f56fc78a
commit 39d874a7e2
19 changed files with 643 additions and 221 deletions
Download Patch File Download Diff File Expand all files Collapse all files

285
src/server/weekly-metrics.ts
View File

@@ -18,20 +18,33 @@
* whose `.env` carries nothing else. The SQL here is the same shape as the
* analytics summaries, so the weekly read and the dashboard cannot drift
* into different label sets.
* - Every stage carries what it actually measured. Two of the seven agreed
* stages share one label in production today and a third is only a page-view
* proxy; the report states that next to the number instead of inventing a
* figure (see `renderWeeklyMarkdown`).
* - Every stage is measured by its own label. FRE-366 gave the funnel that:
* the download door and the DMG file route record distinct `stage` values
* (`click` / `served`), a created Stripe checkout session records its own
* row, and every row carries the real client address and the host it
* addressed, so the visit stage scopes to the Nook. Rows that predate
* that instrumentation are stated next to the numbers, never silently
* counted (see `renderWeeklyMarkdown`).
*
* No `~/env`, no `~/server/database`: the caller owns the connections, the
* tests own an in-memory SQLite, and the script owns the libsql clients.
*/
import { DOWNLOAD_PATH_PREFIX } from "~/lib/download-asset";
import {
VISITOR_ANALYTICS_ADDED_COLUMNS,
VISITOR_ANALYTICS_DDL
} from "~/server/analytics-schema";
/** The one asset key the Nook lands under — `downloadPath("thenook")`. */
export const NOOK_DOWNLOAD_PATH = `${DOWNLOAD_PATH_PREFIX}thenook`;
/** The host the visit stage scopes to. Recorded per row since FRE-366. */
export const NOOK_HOST = "nook.freno.me";
/** The path a created Stripe checkout session records (FRE-366). */
export const NOOK_CHECKOUT_PATH = "/api/the-nook/checkout";
/** An `execute`-shaped connection: the libsql client and the test adapter both fit. */
export interface MetricsConn {
execute(query: {
@@ -49,7 +62,8 @@ export interface Connections {
export type StageKey =
| "visit"
| "download"
| "downloadClick"
| "dmgServed"
| "trial"
| "checkout"
| "purchase"
@@ -58,35 +72,28 @@ export type StageKey =
export interface StageDef {
key: StageKey;
label: string;
/** True when the number is a proxy or a merged label, not a clean stage. */
approximate: boolean;
}
/**
* The seven agreed stages. `download` deliberately carries two of them: the
* door and the file route both write `/api/downloads/thenook`, so a click and
* the DMG it fetched are one label in the data (proved in the report's caveats).
* The seven agreed stages, each with its own label in the data:
* `downloadClick` and `dmgServed` split what used to be one merged download
* label (the `stage` column, FRE-366), and `checkout` reads created Stripe
* sessions instead of the /checkout page-view proxy.
*/
export const STAGES: StageDef[] = [
{ key: "visit", label: "visit (page views)", approximate: false },
{
key: "download",
label: "download click → DMG served (one label)",
approximate: true
},
{ key: "trial", label: "trial started", approximate: false },
{
key: "checkout",
label: "checkout page visit",
approximate: true
},
{ key: "purchase", label: "purchase (licence issued)", approximate: false },
{ key: "activation", label: "activation", approximate: false }
{ key: "visit", label: `visit (${NOOK_HOST} page views)` },
{ key: "downloadClick", label: "download click" },
{ key: "dmgServed", label: "DMG served" },
{ key: "trial", label: "trial started" },
{ key: "checkout", label: "checkout started (Stripe session)" },
{ key: "purchase", label: "purchase (licence issued)" },
{ key: "activation", label: "activation" }
];
export interface FunnelCounts {
visit: number;
download: number;
downloadClick: number;
dmgServed: number;
trial: number;
checkout: number;
purchase: number;
@@ -103,13 +110,17 @@ export interface CampaignRow {
/** Traffic-shape facts the report has to state next to the funnel. */
export interface WindowTraffic {
analyticsRows: number;
loopbackRows: number;
/** All-time row counts, so "every row ever" is a checked claim, not a guess. */
allTimeRows: number;
allTimeAddressedRows: number;
allTimeLoopbackRows: number;
/** Rows carrying a host — written since the FRE-366 instrumentation. */
hostRows: number;
/** Rows with no host — pre-FRE-366; their address is the proxy's loopback. */
legacyRows: number;
/** Post-instrumentation rows still recording loopback: the fix is not live. */
hostLoopbackRows: number;
downloadRows: number;
downloadLoopbackRows: number;
downloadClickRows: number;
downloadServedRows: number;
/** Download rows with no stage label — they predate FRE-366 and cannot be split. */
downloadUnstagedRows: number;
downloadNonBrowserRows: number;
}
@@ -183,6 +194,28 @@ async function scalar(
return toNumber(Object.values(row)[0]);
}
/**
* Bring the analytics table up to the schema the readers expect, through the
* same idempotent bootstrap path the app server uses (`analytics.ts`): the
* DDL is a no-op against the live table and each post-hoc column is ALTERed
* in only when absent. The weekly job runs against databases the app may not
* have migrated yet — a fresh checkout, a restored backup, or production
* before the next deploy — and must render its seven stages regardless.
*/
export async function ensureAnalyticsSchema(conn: MetricsConn): Promise<void> {
await conn.execute({ sql: VISITOR_ANALYTICS_DDL });
const info = await conn.execute({
sql: `PRAGMA table_info(VisitorAnalytics)`
});
const present = new Set(info.rows.map((row) => String((row as any).name)));
for (const column of VISITOR_ANALYTICS_ADDED_COLUMNS) {
if (!present.has(column.name)) {
await conn.execute({ sql: column.ddl });
}
}
}
async function countInWindow(
conn: MetricsConn,
sql: string,
@@ -200,30 +233,43 @@ export async function readFunnel(
const { analytics, nook } = conns;
const { start, end } = window;
const visit = await countInWindow(
// Scoped to the Nook host (FRE-366): before the host column existed, a
// page view of `/` on nook.freno.me was the same row as freno.me's root.
// Pre-instrumentation rows carry no host and are excluded — the traffic
// section states how many.
const visit = await scalar(
analytics,
`SELECT COUNT(*) AS n FROM VisitorAnalytics
WHERE created_at >= ? AND created_at < ?
AND path NOT LIKE '/api/%'`,
start,
end
AND path NOT LIKE '/api/%'
AND host = ?`,
[start, end, NOOK_HOST]
);
const download = await scalar(
const downloadClick = await scalar(
analytics,
`SELECT COUNT(*) AS n FROM VisitorAnalytics
WHERE created_at >= ? AND created_at < ?
AND path = ? AND stage = 'click'`,
[start, end, NOOK_DOWNLOAD_PATH]
);
const dmgServed = await scalar(
analytics,
`SELECT COUNT(*) AS n FROM VisitorAnalytics
WHERE created_at >= ? AND created_at < ?
AND path = ? AND stage = 'served'`,
[start, end, NOOK_DOWNLOAD_PATH]
);
// Created Stripe checkout sessions, recorded by the checkout route itself
// (FRE-366) — not the /checkout page view that used to proxy for them.
const checkout = await scalar(
analytics,
`SELECT COUNT(*) AS n FROM VisitorAnalytics
WHERE created_at >= ? AND created_at < ?
AND path = ?`,
[start, end, NOOK_DOWNLOAD_PATH]
);
const checkout = await countInWindow(
analytics,
`SELECT COUNT(*) AS n FROM VisitorAnalytics
WHERE created_at >= ? AND created_at < ?
AND (path LIKE '/checkout%' OR path LIKE '/nook/checkout%')`,
start,
end
[start, end, NOOK_CHECKOUT_PATH]
);
// The licence DB stores ISO-8601 (`2026-10-28T20:28:46.806Z`); the analytics
@@ -252,7 +298,8 @@ export async function readFunnel(
return {
visit,
download,
downloadClick,
dmgServed,
trial,
checkout,
purchase,
@@ -261,9 +308,11 @@ export async function readFunnel(
}
/**
* Traffic shape for one window: how much of it came from this host, and how
* much of the download label was automation. Without these two facts the
* numbers read as demand, which they are not.
* Traffic shape for one window: how much of it predates the FRE-366
* instrumentation (no host, proxy's address), whether rows recorded since
* still see loopback (which would mean the forwarded-header fix is not
* live), and how the download label splits. Without these facts the numbers
* read as demand, which they are not.
*/
export async function readTraffic(
conns: Connections,
@@ -273,25 +322,6 @@ export async function readTraffic(
const { start, end } = window;
const placeholders = LOOPBACK.map(() => "?").join(", ");
const allTimeRows = await scalar(
analytics,
`SELECT COUNT(*) AS n FROM VisitorAnalytics`,
[]
);
const allTimeAddressedRows = await scalar(
analytics,
`SELECT COUNT(*) AS n FROM VisitorAnalytics WHERE ip_address IS NOT NULL`,
[]
);
const allTimeLoopbackRows = await scalar(
analytics,
`SELECT COUNT(*) AS n FROM VisitorAnalytics
WHERE ip_address IN (${placeholders})`,
[...LOOPBACK]
);
const analyticsRows = await countInWindow(
analytics,
`SELECT COUNT(*) AS n FROM VisitorAnalytics
@@ -300,10 +330,27 @@ export async function readTraffic(
end
);
const loopbackRows = await scalar(
const hostRows = await scalar(
analytics,
`SELECT COUNT(*) AS n FROM VisitorAnalytics
WHERE created_at >= ? AND created_at < ?
AND host IS NOT NULL`,
[start, end]
);
const legacyRows = await scalar(
analytics,
`SELECT COUNT(*) AS n FROM VisitorAnalytics
WHERE created_at >= ? AND created_at < ?
AND host IS NULL`,
[start, end]
);
const hostLoopbackRows = await scalar(
analytics,
`SELECT COUNT(*) AS n FROM VisitorAnalytics
WHERE created_at >= ? AND created_at < ?
AND host IS NOT NULL
AND ip_address IN (${placeholders})`,
[start, end, ...LOOPBACK]
);
@@ -315,12 +362,25 @@ export async function readTraffic(
[start, end, NOOK_DOWNLOAD_PATH]
);
const downloadLoopbackRows = await scalar(
const downloadClickRows = await scalar(
analytics,
`SELECT COUNT(*) AS n FROM VisitorAnalytics
WHERE created_at >= ? AND created_at < ? AND path = ?
AND ip_address IN (${placeholders})`,
[start, end, NOOK_DOWNLOAD_PATH, ...LOOPBACK]
WHERE created_at >= ? AND created_at < ? AND path = ? AND stage = 'click'`,
[start, end, NOOK_DOWNLOAD_PATH]
);
const downloadServedRows = await scalar(
analytics,
`SELECT COUNT(*) AS n FROM VisitorAnalytics
WHERE created_at >= ? AND created_at < ? AND path = ? AND stage = 'served'`,
[start, end, NOOK_DOWNLOAD_PATH]
);
const downloadUnstagedRows = await scalar(
analytics,
`SELECT COUNT(*) AS n FROM VisitorAnalytics
WHERE created_at >= ? AND created_at < ? AND path = ? AND stage IS NULL`,
[start, end, NOOK_DOWNLOAD_PATH]
);
// Anything whose UA is not a browser engine: curl, Sparkle's updater,
@@ -335,12 +395,13 @@ export async function readTraffic(
return {
analyticsRows,
loopbackRows,
allTimeRows,
allTimeAddressedRows,
allTimeLoopbackRows,
hostRows,
legacyRows,
hostLoopbackRows,
downloadRows,
downloadLoopbackRows,
downloadClickRows,
downloadServedRows,
downloadUnstagedRows,
downloadNonBrowserRows
};
}
@@ -483,8 +544,7 @@ export function renderWeeklyMarkdown(report: WeeklyReport): string {
for (const stage of STAGES) {
const now = counts[stage.key];
const before = previousCounts[stage.key];
const label = stage.approximate ? `${stage.label} †` : stage.label;
lines.push(`| ${label} | ${now} | ${before} | ${signed(now - before)} |`);
lines.push(`| ${stage.label} | ${now} | ${before} | ${signed(now - before)} |`);
}
lines.push("");
@@ -528,58 +588,45 @@ export function renderWeeklyMarkdown(report: WeeklyReport): string {
lines.push("## Before acting on these numbers");
lines.push("");
if (traffic.analyticsRows > 0 && traffic.loopbackRows > 0) {
if (traffic.legacyRows > 0) {
lines.push(
`- **${traffic.loopbackRows} of ${traffic.analyticsRows} rows in the window came from a loopback address** ` +
`(127.0.0.1 / ::1)` +
(traffic.allTimeAddressedRows > 0 &&
traffic.allTimeAddressedRows === traffic.allTimeLoopbackRows
? `; every one of the ${traffic.allTimeAddressedRows} addressed rows the table ` +
`has ever held is loopback (${traffic.allTimeRows - traffic.allTimeAddressedRows} carry no address)`
: `; across the table's history, ${traffic.allTimeLoopbackRows} of ` +
`${traffic.allTimeAddressedRows} addressed rows are loopback`) +
`. Either this host is the only audience, or the app sits behind a local proxy ` +
`and \`getRequestIP\` records the proxy, not the visitor. Either way the address ` +
`identifies nobody — "unique visitors" collapses to one host and per-channel ` +
`reach cannot be read from it.`
`- **${traffic.legacyRows} of ${traffic.analyticsRows} rows in the window predate the FRE-366 instrumentation.** ` +
`They carry no host, and their recorded address is the production proxy's loopback ` +
`(127.0.0.1 / ::1) — the writers then recorded the direct peer, not the visitor. ` +
`They are excluded from every host-scoped stage above (visits, and the download ` +
`click/served split), and "unique visitors" over them still collapses to one host. ` +
`Rows recorded since carry the real client address and the site they addressed.`
);
}
if (traffic.hostLoopbackRows > 0) {
lines.push(
`- **${traffic.hostLoopbackRows} rows recorded since the instrumentation still carry a ` +
`loopback address** — the forwarded-header fix is not taking effect in production ` +
`(or the proxy is not setting \`x-forwarded-for\`). Do not trust per-channel reach ` +
`until this is zero.`
);
}
if (traffic.downloadRows > 0) {
lines.push(
`- **The download stage is one label, not two.** The landing-door route and the file ` +
`route that serves the DMG both write \`${NOOK_DOWNLOAD_PATH}\`, so ` +
`"download click → DMG served" cannot be split: one served click writes two rows, ` +
`a click that fails to resolve writes one, and a Sparkle update writes one. ` +
`Of this window's ${traffic.downloadRows} download rows, ` +
`${traffic.downloadLoopbackRows} are loopback and ` +
`${traffic.downloadNonBrowserRows} are not browser user agents (curl, Sparkle, ` +
`link-preview crawlers).`
`- **Download rows are automation-heavy.** Of this window's ${traffic.downloadRows} ` +
`download rows, ${traffic.downloadClickRows} are clicks and ` +
`${traffic.downloadServedRows} are served files` +
(traffic.downloadUnstagedRows > 0
? `; the remaining ${traffic.downloadUnstagedRows} predate the click/served stage ` +
`label and cannot be split, so they count in neither stage`
: "") +
`. ${traffic.downloadNonBrowserRows} of all ${traffic.downloadRows} are not browser ` +
`user agents (curl, Sparkle, link-preview crawlers).`
);
}
lines.push(
`- **Checkout start is not recorded.** \`POST /api/the-nook/checkout\` — the call that ` +
`actually creates the Stripe session — writes nothing to analytics; the only trace is ` +
`the /checkout page view reported above, which is a page, not a started checkout.`
);
lines.push(
`- **There is no host dimension.** \`VisitorAnalytics\` records a path, not the site, so ` +
`the Nook's landing page (\`/\` on nook.freno.me) is the same row as freno.me's root. ` +
`"visit" above is site-wide page views, not Nook visits.`
);
lines.push(
`- **No conversion rate before the cohort matures.** Trial→paid belongs to the mature ` +
`Day 28–35 read (FRE-197); a ratio from a partial cohort is not a metric.`
);
lines.push("");
lines.push(
"† Proxy or merged label — see *Before acting on these numbers* above."
);
lines.push("");
lines.push("## Sources");
lines.push("");
lines.push("- `VisitorAnalytics` (freno.me Turso DB) — visits, downloads, campaigns.");