forked from Mike/freno-dev
Michael Freno
a524f23be2
The analytics page could not answer "how many downloads, of which
product". Three of the four download doors recorded nothing: the tRPC
click proxy only left the generic request row, whose path is the raw URL
with the asset buried in `?batch=1&input={"asset_name":...}`, and the S3
file route that serves Sparkle updates and the Nook app logged nothing
at all. The one door that did log (the Nook redirect) kept a private copy
of the logVisit/enrich/flush sequence, under its own path.
All doors now go through trackDownload, which writes one row per event
under /api/downloads/<asset> and flushes it inside the request — a click
is low-volume and serverless is not guaranteed to run the 30s batch
timer after the response. The file route maps its prefix allowlist to
asset keys, so a version bump stays one asset rather than one per
release. getAnalyticsSummary groups those rows into a downloads block and
the freno.me panel renders it under the overview cards.
History is recovered to the degree it exists: the Nook door's rows and
the tRPC clicks still carry enough to derive from (asset name in the URL).
scripts/backfill-downloads.ts inserts those derived rows with a
`backfill-<source id>` primary key and INSERT OR IGNORE, leaving the
originals untouched, so re-running is a no-op. Run against the live table
it added 66 rows: gaze 42, thenook 19, inputhalo 4, lineage 1.
Tests: new backfill suite (derivation, visitor detail carried over,
idempotency, empty history) and summariseDownloads cases; the Nook route
test now pins the canonical path, and the NOOK-337 allowlist audit reads
the map keys instead of quoted strings. Red-checked by stubbing the
downloads block back to zeros — the summary case fails, then passes.
Full suite matches master.
Co-Authored-By: nw-flash <noreply@nw-flash.local>
MISC-48
96 lines
3.1 KiB
TypeScript
96 lines
3.1 KiB
TypeScript
import { VISITOR_ANALYTICS_COLUMNS } from "~/server/analytics-schema";
|
|
import { downloadPath } from "~/lib/download-asset";
|
|
|
|
/**
|
|
* Backfill canonical `downloadPath(asset)` rows from download history that was
|
|
* recorded before `trackDownload` existed.
|
|
*
|
|
* Two shapes of history are recoverable, and only these two:
|
|
*
|
|
* - the Nook door's own rows (`/api/the-nook/download`), and
|
|
* - tRPC clicks, where the asset name still sits in the request URL as
|
|
* `?input={"0":{"asset_name":"gaze"}}` (percent-encoded in practice).
|
|
*
|
|
* Every other door wrote nothing, so there is nothing to derive for them.
|
|
*
|
|
* Derived rows keep the source row's timestamp, visitor, campaign and country,
|
|
* get `backfill-<source id>` as primary key and are written with
|
|
* `INSERT OR IGNORE` — so a second run adds nothing, and the original rows are
|
|
* left untouched (they are genuine API traffic, and the analytics page only
|
|
* counts the canonical path).
|
|
*/
|
|
|
|
const LEGACY_NOOK_PATH = "/api/the-nook/download";
|
|
const TRPC_DOOR_PREFIX = "/api/trpc/downloads.getDownloadUrl";
|
|
|
|
/** The two libsql calls used here, narrow enough for an in-memory test client. */
|
|
export interface DownloadBackfillConn {
|
|
execute(query: {
|
|
sql: string;
|
|
// `any[]` not `unknown[]`: the libsql client types its bind args as InArg[],
|
|
// and a narrower array type here would reject the real connection.
|
|
args?: any[];
|
|
}): Promise<{ rows: unknown[]; rowsAffected?: number }>;
|
|
}
|
|
|
|
export interface DownloadBackfillResult {
|
|
asset: string;
|
|
added: number;
|
|
}
|
|
|
|
/** One statement: copy every column of the source row, override id and path. */
|
|
const INSERT_DERIVED = `INSERT OR IGNORE INTO VisitorAnalytics (${VISITOR_ANALYTICS_COLUMNS.join(", ")})
|
|
SELECT ${VISITOR_ANALYTICS_COLUMNS.map((column) =>
|
|
column === "id"
|
|
? "'backfill-' || id"
|
|
: column === "path"
|
|
? "?"
|
|
: `"${column}"`
|
|
).join(", ")}
|
|
FROM VisitorAnalytics
|
|
WHERE id = ?`;
|
|
|
|
/** Pull the asset name back out of a logged tRPC request URL. */
|
|
export function assetFromLegacyPath(path: string): string | null {
|
|
if (path === LEGACY_NOOK_PATH) return "thenook";
|
|
|
|
let decoded = path;
|
|
try {
|
|
decoded = decodeURIComponent(path);
|
|
} catch {
|
|
// A malformed escape is not a reason to drop the row — match on the raw.
|
|
}
|
|
|
|
return /"asset_name"\s*:\s*"([^"]+)"/.exec(decoded)?.[1] ?? null;
|
|
}
|
|
|
|
export async function backfillDownloads(
|
|
conn: DownloadBackfillConn
|
|
): Promise<DownloadBackfillResult[]> {
|
|
const legacy = await conn.execute({
|
|
sql: `SELECT id, path FROM VisitorAnalytics WHERE path = ? OR path LIKE ?`,
|
|
args: [LEGACY_NOOK_PATH, `${TRPC_DOOR_PREFIX}%`]
|
|
});
|
|
|
|
const added = new Map<string, number>();
|
|
|
|
for (const row of legacy.rows) {
|
|
const { id, path } = row as { id: string; path: string };
|
|
const asset = assetFromLegacyPath(path);
|
|
if (!asset) continue;
|
|
|
|
const result = await conn.execute({
|
|
sql: INSERT_DERIVED,
|
|
args: [downloadPath(asset), id]
|
|
});
|
|
|
|
if (result.rowsAffected) {
|
|
added.set(asset, (added.get(asset) ?? 0) + result.rowsAffected);
|
|
}
|
|
}
|
|
|
|
return [...added.entries()]
|
|
.map(([asset, count]) => ({ asset, added: count }))
|
|
.sort((a, b) => b.added - a.added);
|
|
}
|