i think this is the one
This commit is contained in:
@@ -0,0 +1,60 @@
|
||||
# AudiobookPipeline Project
|
||||
|
||||
**Status:** Active
|
||||
**Role:** Founding Engineer
|
||||
**Company:** FrenoCorp
|
||||
|
||||
## Current State
|
||||
|
||||
MVP pipeline development in progress. Core infrastructure complete:
|
||||
|
||||
- ✅ Clerk authentication (FRE-39)
|
||||
- ✅ Dashboard UI with job management (FRE-45)
|
||||
- ✅ File upload with S3/minio storage (FRE-31)
|
||||
- ✅ Redis queue integration (FRE-12)
|
||||
- ✅ Turso database integration
|
||||
|
||||
## Recent Completions
|
||||
|
||||
### FRE-31: File Upload with S3/minio Storage (2026-03-09)
|
||||
|
||||
Implemented complete file upload system:
|
||||
|
||||
- S3 client with minio support
|
||||
- Multipart upload for large files
|
||||
- Pre-signed URL generation
|
||||
- 100MB file size limit
|
||||
- File extension validation (.epub, .pdf, .mobi)
|
||||
- Graceful fallback when S3 not configured
|
||||
|
||||
### FRE-14: Filter Components Library (Firesoft) (2026-03-09)
|
||||
|
||||
Created reusable filter components for incident list screens:
|
||||
|
||||
- DateRangeFilter component
|
||||
- MultiSelectFilter component
|
||||
- Priority filter in FilterRow
|
||||
- Integrated into incidents/index.tsx
|
||||
|
||||
## In Progress
|
||||
|
||||
None - awaiting prioritization from board.
|
||||
|
||||
## Backlog (Assigned to Atlas)
|
||||
|
||||
- FRE-16: Optimize Batch Processing (low priority)
|
||||
- FRE-17: Add Progress Tracking to Job Processor
|
||||
- FRE-21: Implement Worker Auto-scaling
|
||||
- FRE-22: Add Integration Tests for API Endpoints
|
||||
- FRE-23: Set Up CI/CD Pipeline
|
||||
- FRE-27: Add Comprehensive Logging and Monitoring
|
||||
- FRE-28: Optimize Database Queries
|
||||
- FRE-29: Implement Caching Layer
|
||||
|
||||
## Blockers
|
||||
|
||||
- FRE-33 (CTO permissions) blocked - affects company-wide coordination
|
||||
|
||||
## Notes
|
||||
|
||||
Working independently with local task files due to Paperclip API auth issues. All completed work documented in daily notes and PARA memory.
|
||||
@@ -0,0 +1,23 @@
|
||||
- id: FRE-11
|
||||
type: task
|
||||
status: in_progress
|
||||
priority: high
|
||||
created: 2026-03-08
|
||||
owner: Atlas (Founding Engineer)
|
||||
agent_id: 38bc84c9-897b-4287-be18-bacf6fcff5cd
|
||||
|
||||
- id: dashboard-ui
|
||||
type: deliverable
|
||||
status: in_progress
|
||||
description: SolidJS dashboard component with job submission and tracking
|
||||
|
||||
- id: api-integration
|
||||
type: deliverable
|
||||
status: complete
|
||||
description: Hono API endpoints (POST /api/jobs, GET /api/jobs)
|
||||
|
||||
- id: turso-dependency
|
||||
type: blocker
|
||||
status: pending
|
||||
assigned_to: Hermes (Junior Engineer)
|
||||
description: Turso database integration required for user auth and job persistence
|
||||
@@ -0,0 +1,63 @@
|
||||
# FRE-11: Dashboard Component (MVP Sprint Week 2)
|
||||
|
||||
**Status:** Done
|
||||
**Started:** 2026-03-08
|
||||
**Completed:** 2026-03-08
|
||||
**Owner:** Atlas (Founding Engineer)
|
||||
**Company:** FrenoCorp
|
||||
|
||||
## Objective
|
||||
|
||||
Build SolidJS dashboard component for job submission and status tracking as part of MVP sprint.
|
||||
|
||||
## Scope
|
||||
|
||||
- Job submission form with file upload
|
||||
- Status dashboard showing active/completed jobs
|
||||
- Integration with Hono API endpoints
|
||||
- Real-time polling for job status updates
|
||||
|
||||
## Completed
|
||||
|
||||
### Dashboard.jsx
|
||||
|
||||
- Real-time job fetching with 5-second polling interval
|
||||
- File upload component calling POST /api/jobs
|
||||
- Job status display with color-coded badges (pending/processing/completed/failed)
|
||||
- Progress bars showing completion percentage
|
||||
- Summary cards: credits, books generated, active jobs
|
||||
- Error handling and loading states
|
||||
|
||||
### Jobs.jsx
|
||||
|
||||
- Full job list with refresh button
|
||||
- Status labels with proper formatting
|
||||
- Progress bars with percentage display
|
||||
- Empty state with navigation to Dashboard
|
||||
- Timestamp display for created_at
|
||||
|
||||
### API Enhancements (FRE-12)
|
||||
|
||||
- Added redis package for queue integration
|
||||
- POST /api/jobs enqueues to Redis 'audiobook_jobs' queue
|
||||
- GET /api/jobs/:id for individual job lookup
|
||||
- PATCH /api/jobs/:id/status for worker status updates
|
||||
- Graceful Redis fallback if not connected
|
||||
|
||||
## Testing
|
||||
|
||||
Requires local setup:
|
||||
|
||||
```bash
|
||||
docker-compose up -d redis
|
||||
npm run server
|
||||
```
|
||||
|
||||
## Dependencies
|
||||
|
||||
✅ Turso database integration complete
|
||||
✅ Redis queue integration complete (FRE-12)
|
||||
|
||||
## Notes
|
||||
|
||||
Task completed 2026-03-08. Dashboard and Jobs pages now fully functional with API integration. Ready for end-to-end testing with worker pipeline.
|
||||
@@ -0,0 +1,54 @@
|
||||
# Atomic facts for FRE-31
|
||||
|
||||
- {
|
||||
type: task,
|
||||
id: FRE-31,
|
||||
title: "Implement File Upload with S3/minio Storage",
|
||||
status: done,
|
||||
completed_on: "2026-03-09",
|
||||
assignee: Atlas,
|
||||
priority: high,
|
||||
}
|
||||
- {
|
||||
type: feature,
|
||||
name: file_upload,
|
||||
storage_backend: s3_minio,
|
||||
fallback: in_memory_mock,
|
||||
}
|
||||
- {
|
||||
type: constraint,
|
||||
name: max_file_size,
|
||||
value: 104857600,
|
||||
unit: bytes,
|
||||
display: "100MB",
|
||||
}
|
||||
- {
|
||||
type: constraint,
|
||||
name: allowed_extensions,
|
||||
values: [".epub", ".pdf", ".mobi"],
|
||||
}
|
||||
- { type: package, name: "@aws-sdk/client-s3", version: "^3.1004.0" }
|
||||
- { type: package, name: "@aws-sdk/lib-storage", version: "^3.1004.0" }
|
||||
- { type: package, name: "@aws-sdk/s3-request-presigner", version: "^3.1004.0" }
|
||||
- {
|
||||
type: endpoint,
|
||||
path: "/api/jobs",
|
||||
method: POST,
|
||||
handles: ["multipart/form-data", "application/json"],
|
||||
}
|
||||
- {
|
||||
type: module,
|
||||
path: "/home/mike/code/AudiobookPipeline/web/src/server/storage.js",
|
||||
functions:
|
||||
[
|
||||
uploadFile,
|
||||
getFileUrl,
|
||||
deleteFile,
|
||||
getUploadUrl,
|
||||
initiateMultipartUpload,
|
||||
uploadPart,
|
||||
completeMultipartUpload,
|
||||
abortMultipartUpload,
|
||||
storeFileMetadata,
|
||||
],
|
||||
}
|
||||
@@ -0,0 +1,67 @@
|
||||
# FRE-31: Implement File Upload with S3/minio Storage
|
||||
|
||||
**Status:** Done
|
||||
**Completed:** 2026-03-09
|
||||
**Owner:** Atlas (Founding Engineer)
|
||||
**Company:** FrenoCorp
|
||||
|
||||
## Objective
|
||||
|
||||
Add actual file upload support to web platform with S3/minio storage integration.
|
||||
|
||||
## Scope
|
||||
|
||||
- File upload with multipart form data
|
||||
- S3/minio integration for production
|
||||
- Graceful fallback for local development
|
||||
- 100MB file size limit enforcement
|
||||
|
||||
## Completed
|
||||
|
||||
### Storage Module (storage.js)
|
||||
|
||||
- S3 client initialization with minio support (forcePathStyle: true)
|
||||
- uploadFile() - handles Blob/File to Buffer conversion
|
||||
- getFileUrl() - returns download URLs
|
||||
- deleteFile() - removes files from storage
|
||||
- getUploadUrl() - generates pre-signed URLs for client-side uploads
|
||||
- Multipart upload support for large files (initiate/uploadPart/complete/abort)
|
||||
- storeFileMetadata() - persists file info to Turso database
|
||||
- Graceful fallback when S3 not configured (returns mock URLs)
|
||||
|
||||
### API Integration (jobs.js)
|
||||
|
||||
- POST /api/jobs handles multipart/form-data
|
||||
- File size validation (100MB limit)
|
||||
- File extension validation (.epub, .pdf, .mobi)
|
||||
- Uploads file to storage before enqueuing job
|
||||
- Stores file URL in job record
|
||||
|
||||
### Frontend (Dashboard.jsx)
|
||||
|
||||
- Sends files via FormData
|
||||
- Displays upload status
|
||||
- Error handling for failed uploads
|
||||
|
||||
## Acceptance Criteria Met
|
||||
|
||||
- ✅ File upload works with multipart form data
|
||||
- ✅ S3 integration when credentials configured
|
||||
- ✅ Graceful fallback when S3 not available
|
||||
- ✅ 100MB file size limit enforced
|
||||
|
||||
## Files Modified
|
||||
|
||||
- `/home/mike/code/AudiobookPipeline/web/src/server/storage.js` - Created
|
||||
- `/home/mike/code/AudiobookPipeline/web/src/server/api/jobs.js` - Added file validation
|
||||
- `/home/mike/code/AudiobookPipeline/web/src/routes/Dashboard.jsx` - Already integrated
|
||||
|
||||
## Dependencies
|
||||
|
||||
- @aws-sdk/client-s3
|
||||
- @aws-sdk/lib-storage
|
||||
- @aws-sdk/s3-request-presigner
|
||||
|
||||
## Notes
|
||||
|
||||
S3 not configured in .env - graceful fallback active. Set S3_ENDPOINT, S3_ACCESS_KEY, S3_SECRET_KEY, and S3_BUCKET to enable production storage.
|
||||
Reference in New Issue
Block a user