Kordant/tasks/core-services-implementation/10-hometitle-county-scrapers.md

10. County Recorder Web Scrapers for Top 100 US Counties

meta: id: core-services-10 feature: core-services-implementation priority: P2 depends_on: [core-services-09] tags: [hometitle, scraping, county-records, fallback, coverage]

objective:

• Build Playwright-based web scrapers for county recorder websites in the top 100 US counties by population, providing a fallback for counties not covered by Attom API and reducing API costs.

deliverables:

• Scrapers for 100 US county recorder websites (starting with top 50)
• Unified property record parser that normalizes disparate HTML formats
• Fallback logic: Attom API → county scraper → manual request (in order)
• scraper health monitoring and breakage detection

steps:

1. Identify top 100 US counties by population (start with top 50):
   • Los Angeles County, CA; Cook County, IL; Harris County, TX; Maricopa County, AZ; etc.
2. Research each county's recorder website:
   • Search URL pattern (usually https://{county}.gov/recorder or similar)
   • Record search interface (by owner name, parcel ID, or address)
   • Result format (HTML table, PDF, JSON API, proprietary system)
3. Create hometitle/county-scrapers/ directory with one module per county
4. Implement base scraper interface:
   • searchByAddress(address): Promise<CountyRecord[]>
   • searchByParcelId(parcelId): Promise<CountyRecord | null>
   • parseResults(html): CountyRecord[]
5. Implement scrapers for each county using Playwright:
   • Navigate to recorder website
   • Fill search form (address or parcel ID)
   • Submit and wait for results
   • Parse HTML table or detail page
   • Extract: owner name, deed date, tax info, lien status
6. Implement unified parseDeedRecords(html) that handles common formats:
   • HTML tables with standard columns
   • Detail pages with labeled fields
   • PDF records (download + text extraction)
7. Add fallback chain in scanner.ts:
   • Try Attom API first (fastest, most reliable)
   • If Attom returns null/empty, try county scraper
   • If scraper fails, queue for manual request (email to user)
8. Add scraper monitoring:
   • Track success/failure rate per county
   • Alert when >20% of scrapers fail in 24h (site changes)
   • Auto-disable broken scrapers, fall back to Attom/manual
9. Handle rate limiting:
   • Throttle requests to county sites (max 1 req/5 sec per county)
   • Use residential proxies if county blocks datacenter IPs
   • Respect robots.txt and terms of service

tests:

• Unit: Mock HTML responses for common county formats, verify parser normalization
• Integration: Test 5 representative county scrapers against live sites
• E2E: Property in county without Attom coverage → scraper fetches real data → snapshot created

acceptance_criteria:

• 50+ county recorder scrapers implemented and tested against live sites
• parseDeedRecords() parses real HTML and returns structured CountyRecord objects
• Fallback chain works: Attom → county scraper → manual request
• Each scraper handles the county's specific search interface and result format
• Rate limiting respects county sites (max 1 request per 5 seconds)
• Broken scrapers are auto-detected within 24 hours and disabled
• Scraper success rate > 70% across all implemented counties
• Property records from scrapers match Attom data quality (owner name, deed date, liens)
• Failed scraper attempts fall back to manual queue with user notification
• No county site is overwhelmed by scraping (responsible rate limits)

validation:

• Run vitest run hometitle.test.ts — extended tests for county scrapers
• Manual: Search property in Cook County IL, verify scraper returns real owner data
• Check fallback: Disable Attom API key, trigger scan, verify county scraper activates
• Monitor health: Dashboard shows per-county scraper success rate

notes:

• County recorder sites are notoriously fragile — expect 30–40% of scrapers to break per quarter
• Many counties use proprietary systems (e.g., Tyler Technologies, Fidlar Technologies) with complex JavaScript
• Some counties require payment per record ($1–$5) — flag these for manual processing
• Consider partnering with Attom for counties they don't cover rather than building scrapers
• Legal: Ensure scraping complies with each county's terms of service and state public records laws
• The existing parseDeedRecords() currently logs "not yet implemented" — replace with real parsing