Architecture

Components
- Crawler (Lambda by default; optional EC2 for long runs): Fetches pages from allowlisted domains, respects robots.txt, stores raw HTML in S3.
- Extractor (Lambda): Parses HTML to article text and metadata; writes structured records to DynamoDB (or S3 JSON if configured).
- Features (Lambda): Applies rule-based extraction using features.yaml from S3.
- Classifier (Lambda): Uses scikit-learn model to classify documents; saves results to DynamoDB/S3 with explainability.
- Storage: S3 and DynamoDB adapters for consistent IO, tagging, and lifecycle.
- Web UI (Optional): Static site in S3 (optional CloudFront) to browse results.

Data Flow
Seeds/Allowlist -> Crawler -> S3 Raw -> Extractor -> DynamoDB/S3 JSON -> Features -> S3/DynamoDB -> Classifier -> DynamoDB/S3 -> Web UI

Cost Guards
- AWS Budget ($5 default) and SNS alert
- S3 lifecycle to archive/delete after 30 days
- On-demand DynamoDB; small Lambda memory and timeouts

Configuration
- Runtime-editable files in S3: features.yaml, suggestions.yaml, allowlist.txt, crawler-config.json
- Local repo copies are synced on deploy and may be edited for defaults


