Testing

Unit Tests
- Crawler: mock HTTP responses; ensure allowlist filtering and S3 key schema
- Extractor: feed sample HTML; verify title/text extraction and language detection
- Features: verify keyword/regex matching from features.yaml
- Classifier: train small model on toy data; verify predictions and weights
- AWS scripts: run deploy in dry-run (CONFIRM_DEPLOY=false), teardown with test tags

Running Tests
pip install -e .[dev]
pytest -q


