Ransomware Analysis Pipeline (Simplified, S3-Only)

IMPORTANT: Defensive research only. This project only crawls and analyzes public, non-sensitive sources (news, vendor blogs, public incident writeups). It must never be used to retrieve malware binaries, private/internal documents, credentials, or to perform intrusive scanning. See SAFETY.md.

Overview
This repository provides a simplified, defensive ransomware‑analysis pipeline that runs continuously and stores data in S3. It uses keyword-based filtering instead of allowlists and can be terminated via GitHub workflows.

Key Capabilities
- Simplified deployment using only S3 storage (no DynamoDB)
- Continuous crawler with keyword-based filtering (no allowlist required)
- Duplicate detection prevents re-crawling of existing websites
- Data stored with website names in month/day folder structure
- GitHub workflow for graceful crawler shutdown
- Daily data combination for LLM processing
- Feature extraction from editable features.yaml; rule‑based keywords/regex
- Model and configuration artifacts stored/versioned in S3; runtime‑editable configs (features.yaml, suggestions.yaml, crawler-config.json)
- Budget guard: default $5 monthly AWS Budget; all resources tagged and deletable

Safety & Compliance
- Public information only; respect robots.txt and site TOS
- No malware samples, credential harvesting, or intrusive activities
- Logging redacts secrets and PII; avoid printing full raw articles in CI logs

Architecture (Simplified)
ASCII Diagram

  [Keyword Discovery] --> [Continuous Crawler] --> [Keyword Filtering] --> [Duplicate Detection] --> [S3 Processed Bucket]
                                                      |                    |                           |
                                                      v                    v                           v
                                              [Feature Extraction] --> [Daily Data Combiner] --> [LLM-Ready JSON Files]
                                                      |                    |
                                                      v                    v
                                              [Relevance Scoring] --> [LLM Analysis Pipeline]

The crawler runs continuously with duplicate detection and can be terminated via GitHub workflows.

Cost Controls
- Default budget cap: $5/month via AWS Budgets API (configurable in deploy workflow)
- S3-only storage with lifecycle policy (archive or delete after 30 days by default)
- No DynamoDB costs

Repository Layout
- features.yaml, suggestions.yaml: editable, synced to S3
- crawler/: crawler workers and configuration adapters
- extractor/: article parsing and normalization
- features/: rule‑based feature extraction
- classifier/: training and inference with explainability
- storage/: S3 adapter only
- aws/: deployment, teardown, shutdown, and data combination scripts
- .github/workflows/: deploy.yml, destroy.yml, shutdown-crawler.yml, combine-daily-data.yml
- tests/: pytest unit tests and dry‑run AWS tests
- examples/: sample excerpts and expected outputs
- DOCS/: deeper documentation for modules and operations

Quickstart
1) Set GitHub Secrets in your repository:
   - AWS_ACCESS_KEY_ID
   - AWS_SECRET_ACCESS_KEY
   - AWS_REGION (e.g., us-east-1)
   - OPTIONAL: EMAIL_FOR_ALERTS (SNS budget/alarms)
   - OPTIONAL: Paid API keys (only if you opt‑in): OPENAI_API_KEY, ANTHROPIC_API_KEY

2) Dry‑run deploy (no AWS changes):
   - Trigger the GitHub Action: .github/workflows/deploy.yml using workflow_dispatch
   - Inputs: confirm_dry_run: true (default) — prints estimated monthly cost and IAM policies

3) Full deploy (AWS changes):
   - Re‑run the deploy workflow with confirm_deploy: true
   - Optional inputs: enable_ec2 (requires explicit confirmation), budget_cap_usd, enable_paid_apis

4) Run a crawl:
   - The crawler runs continuously and can be started via Lambda or EC2
   - Uses keyword-based discovery to find cybersecurity content organically

5) Shutdown crawler:
   - Use .github/workflows/shutdown-crawler.yml with confirm: "SHUTDOWN" — gracefully stops the crawler

6) Combine daily data:
   - Use .github/workflows/combine-daily-data.yml to combine all data for a specific date into LLM-ready format

7) Tear down:
   - Use .github/workflows/destroy.yml with confirm: "I want to destroy" — runs aws/teardown.py to remove all tagged resources

Local Development (Reduced Capability)
- You can run components locally with --no-aws flags for dry‑run and testing
- S3 persistence is stubbed locally

Editing Runtime Terms and Features
- Edit features.yaml and suggestions.yaml locally or update their S3 copies post‑deploy
- Update crawler-config.json in S3 to change crawling behavior without code changes

Paid APIs (Optional, Off by Default)
- Adapters are provided for LLMs and commercial feeds; disabled unless secrets are present and enable flags are set
- Cost warnings are shown in README and deploy workflow

Example Output (truncated)
{
  "url": "https://example.com/vendor-blog/ransomware-update",
  "title": "New Ransomware Tactics",
  "date": "2024-07-01",
  "extracted_text": "Attackers employ double extortion...",
  "features": {"double_extortion": true, "encryption": true},
  "relevant_features": ["double_extortion", "encryption"],
  "relevance_score": 0.87,
  "s3_path": "s3://<bucket>/2024/07/01/example_com_hash.json"
}

Troubleshooting
- Missing AWS permissions: review printed IAM policy in the deploy dry‑run, ensure GitHub OIDC role or access keys are set correctly
- Budget API unavailable: the workflow prints instructions to create the budget manually or aborts deploy
- Newspaper3k download failures: the extractor falls back to BeautifulSoup; ensure network egress from Lambda is allowed

License
Apache‑2.0 (or similar permissive license). See LICENSE file if provided.


