# Repository Structure & File Tree Guide

This document explains the purpose and organization of each file and directory in the ransomware analysis pipeline repository.

## Top-Level Files

### Core Documentation
- **README.md** - Main project documentation with quickstart, architecture, and usage instructions
- **SAFETY.md** - Safety guidelines and legal compliance requirements for defensive research only
- **LOGGING.md** - Logging policies and redaction rules to prevent credential/PII leakage
- **README2.md** - This file tree guide (current document)

### Configuration & Dependencies
- **pyproject.toml** - Python project configuration with pinned dependencies for reproducibility
- **.gitignore** - Git ignore patterns for Python artifacts, logs, and sensitive files

## Configuration Directory (`config/`)

Runtime-editable configuration files that are synced to S3 during deployment:

- **features.yaml** - Feature extraction rules with keywords and regex patterns for ransomware indicators
  - Categories: locking, encryption, exfiltration, ransom_note, initial_access, lateral_movement, C2, ransom_currency, double_extortion, file_extensions_targeted, ransom_family_names
- **suggestions.yaml** - Maps detected features to defensive action recommendations
- **allowlist.txt** - Domain allowlist for crawler (public security sources only)
- **crawler-config.json** - Crawler settings: max_pages, concurrency, delays, user_agent, seed URLs

## Core Modules

### Storage Layer (`storage/`)
- **config.py** - AWS configuration dataclass and environment variable loading
- **s3_adapter.py** - S3 storage operations with tagging, lifecycle policies, and raw HTML persistence
- **dynamodb_adapter.py** - DynamoDB operations for document metadata and structured data

### Crawler Module (`crawler/`)
- **__init__.py** - Module exports
- **settings.py** - Scrapy configuration with robots.txt compliance and rate limiting
- **lambda_handler.py** - AWS Lambda entry point for crawler execution
- **spiders/__init__.py** - Scrapy spiders module
- **spiders/public_spider.py** - Main crawler spider with allowlist filtering and S3 persistence

### Content Extraction (`extractor/`)
- **__init__.py** - Module exports
- **extract.py** - Article extraction using Newspaper3k with BeautifulSoup fallback
  - Handles title, text, author, date, canonical URL extraction
  - Language detection and text normalization
- **lambda_handler.py** - Lambda handler for extraction pipeline

### Feature Extraction (`features/`)
- **__init__.py** - Module exports
- **rules.py** - Rule-based feature extraction from features.yaml configuration
- **lambda_handler.py** - Lambda handler for feature extraction
- **embeddings.py** - Optional ML embeddings using sentence-transformers (flag-driven)

### Classification (`classifier/`)
- **__init__.py** - Module exports
- **model.py** - Scikit-learn pipeline with TF-IDF vectorization and logistic regression
  - Model training, versioning, and S3 persistence
  - Explainability through feature weights
- **lambda_handler.py** - Lambda handler for classification inference

## AWS Infrastructure (`aws/`)

### Deployment & Management
- **deploy.py** - AWS resource creation script
  - Creates S3 buckets with lifecycle policies
  - Sets up DynamoDB tables with on-demand billing
  - Creates AWS Budgets with SNS alerts
  - Tags all resources for tracking
- **teardown.py** - Complete resource cleanup script
  - Deletes all tagged resources (S3, DynamoDB, etc.)
  - Ensures no orphaned resources remain

## GitHub Actions (`.github/workflows/`)

All workflows are manual-only (`workflow_dispatch`) for safety:

- **deploy.yml** - Manual deployment workflow
  - Dry-run cost estimation
  - Resource creation with budget controls
  - Requires explicit confirmation
- **destroy.yml** - Manual teardown workflow
  - Requires "I want to destroy" confirmation
  - Runs aws/teardown.py
- **run-crawl.yml** - Manual crawl execution
  - Configurable page limits and run modes (lambda/EC2)

## Documentation (`DOCS/`)

- **architecture.md** - High-level system architecture and component interactions
- **deployment.md** - IAM permissions, workflow usage, and security considerations
- **features.md** - Feature configuration editing and runtime behavior
- **testing.md** - Test organization and execution instructions

## Optional Web UI (`webui/`)

- **index.html** - Minimal static viewer for analysis results
  - Read-only interface for browsing classifications
  - Displays confidence scores, features, and suggestions
  - Designed for S3 static hosting with optional CloudFront

## Testing (`tests/`)

Unit tests for all major components:

- **test_features.py** - Feature extraction rule testing
- **test_classifier.py** - Model training and prediction testing
- **test_crawler_allowlist.py** - Crawler allowlist filtering
- **test_deploy_dryrun.py** - Deployment cost estimation testing

## Examples (`examples/`)

Sample data and expected outputs:

- **sample1.txt** - Example article text for testing
- **expected1.json** - Expected JSON output format showing:
  - URL, title, date, extracted text
  - Feature detection results
  - Classification with confidence
  - Defensive suggestions
  - Model version and S3 paths

## Module Dependencies

### Core Dependencies (requirements.txt)
- **boto3** - AWS SDK for Python
- **scrapy** - Web crawling framework
- **beautifulsoup4** - HTML parsing
- **newspaper3k** - Article extraction
- **scikit-learn** - Machine learning pipeline
- **pyyaml** - Configuration file parsing

### Optional Dependencies
- **sentence-transformers** - ML embeddings (flag-driven)
- **spacy** - NLP processing (flag-driven)
- **pytest** - Testing framework
- **moto** - AWS service mocking for tests

## Data Flow Architecture

```
Seeds/Allowlist → Crawler → S3 Raw → Extractor → DynamoDB → Features → S3 → Classifier → DynamoDB/S3 → Web UI
```

1. **Crawler** reads allowlist and seeds, respects robots.txt, stores raw HTML in S3
2. **Extractor** processes HTML, extracts article content, stores metadata in DynamoDB
3. **Features** applies rule-based extraction using features.yaml, stores results in S3
4. **Classifier** runs ML inference, stores predictions with explainability in DynamoDB/S3
5. **Web UI** provides read-only access to results

## Security & Compliance

- All AWS resources are tagged for tracking and cleanup
- Budget controls prevent cost overruns ($5 default cap)
- Logging redacts secrets and PII
- Only public, non-sensitive sources are crawled
- No malware samples or private data collection

## Cost Controls

- S3 lifecycle policies (30-day archival/deletion)
- DynamoDB on-demand billing
- Small Lambda sizes (128-512 MB)
- Optional EC2 (t3.micro) with explicit confirmation
- AWS Budgets with SNS alerts

This structure provides a complete, cloud-first ransomware analysis pipeline that can be deployed, configured, and torn down safely while maintaining cost controls and security best practices.
