# LLM Integration for Cybersecurity Analysis

This system retrieves processed JSON files from S3 and feeds them into LLMs for advanced feature matching and threat intelligence analysis.

## 🎯 **What This System Does**

1. **📁 Retrieves JSON files** from your S3 processed bucket
2. **📦 Batches content** for efficient LLM processing
3. **🤖 Analyzes with LLM** for advanced threat intelligence
4. **📊 Extracts insights** about ransomware families, attack techniques, and indicators
5. **💾 Saves results** for further analysis

## 🚀 **Quick Start**

### **1. Basic Analysis (Mock Mode)**
```bash
# Run with mock LLM (no API keys needed)
python run_llm_analysis.py --limit 5 --batch-size 2 --provider mock
```

### **2. OpenAI Analysis**
```bash
# Set your OpenAI API key
export OPENAI_API_KEY="your-api-key-here"

# Run with OpenAI GPT-4
python run_llm_analysis.py --limit 10 --batch-size 3 --provider openai
```

### **3. Anthropic Analysis**
```bash
# Set your Anthropic API key
export ANTHROPIC_API_KEY="your-api-key-here"

# Run with Claude
python run_llm_analysis.py --limit 10 --batch-size 3 --provider anthropic
```

## 📁 **Files Created**

### **Core Components**
- **`process_s3_json_for_llm.py`** - Retrieves and prepares JSON files from S3
- **`llm_feature_matcher.py`** - LLM integration and analysis
- **`run_llm_analysis.py`** - Complete pipeline orchestrator
- **`llm_processor/lambda_handler.py`** - AWS Lambda version

### **Generated Files**
- **`llm_analysis_data_TIMESTAMP.json`** - Prepared data for LLM
- **`complete_llm_analysis_TIMESTAMP.json`** - Final analysis results

## 🔧 **Configuration Options**

### **Environment Variables**
```bash
# S3 Configuration
S3_BUCKET_PROCESSED=your-processed-bucket
AWS_ACCESS_KEY_ID=your-access-key
AWS_SECRET_ACCESS_KEY=your-secret-key
AWS_DEFAULT_REGION=us-east-1

# LLM Configuration
OPENAI_API_KEY=your-openai-key
ANTHROPIC_API_KEY=your-anthropic-key

# Analysis Configuration
LLM_ANALYSIS_LIMIT=20        # Max files to process
LLM_BATCH_SIZE=5             # Documents per batch
```

### **Command Line Options**
```bash
--bucket BUCKET_NAME         # S3 bucket (defaults to processed bucket)
--limit N                    # Max files to process (default: 10)
--batch-size N               # Documents per batch (default: 3)
--provider {openai|anthropic|mock}  # LLM provider (default: mock)
--output FILENAME            # Output file (default: auto-generated)
```

## 📊 **Analysis Output**

The system provides comprehensive analysis including:

### **Threat Intelligence**
- **Ransomware families** identified
- **Attack techniques** and TTPs
- **Target industries** and organizations
- **Technical indicators** (file extensions, encryption methods)
- **Timeline information** and events
- **Threat actor** identification

### **Content Analysis**
- **Language detection** across documents
- **Domain analysis** of sources
- **Feature frequency** analysis
- **Relevance scoring** and prioritization

### **JSON Output Structure**
```json
{
  "pipeline_metadata": {
    "created_at": "2025-01-09T12:00:00",
    "bucket_name": "ransom-analysis-processed",
    "llm_provider": "openai"
  },
  "s3_analysis": {
    "total_files": 50,
    "successful_retrievals": 48,
    "high_relevance_files": 25,
    "feature_counts": {"ransomware": 15, "lockbit": 8}
  },
  "llm_results": {
    "batch_results": [
      {
        "batch_id": 0,
        "llm_response": {
          "analysis": {
            "batch_analysis": {
              "ransomware_families_found": ["LockBit", "CL0P"],
              "attack_techniques": ["Phishing", "RDP Brute Force"],
              "target_industries": ["Healthcare", "Finance"]
            }
          }
        }
      }
    ]
  }
}
```

## 🧪 **Testing**

### **Test S3 Retrieval**
```bash
python process_s3_json_for_llm.py
```

### **Test LLM Processing**
```bash
# Create test data first
python process_s3_json_for_llm.py --limit 3

# Then process with LLM
python llm_feature_matcher.py --input-file llm_analysis_data_*.json --provider mock
```

### **Test Complete Pipeline**
```bash
python run_llm_analysis.py --limit 5 --provider mock
```

## 🔄 **Integration with Crawler**

The LLM analysis system works seamlessly with your keyword-based crawler:

1. **Crawler runs** → Stores processed JSON in S3
2. **LLM processor** → Retrieves JSON files from S3
3. **LLM analysis** → Provides advanced threat intelligence
4. **Results** → Can be fed back into your analysis pipeline

## 📈 **Advanced Features**

### **Batch Processing**
- **Efficient batching** for large datasets
- **Configurable batch sizes** for different LLM limits
- **Error handling** for failed batches

### **Multiple LLM Support**
- **OpenAI GPT-4** for comprehensive analysis
- **Anthropic Claude** for detailed reasoning
- **Mock mode** for testing without API costs

### **Content Optimization**
- **Text truncation** for LLM token limits
- **Relevance filtering** to focus on high-value content
- **Language detection** for multilingual analysis

## 🛠️ **Customization**

### **Custom Prompts**
Modify `create_llm_analysis_prompt()` in `process_s3_json_for_llm.py` to customize the analysis prompt for your specific needs.

### **Custom Features**
Update the feature keywords list in `llm_feature_matcher.py` to focus on specific threat indicators.

### **Custom Output Format**
Modify the JSON output structure in `run_llm_analysis.py` to match your downstream analysis tools.

## 🚨 **Troubleshooting**

### **Common Issues**
1. **No files found**: Check S3 bucket name and permissions
2. **LLM API errors**: Verify API keys and quotas
3. **JSON parsing errors**: Check LLM response format
4. **Memory issues**: Reduce batch size or limit

### **Debug Mode**
```bash
# Enable debug logging
export LOG_LEVEL=DEBUG
python run_llm_analysis.py --limit 3 --provider mock
```

## 🎯 **Benefits**

- **🤖 Advanced Analysis**: LLM-powered threat intelligence extraction
- **📊 Structured Output**: JSON format for easy integration
- **🔄 Automated Pipeline**: End-to-end processing from S3 to insights
- **💡 Actionable Intelligence**: Identifies specific ransomware families and techniques
- **📈 Scalable**: Handles large volumes of processed content
- **🔧 Flexible**: Supports multiple LLM providers and configurations

Your processed JSON files from the crawler are now ready for advanced LLM analysis to extract detailed threat intelligence insights!
