# GitHub Actions LLM Integration

This document explains how to use the GitHub Actions workflows for LLM analysis of your cybersecurity crawler data.

## 🚀 **Available Workflows**

### **1. LLM Analysis Pipeline** (`.github/workflows/llm-analysis.yml`)
**Purpose**: Run LLM analysis on processed JSON files from S3

**Features**:
- ✅ **Manual trigger** with configurable parameters
- ✅ **Multiple LLM providers** (OpenAI, Anthropic, Mock)
- ✅ **S3 connection testing** before analysis
- ✅ **Comprehensive result verification**
- ✅ **Threat intelligence extraction**

### **2. Scheduled LLM Analysis** (`.github/workflows/scheduled-llm-analysis.yml`)
**Purpose**: Automated LLM analysis with smart scheduling

**Features**:
- ✅ **Smart data checking** - only runs if new data is available
- ✅ **Multiple analysis modes** (quick, standard, comprehensive)
- ✅ **Threat intelligence summary** generation
- ✅ **Automated reporting**

### **3. Enhanced Crawler Workflow** (`.github/workflows/run-crawl.yml`)
**Purpose**: Crawler with optional LLM analysis integration

**Features**:
- ✅ **Crawl + LLM analysis** in one workflow
- ✅ **Automatic LLM processing** of newly crawled data
- ✅ **Integrated reporting**

## 🔧 **Setup Requirements**

### **GitHub Secrets Required**

#### **AWS Configuration** (Required for all workflows)
```
AWS_ACCESS_KEY_ID=your_aws_access_key
AWS_SECRET_ACCESS_KEY=your_aws_secret_key
AWS_DEFAULT_REGION=us-east-1
S3_BUCKET_PROCESSED=your-processed-bucket-name
```

#### **LLM API Keys** (Required for real LLM providers)
```
OPENAI_API_KEY=your_openai_api_key
ANTHROPIC_API_KEY=your_anthropic_api_key
```

## 🎯 **How to Use**

### **Option 1: Manual LLM Analysis**

1. **Go to Actions** → **LLM Analysis Pipeline**
2. **Click "Run workflow"**
3. **Configure parameters**:
   - **Max files**: Number of files to analyze (default: 20)
   - **Batch size**: Documents per batch (default: 3)
   - **LLM provider**: Choose your provider
   - **Bucket name**: Leave empty for default
4. **Click "Run workflow"**

### **Option 2: Crawler + LLM Analysis**

1. **Go to Actions** → **Run Keyword-Based Crawler**
2. **Click "Run workflow"**
3. **Configure parameters**:
   - **Max pages**: Pages to crawl
   - **Run LLM analysis**: ✅ Check this box
   - **LLM provider**: Choose your provider
4. **Click "Run workflow"**

### **Option 3: Scheduled Analysis**

1. **Go to Actions** → **Scheduled LLM Analysis**
2. **Click "Run workflow"**
3. **Configure parameters**:
   - **Analysis mode**: quick/standard/comprehensive
   - **LLM provider**: Choose your provider
4. **Click "Run workflow"**

## 📊 **Analysis Modes**

### **Quick Mode**
- **Files**: 10
- **Batch size**: 2
- **Use case**: Fast analysis of recent data

### **Standard Mode**
- **Files**: 25
- **Batch size**: 3
- **Use case**: Regular analysis of recent data

### **Comprehensive Mode**
- **Files**: 50
- **Batch size**: 5
- **Use case**: Deep analysis of all available data

## 🤖 **LLM Providers**

### **Mock Mode** (No API Key Required)
- ✅ **Free to use**
- ✅ **Good for testing**
- ❌ **Returns mock data only**

### **OpenAI GPT-4** (Requires API Key)
- ✅ **High-quality analysis**
- ✅ **Good for complex threat intelligence**
- 💰 **Costs API credits**

### **Anthropic Claude** (Requires API Key)
- ✅ **Excellent reasoning**
- ✅ **Good for detailed analysis**
- 💰 **Costs API credits**

## 📈 **What You Get**

### **Analysis Results**
- **Threat intelligence** extracted from your data
- **Ransomware families** identified
- **Attack techniques** discovered
- **Target industries** analyzed
- **Technical indicators** found

### **Output Files**
- **`llm_analysis_results.json`** - Complete analysis results
- **`threat_intelligence_summary.json`** - Structured threat intel
- **Log files** - Detailed execution logs

### **GitHub Summary Reports**
- **Analysis statistics**
- **Key findings**
- **Success/failure status**
- **Download links for artifacts**

## 🔍 **Example Results**

### **Threat Intelligence Summary**
```json
{
  "analysis_date": "2025-01-09T12:00:00",
  "summary": {
    "total_files_retrieved": 25,
    "successful_llm_batches": 8,
    "total_documents_analyzed": 24
  },
  "threat_findings": [
    {
      "ransomware_families": ["LockBit", "CL0P"],
      "attack_techniques": ["Phishing", "RDP Brute Force"],
      "target_industries": ["Healthcare", "Finance"],
      "technical_indicators": [".locked", "bitcoin"],
      "confidence_level": 8
    }
  ]
}
```

## 🚨 **Troubleshooting**

### **Common Issues**

#### **1. No Files Found**
```
❌ No processed JSON files found - crawler may need to run first
```
**Solution**: Run the crawler workflow first to generate processed data

#### **2. API Key Issues**
```
❌ OpenAI API key not provided
```
**Solution**: Add the required API key to GitHub Secrets

#### **3. S3 Connection Failed**
```
❌ S3 connection failed: Access Denied
```
**Solution**: Check AWS credentials and bucket permissions

### **Debug Steps**
1. **Check the logs** in the workflow run
2. **Verify GitHub Secrets** are set correctly
3. **Test S3 connection** manually
4. **Run with mock provider** first to test the pipeline

## 🎉 **Benefits**

### **Automation**
- ✅ **No manual setup** required
- ✅ **Consistent analysis** every time
- ✅ **Scalable processing**

### **Integration**
- ✅ **Works with your crawler** seamlessly
- ✅ **Uses your existing AWS infrastructure**
- ✅ **Generates actionable intelligence**

### **Flexibility**
- ✅ **Multiple LLM providers** supported
- ✅ **Configurable parameters** for different needs
- ✅ **Manual and scheduled** execution options

## 🔄 **Workflow Integration**

### **Complete Pipeline**
1. **Crawler runs** → Stores processed JSON in S3
2. **LLM analysis** → Processes JSON files for threat intelligence
3. **Results stored** → Available for download and further analysis
4. **Reporting** → GitHub summary with key findings

### **Scheduling Options**
- **Manual trigger** - Run when needed
- **After crawler** - Automatic analysis of new data
- **Scheduled runs** - Regular analysis of accumulated data

Your LLM analysis is now fully integrated into GitHub Actions, providing automated threat intelligence extraction from your cybersecurity crawler data!
