# GitHub Actions Setup for Keyword-Based Crawler

This document explains how to set up GitHub Actions to automatically run your keyword-based cybersecurity crawler.

## 🚀 Workflows Created

### 1. **Scheduled Crawler** (`.github/workflows/scheduled-crawl.yml`)
- **Runs daily at 3 AM UTC**
- **Manual trigger available**
- **Keyword-based filtering enabled**
- **Intelligent content discovery**

### 2. **CI/CD Pipeline** (`.github/workflows/crawler-ci.yml`)
- **Runs on code changes**
- **Tests keyword filtering**
- **Production deployment on main branch**

### 3. **Manual Crawler** (`.github/workflows/run-crawl.yml`)
- **On-demand execution**
- **Configurable parameters**
- **Test mode available**

## 🔐 Required GitHub Secrets

Set these in your repository: **Settings → Secrets and variables → Actions**

### AWS Configuration
```
AWS_ACCESS_KEY_ID=your_aws_access_key
AWS_SECRET_ACCESS_KEY=your_aws_secret_key
AWS_DEFAULT_REGION=us-east-1
```

### S3 Buckets
```
S3_BUCKET_RAW=your-raw-bucket-name
S3_BUCKET_PROCESSED=your-processed-bucket-name
```

### DynamoDB
```
DYNAMODB_TABLE=your-dynamodb-table-name
```

## 🎯 Keyword-Based Filtering Features

The crawler now includes intelligent filtering:

### ✅ **Allowed Content**
- **High-value keywords**: `ransomware`, `lockbit`, `cl0p`, `encryption`, `bitcoin`
- **Medium-value keywords**: `malware`, `breach`, `attack`, `healthcare`, `finance`
- **Trusted domains**: `cisa.gov`, `bleepingcomputer.com`, `krebsonsecurity.com`
- **Security vendors**: `mandiant.com`, `crowdstrike.com`, `paloaltonetworks.com`

### ❌ **Blocked Content**
- Social media: `facebook.com`, `twitter.com`, `instagram.com`
- Shopping: `amazon.com`, `ebay.com`, `walmart.com`
- General news: `cnn.com`, `foxnews.com`, `bbc.com`

### 📊 **Content Scoring**
- **Relevance score**: 0.0 to 1.0 based on keyword density
- **High-value keywords**: +0.1 points each
- **Medium-value keywords**: +0.05 points each
- **Substantial content**: +0.1 bonus for long articles
- **Feature matches**: +0.05 points each

## 🧪 Testing the Workflow

### Local Testing
```bash
# Test the workflow simulation
python run_crawler_workflow.py --mode test --max-pages 5

# Test keyword filtering
python test_keyword_filtering_safe.py

# Run production simulation
python run_crawler_workflow.py --mode production --max-pages 20
```

### GitHub Actions Testing
1. **Push to main branch** → Triggers CI/CD pipeline
2. **Manual trigger** → Go to Actions tab → Run workflow
3. **Scheduled run** → Automatic daily at 3 AM UTC

## 📊 Workflow Parameters

### Crawl Modes
- **`test`**: 10 pages, conservative settings
- **`production`**: 100 pages, balanced settings  
- **`discovery`**: 200 pages, aggressive discovery

### Configurable Parameters
- **`max_pages`**: Maximum pages to crawl
- **`crawl_mode`**: Test/production/discovery
- **`CRAWLER_CONCURRENCY`**: Concurrent requests
- **`CRAWLER_PER_DOMAIN_DELAY`**: Delay between requests

## 🔍 What the Crawler Does

1. **🌐 Intelligent Discovery**: Finds new cybersecurity sources automatically
2. **🎯 Smart Filtering**: Only crawls relevant cybersecurity content
3. **📈 Quality Scoring**: Prioritizes high-value content
4. **🚫 Waste Prevention**: Blocks irrelevant domains
5. **💾 Structured Storage**: Saves content in DynamoDB and S3
6. **📊 Feature Detection**: Identifies ransomware indicators

## 📋 Expected Results

After running, you'll have:
- **Raw HTML** stored in S3 for reference
- **Processed content** in DynamoDB with extracted text
- **Feature analysis** showing ransomware indicators found
- **Relevance scores** for content quality assessment
- **Automated discovery** of new cybersecurity sources

## 🚨 Troubleshooting

### Common Issues
1. **AWS Credentials**: Ensure all secrets are set correctly
2. **Bucket Permissions**: Verify S3 bucket access
3. **DynamoDB Permissions**: Check table access
4. **Rate Limiting**: Adjust delays if getting blocked

### Logs and Debugging
- Check **Actions** tab for workflow logs
- Use **test mode** for debugging
- Review **artifact uploads** for detailed logs

## 🎉 Benefits

- **🔄 Automated**: Runs daily without manual intervention
- **🎯 Intelligent**: Finds relevant content automatically
- **📈 Scalable**: Discovers new sources as they emerge
- **🛡️ Secure**: Uses GitHub Secrets for credentials
- **📊 Monitored**: Full logging and result tracking
- **⚡ Efficient**: Only processes relevant cybersecurity content

Your crawler is now a fully automated, intelligent cybersecurity intelligence gathering system!
