# Synthetic Data Generator

A professional Flask web application for generating synthetic financial bid/ask data using TimeGAN and validating it with the Shapiro-Wilk normality test.

## Features

- **TimeGAN-based Generation**: Uses deep learning to generate realistic synthetic time series data
- **Customizable Parameters**: Control simulation length and volatility
- **Statistical Validation**: Automatic Shapiro-Wilk test on bid-ask spread
- **Professional Dark Theme UI**: Clean, modern interface with interactive visualizations
- **Real-time Results**: Displays generated data, statistics, and validation results
- **Data Persistence**: Saves generated data as parquet files

## Installation

### 1. Install Dependencies

```bash
cd /Volumes/BAT_Systems/synthetic_generator
pip3 install -r requirements.txt
```

### 2. Verify Installation

```bash
python3 -c "import flask, pandas, ydata_synthetic; print('Installation successful!')"
```

## Usage

### Start the Flask Application

```bash
cd /Volumes/BAT_Systems/synthetic_generator/src
python3 app.py
```

The application will start on `http://localhost:5000`

### Web Interface

1. Open your browser and navigate to `http://localhost:5000`
2. Set your parameters:
   - **Simulation Length**: Number of data points (100 - 100,000)
   - **Volatility Multiplier**: Scale factor for price volatility (0.1 - 10.0)
3. Click "Generate Synthetic Data"
4. View results:
   - Summary statistics
   - Shapiro-Wilk normality test results
   - Time series plots
   - Spread distribution
   - Data preview (first and last 10 rows)

## Project Structure

```
synthetic_generator/
├── data/
│   ├── sample_data.parquet          # Reference data (50,000 XAUUSD records)
│   └── synthetic_data/              # Generated synthetic data outputs
├── src/
│   ├── app.py                       # Flask application
│   ├── generator.py                 # TimeGAN data generation
│   ├── validator.py                 # Shapiro-Wilk validation
│   ├── templates/
│   │   └── index.html              # Main UI
│   └── static/
│       └── style.css               # Dark theme styling
├── requirements.txt                 # Python dependencies
└── README.md                       # This file
```

## API Endpoints

### `GET /`
Returns the main web interface

### `POST /generate`
Generates synthetic data

**Request Body:**
```json
{
  "simulation_length": 5000,
  "volatility": 1.5
}
```

**Response:**
```json
{
  "statistics": {...},
  "validation": {...},
  "visualization": {...},
  "preview": {...},
  "filename": "synthetic_5000_1.5x_20250101_120000.parquet",
  "filepath": "/path/to/file.parquet"
}
```

### `GET /health`
Health check endpoint

## Technical Details

### TimeGAN Configuration
- Window size: 24 bars
- Batch size: 128
- Learning rate: 5e-4
- Noise dimension: 32
- Layer dimension: 64
- Training epochs: 50-100 (adjustable)

### Shapiro-Wilk Test
- Tests normality of bid-ask spread
- Significance level (α): 0.05
- Uses sampling for datasets > 5000 points

### Output Files
Generated files are saved in `data/synthetic_data/` with naming format:
```
synthetic_{length}_{volatility}x_{timestamp}.parquet
```

## Requirements

- Python 3.8+
- Flask 3.0+
- TensorFlow 2.15+
- pandas, numpy, scipy
- ydata-synthetic 1.4+
- plotly (for visualizations)

## Notes

- First generation may take longer as TimeGAN model trains
- Subsequent generations are faster using the cached model
- GPU acceleration supported (if TensorFlow GPU available)
- Output files include timestamp index for time series analysis

## License

Internal use only - BAT Systems
