# Loan Application ML Pipeline

> *Machine-learning pipeline deliverable for the FIU Capstone II Data Science & AI team. This repository contains only the backend automation that ingests loan packets, runs OCR + KPI extraction, validates data, and produces risk scores. The companion web application was implemented and delivered separately by the Computer Science team.*

---

## Project Purpose

| Project Purpose |
| --- |
| Provide lenders with a reproducible automation layer that validates every applicant packet before disbursement. |
| Extract standardized KPIs from applicant documents and enforce consistency checks. |
| Produce transparent prediction artifacts that auditors can trace from document ingestion through risk scoring. |

---

## Core Capabilities

| Core Capabilities |
| --- |
| **Document OCR** – Upload PDFs from Google Cloud Storage (GCS), run Vision OCR, and persist JSON outputs for reuse. |
| **KPI Extraction** – Parse OCR text using the KPI checklist provided by the business, writing the structured data back to GCS. |
| **Applicant Validation** – Run rule-based validation across extracted KPIs to flag missing or tampered fields. |
| **Risk Modeling** – Score each validated applicant with the trained model saved in `models/`. |
| **FastAPI service** – `ml-pipeline-backend/main.py` exposes a REST entry point that orchestrates the full flow per applicant ID. |

---

## Project Directory Structure

```text
.
├── artifacts/                     # Saved OCR + KPI outputs produced while testing
├── data/                          # Raw datasets used for experimentation
├── environment.yml                # Conda environment specification for reproducibility
├── ml-pipeline-backend/           # FastAPI service and pipeline modules
│   ├── main.py                    # /health and /applicants/{id} endpoints
│   ├── ocr_pipeline.py            # GCS + OCR helpers
│   ├── kpi_extraction.py          # KPI parsing logic
│   ├── applicant_validation.py    # Data quality checks
│   └── risk_model.py              # Model loading + inference
├── models/                        # Trained model artifacts (joblib)
├── notebooks/                     # Exploratory notebooks (e.g., spam detection RNN prototype)
├── Pre Disbursement Checklist.xlsx# KPI source of truth (included in deliverable)
├── turing-agent-*.json            # Example service-account credential (see below)
└── README.md
```

---

## Required Files

| Required Files |
| --- |
| **Service-account JSON for GCS/Vision** – During development this file lived outside the repo (set in `ml-pipeline-backend/config.py` as `SERVICE_ACCOUNT_JSON`). The deliverable includes `turing-agent-358210-44bf93602660.json` inside the repo solely as an example because the backing GCP project will be deleted after the semester. Replace it with your own credentials and, in production, keep it outside the repo for security. |
| **Pre Disbursement Checklist.xlsx** – Business KPI dictionary consumed by `ml-pipeline-backend/main.py` at startup. Maintain its structure (`document_type`, `kpi_description`, ...). The `Pre-Disbursement` sheet provides the authoritative KPI list used by the pipeline. |

---

## Installation Guide

1. **Clone the repo** into your working directory.
2. **Create the Python environment** (recommended: Conda because the dependencies are captured in `environment.yml`).
   ```bash
   conda env create -f environment.yml
   conda activate loan-ml-pipeline
   ```
   If you prefer `venv`, install with `pip install -r requirements.txt` (create the file from `environment.yml` dependencies if needed).
3. **Provide Google Cloud credentials.**
   - Place your service-account JSON outside the repo (e.g., `~/secrets/gcs-sa.json`).
   - Update `SERVICE_ACCOUNT_JSON` in `ml-pipeline-backend/config.py` to point to that file (see section below).
4. **Configure the KPI checklist.** Keep `Pre Disbursement Checklist.xlsx` in the repo root or update `_CONFIG_PATH` in `ml-pipeline-backend/main.py` to wherever the product owner stores it.
5. **Start the FastAPI app.**
   ```bash
   cd ml-pipeline-backend
   uvicorn main:app --reload
   ```
6. **(Optional) Run notebooks** located in `notebooks/` for experimentation by launching Jupyter Notebook or Lab from the project root.

---

## API Use Guidelines

> **Base URL** – `http://localhost:8000` when running locally.
>
> **Health probe** – `GET /health` returns `{"status": "ok"}`.
>
> **Main inference endpoint** – `GET /applicants/{applicant_id}` orchestrates OCR → KPI extraction → validation → risk prediction.
>
> - Query parameters:
>   - `force_rerun_ocr` (bool, default `false`) to bypass cached OCR output and re-run OCR.
>   - `force_rerun_kpi` (bool, default `false`) to rebuild KPI outputs even when cached rows exist.
> - Response payload:
>   ```json
>   {
>     "applicant_id": "...",
>     "ocr_uri": "gs://.../ocr-results-json/<applicant>.json",
>     "kpi_rows": 42,
>     "validated_rows": 42,
>     "model_output": {"default_probability": 0.11}
>   }
>   ```
>
> **Recommended workflow**
> 1. Upload applicant PDFs to the GCS bucket under `PDF_INPUT_PREFIX`.
> 2. Call `/applicants/{id}` once to process the applicant end-to-end.
> 3. Re-use the cached OCR and KPI outputs by letting the defaults handle them unless data changes.
>
> **Error handling** – Each stage raises a `500` with a detailed message (OCR, KPI extraction, validation, risk). Inspect server logs for stack traces during debugging.

---

## Changing the GCS Bucket or Credential Directory

All storage-related settings live in `ml-pipeline-backend/config.py`.

1. **New bucket:** update
   ```python
   GCP_PROJECT_ID = "new-project"
   GCS_BUCKET_NAME = "new-bucket"
   ```
   Adjust the prefix constants (`PDF_INPUT_PREFIX`, `VISION_OUTPUT_PREFIX`, etc.) if the folder layout differs in the new bucket.
2. **Model artifact location:** update `MODEL_PATH` to point at the correct file inside `models/` or any other folder if you move it.
3. **Service-account JSON path:** change
   ```python
   SERVICE_ACCOUNT_JSON = Path("/absolute/path/to/new/sa.json")
   ```
   Prefer storing credentials outside the repository and ensure the environment variable `GOOGLE_APPLICATION_CREDENTIALS` is set (the config already exports it when the file exists).
4. **Environment variables:** Alternatively, skip editing the file by providing `GOOGLE_APPLICATION_CREDENTIALS` and overriding bucket/project values via env vars before starting `uvicorn`.

---

## User Manual

1. **Prepare data sources.**
   - Confirm applicant PDFs are uploaded to the configured GCS bucket under `PDF_INPUT_PREFIX`.
   - Keep the KPI Excel file up to date with business rules (sheet `Pre-Disbursement`).
2. **Launch the backend.**
   - Activate the environment, navigate to `ml-pipeline-backend/`, and run `uvicorn main:app --reload`.
   - Monitor console output for step-by-step logs (OCR, KPI extraction, validation, risk scoring).
3. **Process an applicant.**
   ```bash
   curl "http://localhost:8000/applicants/12345?force_rerun_ocr=false&force_rerun_kpi=false"
   ```
   Review the JSON response for artifact URIs and default probability.
4. **Inspect generated artifacts.**
   - OCR JSON and KPI CSVs are uploaded back to the bucket under `VISION_OUTPUT_PREFIX`, `OCR_RESULTS_PREFIX`, and `KPI_RESULTS_PREFIX`.
   - Validation outputs live in `VALIDATION_RESULTS_PREFIX`, while model predictions are persisted in `RISK_RESULTS_PREFIX`.
5. **Update artifacts or rerun steps.**
   - Re-upload corrected documents, then call the endpoint with `force_rerun_ocr=true`.
   - To refresh KPI parsing only, set `force_rerun_kpi=true` while leaving OCR cached.
6. **Handover to stakeholders.**
   - Share the GCS URIs or download artifacts for auditors.
   - Communicate any validation errors surfaced in `validated_df` for manual remediation.

---

## Team

| Team Members |
| --- |
| Luis D. Jimenez – Team Leader, Principal Developer & Data Scientist |
| Carlos Hernandez – Data Science Support |
