# DCS Credit Application Processor

## Overview
System that extracts customer data from DCS credit applications and fills AutoKing PDF templates. Two processing methods available:
- **Chrome Extension (Primary)**: 95%+ accuracy via direct web page extraction
- **File Watcher (Legacy)**: 60% accuracy via OCR from PDF files

## Quick Start

### Chrome Extension Setup
```bash
# Install dependencies
pip install fastapi uvicorn pydantic python-multipart PyPDF2 PyMuPDF

# Start API server
python standalone_api.py

# Load extension in Chrome
1. Navigate to chrome://extensions/
2. Enable Developer mode
3. Load unpacked → select 'extension' folder
4. Visit DCS credit application page
5. Click extension icon → Process application
```

### Odoo CRM Integration Setup
```bash
# 1. Configure Odoo connection in extension
- Click "🔗 Odoo Integration →" in extension popup
- Enter Odoo server URL, database, username, API key/password
- Enable "Enable Odoo sync" checkbox
- Click "Save & Test" to verify connection

# 2. Workflow
1. Process DCS application (creates filled AutoKing PDF + original DCS PDF + JSON)
2. System searches Odoo for leads by phone number
3. Attaches files to matching lead automatically
4. Adds "synced" tag to lead for tracking

# 3. Lead Matching Strategy
- Priority: Phone number matching (handles vehicle-named leads)
- Fallback: Customer name matching
- Works with existing ADF → Odoo lead import workflow
```

### Legacy File Watcher Setup
```bash
# Install dependencies
pip install watchdog pdfplumber PyPDF2 PyMuPDF reportlab

# Optional: OCR support for problematic PDFs
pip install pytesseract pillow

# Run
python main.py
# Drop PDFs into ./input folder
```

## Important Runtime Notes

### Service Execution Environment
- **File watcher service (file_watcher_service.py)**: Runs on native Windows, NOT in WSL
- **API server (standalone_api.py)**: Runs on native Windows, NOT in WSL
- **To restart services**: Must be done from Windows terminal/PowerShell, not from WSL
- **Process monitoring**: WSL cannot see Windows native Python processes

### After Code Changes
1. **File watcher**: Stop with Ctrl+C in Windows terminal, then restart with `python file_watcher_service.py`
2. **API server**: Stop with Ctrl+C in Windows terminal, then restart with `python standalone_api.py`

## Architecture

### Chrome Extension System
- **Detection**: Auto-detects DCS pages, handles React SPA loading
- **Extraction**: 29+ fields from DOM elements (SSN, names, employment, vehicle)
- **Processing**: FastAPI backend fills AutoKing template
- **Output**: Customer folders in `D:\Customer Docs\2025\Month\Name\`

### File Structure
```
credit-app-processor/
├── standalone_api.py          # FastAPI server (main API)
├── extension/
│   ├── content.js            # DOM extraction engine
│   ├── popup.html/js         # User interface
│   └── manifest.json         # Extension config
├── template/
│   └── autoking_template.pdf # PDF form to fill
└── main.py                   # Legacy file watcher
```

### Output Organization
```
D:\Customer Docs\2025\08 - August\John Smith\
├── DCS John Smith Credit App.pdf    # Original
├── John Smith Credit App.pdf        # Filled template
└── John Smith Credit App.json       # Extracted data
```

## Critical Technical Details

### Data Extraction Fields (29+)
- **Applicant**: Name, address, phone, email, SSN, DOB
- **Employment**: Employer, income, occupation, work phone, time on job
- **Vehicle**: Year, make, model, VIN, mileage, sales price
- **Co-Applicant**: 22+ fields with full extraction support

### Key Regex Patterns (Fixed Issues)
```javascript
// SSN - handles both dashed and non-dashed
/SSN[:\s]*(\d{3}-?\d{2}-?\d{4}|\d{9})/

// Multi-word last names (e.g., "Giron Ozuna")
/Last Name[:\s]*([A-Za-z\s-]+?)\s*(?:First|Middle|$)/

// Numeric vehicle models (e.g., "Chrysler 200")
/Model[:\s]*([A-Za-z0-9\s\-]+)/

// Work phone with formatting
/((?:\(?\d{3}\)?[-.\s]*\d{3}[-.\s]*\d{4}|\d{10}))/

// Employer with numbers (e.g., "New Beginnings 2.0")
/Employer[:\s]*([A-Za-z0-9\s&.'"-]+)/
```

### Configuration
```python
# Key paths
CUSTOMER_DOCS_BASE = r"D:\Customer Docs"
TEMPLATE_PATH = "./template/autoking_template.pdf"
WATCH_FOLDER = "./input"      # Legacy system
EXPORT_FOLDER = "./output"    # Legacy system

# API settings
API_HOST = "127.0.0.1"
API_PORT = 8000
```

## Known Issues & Roadmap

### Known Limitations
1. **Trade-In Vehicle**: Not applicable - DCS credit applications don't collect trade-in information
2. **Field Mapping**: Some AutoKing PDF fields may not map correctly

### Completed Fixes (Production Ready)
- ✅ SSN extraction (dashed format support)
- ✅ Multi-word last names
- ✅ Numeric vehicle models
- ✅ Hyphenated name capitalization
- ✅ Work phone formatting
- ✅ Employer names with numbers
- ✅ Complete employer address
- ✅ **Middle name/initial support** (2025-09-25)
- ✅ User-friendly error messages for file locks
- ✅ Reduced log noise for history folder files

### Recent Completions (2025-09-25): Middle Name Implementation
- ✅ **Chrome Extension**: Extracts middle names from DCS full names ("John William Smith" → first: "John", middle: "William", last: "Smith")
- ✅ **PDF Mapping**: Full AutoKing template support (`Buyer Middle Initial`, `Co-buyer middle initial`, etc.)
- ✅ **Odoo CRM Fields**: Added `x_middle_name` and `x_co_applicant_middle_name` with UI integration
- ✅ **PDF Regeneration**: Manual regeneration from Odoo includes middle names
- ✅ **Error Handling**: Permission denied errors now suggest closing PDF viewers
- ✅ **Log Cleanup**: History folder file warnings changed to debug level

### Development Roadmap
- **Future (if needed)**: 2-way sync system (Adobe ↔ System ↔ Odoo)
- **Consideration**: ADF email middle name extraction (if needed)
- **Enhancement**: Batch processing capabilities
- **Analytics**: Processing statistics and reporting dashboard

## Future Development - 2-Way PDF Sync

### Field Editing with Live PDF Updates
- **Architecture**: New field editor interface with real-time PDF preview
- **Layout**: Two-column design - editable fields (left), PDF preview (right)
- **Updates**: Debounced 500ms updates for smooth PDF regeneration
- **Workflow**: Extract → Edit Fields → Live Preview → Save & Process

### 2-Way Sync Architecture
Enables editing from Adobe Acrobat, Odoo CRM, or web interface with automatic synchronization:

#### Sync Points
1. **Adobe → System**: 
   - File watcher detects PDF modifications in `D:\Customer Docs\`
   - Extracts updated field values using PyMuPDF
   - Updates system data and syncs to Odoo

2. **Odoo → System**:
   - Polls Odoo API or uses webhooks for lead/opportunity changes
   - Regenerates PDF with updated data
   - Replaces file in Customer Docs folder

3. **System → Both**:
   - Web field editor updates trigger PDF regeneration
   - Saves to Customer Docs and syncs to Odoo

#### Version Control Strategy
- **Main File**: `John Smith Credit App.pdf` (always current)
- **History**: `D:\Customer Docs\2025\08\John Smith\versions\`
- **Naming**: `John Smith Credit App_20250828_143022.pdf`
- **Retention**: Keep last 5-10 versions for recovery

#### Conflict Resolution
- **Strategy**: Last write wins with automatic versioning
- **Protection**: Previous version auto-saved before overwrite
- **Notifications**: Optional conflict alerts
- **Future**: Field-level merge capabilities

#### Technical Implementation Notes
- **PDF Reading**: PyMuPDF extracts form field values from Adobe-edited PDFs
- **Field Mapping**: Reverse mapping for PDF → data extraction
- **File Handling**: Lock detection with retry logic and queue system
- **Change Detection**: Compare field values to identify modifications
- **Selective Sync**: Only sync changed fields to minimize API calls

## Troubleshooting

### Chrome Extension Issues
- **Page not detected**: Refresh page, ensure React content loaded
- **No data extracted**: Check console for errors (F12)
- **PDF not generated**: Verify API server running on port 8000

### Legacy System Issues
- **Low extraction rate**: Install Tesseract OCR for PDFCreator PDFs
- **Printing issues**: Install SumatraPDF, check printer name in config
- **File locks**: System uses copy-fallback for locked files

### PDF Extraction Methods (Legacy)
1. Direct text extraction (pdfplumber, PyMuPDF, PyPDF2)
2. Character decoding for PDFCreator encoding
3. OCR fallback via Tesseract
4. Regex pattern matching on extracted text

## Dependencies

### Core Requirements
```
# Chrome Extension System
fastapi uvicorn pydantic python-multipart
PyPDF2 PyMuPDF reportlab

# Legacy File Watcher
watchdog pdfplumber

# Optional OCR Support
pytesseract pillow
```

### External Software
- **Chrome Browser**: For extension
- **Tesseract OCR**: For problematic PDFs (optional)
- **SumatraPDF**: Silent printing for legacy system (optional)
- **Adobe Acrobat**: Review filled forms

## Recent Major Updates

### 2025-08-28: Time Field Combination Implementation (IN PROGRESS)
- ✅ **Time combination logic**: Added `combine_time_fields()` function in both create_lead/update_lead
- ✅ **Residence time working**: Both applicant and co-applicant time at residence now combine YR/MO fields correctly
- ❌ **Job time issue discovered**: PDF field name conflict - both residence and job time use same field names (YR/MO)
- ✅ **Enhanced field mappings**: Added 100+ comprehensive PDF field mappings including co-applicant variations
- ✅ **Multi-file monitoring**: Credit apps now trigger immediate folder scanning + ongoing file monitoring
- ✅ **Co-applicant data**: Email and phone number mappings improved

#### Critical Issue Identified
**PDF Field Name Overlap Problem**: In `standalone_api.py`, both residence and job time use identical field names:
- Applicant: `YR`/`MO` used for both residence AND job time (lines 1097-1099, 1193-1195)
- Co-applicant: `YR_2`/`MO_2` used for both residence AND job time
- Result: Only one time type can be extracted per field (residence time currently wins)

#### Next Sprint Tasks
- [ ] Investigate AutoKing PDF template for separate job time field names
- [ ] Review unmapped `fill_XX` fields that might contain job time data
- [ ] Determine if PDF design issue requires different extraction strategy

### 2025-09-25: Manual PDF Regeneration Implementation (COMPLETED)
- ✅ **Odoo Button Integration**: Added `🔄 Regenerate PDF` button in lead header for credit applications
- ✅ **Two-Way Data Sync**: Implemented `/fill-pdf-from-data` API endpoint for Odoo → PDF generation
- ✅ **Customer Docs Integration**: Regenerated PDFs save to both Odoo attachments AND Customer Docs folder
- ✅ **History Management**: Existing PDFs automatically archived to history folder before replacement
- ✅ **Comprehensive Changelog**: Full audit trail in `CHANGELOG.md` tracking all PDF regenerations
- ✅ **Error Handling**: Complete field mapping fixes (x_email vs x_email_address, date fields)
- ✅ **Production Ready**: Tested and verified working in production environment

#### Technical Implementation Details
- **Odoo View**: `/mnt/c/odoo/custom/addons/crm_adf_lead_processor/views/adf_lead_view.xml:11-16`
  - Button only visible for credit applications (`x_lead_type != 'credit_app'`)
  - Triggers `regenerate_pdf_from_lead` method in Odoo model

- **Odoo Model**: `/mnt/c/odoo/custom/addons/crm_adf_lead_processor/models/adf_lead.py:1026-1173`
  - Complete PDF regeneration workflow with Customer Docs saving
  - Automatic history archival with timestamp and source tracking
  - Comprehensive changelog management in local folders

- **API Endpoint**: `/mnt/e/projects/deployed/credit-app-processor/standalone_api.py`
  - New `/fill-pdf-from-data` endpoint for Odoo integration
  - Fixed Response import and method calls
  - Uses existing PDF generation infrastructure

#### Workflow Process
1. User clicks `🔄 Regenerate PDF` in Odoo lead header
2. System extracts current field values from Odoo lead
3. FastAPI generates new PDF from current data via `/fill-pdf-from-data`
4. Existing PDF archived to `D:\Customer Docs\YYYY\MM\Customer Name\history\`
5. New PDF saved to Customer Docs folder structure
6. Odoo attachment updated with regenerated PDF
7. History changelog updated with regeneration details and timestamp

### 2025-08-27: Odoo CRM Integration (COMPLETED)
- ✅ Complete Odoo 18 Community Edition integration
- ✅ Phone-prioritized lead search (handles vehicle-named leads)
- ✅ Auto-sync credit applications to existing leads
- ✅ File attachment (PDF + JSON) to Odoo leads
- ✅ Configurable sync settings with test capabilities
- ✅ "Synced" tag management for processed applications

### 2025-08-22: Critical Data Fixes
- Fixed SSN extraction for dashed format (123-45-6789)
- Added support for numeric vehicle models (200, 335i, F-150)
- Fixed hyphenated name capitalization (Westlund-Cartie)

### 2025-08-04: UI & Extraction Improvements
- Single-button workflow in extension
- Employer names with numbers support
- Complete employer address field combination

### 2025-07-25: Chrome Extension Launch
- Complete transformation from file-based to web-based
- 95%+ accuracy through DOM extraction
- FastAPI backend with real-time processing

## Important Notes
- Working directory: `/credit-app-processor/`
- Primary system: Chrome extension (95%+ accuracy)
- Legacy available for backward compatibility (60% accuracy)
- All fixes tested in production environment