PDF Scanner Python Server

A FastAPI-based Python server for PDF processing, PII detection, redaction, and analytics with ClickHouse integration.

🚀 Features

Core Functionality

• PDF Parsing: Text extraction using PyMuPDF
• PII Detection: Advanced detection of emails, SSNs, and sensitive data
• PDF Redaction: Secure removal of PII with visual indicators
• Analytics Engine: Real-time metrics and P95 performance monitoring
• File Management: Download both original and redacted PDF versions

Technical Features

• ClickHouse Integration: High-performance analytics database
• RESTful API: Comprehensive endpoints for all operations
• CORS Support: Configured for React frontend integration
• Background Processing: Async PDF processing and redaction
• File Storage: Organized storage with UUID-based directories

Setup

1. Create a virtual environment:

python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

2. Install dependencies:

pip install -r requirements.txt

3. Ensure ClickHouse is running:

# From project root
./setup-clickhouse.sh

Running the Server

Development Mode

python main.py

Using Uvicorn directly

uvicorn main:app --host 0.0.0.0 --port 8080 --reload

The server will start on http://localhost:8080

API Endpoints with Sample Responses

Health & Status

GET /api/health

Sample Response:

{
  "status": "healthy",
  "message": "PDF Redactor server is running",
  "server": "FastAPI",
  "version": "1.0.0"
}

GET /api/python-server-status

Sample Response:

{
  "status": "running",
  "server": "PDF Redactor Python Server",
  "port": 8080,
  "endpoints": [
    "/api/health",
    "/api/findings",
    "/api/statistics",
    "/api/upload-history",
    "/api/upload-pdf",
    "/api/download-pdf/{upload_id}",
    "/api/download-redacted-pdf/{upload_id}"
  ],
  "capabilities": [
    "PDF parsing and text extraction",
    "PII detection (emails, SSNs, phones, credit cards)",
    "PDF redaction with coordinate-based detection",
    "ClickHouse analytics integration",
    "Real-time processing metrics",
    "P95 performance monitoring",
    "Secure file download management"
  ]
}

PDF Processing & Management

POST /api/upload-pdf

Request: Multipart form data with PDF file Sample Response:

{
  "upload_id": "195fe7a3-b850-4d4b-a54c-85e229837ba8",
  "status": "uploaded",
  "message": "PDF uploaded successfully and queued for processing"
}

GET /api/upload-history

Sample Response:

{
  "uploads": [
    {
      "upload_id": "195fe7a3-b850-4d4b-a54c-85e229837ba8",
      "filename": "document.pdf",
      "upload_date": "2024-01-15T10:30:00",
      "status": "complete",
      "file_size": 245760,
      "pages_processed": 3,
      "email_count": 3,
      "ssn_count": 1,
      "processing_time": 0.002,
      "is_clean": false,
      "emails": ["j***@example.com", "a***@company.org", "u***@test.net"],
      "ssns": ["123-**-4567"],
      "redaction_applied": true,
      "redacted_file_available": true,
      "total_redactions": 4
    }
  ],
  "total_count": 1
}

GET /api/upload-status/{upload_id}

Sample Response:

{
  "upload_id": "195fe7a3-b850-4d4b-a54c-85e229837ba8",
  "status": "complete",
  "emails": ["j***@example.com", "a***@company.org", "u***@test.net"],
  "ssns": ["123-**-4567"],
  "text_length": 15420,
  "pages_processed": 3,
  "processing_time": 0.002,
  "error": null
}

GET /api/download-pdf/{upload_id}

Response: Binary PDF file with appropriate headers

GET /api/download-redacted-pdf/{upload_id}

Response: Binary redacted PDF file with appropriate headers

Analytics & Metrics

GET /api/findings

Sample Response:

{
  "total_pdfs_processed": 150,
  "total_pii_items": 342,
  "success_rate": 98.5,
  "avg_processing_time": 0.003,
  "p95_processing_time": 0.008,
  "pii_types": {
    "emails": 245,
    "ssns": 97
  },
  "processing_trends": [
    {
      "date": "2024-01-15",
      "uploads": 12,
      "avg_time": 0.003
    },
    {
      "date": "2024-01-14",
      "uploads": 8,
      "avg_time": 0.004
    }
  ],
  "recent_uploads": [
    {
      "upload_id": "195fe7a3-b850-4d4b-a54c-85e229837ba8",
      "filename": "document.pdf",
      "upload_date": "2024-01-15T10:30:00",
      "status": "complete",
      "email_count": 3,
      "ssn_count": 1,
      "processing_time": 0.002,
      "redaction_applied": true,
      "redacted_file_available": true,
      "total_redactions": 4
    }
  ]
}

GET /api/statistics

Sample Response:

{
  "total_uploads": 150,
  "clean_pdfs": 45,
  "pii_pdfs": 105,
  "total_emails": 245,
  "total_ssns": 97,
  "avg_processing_time": 0.003
}

Testing

POST /api/run-tests

Sample Response:

{
  "total_tests": 15,
  "passed": 14,
  "failed": 1,
  "success_rate": 93.33,
  "details": {
    "pdf_parsing": {
      "passed": 5,
      "failed": 0
    },
    "pii_detection": {
      "passed": 6,
      "failed": 1
    },
    "redaction": {
      "passed": 3,
      "failed": 0
    }
  }
}

Database Schema

ClickHouse Tables

-- Main PDF uploads table with redaction support
CREATE TABLE pdf_uploads (
    upload_id String,
    filename String,
    file_path String,
    file_size UInt64,
    upload_date DateTime,
    processing_date Nullable(DateTime),
    status String,
    pages_processed UInt32,
    text_length UInt32,
    processing_time Float32,
    emails Array(String),
    ssns Array(String),
    error_message Nullable(String),
    redaction_applied UInt8 DEFAULT 0,
    redacted_file_available UInt8 DEFAULT 0,
    total_redactions UInt32 DEFAULT 0,
    redacted_file_path Nullable(String)
) ENGINE = MergeTree()
ORDER BY (upload_id, upload_date);

-- Materialized views for fast lookups
CREATE MATERIALIZED VIEW email_index AS
SELECT upload_id, upload_date, email, filename
FROM pdf_uploads
ARRAY JOIN emails AS email;

CREATE MATERIALIZED VIEW ssn_index AS
SELECT upload_id, upload_date, ssn, filename
FROM pdf_uploads
ARRAY JOIN ssns AS ssn;

Data Models

PDFProcessingResult

class PDFProcessingResult(BaseModel):
    upload_id: str
    filename: str
    file_path: str
    file_size: int
    upload_date: datetime
    processing_date: Optional[datetime] = None
    status: ProcessingStatus
    pages_processed: int
    text_length: int
    processing_time: float
    emails: List[str]
    ssns: List[str]
    error_message: Optional[str] = None
    redaction_result: Optional[RedactionResult] = None

RedactionResult

class RedactionResult(BaseModel):
    redacted_file_path: str
    redaction_summary: dict
    total_redactions: int
    redaction_time: float

Processing Pipeline

1. File Upload: PDF received and stored with UUID
2. Text Extraction: PyMuPDF extracts text content
3. PII Detection: Regex patterns detect emails and SSNs
4. PDF Redaction: PII removed and redacted file created
5. Data Storage: Results stored in ClickHouse
6. Analytics: P95 calculations and metrics computed

Performance Metrics

Processing Times

• Average: ~1-5ms per document
• P95: ~8-15ms for 95th percentile
• Redaction: Additional 2-5ms per document

Analytics Data

• Total PDFs processed: Count of all documents
• Success rate: Percentage of successful processing
• PII detection: Emails and SSNs found
• Redaction statistics: Files redacted and items removed

Testing

Run Test Suite

# Run all tests
python run_tests.py

# Run specific test
python tests/test_simple_redaction.py

# Test ClickHouse connection
python test-clickhouse.py

Test Coverage

• Unit Tests: PDF parsing, PII detection, redaction
• Integration Tests: API endpoints, database operations
• Performance Tests: Processing time and database queries
• Security Tests: PII masking and redaction effectiveness

File Structure

python_server/
├── main.py                 # FastAPI application and endpoints
├── pdf_parser.py           # PDF text extraction
├── clickhouse_service.py   # Database operations
├── file_storage.py         # File management
├── models.py               # Data models and schemas
├── requirements.txt        # Python dependencies
├── tests/                  # Test files
│   ├── test_simple_redaction.py
│   └── ...
├── uploads/                # Uploaded PDF storage
└── redacted/               # Redacted PDF storage

Integration with React App

The server is configured to work with the React app running on http://localhost:3000. The React app's proxy configuration in src/setupProxy.js forwards all /api requests to this Python server.

CORS Configuration

app.add_middleware(
    CORSMiddleware,
    allow_origins=["http://localhost:3000"],
    allow_credentials=True,
    allow_methods=["*"],
    allow_headers=["*"],
)

Error Handling

Common Error Scenarios

• File not found: 404 errors for missing uploads
• Processing failures: 500 errors with detailed messages
• Invalid files: 400 errors for non-PDF uploads
• Database errors: Graceful fallbacks with logging

Logging

• Processing logs: Upload and processing events
• Error logs: Detailed error information
• Performance logs: Processing time tracking
• Security logs: PII detection and redaction events

Security Features

PII Protection

• Data masking: Sensitive data masked in API responses
• Secure redaction: Permanently removes PII from PDFs
• File isolation: Each upload in separate directory
• Audit trail: Complete processing history

API Security

• Input validation: All inputs validated and sanitized
• Error handling: No sensitive data in error messages
• File type validation: Only PDF files accepted
• Size limits: Configurable file size restrictions

Monitoring & Analytics

Real-time Metrics

• Processing queue: Number of files being processed
• Success rates: Processing success percentages
• Performance trends: P95 processing times
• Error rates: Failed processing attempts

Database Queries

# Check P95 processing times
SELECT quantile(0.95)(processing_time) FROM pdf_uploads WHERE processing_time > 0;

# View redaction statistics
SELECT COUNT(*) as total_redacted, SUM(total_redactions) as total_items_redacted 
FROM pdf_uploads WHERE redaction_applied = 1;

# Get recent uploads
SELECT filename, status, processing_time, total_redactions 
FROM pdf_uploads ORDER BY upload_date DESC LIMIT 10;

Future Enhancements

Planned Features

• Advanced PII detection: Machine learning-based detection
• Batch processing: Multiple file uploads
• Cloud storage: S3/Google Cloud integration
• Advanced analytics: Custom reporting and dashboards
• API rate limiting: Request throttling and quotas
• Webhook support: Real-time notifications
• Multi-format support: DOC, DOCX, TXT processing