Async Document Processing Workflow (FastAPI + Celery + Next.js)
This project implements a production-style asynchronous document pipeline:
Upload one or more files
Persist document + job state in PostgreSQL
Process in background using Celery workers
Publish progress events from workers through Redis Pub/Sub
Stream near-real-time progress in the UI
Review/edit extracted output, finalize, and export as JSON/CSV
Tech Stack
Frontend: Next.js + React + TypeScript
Backend: FastAPI (Python)
Database: PostgreSQL
Queue/Workers: Celery
Messaging/State: Redis (broker + result backend + Pub/Sub)
Architecture Overview
Backend (FastAPI)
Handles upload, list, detail, retry, review update, finalize, export.
Stores metadata and processing state in PostgreSQL.
Exposes SSE endpoint to stream progress events to frontend.
Worker Layer (Celery)
Receives jobs from Redis broker.
Executes multi-stage background workflow.
Publishes progress events at each stage via Redis Pub/Sub.
Redis
Celery broker/result backend
Pub/Sub channel for per-job progress events (job_progress:{job_id})
Frontend (Next.js)
Upload page
Jobs dashboard with search/filter/sort
Job detail page for progress, review/edit, finalize, export
Document Processing Stages
Each job emits progress events for:
job_queued
job_started
document_parsing_started
document_parsing_completed
field_extraction_started
field_extraction_completed
final_result_stored
job_completed / job_failed
API Surface
POST /api/upload — upload one or more files and enqueue jobs
GET /api/jobs — list jobs with search/filter/sort
GET /api/jobs/{job_id} — job detail
GET /api/jobs/{job_id}/events — SSE progress stream
POST /api/jobs/{job_id}/retry — retry failed job
PATCH /api/jobs/{job_id}/reviewed — save reviewed JSON
POST /api/jobs/{job_id}/finalize — finalize reviewed result
GET /api/jobs/{job_id}/export?format=json|csv — export finalized resultThis project implements a production-style asynchronous document pipeline:
Upload one or more files
Persist document + job state in PostgreSQL
Process in background using Celery workers
Publish progress events from workers through Redis Pub/Sub
Stream near-real-time progress in the UI
Review/edit extracted output, finalize, and export as JSON/CSV
Tech Stack
Frontend: Next.js + React + TypeScript
Backend: FastAPI (Python)
Database: PostgreSQL
Queue/Workers: Celery
Messaging/State: Redis (broker + result backend + Pub/Sub)
Architecture Overview
Backend (FastAPI)
Handles upload, list, detail, retry, review update, finalize, export.
Stores metadata and processing state in PostgreSQL.
Exposes SSE endpoint to stream progress events to frontend.
Worker Layer (Celery)
Receives jobs from Redis broker.
Executes multi-stage background workflow.
Publishes progress events at each stage via Redis Pub/Sub.
Redis
Celery broker/result backend
Pub/Sub channel for per-job progress events (job_progress:{job_id})
Frontend (Next.js)
Upload page
Jobs dashboard with search/filter/sort
Job detail page for progress, review/edit, finalize, export
Document Processing Stages
Each job emits progress events for:
job_queued
job_started
document_parsing_started
document_parsing_completed
field_extraction_started
field_extraction_completed
final_result_stored
job_completed / job_failed
API Surface
POST /api/upload — upload one or more files and enqueue jobs
GET /api/jobs — list jobs with search/filter/sort
GET /api/jobs/{job_id} — job detail
GET /api/jobs/{job_id}/events — SSE progress stream
POST /api/jobs/{job_id}/retry — retry failed job
PATCH /api/jobs/{job_id}/reviewed — save reviewed JSON
POST /api/jobs/{job_id}/finalize — finalize reviewed result
GET /api/jobs/{job_id}/export?format=json|csv — export finalized result