About Document Intelligence Lab

Document Intelligence Lab explores a practical question: how can we turn PDFs into structured data and useful AI answers while keeping the original evidence easy to inspect?

This publication shares the engineering behind an MIT-licensed open-source project for document extraction, validation, retrieval, and human review. It covers general business documents, insurance wording, and fictional payroll and banking examples. The focus is on understandable workflows, reproducible experiments, and the limits of what the software can establish.

Start small or explore the full workspace

The standalone Python library reads local PDFs through Docling and normalizes pages, tables, and source references. Its validation tools check supported currency formats, required literal labels, exact quotations, and citation identifiers. The core and optional Docling reader can be used without Java, Docker, authentication, or an LLM. The alpha is now published on PyPI as document-evidence-lab 0.1.0a1, with source installation also available. It requires Python 3.10 or newer; the optional Docling extra adds the PDF parser, while the lightweight core and SDK have no required third-party runtime dependencies.

The research workspace combines Java, Spring Boot, and Next.js with document processing, authenticated dossiers, evidence inspection, reviewer tasks, and audit history. It investigates hybrid retrieval with MiniLM, Qdrant, and BM25, plus grounded generation through Nova. A separate Python SDK connects scripts to an existing authenticated workspace.

Local and Azure processing are explicit alternatives. Local workflows use Docling and Ollama; Azure workflows use separately configured Document Intelligence and Azure OpenAI resources. The documented local profile does not silently switch to cloud inference.

What you will find here

Articles explain architecture choices, extraction and OCR, retrieval and chunking experiments, prompts, evidence checks, and operational lessons. Data engineering topics include DuckDB, dbt, Parquet, and Bronze/Silver/Gold layers, with reviewer-confirmed amounts required for Gold.

An important distinction runs through every article: a source link supports inspection, but it does not automatically prove an answer correct. Exact quote checks are not semantic verification, and a successful extraction is not an accuracy guarantee. Human review remains necessary for consequential use.

The project is research software. It does not make credit, underwriting, settlement, or payment decisions.

Explore the project

Start with the architecture article, then visit the GitHub repository for current setup instructions, examples, tests, and known limits. Technical questions, reproducible bug reports, and focused contributions are welcome through the repository.