LLM Contract Analysis

The Problem

An enterprise legal department reviewed 10,000+ vendor and customer contracts annually, averaging 4 hours per contract. Three paralegals and two attorneys spent 80% of their time on routine clause identification and risk scoring. High-risk clauses (indemnification, liability caps, IP assignment) were missed in 12% of reviews due to volume fatigue.

The Dataset

15K annotated contracts with 340K labeled clauses across 28 clause types (indemnification, limitation of liability, IP, termination, force majeure, etc.). Attorney-verified risk scores for each clause. 800 precedent memos mapping clauses to acceptable/unacceptable thresholds.

Model & Approach

Fine-tuned Llama 3 70B using QLoRA (4-bit quantization with LoRA adapters) on the annotated contract corpus. Multi-task training: clause extraction, risk classification, obligation identification, and redline suggestion generation.

• Base Model: Llama 3 70B (Meta) — chosen for open-source licensing allowing on-premise deployment without data leaving the network.
• Fine-Tuning: QLoRA with rank-64 adapters, trained for 3 epochs on 8× A100 80GB cluster. Custom loss function weighting high-risk clauses 3× higher.
• Inference: GPTQ 4-bit quantization on 4× A100 80GB production server. Average inference: 12 seconds per contract page.
• RAG Augmentation: Precedent memo retrieval for risk scoring context — ensures scores align with the firm's specific risk appetite.

Architecture

PDF/DOCX ingestion → OCR fallback (Tesseract) → chunking → Llama 3 clause extraction → risk scorer with RAG precedent retrieval → obligation tracker → redline generator → attorney review dashboard. All on-premise: air-gapped from internet, TLS-encrypted internal traffic, RBAC access controls.

Deployment

On-premise Kubernetes cluster with 4× A100 GPUs. vLLM inference server with continuous batching for throughput. Integration with the firm's DMS (iManage) via REST API. Secure enclave for model weights and contract data. Two-week attorney training program for the review dashboard.

Results

ROI

$480K annual savings. Reduced paralegal review hours by 75%. Attorneys now focus on negotiation strategy instead of clause hunting. Risk exposure reduced by eliminating 90% of missed high-risk clauses. Infrastructure cost: $60K/year for on-premise GPU cluster vs. $150K+ estimated cloud API costs.

Why It Was Hard

Legal language is dense and highly contextual. A "limitation of liability" clause in a SaaS agreement reads completely differently from one in a construction contract. The model needed to understand contract type context to score risk correctly.

On-premise deployment added 6 weeks to the timeline. No cloud GPUs, no managed inference services—everything from CUDA drivers to model serving had to be configured and hardened by hand.

What We Learned

QLoRA is production-viable for domain-specific LLM fine-tuning. The 70B model significantly outperformed the 8B variant on clause extraction—worth the extra infrastructure cost. RAG augmentation with precedent memos was the key to achieving attorney-acceptable risk scores.

Attorney buy-in required showing the model's reasoning, not just its output. Explainable highlights (which tokens contributed to risk score) were essential for adoption.

FAQ