Multi-format ingestion, semantic chunking, and dual vector + graph retrieval for accurate AI responses
The OpenRails RAG (Retrieval-Augmented Generation) Pipeline is the intelligence backbone of the platform. It transforms raw documents into searchable, contextual knowledge that powers AI chat responses, agent workflows, and automated analysis. Unlike basic vector-only RAG implementations, OpenRails uses a dual retrieval architecture combining vector search with knowledge graph RAG for relationship-aware context retrieval.
Key Differentiator: Dual retrieval means OpenRails understands not just semantic similarity (what things are about) but also entity relationships (how things connect). This produces significantly more accurate and complete answers than vector-only systems.
Process PDF, DOCX, PPTX, TXT, HTML, Markdown, images (via OCR), video (frame extraction + transcription), and audio (speech-to-text transcription). Each format has optimized extraction logic.
Intelligent document splitting that respects paragraph boundaries, section headers, and semantic coherence. Configurable chunk sizes with overlap for context continuity across chunk boundaries.
High-performance embedding storage and similarity search. Supports multiple embedding models, configurable distance metrics, and filtered search by metadata, collection, or security level.
Entity and relationship extraction builds a knowledge graph. Queries traverse entity connections to find contextually relevant information that vector search alone would miss.
Organize ingested content into collections (data lakes) for scoped retrieval. Control which knowledge bases are available to which projects, users, or agents.
Documents are automatically chunked and embedded upon ingestion. Background processing handles large batches without blocking the UI. Progress tracking for bulk uploads.
| Strategy | Best For | Chunk Size | Overlap |
|---|---|---|---|
| Semantic | Long-form documents, reports, manuals | Configurable | Adjustable based on content type |
| Paragraph | Well-structured documents with clear sections | Variable | Adjustable based on content type |
| Fixed-Size | Unstructured text, logs, transcripts | Configurable | Configurable |
| Recursive | Nested documents, code files | Configurable | Configurable |
Query Flow:
User Query → Embedding Generation → parallel: [ Vector Search | Knowledge Graph Traversal ] → Result Fusion → Ranked Context → LLM Prompt
Finds chunks semantically similar to the query. Excels at "what is X?" and "tell me about Y" questions where topical relevance is key.
Traverses entity relationships to find connected information. Excels at "how does X relate to Y?" and "what are the dependencies of Z?" questions.
| Component | Technology | Configuration |
|---|---|---|
| Vector Store | Vector Database | Optimized similarity matching with filtering |
| Graph Store | Knowledge Graph Database | Entity extraction, relationship mapping, graph queries |
| Embedding Models | Configurable | OpenAI, local self-hosted models, custom models |
| Chunk Storage | Relational Database + Object Storage | Metadata, lineage tracking, version history |
| Processing Queue | High-performance processing pipeline | Async ingestion, retry logic, progress tracking |
Turn thousands of documents into a searchable AI-powered knowledge base with cited answers
Query regulatory documents and internal policies with relationship-aware retrieval
Ingest research papers and find connections across studies using graph traversal