Privacy-First RAG Stack for Financial Institutions
Privacy-First Institutional Memory: Architecting the Obsidian and NIM RAG Stack for Cooperatives
The Crisis of Institutional Memory and Regulatory Imperatives
Modern cooperatives and financial institutions generate immense volumes of unstructured data, spanning board meeting minutes, compliance policies, human resources manuals, vendor service level agreements (SLAs), and intricate IT workflows. Historically, this information has been siloed within hierarchical folder structures, proprietary document formats, and disconnected legacy systems. This fragmentation creates a crisis of institutional memory, where critical knowledge is inaccessible during time-sensitive operations, such as incident response or regulatory audits. When an API goes down or a vendor breaches an SLA, technology leaders waste precious hours manually retrieving and cross-referencing isolated PDFs rather than executing remediation protocols.
Simultaneously, the regulatory landscape governing financial institutions is becoming increasingly stringent regarding data sovereignty and artificial intelligence (AI). Guidelines such as the Nepal Rastra Bank (NRB) IT Guidelines and the Privacy Act 2075 mandate strict data localization, multi-level security for information systems, and rigorous risk management protocols. Furthermore, the NRB’s recent Artificial Intelligence Guidelines explicitly mandate robust cybersecurity measures, ethical AI use, data privacy compliance, and board-level governance for high-risk AI systems. Institutions are required to adopt a risk-based approach to AI, documenting operational, privacy, and model risks in a dedicated registry while maintaining comprehensive audit trails aligned with ISO/IEC 42001 standards.
These frameworks dictate that sensitive financial data, internal decision-making matrices, and customer information cannot be offloaded to public cloud AI application programming interfaces (APIs) such as OpenAI or Google. Adapting a localized, air-gapped Retrieval-Augmented Generation (RAG) stack into an enterprise context resolves this tension. By orchestrating a knowledge architecture comprising Obsidian as the centralized Markdown vault, IBM Docling and Nvidia Nemotron Parse for unstructured data ingestion, Qdrant for vector storage, LightRAG for graph-based retrieval, and Nvidia Inference Microservices (NIM) for local Large Language Model execution—specifically utilizing open-weight models like Hermes—cooperatives can build a living, interconnected graph of institutional knowledge. This architecture ensures that all financial data remains strictly within the organization’s secure perimeter, providing the Fractional Chief Technology Officer (CTO) with a localized “Second Brain” capable of instantly resolving complex operational interdependencies during emergencies.
Markdown as Enterprise Architecture: From Unstructured Data to Knowledge Graphs
The foundational step in constructing a privacy-first institutional memory is liberating data from proprietary, opaque formats like legacy Word documents (DOCX), scattered Portable Document Formats (PDFs), and complex spreadsheets (XLSX). Transforming this data into a centralized, locally hosted Markdown vault creates a durable, vendor-agnostic foundation that serves as the bedrock for all subsequent AI operations. Markdown forces semantic structuring, mapping complex financial and operational documents into universally readable hierarchies that local LLMs can easily parse and index.
Unstructured Data Ingestion via Docling and Nemotron Parse
To transition legacy documents into Markdown, the architecture relies on advanced document parsing pipelines that understand structural semantics, rather than merely performing naive Optical Character Recognition (OCR). Traditional OCR engines view documents as flat sequences of characters; a table is perceived merely as a collection of words that happen to lie close to each other, destroying the structural relationships required for accurate data retrieval.
IBM’s Docling is a highly optimized Python pipeline designed to parse diverse formats into a unified DoclingDocument representation. Operating locally, Docling applies advanced PDF understanding to capture page layout, reading order, complex table structures, and mathematical formulas. Through its Python integration, the DocumentConverter object can batch-process legacy files, emitting structured Markdown and lossless JSON that perfectly preserves the structural hierarchy required for accurate LLM ingestion. The CLI tool allows for rapid conversion of local directories directly into Markdown, natively supporting vision-language models for advanced figure captioning during the extraction phase.
Alternatively, the Nvidia Nemotron Parse microservice offers a transformer-based Vision-Language Model specialized for multimodal document extraction. Nemotron Parse fundamentally shifts the paradigm from character recognition to document understanding, utilizing specific tokens to represent spatial coordinates and semantic classes such as titles, lists, and figures. This model operates highly efficiently on local GPUs and accurately extracts complex tables, yielding LaTeX tabular strings and semantic segmentation with spatial bounding boxes.
| Parsing Engine | Primary Architecture | Key Capabilities | Optimal Use Case |
|---|---|---|---|
| IBM Docling | Python-based hybrid pipeline (OCR + Layout Models) | Multi-format support (PDF, DOCX, HTML, XBRL), table structure recognition, CLI automation. | Broad enterprise document batch conversion to Markdown and JSON. |
| Nemotron Parse v1.1 | Transformer-based Vision-Language Model | Spatial grounding, LaTeX table extraction, visual layout preservation, NIM containerization. | Highly complex PDFs containing dense financial tables and diagrams. |
| PaddleOCR / Tesseract | Traditional & Deep Learning OCR | Fast character transcription, multi-language support. | Legacy flat-text extraction where structural preservation is secondary. |
By executing these parsing pipelines locally, cooperatives can automate the conversion of decades of historical board minutes, hardware audit logs, and compliance documentation into clean, structured Markdown, fully maintaining the contextual integrity of tables and hierarchies without data ever leaving the secure network.
Architecting the Obsidian Vault
Once documents are converted into Markdown, they are ingested into Obsidian, which serves as the local knowledge graph and primary user interface for the enterprise architecture. Obsidian operates entirely on local, plain-text files, ensuring data permanence and circumventing the proprietary schema lock-in associated with cloud-based alternatives.
In a cooperative environment, the Obsidian vault is structured across three distinct layers to maintain data integrity and AI workflow discipline:
- The raw sources layer: Includes the immutable collection of parsed policies, vendor SLAs, and audit logs. The LLM reads from these but does not modify them, preserving a strict, auditable source of truth.
- The synthesis layer: Houses LLM-generated Markdown files containing incident summaries, entity pages, and cross-references. The AI autonomously updates this layer as new data arrives or incidents occur.
- The schema layer: A configuration document that defines the conventions, workflows, and metadata taxonomies the LLM must follow when organizing the vault.
Obsidian’s performance is highly deterministic and resource-efficient. While cloud-synced applications inject network calls on every keystroke, generating significant write amplification and attention residue, Obsidian’s local execution reduces background write amplification by 91% on NVMe Solid State Drives (SSDs). For enterprise vaults scaling to 50,000 notes, indexing performance remains robust, provided that active plugins are minimized, base64 images are extracted to standard files, and periodic index rebuilds are executed.
The Fractional CTO’s Second Brain: Orchestration and Telemetry
The true value of this architecture emerges when it is operationalized as a “Second Brain” for technology leadership. By integrating Obsidian with the underlying AI stack, a Fractional CTO can exert unprecedented control and visibility over the cooperative’s digital infrastructure, instantly querying dependencies between infrastructure logs and business continuity plans.
Telemetry Ingestion via the Obsidian Local REST API
Obsidian is not merely a static repository; it functions as a programmable endpoint via the Obsidian Local REST API plugin. This capability allows automated, Python-based orchestration workflows to interact dynamically with the vault without requiring manual data entry.
For example, a Fractional CTO must monitor continuous API logs, server health metrics, and newly executed vendor SLAs. A local Python webhook architecture is established to monitor secure network drives and internal syslog servers. When a new vendor contract is signed and deposited as a PDF, a Python file watcher detects the file, triggering the local Docling container to parse the document. The resulting structured text is passed to the local embedding model, indexed, and processed for entity extraction. Finally, the Python script formulates a standardized Markdown summary—complete with YAML frontmatter denoting the vendor name, renewal date, and SLA tiers—and uses an HTTPS POST request via the Local REST API to write the note directly into the Obsidian vault.
Similarly, critical IT alerts (such as a database threshold warning) can be automatically appended to a daily incident log within Obsidian, complete with back-links to the specific servers and affected applications.
The vault becomes a self-updating reflection of the enterprise’s operational state.
Dynamic Dashboards with DataviewJS
To visualize this continuous influx of data, the Fractional CTO utilizes Dataview and DataviewJS. These plugins allow the vault to function as an embedded graph database, rendering dynamic tables and lists by querying the YAML frontmatter appended to each Markdown file.
Using the Dataview Query Language (DQL), the CTO can build dynamic dashboards that aggregate active incident logs, monitor SLA countdown timers, and cross-reference compliance matrices. For instance, a dashboard can dynamically render all vendor contracts where SLA_Status = “At Risk” or where Renewal_Date < date(today) + dur(30 days). Paired with tools like the Casefile plugin, Obsidian can effectively function as a local, fully private Security Operations Center (SOC) incident tracker, utilizing Jira-style epic keys, Kanban swimlanes, and shift-handover reports entirely within the Markdown ecosystem.
Air-Gapped Retrieval-Augmented Generation: The NIM Deployment
To comply with regulations preventing the external transmission of financial data, the LLM answering the Fractional CTO’s queries must operate in a completely air-gapped configuration. Nvidia Inference Microservices (NIM) provides prepackaged, optimized containers that deliver scalable, high-throughput LLM inference locally, wrapping the latest open-weight models (such as the Hermes models based on the Llama architecture) with industry-standard APIs.
Executing the Two-Phase Air-Gap Workflow
Deploying NIMs without internet access follows a rigorous two-phase workflow designed to maintain strict network isolation while allowing for the ingestion of optimized model weights.
During the network-connected phase, administrators operate on a secured staging server with outbound access to the Nvidia GPU Cloud (NGC). After setting the NGC_API_KEY and configuring the local cache directory, a Docker container is initialized to pull the specific LLM image. Utilizing the download-to-cache and create-model-store commands, the system downloads all foundation models, optimized TensorRT engines, and runtime dependencies into a localized directory format.
Once the model store is finalized, the asset transfer phase begins. The consolidated model store and container images are securely transferred across the air gap using physical media or secure copy protocols (SCP). In enterprise Kubernetes environments, these images must be mirrored into a private container registry (such as Harbor) operating strictly within the enclave. Administrators must ensure all deployment manifests point to the internal repository and set the imagePullPolicy to IfNotPresent, rigorously verifying that no workloads reference external hosted APIs.
Finally, during the air-gapped phase, the production servers instantiate the NIM container by mounting the transferred model repository. The container operates entirely offline, binding to local ports (e.g., 8000) and utilizing environment variables like NIM_MODEL_PATH and NIM_SERVED_MODEL_NAME to serve the Hermes model via an OpenAI-compatible endpoint. This ensures that all inference happens precisely where the data resides.
Optimizing Inference: TensorRT-LLM vs. vLLM
Nvidia NIM supports multiple inference backends, primarily TensorRT-LLM and vLLM. Selecting the correct backend depends on the operational cadence and specific throughput requirements of the cooperative.
TensorRT-LLM delivers the highest possible throughput and lowest latency, particularly Time to First Token (TTFT). However, it requires extensive build-time engine compilation tailored to the specific GPU architecture. This backend is optimal for stable, production-grade SLA resolution endpoints where the model weights will not change for extended periods, providing a 10-30% performance advantage at high concurrency.
Conversely, vLLM offers near-TensorRT performance but allows direct loading of HuggingFace models without prior compilation, executing in minutes rather than hours. For the Fractional CTO continuously iterating on new regulatory models or integrating fine-tuned Hermes variants for specific compliance parsing, vLLM provides the necessary agility.
| Factor | TensorRT-LLM | vLLM |
|---|---|---|
| Setup Time | Hours (engine compilation per GPU architecture) | Minutes (direct load from HuggingFace/local store) |
| Throughput | Highest baseline performance | 80-90% of TensorRT-LLM |
| Quantization | Build-time (INT8, FP8) | Runtime (AWQ, GPTQ, FP8 KV Cache) |
| Optimal Use Case | Stable production serving maximum concurrent users | Dev, testing, rapid iteration, and fallback |
Hardware Provisioning: Sizing the Data Center for Local AI
Running advanced AI pipelines entirely on-premises necessitates carefully provisioned hardware. The computational demands of parsing documents, generating embeddings, and running multi-turn reasoning workflows dictate the selection of specific data center infrastructure that consumer-grade components cannot support.
Enterprise Server Topologies and GPU Selection
For an air-gapped enterprise environment, a 2U rackmount server such as the Dell PowerEdge R760xa provides the optimal chassis. Engineered specifically for high-density GPU compute, the R760xa supports dual 4th or 5th Generation Intel Xeon Scalable processors, up to 8TB of DDR5 memory, and crucially, up to four double-wide enterprise GPUs. Its MultiVector Cooling architecture and optional Direct Liquid Cooling ensure that passive enterprise GPUs do not succumb to thermal throttling during sustained inference loads, a critical requirement for maintaining SLA uptime.
When selecting the accelerator for LLM inference (specifically for models in the 7B to 30B parameter range, such as open-weight Hermes configurations), the NVIDIA L40S emerges as the enterprise champion. Based on the Ada Lovelace architecture, the L40S is a passive, data-center-licensed GPU featuring 48 GB of GDDR6 ECC memory and a 350W thermal design power (TDP). While the H100 (Hopper) excels in massive-scale training and long-context inference via its HBM3 memory, the L40S provides superior total cost of ownership (TCO) and throughput-per-watt for high-concurrency generative AI inference.
| GPU Model | Memory | TDP | Optimal Inference Workload |
|---|---|---|---|
| NVIDIA H100 | 80 GB HBM3 | 700W | Large-scale LLMs (>70B), massive concurrency |
| NVIDIA L40S | 48 GB GDDR6 ECC | 350W | High-throughput 7B-30B inference, cost-sensitive GenAI ops |
| RTX 6000 Ada | 48 GB GDDR6 ECC | 300W | Active-cooled workstation prototyping, quiet environments |
Consumer cards like the RTX 4090 or RTX 5090 are explicitly prohibited in commercial data centers by Nvidia’s GeForce driver End User License Agreement (EULA). Furthermore, consumer GPUs possess active cooling fans designed for open cases with side airflow; when placed in a high-static-pressure rack server like the R760xa, they rapidly overheat and throttle, destabilizing the entire AI pipeline.
VRAM and Memory Capacity Calculations for Long Context
Ensuring that a selected LLM fits within the hardware involves calculating both the static model weights and the dynamic Key-Value (KV) cache utilized during inference. Modern engines like vLLM preallocate this memory, meaning capacity planning must account for maximum context lengths (max_model_len) and concurrent users.
The memory required for model parameters is calculated as the parameter count multiplied by the precision bytes. For an 8-billion (8B) parameter model utilizing FP8 (8-bit floating point) quantization, the weight memory is approximately 8.5 GB. However, the dynamic KV cache, which stores attention states for every token in the context window to prevent re-computation, expands linearly.
The KV cache footprint is calculated based on several variables:
- Number of transformer layers
- Number of KV heads
- Head dimension (hidden_size / num_attention_heads)
- Sequence/Context length
- Precision bytes (1 for FP8, 2 for FP16)
For a 32,000-token context window running a typical 8B model in FP16, the KV cache consumes approximately 4.1 GB per concurrent request. Given that vLLM begins evicting cache when GPU memory utilization reaches approximately 85%, an L40S with 48 GB of VRAM hosting an 8.5 GB model leaves roughly 32 GB for active context. Utilizing FP8 quantization for the KV cache halves this memory footprint to 2 GB per request, effectively doubling the concurrent user capacity or allowing the deployment of larger 30B parameter models alongside their associated context windows without triggering out-of-memory (OOM) failures.
Graph-Enhanced Semantic Retrieval: LightRAG and Qdrant
Traditional RAG architectures rely on naive semantic chunking, where documents are cut into arbitrary text blocks and matched via vector similarity. This approach fails to capture complex interdependencies, resulting in fragmented answers when querying dense institutional data, such as determining how a human resources policy change impacts specific IT security access controls.
Dual-Level Graph Retrieval with LightRAG
LightRAG fundamentally upgrades this process by integrating graph structures directly into the text indexing and retrieval pipeline. When an unstructured Markdown document from the Obsidian vault is ingested, LightRAG prompts the local LLM to extract multi-granularity entities (nodes) and their relationships (edges) in a single pass.
This creates a highly efficient, dual-level retrieval paradigm:
- Low-Level Retrieval: Focuses on fetching precise information about specific entities and their attributes. For example, extracting the exact uptime penalty clause from a specific vendor SLA.
- High-Level Retrieval: Aggregates information across multiple related entities to address broader thematic queries.
For instance, summarizing the overarching cybersecurity risk posture across all third-party vendors. To handle polysemy—where a single term holds multiple meanings depending on context—advanced iterations like LiteSemRAG implement clustering-based semantic induction. By applying density-based clustering algorithms (such as HDBSCAN) to contextual embeddings, the system ensures that an entity like “server” is contextually differentiated between a physical hardware asset in an audit log and a software application server in an API manual. Because it constructs this graph locally and allows for incremental updates without rebuilding the entire index, LightRAG seamlessly synchronizes with the Obsidian vault’s file watcher, ensuring that the knowledge graph is updated in near real-time as new Markdown notes are saved.
Qdrant for Secure Vector Storage and Multi-Tenancy
The vector embeddings and graph structures generated by LightRAG require a highly performant storage backend. Qdrant, an open-source vector search engine written in Rust, provides fast and scalable similarity search while accommodating the strict data siloing required in a cooperative environment. In financial institutions, human resources must not have access to unredacted IT security vulnerability reports, and customer service teams should not view executive board minutes.
Rather than provisioning computationally expensive, separate vector collections for every department, Qdrant natively supports sophisticated multitenancy and Role-Based Access Control (RBAC) through payload filtering. During ingestion, each Markdown chunk is tagged with a JSON payload containing metadata (e.g., {“department”: “HR”, “tenant_id”: “Confidential”}). Qdrant supports multiple data types for these payloads, including 64-bit integers, floats, keywords, and RFC 3339 datetimes.
| Payload Type | Qdrant Implementation | Financial Data Use Case |
|---|---|---|
| Keyword | String value (keyword) | Department routing, tenant_id, strict access control categories. |
| Datetime | RFC 3339 format (datetime) | Filtering SLA expiration dates, audit timestamps, or contract renewals. |
| Float | 64-bit floating point (float) | Financial thresholds, uptime percentages, penalty multipliers. |
When the LLM queries the database, the API request enforces a hard filter on these payloads before executing the semantic similarity search. Because Qdrant can isolate tenants physically at the shard level while maintaining a single collection, a query scoped to the IT department never touches the HR shard, effectively eliminating the “noisy neighbor” problem and guaranteeing data privacy at the foundational vector level.
AI Governance and Data Sovereignty with NeMo Guardrails
Deploying autonomous LLMs within a heavily regulated financial entity introduces significant risks, including hallucinated financial data, prompt injection attacks, and the inadvertent exposure of Personally Identifiable Information (PII). The NRB Artificial Intelligence Guidelines demand continuous performance tracking, ethical AI utilization, and strict privacy protection mechanisms.
To satisfy these mandates, the architecture integrates Nvidia NeMo Guardrails, an open-source runtime orchestration layer that intercepts requests and evaluates them against strict, programmable policies before they reach the main Hermes LLM. NeMo Guardrails operates using Colang 2.0, a highly flexible modeling language that defines conversational logic, state tracking, and boundary constraints through natural language constructs.
The system implements multiple layers of protection across the conversational pipeline:
- Input Rails: Scrutinize raw user prompts to detect and block jailbreak attempts or adversarial injections using fast, lightweight classifier models. For example, Llama Prompt Guard 2 (an 86M parameter model) can catch obvious injection attempts in 20-50 milliseconds on an L40S before invoking larger, more expensive classifiers.
- Dialog Rails: Control conversational flow based on predefined Colang 2.0 flows, ensuring the model refuses to engage in off-topic discussions (e.g., politely declining to provide investment advice when configured solely for IT infrastructure queries).
- Retrieval Rails: Validate the context retrieved from Qdrant, ensuring that only highly relevant and authorized data is passed to the generation phase.
- Output Rails: Evaluate the final LLM generation for factual grounding, ensuring it strictly adheres to the retrieved context and preventing hallucinated SLA terms from being presented to the user.
Crucially for cooperatives processing customer data, NeMo Guardrails integrates advanced PII masking capabilities utilizing models like the NVIDIA GLiNER-PII NIM or Microsoft Presidio. A Colang 2.0 masking flow evaluates the input stream; if an employee pastes a customer’s account number, email address, or social security number into the prompt, the input rail automatically detects and masks the entity (e.g., replacing it with
Emergency Incident Response: Operationalizing the Stack
The culmination of this architecture is its operationalization during critical events. Consider a scenario where a core database server experiences an unexpected outage. The Fractional CTO must immediately determine the blast radius, specifically how the outage impacts the Mobile Banking SLA, and identify which third-party vendors must be notified.
In a traditional setup, this requires manually cross-referencing disparate IT architecture diagrams, vendor contracts, and compliance thresholds while the system remains down. Within the Obsidian and NIM RAG stack, the CTO submits a natural language query directly within their Obsidian interface: “Map the dependencies of Server DB-01 to the Mobile Banking service and outline our SLA notification obligations.”
The resolution unfolds autonomously within milliseconds:
- Authentication and Guardrails: NeMo Guardrails authenticates the query, ensuring the user’s role permits access to infrastructure data, while simultaneously scanning the input for prompt injections using the local classifier.
- Semantic Graph Traversal: LightRAG queries the knowledge graph stored in Qdrant. The low-level retrieval identifies the direct edge connecting “DB-01” to the “Authentication Microservice.” The high-level retrieval links “Authentication Microservice” to the broad “Mobile Banking Application” node, subsequently retrieving the specific uptime thresholds from the associated vendor SLAs mapped in the Markdown vault.
- Inference and Synthesis: The air-gapped Hermes model, running via vLLM on the Dell R760xa’s L40S GPUs, processes the retrieved paths within its 32,000-token context window (optimized via FP8 KV cache). It synthesizes a coherent, cited response detailing the exact systems impacted, the 15-minute notification window required by the vendor SLA, and the specific compliance personnel to contact.
- Automated Telemetry: Utilizing Obsidian’s Dataview DQL capabilities, the vault automatically populates a real-time incident dashboard. As the AI generates the incident report, Dataview aggregates metadata tags (e.g., status: active_incident, severity: high), updating dynamic Kanban boards and SLA countdown timers directly within the Markdown interface.
Conclusion
The convergence of local Markdown vaults, advanced document parsing, graph-enhanced vector retrieval, and sovereign LLM inference represents a paradigm shift for enterprise knowledge management. For cooperatives navigating the complex intersection of massive unstructured data and stringent regulatory compliance, the traditional reliance on fragmented folders and public cloud APIs is no longer viable.
By architecting a privacy-first stack utilizing Obsidian for knowledge structuring, IBM Docling and Nemotron Parse for unstructured ingestion, Qdrant and LightRAG for advanced retrieval, and Hermes via Nvidia NIM for local inference on robust hardware like the Dell PowerEdge R760xa, institutions can reclaim total control over their data. This air-gapped infrastructure not only satisfies the rigorous demands of central banking guidelines and data privacy laws but fundamentally transforms how organizational knowledge is utilized. It empowers technology leaders with an autonomous, highly secure “Second Brain,” capable of instantaneously synthesizing decades of institutional memory to drive precise, compliant, and decisive operational action in the face of complexity.
