Enterprise Grounded RAG on SAP BTP: Generative AI Hub & HANA Vector Engine

A step-by-step architectural guide to designing compliance-first, zero-retention Grounded Retrieval-Augmented Generation (RAG) architectures on SAP Business Technology Platform.

1. The 3-Tier Enterprise AI Architecture

In a Clean Core paradigm, generative AI capabilities are deployed side-by-side on SAP BTP (Tier 3) rather than executing direct unmanaged Python processes inside S/4HANA:

• Tier 1 (S/4HANA Core): Exposes released CDS View Entities and OData V4 services (e.g., I_PurchaseOrderAPI01) acting as authoritative ground truth.
• Tier 2 (SAP BTP HANA Cloud Vector Engine): Stores multi-dimensional document chunks and embeddings (1536-dim text-embedding-ada-002 or text-embedding-3-small) using the native REAL_VECTOR data type.
• Tier 3 (SAP Generative AI Hub): Orchestrates model endpoints (GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro) with tenant isolation, rate limiting, and audit logging.

2. SAP HANA Cloud Vector DDL

Create the native vector storage table in SAP HANA Cloud using the REAL_VECTOR datatype and an HNSW vector index:

-- SAP HANA Cloud Vector Engine DDL
CREATE COLUMN TABLE SAP_ENTERPRISE_KNOWLEDGE_BASE (
  DOCUMENT_ID     VARCHAR(64) NOT NULL,
  CHUNK_INDEX     INTEGER NOT NULL,
  SOURCE_SYSTEM   VARCHAR(32) DEFAULT 'S4HANA_CLOUD',
  DOCUMENT_TITLE  NVARCHAR(256),
  CHUNK_CONTENT   NCLOB,
  EMBEDDING       REAL_VECTOR(1536), -- 1536-dim embeddings
  CREATED_AT      TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
  PRIMARY KEY (DOCUMENT_ID, CHUNK_INDEX)
);

-- Fast Approximate Nearest Neighbor (ANN) Vector Index
CREATE HNSW INDEX IDX_HANA_VECTOR_SEARCH 
  ON SAP_ENTERPRISE_KNOWLEDGE_BASE (EMBEDDING)
  COSINE SIMILARITY;

3. Cosine Similarity Hybrid Search Query

Perform cosine similarity ranking directly inside the HANA Cloud SQL engine:

-- Top-K Similarity Search with Score Thresholding (> 0.75)
SELECT TOP 3
  DOCUMENT_ID,
  CHUNK_INDEX,
  DOCUMENT_TITLE,
  CHUNK_CONTENT,
  COSINE_SIMILARITY(EMBEDDING, TO_REAL_VECTOR(?)) AS RELEVANCE_SCORE
FROM SAP_ENTERPRISE_KNOWLEDGE_BASE
WHERE COSINE_SIMILARITY(EMBEDDING, TO_REAL_VECTOR(?)) > 0.75
ORDER BY RELEVANCE_SCORE DESC;

4. Orchestration with SAP CAP & Generative AI Hub

Connect to the Generative AI Hub destination using the SAP Cloud Application Programming model (CAP) in Node.js:

const cds = require('@sap/cds');

module.exports = cds.service.impl(async function() {
  this.on('askGroundedAssistant', async (req) => {
    const { userPrompt } = req.data;
    const db = await cds.connect.to('db');

    // 1. Generate query embedding via GenAI Hub
    const aiCore = await cds.connect.to('SAP_AI_CORE_DESTINATION');
    const embedRes = await aiCore.send('POST', '/v2/inference/deployments/embed-deploy-id/embeddings', {
      input: [userPrompt]
    });
    const queryVector = embedRes.data[0].embedding;

    // 2. Perform Vector Search on HANA Cloud
    const chunks = await db.run(
      SELECT.from('SAP_ENTERPRISE_KNOWLEDGE_BASE')
        .columns('DOCUMENT_TITLE', 'CHUNK_CONTENT', 'COSINE_SIMILARITY(EMBEDDING, TO_REAL_VECTOR(?)) as SCORE')
        .where('COSINE_SIMILARITY(EMBEDDING, TO_REAL_VECTOR(?)) >', 0.75)
        .orderBy('SCORE desc')
        .limit(3),
      [queryVector, queryVector]
    );

    const contextText = chunks.map(c => c.CHUNK_CONTENT).join('\n---\n');

    // 3. Invoke LLM with strict grounding instructions
    const completion = await aiCore.send('POST', '/v2/inference/deployments/chat-deploy-id/chat/completions', {
      messages: [
        { role: 'system', content: 'You are an SAP Enterprise AI Assistant. Answer questions ONLY based on the provided context.' },
        { role: 'user', content: `CONTEXT:\n${contextText}\n\nQUESTION:\n${userPrompt}` }
      ],
      temperature: 0.1
    });

    return completion.choices[0].message.content;
  });
});