Advanced RAG with Azure AI Search and Claude Code

Introduction

Retrieval Augmented Generation (RAG) has become a cornerstone for building sophisticated AI applications that leverage external knowledge. By combining the power of Large Language Models (LLMs) with a dedicated retrieval system, we can ground AI responses in factual, up-to-date information, thereby reducing hallucinations and improving accuracy. This chapter focuses on constructing advanced RAG pipelines using Azure AI Search as the retrieval engine and Claude Code as the generative model. We’ll explore architectural patterns, practical implementation details, and common pitfalls to avoid.

TL;DR

• Learn to architect and build RAG pipelines integrating Azure AI Search for retrieval and Claude Code for generation.
• Explore practical implementation patterns in .NET and Java, including data indexing, query translation, and response synthesis.
• Understand how to leverage Azure AI Search’s advanced features for nuanced retrieval.
• Identify common anti-patterns and best practices for robust and scalable RAG systems.

Architectural Considerations

When designing a RAG pipeline with Azure AI Search and Claude Code, several architectural decisions are critical for scalability, maintainability, and performance.

Data Ingestion and Indexing Strategy

The effectiveness of your RAG system is directly proportional to the quality and structure of your indexed data. Azure AI Search offers a flexible schema and various indexing capabilities.

• Chunking: Documents must be broken down into meaningful chunks. The optimal chunk size depends on your data and retrieval strategy. Too small, and context is lost; too large, and retrieval becomes less precise. Consider overlapping chunks to preserve context across boundaries.
• Metadata: Enriching chunks with relevant metadata (e.g., source document, date, author, section title) is crucial. This metadata can be used for advanced filtering during retrieval, improving precision and relevance.
• Vectorization: For semantic search, each chunk needs to be embedded into a vector. You can use Azure OpenAI Embeddings or other embedding models. Azure AI Search supports vector search directly, enabling efficient similarity searches.
• Index Design: Design your Azure AI Search index with appropriate fields for text content, vectors, and metadata. Consider using a hybrid search approach that combines keyword (full-text) search with vector search for robust retrieval.

Retrieval Strategy

Beyond simple keyword or vector similarity, Azure AI Search provides advanced features that can significantly enhance retrieval accuracy.

• Hybrid Search: A combination of keyword search (e.g., BM25) and vector search often yields the best results. Keyword search excels at exact matches, while vector search captures semantic meaning.
• Reranking: After an initial retrieval, a reranking step can further refine the order of results. This can be done using a separate, more powerful model or by incorporating additional scoring mechanisms based on metadata.
• Query Expansion/Translation: The user’s query might not directly map to terms or concepts in your knowledge base. Employing techniques to expand or translate the query before sending it to Azure AI Search can improve recall. This can involve using an LLM to rephrase the query or generate synonyms.

Generation Strategy

Claude Code excels at synthesizing information and generating coherent responses based on provided context.

• Prompt Engineering: The prompt sent to Claude Code is paramount. It should clearly instruct the model to answer the user’s question based only on the provided retrieved documents. Specify the desired output format, tone, and any constraints.
• Context Window Management: Claude Code has a significant context window, but it’s not infinite. Carefully manage the amount of retrieved text you include in the prompt. Prioritize the most relevant chunks based on the reranking score.
• Citation: To maintain trust and verifiability, ensure Claude Code can cite its sources. This can be achieved by including source information within the retrieved chunks and instructing the model to reference these sources in its answer.

Practical Implementation with .NET

This section provides practical code examples for building a RAG pipeline in .NET.

Setting up Azure AI Search

First, ensure you have an Azure AI Search service provisioned. You’ll need the service name and an API key. Install the Azure.Search.Documents NuGet package.

using Azure.Search.Documents;
using Azure.Search.Documents.Indexes;
using Azure.Search.Documents.Indexes.Models;
using Azure.Search.Documents.Models;
using Azure.Core;

public class SearchService
{
    private readonly SearchIndexClient _indexClient;
    private readonly SearchClient _searchClient;
    private readonly string _indexName = "my-rag-index"; // Replace with your index name

    public SearchService(string searchServiceName, string apiKey)
    {
        var credential = new AzureKeyCredential(apiKey);
        var endpoint = $"https://{searchServiceName}.search.windows.net";

        _indexClient = new SearchIndexClient(new Uri(endpoint), credential);
        _searchClient = new SearchClient(new Uri(endpoint), _indexName, credential);
    }

    // Method to create the index (run this once)
    public async Task CreateIndexAsync()
    {
        var index = new SearchIndex(_indexName, new Field[]
        {
            new SimpleField("id", SimpleFieldType.String) { IsKey = true },
            new SearchableField("content") { IsFilterable = true, IsSortable = true, IsFacetable = true },
            new SearchField("contentVector", SearchFieldDataType.Collection(SearchFieldDataType.Single))
            {
                IsSearchable = true, IsFilterable = false, IsSortable = false, IsFacetable = false, VectorSearchDimensions = 1536 // Example dimension
            },
            new SimpleField("sourceDocument", SimpleFieldType.String) { IsFilterable = true, IsSortable = true, IsFacetable = true },
            // Add other metadata fields as needed
        })
        {
            VectorSearch = new VectorSearch()
            {
                AlgorithmConfigurations = { new HnswAlgorithmConfiguration("my-hnsw-config") },
                VectorSearchProfile = new VectorSearchProfile("my-vector-profile", "my-hnsw-config")
            }
        };

        await _indexClient.CreateIndexAsync(index);
    }

    // Method to index documents
    public async Task IndexDocumentsAsync(IEnumerable<MyDocument> documents)
    {
        var searchableDocuments = documents.Select(doc => new IndexAction<MyDocument>(IndexActionType.Upload, doc));
        await _searchClient.IndexDocumentsAsync(searchableDocuments);
    }

    // Method for hybrid search
    public async Task<SearchResults<MyDocument>> SearchAsync(string query, string? vectorQuery = null, int topK = 5)
    {
        var searchOptions = new SearchOptions
        {
            Size = topK,
            IncludeTotalCount = true,
            QueryType = SearchMode.All, // Or SearchMode.Any
            Select = { "id", "content", "sourceDocument" } // Select desired fields
        };

        // Configure hybrid search
        searchOptions.SemanticConfiguration = "my-semantic-config"; // Assuming a semantic configuration is set up in Azure AI Search
        searchOptions.VectorQueries.Add(new VectorQuery("contentVector", Convert.FromBase64String(vectorQuery)) { KNearestNeighborsCount = topK, Fields = { "contentVector" } });

        var results = await _searchClient.SearchAsync<MyDocument>(query, searchOptions);
        return results;
    }
}

// Example document structure for indexing
public class MyDocument
{
    public string Id { get; set; }
    public string Content { get; set; }
    public float[] ContentVector { get; set; } // For vector indexing
    public string SourceDocument { get; set; }
    // Add other metadata properties
}

Interacting with Claude Code

For interacting with Claude Code, you’ll typically use the claude CLI or an SDK if available. Assuming you have the claude CLI installed and configured with your API key.

# Example CLI command to interact with Claude
claude --model claude-3-opus-20240229 --message "User: Summarize the following text. \n\n Context: [retrieved text content] \n\n Question: [user's original question]"

Building the RAG Pipeline Logic

The core RAG logic involves fetching documents from Azure AI Search and then using them to construct a prompt for Claude Code.

using Azure.Search.Documents.Models;
using System.Text;

public class RagPipeline
{
    private readonly SearchService _searchService;
    // Assuming you have an EmbeddingService to convert text to vectors
    // private readonly EmbeddingService _embeddingService;

    public RagPipeline(SearchService searchService /*, EmbeddingService embeddingService*/)
    {
        _searchService = searchService;
        // _embeddingService = embeddingService;
    }

    public async Task<string> AnswerQuestionAsync(string question)
    {
        // 1. Generate vector for the query (if using vector search)
        // var queryVector = await _embeddingService.GetEmbeddingsAsync(question);
        // var vectorQueryBase64 = Convert.ToBase64String(queryVector); // Or appropriate serialization

        // 2. Perform search in Azure AI Search
        // Replace with actual vector query generation if needed
        var searchResults = await _searchService.SearchAsync(question, null); // Pass vectorQueryBase64 if available

        // 3. Construct the prompt for Claude Code
        var contextBuilder = new StringBuilder();
        var citations = new List<string>();
        int chunkCount = 0;
        const int maxContextChunks = 5; // Limit context to a reasonable number of chunks

        foreach (var result in searchResults.GetResults().Take(maxContextChunks))
        {
            var doc = result.Document;
            contextBuilder.AppendLine($"--- Document Chunk {chunkCount + 1} ---");
            contextBuilder.AppendLine($"Source: {doc.SourceDocument}");
            contextBuilder.AppendLine($"Content: {doc.Content}");
            contextBuilder.AppendLine();
            citations.Add($"{doc.SourceDocument} (Chunk {chunkCount + 1})");
            chunkCount++;
        }

        if (chunkCount == 0)
        {
            return "I couldn't find any relevant information to answer your question.";
        }

        var prompt = $@"You are an AI assistant. Answer the question based ONLY on the provided context. If the answer is not found in the context, state that you cannot find the answer.
Do not use any prior knowledge.
Format your answer clearly and provide citations for the information used.

Context:
{contextBuilder.ToString()}

Question: {question}

Answer:";

        // 4. Call Claude Code
        // This is a conceptual representation of calling Claude Code CLI
        // In a real application, you would execute this as a process or use an SDK if available.
        // For demonstration, we'll simulate the output.
        Console.WriteLine("--- Sending to Claude Code ---");
        Console.WriteLine(prompt);
        Console.WriteLine("------------------------------");

        // Replace with actual Claude Code invocation and response parsing
        string claudeResponse = await SimulateClaudeCodeResponse(prompt);

        // 5. Post-processing (add citations)
        return PostProcessClaudeResponse(claudeResponse, citations);
    }

    // Placeholder for simulating Claude Code response
    private async Task<string> SimulateClaudeCodeResponse(string prompt)
    {
        // In a real scenario, this would involve:
        // 1. Executing the `claude` CLI command.
        // 2. Capturing stdout.
        // 3. Parsing the output.
        // For this example, we'll return a mock response.
        await Task.Delay(100); // Simulate network latency
        return "Based on the provided context, the answer to your question is [Synthesized Answer from Context]. Citations: [Placeholder for citations].";
    }

    private string PostProcessClaudeResponse(string claudeResponse, List<string> citations)
    {
        // Enhance the response with actual citations if Claude Code didn't inline them perfectly
        var finalAnswer = claudeResponse;
        if (!finalAnswer.Contains("Citations:"))
        {
            finalAnswer += $"\n\nReferences:\n{string.Join("\n", citations)}";
        }
        return finalAnswer;
    }
}

Practical Implementation with Java and Spring Boot

This section outlines building a RAG pipeline using Java with Spring Boot and Azure AI Search.

Maven Dependencies

Ensure you have the necessary Azure SDK for Java dependencies in your pom.xml:

<dependency>
    <groupId>com.azure</groupId>
    <artifactId>azure-search-documents</artifactId>
    <version>11.6.0</version> <!-- Use the latest version -->
</dependency>
<!-- Add dependencies for embedding models if needed -->

Azure AI Search Configuration and Service

Configure your Azure AI Search client within your Spring Boot application.

import com.azure.core.credential.AzureKeyCredential;
import com.azure.search.documents.SearchClient;
import com.azure.search.documents.SearchClientBuilder;
import com.azure.search.documents.SearchIndexClient;
import com.azure.search.documents.SearchIndexClientBuilder;
import com.azure.search.documents.indexes.models.*;
import com.azure.search.documents.util.SearchDocumentStream;
import com.azure.search.documents.util.SearchPagedIterable;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

import java.util.Arrays;
import java.util.List;
import java.util.stream.Collectors;

@Configuration
public class AzureSearchConfig {

    @Value("${azure.search.service.name}")
    private String searchServiceName;

    @Value("${azure.search.api.key}")
    private String searchApiKey;

    @Value("${azure.search.index.name:my-rag-index}")
    private String indexName;

    @Bean
    public SearchIndexClient searchIndexClient() {
        return new SearchClientBuilder()
            .endpoint(String.format("https://%s.search.windows.net", searchServiceName))
            .credential(new AzureKeyCredential(searchApiKey))
            .build();
    }

    @Bean
    public SearchClient searchClient(SearchIndexClient searchIndexClient) {
        // Cast is safe here as SearchClientBuilder builds SearchClient
        return new SearchClientBuilder()
            .endpoint(String.format("https://%s.search.windows.net", searchServiceName))
            .credential(new AzureKeyCredential(searchApiKey))
            .indexName(indexName)
            .build();
    }

    // Method to create index (call this during application startup or via an admin process)
    public void createIndexIfNotExists() {
        SearchIndexClient indexClient = searchIndexClient();
        try {
            indexClient.getIndex(indexName);
            System.out.println("Index already exists: " + indexName);
        } catch (Exception e) {
            System.out.println("Creating index: " + indexName);
            SearchIndex index = new SearchIndex(indexName, Arrays.asList(
                new SimpleField("id", SimpleFieldType.STRING).setKey(true),
                new SearchableField("content").setFilterable(true).setSortable(true).setFacetable(true),
                new SearchField("contentVector", SearchFieldDataType.COLLECTION_SINGLE).setSearchable(true), // Vector field
                new SimpleField("sourceDocument", SimpleFieldType.STRING).setFilterable(true).setSortable(true).setFacetable(true)
                // Add other metadata fields
            ))
            .setVectorSearch(new VectorSearch()
                .setAlgorithmConfigurations(Arrays.asList(new HnswAlgorithmConfiguration("my-hnsw-config")))
                .setVectorSearchProfile(new VectorSearchProfile("my-vector-profile", "my-hnsw-config"))
            );
            indexClient.createIndex(index);
        }
    }

    // Method for hybrid search
    public SearchDocumentStream search(String query, String vectorQueryBase64, int topK) {
        SearchClient searchClient = searchClient(null); // Re-fetch or use injected instance

        SearchOptions searchOptions = new SearchOptions()
            .setIncludeTotalCount(true)
            .setTopK(topK)
            .setQueryType(SearchMode.ALL) // Or SearchMode.ANY
            .setSelect("id", "content", "sourceDocument"); // Select desired fields

        // Configure hybrid search
        searchOptions.setSemanticConfiguration("my-semantic-config"); // Assuming semantic config is set up

        // Add vector query if available
        if (vectorQueryBase64 != null && !vectorQueryBase64.isEmpty()) {
            byte[] vectorBytes = java.util.Base64.getDecoder().decode(vectorQueryBase64);
            searchOptions.getVectorQueries().add(new VectorQuery("contentVector", vectorBytes)
                .setKNearestNeighborsCount(topK)
                .setFields(Arrays.asList("contentVector")));
        }

        return searchClient.search(query, searchOptions);
    }
}

Claude Code Integration (Conceptual)

Interacting with claude CLI from Java typically involves using ProcessBuilder or an external command execution library.

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.util.List;
import java.util.stream.Collectors;

public class ClaudeService {

    // Placeholder for calling Claude Code CLI
    public String generateResponse(String prompt) throws IOException, InterruptedException {
        ProcessBuilder processBuilder = new ProcessBuilder(
            "claude",
            "--model", "claude-3-opus-20240229",
            "--message", prompt
        );

        Process process = processBuilder.start();
        StringBuilder output = new StringBuilder();
        try (BufferedReader reader = new BufferedReader(new InputStreamReader(process.getInputStream()))) {
            String line;
            while ((line = reader.readLine()) != null) {
                output.append(line).append("\n");
            }
        }

        int exitCode = process.waitFor();
        if (exitCode != 0) {
            // Handle error: log stderr, throw exception
            try (BufferedReader errorReader = new BufferedReader(new InputStreamReader(process.getErrorStream()))) {
                String errorLine;
                StringBuilder errorOutput = new StringBuilder();
                while ((errorLine = errorReader.readLine()) != null) {
                    errorOutput.append(errorLine).append("\n");
                }
                throw new IOException("Claude CLI exited with code " + exitCode + ": " + errorOutput.toString());
            }
        }
        return output.toString().trim();
    }
}

RAG Pipeline Logic in Java

Combine the search and generation steps.

import com.azure.search.documents.models.SearchResult;
import com.azure.search.documents.util.SearchDocumentStream;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;

import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import java.util.stream.Collectors;

@Service
public class RagPipelineJava {

    @Autowired
    private AzureSearchConfig azureSearchConfig; // Inject your SearchClient/IndexClient bean

    @Autowired
    private ClaudeService claudeService; // Inject your ClaudeService

    // Assuming an EmbeddingService is available/injected

    public String answerQuestion(String question) {
        // 1. Generate vector for the query (if using vector search)
        // String vectorQueryBase64 = embeddingService.getEmbeddingsBase64(question);

        // 2. Perform search in Azure AI Search
        SearchDocumentStream searchResultsStream = azureSearchConfig.search(question, null, 5); // Pass vector query if available

        // 3. Construct the prompt for Claude Code
        StringBuilder contextBuilder = new StringBuilder();
        List<String> citations = new ArrayList<>();
        int chunkCount = 0;
        final int maxContextChunks = 5;

        for (SearchResult result : searchResultsStream) {
            if (chunkCount >= maxContextChunks) break;

            String content = (String) result.getDocument().get("content");
            String sourceDocument = (String) result.getDocument().get("sourceDocument");

            contextBuilder.append("--- Document Chunk ").append(chunkCount + 1).append(" ---\n");
            contextBuilder.append("Source: ").append(sourceDocument).append("\n");
            contextBuilder.append("Content: ").append(content).append("\n\n");
            citations.add(sourceDocument + " (Chunk " + (chunkCount + 1) + ")");
            chunkCount++;
        }

        if (chunkCount == 0) {
            return "I couldn't find any relevant information to answer your question.";
        }

        String prompt = String.format(
            "You are an AI assistant. Answer the question based ONLY on the provided context. If the answer is not found in the context, state that you cannot find the answer.\n" +
            "Do not use any prior knowledge.\n" +
            "Format your answer clearly and provide citations for the information used.\n\n" +
            "Context:\n%s\n\n" +
            "Question: %s\n\n" +
            "Answer:",
            contextBuilder.toString(), question
        );

        // 4. Call Claude Code
        String claudeResponse = "Error generating response from Claude.";
        try {
            claudeResponse = claudeService.generateResponse(prompt);
        } catch (IOException | InterruptedException e) {
            e.printStackTrace(); // Log the error properly
            return "An error occurred while generating the response.";
        }

        // 5. Post-processing (add citations)
        return postProcessClaudeResponse(claudeResponse, citations);
    }

    private String postProcessClaudeResponse(String claudeResponse, List<String> citations) {
        // Enhance the response with actual citations if Claude Code didn't inline them perfectly
        String finalAnswer = claudeResponse;
        if (!finalAnswer.contains("Citations:") && !finalAnswer.contains("References:")) {
            finalAnswer += "\n\nReferences:\n" + String.join("\n", citations);
        }
        return finalAnswer;
    }
}

Advanced Retrieval Techniques

Azure AI Search offers powerful features to fine-tune your retrieval.

• Semantic Search: Configure semantic ranking in Azure AI Search to improve the relevance of search results beyond vector similarity. This often involves training a semantic model on your data.
• Query Intentions: Analyze user queries to determine if they require factual answers, summarization, or creative generation. Route queries accordingly.
• Contextual Filtering: Use metadata filters in your Azure AI Search queries to narrow down results to specific sources, dates, or categories, ensuring that the LLM is provided with the most relevant and scoped information.

Anti-patterns

1. Over-reliance on Exact Keyword Matching

Problem: Only using basic keyword search or vector similarity without considering semantic nuances or hybrid approaches. Why it’s wrong: Users rarely formulate queries that exactly match the language in your knowledge base. Relying solely on exact matches leads to poor recall, as semantically similar but lexically different queries will fail to find relevant documents. Solution: Implement hybrid search (keyword + vector) in Azure AI Search. Consider query expansion or using an LLM to rephrase user queries to better match document content.

2. Returning Too Much Context

Problem: Including a large number of retrieved documents or excessively long chunks in the prompt sent to Claude Code. Why it’s wrong: LLMs have finite context windows. Sending too much information can lead to: * “Lost in the Middle” Phenomenon: The model might ignore information that is neither at the beginning nor at the end of the context. * Increased Latency and Cost: Larger prompts take longer to process and are more expensive. * Degraded Relevance: The LLM might struggle to distill the answer from a vast amount of irrelevant or tangential information. Solution: Implement a strict limit on the number of retrieved chunks sent to the LLM. Use reranking to ensure the most relevant chunks are prioritized. Consider summarizing or extracting key information from retrieved chunks before including them in the final prompt.

3. Not Handling “No Answer Found” Gracefully

Problem: Assuming Claude Code will always find an answer in the provided context, or not explicitly instructing it to state when it cannot. Why it’s wrong: Users expect accurate and honest responses. If the RAG pipeline fails to retrieve relevant information, and the LLM is not instructed to admit it, it might hallucinate an answer. Solution: Explicitly instruct Claude Code in the prompt to state when an answer cannot be found within the provided context. Also, implement fallback mechanisms: if no documents are retrieved, inform the user directly.

Common Pitfalls and How to Avoid Them

• Indexing Performance: Large-scale indexing can be time-consuming. Optimize your data ingestion process by using batching and efficient data transformations. Monitor your Azure AI Search ingestion rate and consider scaling options.
• Vectorization Costs: Generating embeddings for a large corpus can incur significant costs. Choose an embedding model that balances accuracy with cost-effectiveness for your use case. Consider incremental updates for existing embeddings.
• Prompt Injection: Be mindful of potential prompt injection attacks, where malicious users try to manipulate the LLM’s behavior by including special instructions within their input. Input validation and sanitization are crucial.
• Latency: The RAG pipeline involves multiple steps (query embedding, search, LLM inference). Optimize each step to minimize end-to-end latency. Consider caching common query results or using faster LLM models for less complex tasks.
• Evaluation: Regularly evaluate the performance of your RAG system. This includes assessing retrieval relevance and the accuracy and coherence of generated responses. Implement automated evaluation metrics and human review processes.

Conclusion

Building advanced RAG pipelines with Azure AI Search and Claude Code empowers developers to create highly sophisticated and context-aware AI applications. By carefully considering architectural choices, implementing robust retrieval and generation strategies, and being aware of common pitfalls, you can leverage these powerful tools to their full potential. The integration of Azure AI Search’s intelligent retrieval capabilities with Claude Code’s advanced generative power opens up new frontiers for building intelligent systems grounded in reliable information.