Enhancing LLM accuracy with Coveo Passage Retrieval on Amazon Bedrock

This post is co-written with Keith Beaudoin and Nicolas Bordeleau from Coveo.

As generative AI transforms business operations, enterprises face a critical challenge: how can they help large language models (LLMs) provide accurate and trustworthy responses? Without reliable data foundations, these AI models can generate misleading or inaccurate responses, potentially reducing user trust and organizational credibility.

As an AWS Partner, Coveo addresses this challenge with its Passage Retrieval API. This solution enhances the reliability of LLM-powered applications by providing them with relevant, context-aware enterprise knowledge to inform generated responses. In Retrieval Augmented Generation (RAG) systems, the retrieval process is the most complex component. It requires extracting the most relevant, precise information from enterprise data sources. By integrating the Coveo AI-Relevance Platform with Amazon Bedrock Agents, organizations gain access to an industry-leading enterprise search service featuring a secured, unified hybrid index that respects enterprise permission models and offers robust connectivity. The Coveo AI-Relevance Platform uses machine learning (ML) and in-depth usage analytics to continuously optimize relevance. This enables Amazon Bedrock Agents to deliver grounded, contextually relevant responses tailored to complex enterprise content.

The Coveo AI-Relevance Platform is an industry-leading service that connects and unifies the content of cloud and on-premises repositories in a single index, making it fast and simple to find relevant content quickly. Its ML algorithms analyze user behavior, in-app context, as well as profile and permissions data to retrieve personalized search results and recommendations. It then aggregates and reports insights back to content and experience managers. By integrating seamlessly with enterprise systems (such as websites, knowledge bases, and CRM) and enforcing security permissions, Coveo helps users get the most pertinent information while maintaining data protection.

In this post, we show how to deploy Coveo’s Passage Retrieval API as an Amazon Bedrock Agents action group to enhance response accuracy, so Coveo users can use their current index to rapidly deploy new generative experiences across their organization.

Coveo’s Passage Retrieval API

The Coveo Passage Retrieval API enhances LLM applications by passing ranked text passages (or chunks) retrieved from the unified index, along with appropriate metadata such as source URLs for citations, so that generated answers are grounded in an organization’s proprietary knowledge. Built on Coveo’s unified hybrid index, the Passage Retrieval API applies a two-stage retrieval process to extract the most relevant passages from structured and unstructured content sources, providing accuracy, security, and real-time relevance. The following diagram illustrates these stages.

The process consists of the following key components:

• Relevant passage extraction using a two-stage retrieval process – In the first retrieval stage, Coveo’s hybrid search system is used to identify the most relevant documents. Then, it extracts the most relevant text passages from these documents, along with ranking scores, citation links, and other key metadata. This two-stage approach allows Coveo to identify accessible and relevant documents from the sources and then more precisely identify the most relevant passages.
• Enhanced search results with hybrid ranking – Combining semantic (vector) search, and lexical (keyword) matching helps Coveo retrieve the right information in the right context.
• ML relevancy – AI continuously learns from user interactions, tailoring retrieval to each user’s journey, behavior, and profile for context-aware responses.
• Content connected across sources with a unified index – A centralized hybrid index connects structured and unstructured content across sources. This unified hybrid index provides better multi-source relevancy than a federated search approach by applying the ranking function across sources. Coveo also provides a robust library of pre-built connectors to maintain seamless integration with third-party services (such as Salesforce, SharePoint, and Google Drive), facilitating data freshness with automatic updates for real-time retrieval.
• Analytics and insights for performance tracking – With events tracking through the Data Platform and Knowledge Hub, you can see exactly how your generated answers perform, where information is missing or underused, and which content needs tuning. With those insights, you can boost answer quality and drive measurable business impact.
• Enterprise-grade security – Coveo provides the native permission model of each connected content source by importing item‑level permissions at crawl time through an early‑binding approach. Resolving access rights before indexing helps prevent data leakage and boosts search performance by filtering out items a user can’t see before a query is run.
• Redefining enterprise-ready RAG – Coveo reimagines what RAG systems can achieve by going beyond basic vector search, offering a dynamic and intelligent retrieval pipeline designed for enterprise needs. At its core is a unified hybrid index that seamlessly connects structured, unstructured, and permission-sensitive data. This foundation, combined with real-time ML-driven tuning, helps validate that every response delivered to an LLM is relevant and securely grounded in the right context.

Through native access control enforcement, behavior-based relevance adjustment, and deep analytics into content usage and performance, Coveo empowers organizations to continuously refine their generative AI experiences. Backed by consistent recognition from leading analyst firms such as Gartner, Forrester, and IDC, Coveo delivers a reliable, secure, and scalable foundation for enterprise-grade generative AI.

Solution overview

This solution demonstrates how you can integrate Coveo’s Passage Retrieval API with Amazon Bedrock Agents to build LLM-powered assistants that deliver accurate, context-aware, and grounded responses. It applies broadly across use cases where agents need to access proprietary knowledge, whether to support customers, assist employees, or empower sales and service teams. This approach helps AI responses stay grounded in your most relevant and trusted information across structured and unstructured content. Amazon Bedrock Agents acts as the intelligent backbone of this solution, interpreting natural language queries and orchestrating the retrieval process to deliver grounded, contextually relevant insights. For this use case, the agent is equipped to answer a user’s questions on Coveo services capabilities, API documentation, and integration guides. The agent is designed to fetch precise passages from enterprise content in response to user questions, enabling applications such as virtual assistants, support copilots, or internal knowledge bots. By using structured definitions and instructions, the agent understands when and how to trigger Coveo’s Passage Retrieval API, making sure that LLM-generated responses are backed by accurate and trusted content.

The action group defines the structured API operations that the Amazon Bedrock agent can invoke. Using OpenAPI specifications, it defines the interface between the agent and AWS Lambda functions. In this use case, fetching relevant passages based on the user’s search intent is the only available operation.

The following diagram illustrates the solution architecture.

For this demo, the agent is set with the following instructions during creation:

You will act as an expert on Coveo documentation, platform, APIs, analytics, and integration guides.
        Use the CoveoPRAPI Action Group to retrieve relevant information on Coveo documentation, platform, APIs, analytics, and integration guides.
        Summarize the information retrieved from the Action Group response clearly and accurately.
        OuputFormatting guidelines:
        - Provide clear, direct answers without introductory phrases such as “As an expert,” “Sure,” or “Here is...”
        - When appropriate, organize content using:
          - Numbered or bulleted lists
        - Structured sections (e.g., features, steps, key points)
        Keep answers concise, informative, and free of conversational filler.

The Lambda function defined in the action group is essential for enabling the Amazon Bedrock agent to call the Coveo Passage Retrieval API. The Lambda function performs the following tasks:

• Receives incoming requests from the Amazon Bedrock agent
• Queries the Coveo Passage Retrieval API using the user’s input
• Returns the relevant search results back to the Amazon Bedrock agent

Lastly, the Coveo AI-Relevance Platform provides indexed and structured search results through the Passage Retrieval API.

Prerequisites

Before you begin, you must have the following prerequisites:

• An AWS account with AWS Identity and Access Management (IAM) permissions to deploy an AWS CloudFormation stack
• Access to models in Amazon Bedrock
• A Coveo index created and ready to use
• The following Coveo information:
  • Coveo organization ID
  • Coveo search hub
  • Coveo API key

Deploy the solution with AWS CloudFormation

To deploy your agent, complete the following steps:

1. Launch the CloudFormation stack:
   
2. Enter a stack name and values for AgentModelID, AgentName, CoveoApiKey, CoveoOrgID, and CoveoSearchHub.
3. In the Capabilities section, select the acknowledge check box.
4. Choose Create stack.
5. Wait for the stack creation to complete.
6. Verify all resources are created on the stack details page.

Test the solution

To test the solution, complete the following steps:

1. On the Amazon Bedrock console, choose Agents in the navigation pane.
2. Choose the agent created by the CloudFormation stack.

3. Under Test, enter your question in the message box.

For this demo, Coveo technical documentation (from the website) was ingested in an existing Coveo Search index. Let’s start with the query “What is the difference between Coveo Atomic and Headless?”

Here’s how the agent answered the question when Claude 3 Haiku v1 is used as the LLM.

4. Choose Show trace in the right pane and expand Trace step 1 to see the agent’s rationale.

The following screenshot demonstrates how Amazon Bedrock Agents processed and answered the question. First, it formed a rationale:

To understand the difference between Coveo Atomic and Headless, I will need to retrieve relevant information from the Coveo technical documentation using the CoveoPRAPI Action Group.

Then it invokes the CoveoPRAP action group, which is specifically designed to retrieve relevant passages, through a Lambda function to make an API call to /rest/search/v3/passages/retrieve.

This example illustrates the agent’s systematic approach to planning and executing necessary actions through the CoveoPRAP1 action group, and retrieving relevant document chunks before formulating its final response.

The Lambda function code includes a debugging feature that logs each retrieved passage from the Passage Retrieval API. This logging mechanism iterates through the returned chunks, numbering them sequentially for quick reference. These logs are available in Amazon CloudWatch, so you can see exactly which passages were retrieved for each user query, and how they contributed to the final response. To visualize the logs, open the CloudWatch console, and on the Log groups page, locate the Lambda function name.

The following screenshot shows the agent detailed logs in CloudWatch. In the logs, the Coveo Passage Retrieval API returns the five most relevant chunks to the LLM.

Clean up

To avoid ongoing charges, delete the resources deployed as part of the CloudFormation stack:

1. On the AWS CloudFormation console, choose Stacks in the navigation pane.
2. Choose the stack you created, then choose Delete.
3. Choose Delete stack when prompted.

For more information, refer to Deleting a stack on the AWS CloudFormation console.

Conclusion

By integrating Amazon Bedrock capabilities with Coveo’s AI-driven retrieval, organizations can develop AI applications that provide validated responses based on their enterprise content. This approach helps reduce inaccuracies while delivering real-time, secure responses.

We encourage you to explore pre-built examples in the GitHub repository to help you get started with Amazon Bedrock.

To learn more about the Coveo AI-Relevance Platform and how to implement the Passage Retrieval API in your Coveo organization, refer to Passage Retrieval (CPR) implementation overview.

About the authors

Yanick Houngbedji is a Solutions Architect for Independent Software Vendors (ISV) at Amazon Web Services (AWS), based in Montréal, Canada. He specializes in helping customers architect and implement highly scalable, performant, and secure cloud solutions on AWS. Before joining AWS, he spent over 8 years providing technical leadership in data engineering, big data analytics, business intelligence, and data science solutions.

Keith Beaudoin is a Senior Solution Architect at Coveo Labs. He is specialized in designing and implementing intelligent search and AI-powered relevance solutions, with expertise in Agentic solutions, cloud technologies, search architecture, and third-party integrations. Keith helps organizations harness the full potential of Coveo’s platform, optimizing digital transformation strategies for seamless and impactful search implementations that drive business value

Nicolas Bordeleau is Head of Product Relations at Coveo. With over 19 years of experience in the search industry, including 11 years in product management, Nicolas has a keen understanding of enterprises and developers’ needs related to search and information retrieval. He has applied this knowledge to develop award-winning products that fulfill those needs in innovative ways.