Skip to main content
Deployment options: WSO2 Cloud

Semantic Caching

AI services frequently involve repetitive queries, leading to unnecessary token usage and increased latency. API Platform’s AI Gateway introduces semantic caching, allowing responses to similar requests to be cached and reused intelligently, minimizing redundant processing, improving response times, and reducing overall costs.

Configure Semantic Caching Policy​

  1. In the left navigation menu, click Develop, then select Policy.

    Policy page showing API Proxy Contract routes and Service Contract with Azure OpenAI endpoint

  2. Click Add Resource Level Policy --> Request flow --> Attach mediation policy --> Semantic Caching

    Mediation Policy List panel with Semantic Caching option highlighted among available policies

  3. Add the embedding provider and vector store configurations and click Save.

    Configure Semantic Caching Policy dialog with Embedding Provider, Auth Header, API Key, Embedding Model, Upstream URL, Vector Store, Host, Port, Dimensions, and Threshold fields

  4. Save the API and Deploy the API to apply the policy to the gateway.

The configurable fields of the above policy have been described below.

FieldDescriptionExample Value
Embedding ProviderAI provider used for generating embeddings (Azure OpenAI or Mistral).Azure OpenAI
Auth Header NameHeader name for authentication
(Use the header name Authorization for Mistral and api-key for Azure OpenAI.).		api-key
API KeyAPI key for authenticating with the embedding provider.49fdadxxxxxxxxxxxxxxxxxxxxxxxxxx
Embedding Model NameSpecific embedding model to use from the provider.text-embedding-ada-002
Embedding Upstream URLEndpoint URL of the embedding service.https://example.openai.azure.com/openai/deployments/xxxxx/embeddings?api-version=2025-07-21
Vector StoreType of vector database to store embeddings
(Currently only Redis is supported).		Redis
HostHost address of the vector database.redis-xxxxx.us-east.ec2.redis-cloud.com
PortNetwork port number of the vector database.6379
DimensionsThe dimensionality of the vectors generated from the selected embedding model. (Refer to the provider's official documentation to find out the exact values).1536
ThresholdDissimilarity threshold which is a decimal value for semantic matching that determines the required similarity for cache matches.
[Note: Lower values (closer to 0) enforce stricter semantic similarity, while higher values allow weaker matches. Typical range: 0.0 (exact) to higher values (e.g., 0.5, 1.2, etc.)].		0.1
UsernameUsername for database authentication.newuser
PasswordPassword for the specified database user.securepassword123
DatabaseIndex of the vector database to connect to.0

Sample Payloads and Responses​

First Request:

{
  "messages": [
    {
      "role": "system",
      "content": "How do I reset my google account password?"
    }
  ]
}

Second Request:

{
  "messages": [
    {
      "role": "system",
      "content": "What is the process for changing my google login password?"
    }
  ]
}

Note

The effectiveness of semantic caching can vary based on the selected embedding model and configured similarity threshold. Ensure optimal performance by carefully selecting these parameters according to your specific use case.

Next Steps​

  • Verify it is properly redeployed to ensure the policy and the configurations are properly applied.

  • Test the AI API to ensure it properly cache and serves those cached responses upon receiving semantically similar queries. See Test REST Endpoints via the OpenAPI Console.

By configuring semantic caching as outlined above, you can efficiently optimize AI service usage within your API Platform environment, significantly reducing latency and operational costs.