Kong has launched the newest replace to Kong AI Gateway, an answer for securing, governing, and controlling LLM consumption from in style third-party suppliers.
Kong AI Gateway 3.11 introduces a brand new plugin that reduces token prices, a number of new generative AI capabilities, and assist for AWS Bedrock Guardrails.
The brand new immediate compression plugin that removes padding and redundant phrases or phrases. This strategy preserves 80% of the meant semantic that means of the immediate, however the elimination of pointless phrases can result in as much as a 5x discount in price.
In accordance with Kong, the immediate compression plugin enhances different cost-saving measures, similar to Semantic Caching to forestall redundant LLM calls and AI Fee Limiting to handle utilization limits by software or workforce.
This replace additionally provides over 10 new generative AI capabilities, together with batch execution of a number of LLM calls, audio transcription and translation, picture technology, stateful assistants, and enhanced response introspection.
Lastly, Kong AI Gateway 3.11 provides assist for AWS Bedrock Guardrails, which may also help defend AI functions from malicious and unintended penalties, like hallucinations or inappropriate content material. Builders can monitor functions and alter insurance policies in actual time with no need to vary code.
“We’re excited to introduce one among our most vital Kong AI Gateway releases up to now. With options like immediate compression, multimodal assist and guardrails, model 3.11 provides groups the instruments they should construct extra succesful AI techniques—sooner and with far much less operational overhead. It’s a significant step ahead for any group seeking to scale AI reliably whereas holding infrastructure prices beneath management,” stated Marco Palladino, CTO and co-founder of Kong.