@codesoul-co/hypha-serving-cache / middleware/llm-cache-middleware

• Package index: @codesoul-co/hypha-serving-cache
• Source: packages/serving-cache/src/middleware/llm-cache-middleware.ts
• Exports: 3

Using this module

Use the Llm cache middleware module for reading, writing, or coordinating cache state. It exports 1 class, 2 functions.

Import from the package entrypoint

import {
  CachedLLMProvider,
  requestHashForModelRequest,
  servingCacheResponseMetadata,
} from '@codesoul-co/hypha-serving-cache';

Usage patterns

• The module exposes 1 class as constructable runtime implementations. Each symbol entry lists its constructor and public methods.
• The module exposes 2 functions as direct operation entrypoints. Every overload, required/optional parameter, and return type is documented below.

Public exports

Symbol	Kind	Signature	Description
CachedLLMProvider	class	new CachedLLMProvider(inner: ModelProvider<ModelRequest, ModelResponse>, cache: ServingCacheManager, options?: CachedLLMProviderOptions): CachedLLMProvider	Cached LLM Provider class with 6 public constructor or member entries; its exact declarations are listed below.
requestHashForModelRequest	function	requestHashForModelRequest(request: ModelRequest): string	Request Hash For Model Request function with 1 public call signature; parameters and return types are listed below.
servingCacheResponseMetadata	function	servingCacheResponseMetadata(response: ModelResponse): Record<string, unknown> | undefined	Serving Cache Response Metadata function with 1 public call signature; parameters and return types are listed below.

CachedLLMProvider

Cached LLM Provider class with 6 public constructor or member entries; its exact declarations are listed below.

• Kind: class
• Import: import { CachedLLMProvider } from '@codesoul-co/hypha-serving-cache';
• Source module: middleware/llm-cache-middleware

Declaration

export declare class CachedLLMProvider implements ModelProvider<ModelRequest, ModelResponse> {
    readonly id: string;
    constructor(inner: ModelProvider<ModelRequest, ModelResponse>, cache: ServingCacheManager, options?: CachedLLMProviderOptions);
    capabilities(): ReturnType<ModelProvider['capabilities']>;
    generate(request: ModelRequest): Promise<ModelResponse>;
    stream(request: ModelRequest): AsyncIterable<ModelStreamEvent>;
    countTokens(input: unknown): Promise<ModelUsage>;
}

Public members

Member	Kind	Signature	Description
capabilities	method	capabilities(): ReturnType<ModelProvider["capabilities"]>	Public method; parameters and return type are shown in the signature.
constructor	constructor	(inner: ModelProvider<ModelRequest, ModelResponse>, cache: ServingCacheManager, options?: CachedLLMProviderOptions): CachedLLMProvider	Creates an instance of this class.
countTokens	method	countTokens(input: unknown): Promise<ModelUsage>	Public method; parameters and return type are shown in the signature.
generate	method	generate(request: ModelRequest): Promise<ModelResponse>	Public method; parameters and return type are shown in the signature.
id	property	readonly id: string	Public property; its type, readonly modifier and optionality are shown in the signature.
stream	method	stream(request: ModelRequest): AsyncIterable<ModelStreamEvent>	Public method; parameters and return type are shown in the signature.

requestHashForModelRequest

Request Hash For Model Request function with 1 public call signature; parameters and return types are listed below.

• Kind: function
• Import: import { requestHashForModelRequest } from '@codesoul-co/hypha-serving-cache';
• Source module: middleware/llm-cache-middleware

Declaration

export declare function requestHashForModelRequest(request: ModelRequest): string;

Call signature

requestHashForModelRequest(request: ModelRequest): string

Parameters

Parameter	Type	Required	Description
request	ModelRequest<unknown>	Yes	Required parameter; accepted values are defined by the type column.

Returns

• Type: string
• Description: The return contract is defined by the type shown above.

servingCacheResponseMetadata

Serving Cache Response Metadata function with 1 public call signature; parameters and return types are listed below.

• Kind: function
• Import: import { servingCacheResponseMetadata } from '@codesoul-co/hypha-serving-cache';
• Source module: middleware/llm-cache-middleware

Declaration

export declare function servingCacheResponseMetadata(response: ModelResponse): Record<string, unknown> | undefined;

Call signature

servingCacheResponseMetadata(response: ModelResponse): Record<string, unknown> | undefined

Parameters

Parameter	Type	Required	Description
response	ModelResponse<string>	Yes	Required parameter; accepted values are defined by the type column.

Returns

• Type: Record<string, unknown>
• Description: The return contract is defined by the type shown above.