@codesoul-co/hypha-inference / manager

• Package index: @codesoul-co/hypha-inference
• Source: packages/inference/src/manager.ts
• Exports: 6

Using this module

Use the Manager module for using the public contracts and operations for this capability boundary. It exports 4 classes, 2 interfaces.

Import from the package entrypoint

import {
  InferenceCacheCapacityError,
  InferenceManager,
  InMemoryKvCacheProvider,
  InMemoryPrefixCacheProvider,
} from '@codesoul-co/hypha-inference';

import type {
  InMemoryKvCacheProviderOptions,
  InMemoryPrefixCacheProviderOptions,
} from '@codesoul-co/hypha-inference';

Usage patterns

• Use the 2 type/interface exports as static contracts in application code, adapters, or tests. Import them with import type; they do not exist at runtime.
• The module exposes 4 classes as constructable runtime implementations. Each symbol entry lists its constructor and public methods.

Public exports

Symbol	Kind	Signature	Description
InferenceCacheCapacityError	class	new InferenceCacheCapacityError(message: string): InferenceCacheCapacityError	Inference Cache Capacity Error class with 9 public constructor or member entries; its exact declarations are listed below.
InferenceManager	class	new InferenceManager(options?: InferenceManagerOptions): InferenceManager	Inference Manager class with 5 public constructor or member entries; its exact declarations are listed below.
InMemoryKvCacheProvider	class	new InMemoryKvCacheProvider(options?: InMemoryKvCacheProviderOptions): InMemoryKvCacheProvider	In Memory Kv Cache Provider class with 6 public constructor or member entries; its exact declarations are listed below.
InMemoryPrefixCacheProvider	class	new InMemoryPrefixCacheProvider(options?: InMemoryPrefixCacheProviderOptions): InMemoryPrefixCacheProvider	In Memory Prefix Cache Provider class with 5 public constructor or member entries; its exact declarations are listed below.
InMemoryKvCacheProviderOptions	interface	interface InMemoryKvCacheProviderOptions	In Memory Kv Cache Provider Options interface with 3 public fields or methods.
InMemoryPrefixCacheProviderOptions	interface	interface InMemoryPrefixCacheProviderOptions	In Memory Prefix Cache Provider Options interface with 3 public fields or methods.

InferenceCacheCapacityError

Inference Cache Capacity Error class with 9 public constructor or member entries; its exact declarations are listed below.

• Kind: class
• Import: import { InferenceCacheCapacityError } from '@codesoul-co/hypha-inference';
• Source module: manager

Declaration

export declare class InferenceCacheCapacityError extends Error {
    readonly code = "INFERENCE_CACHE_CAPACITY_EXCEEDED";
    constructor(message: string);
}

Public members

Member	Kind	Signature	Description
cause	property	cause?: unknown	Public property; its type, readonly modifier and optionality are shown in the signature.
code	property	readonly code: "INFERENCE_CACHE_CAPACITY_EXCEEDED"	Public property; its type, readonly modifier and optionality are shown in the signature.
constructor	constructor	(message: string): InferenceCacheCapacityError	Creates an instance of this class.
message	property	message: string	Public property; its type, readonly modifier and optionality are shown in the signature.
name	property	name: string	Public property; its type, readonly modifier and optionality are shown in the signature.
stack	property	stack?: string	Public property; its type, readonly modifier and optionality are shown in the signature.
static captureStackTrace	method	static captureStackTrace(targetObject: object, constructorOpt?: Function): void	Creates a .stack property on targetObject, which when accessed returns a string representing the location in the code at which Error.captureStackTrace() was called. js const myObject = {}; Error.captureStackTrace(myObject); myObject.stack; // Similar to `new Error().stack` The first line of the trace will be prefixed with ${myObject.name}: ${myObject.message}. The optional constructorOpt argument accepts a function. If given, all frames above constructorOpt, including constructorOpt, will be omitted from the generated stack trace. The constructorOpt argument is useful for hiding implementation details of error generation from the user. For instance: js function a() { b(); } function b() { c(); } function c() { // Create an error without stack trace to avoid calculating the stack trace twice. const { stackTraceLimit } = Error; Error.stackTraceLimit = 0; const error = new Error(); Error.stackTraceLimit = stackTraceLimit; // Capture the stack trace above function b Error.captureStackTrace(error, b); // Neither function c, nor b is included in the stack trace throw error; } a();
static prepareStackTrace	method	static prepareStackTrace(err: Error, stackTraces: NodeJS.CallSite[]): any	Public method; parameters and return type are shown in the signature.
static stackTraceLimit	property	static stackTraceLimit: number	The Error.stackTraceLimit property specifies the number of stack frames collected by a stack trace (whether generated by new Error().stack or Error.captureStackTrace(obj)). The default value is 10 but may be set to any valid JavaScript number. Changes will affect any stack trace captured after the value has been changed. If set to a non-number value, or set to a negative number, stack traces will not capture any frames.

InferenceManager

Inference Manager class with 5 public constructor or member entries; its exact declarations are listed below.

• Kind: class
• Import: import { InferenceManager } from '@codesoul-co/hypha-inference';
• Source module: manager

Declaration

export declare class InferenceManager {
    constructor(options?: InferenceManagerOptions);
    register(provider: InferenceProvider): void;
    get(providerId: string): InferenceProvider | null;
    infer(providerId: string, request: InferenceRequest): Promise<InferenceResponse>;
    stream(providerId: string, request: InferenceRequest): AsyncIterable<InferenceResponse>;
}

Public members

Member	Kind	Signature	Description
constructor	constructor	(options?: InferenceManagerOptions): InferenceManager	Creates an instance of this class.
get	method	get(providerId: string): InferenceProvider | null	Public method; parameters and return type are shown in the signature.
infer	method	infer(providerId: string, request: InferenceRequest): Promise<InferenceResponse>	Public method; parameters and return type are shown in the signature.
register	method	register(provider: InferenceProvider): void	Public method; parameters and return type are shown in the signature.
stream	method	stream(providerId: string, request: InferenceRequest): AsyncIterable<InferenceResponse>	Public method; parameters and return type are shown in the signature.

InMemoryKvCacheProvider

In Memory Kv Cache Provider class with 6 public constructor or member entries; its exact declarations are listed below.

• Kind: class
• Import: import { InMemoryKvCacheProvider } from '@codesoul-co/hypha-inference';
• Source module: manager

Declaration

export declare class InMemoryKvCacheProvider implements KvCacheProvider {
    constructor(options?: InMemoryKvCacheProviderOptions);
    get(ref: KvCacheRef): Promise<unknown | null>;
    put(ref: KvCacheRef, value: unknown): Promise<void>;
    invalidate(ref: KvCacheRef): Promise<void>;
    size(): number;
    stats(): {
            entries: number;
            totalBytes: number;
        };
}

Public members

Member	Kind	Signature	Description
constructor	constructor	(options?: InMemoryKvCacheProviderOptions): InMemoryKvCacheProvider	Creates an instance of this class.
get	method	get(ref: KvCacheRef): Promise<unknown | null>	Public method; parameters and return type are shown in the signature.
invalidate	method	invalidate(ref: KvCacheRef): Promise<void>	Public method; parameters and return type are shown in the signature.
put	method	put(ref: KvCacheRef, value: unknown): Promise<void>	Public method; parameters and return type are shown in the signature.
size	method	size(): number	Public method; parameters and return type are shown in the signature.
stats	method	stats(): { entries: number; totalBytes: number; }	Public method; parameters and return type are shown in the signature.

InMemoryPrefixCacheProvider

In Memory Prefix Cache Provider class with 5 public constructor or member entries; its exact declarations are listed below.

• Kind: class
• Import: import { InMemoryPrefixCacheProvider } from '@codesoul-co/hypha-inference';
• Source module: manager

Declaration

export declare class InMemoryPrefixCacheProvider implements PrefixCacheProvider {
    constructor(options?: InMemoryPrefixCacheProviderOptions);
    get(ref: PrefixCacheRef): Promise<string | null>;
    put(ref: PrefixCacheRef, content: string): Promise<void>;
    invalidate(ref: PrefixCacheRef): Promise<void>;
    stats(): {
            entries: number;
            totalBytes: number;
        };
}

Public members

Member	Kind	Signature	Description
constructor	constructor	(options?: InMemoryPrefixCacheProviderOptions): InMemoryPrefixCacheProvider	Creates an instance of this class.
get	method	get(ref: PrefixCacheRef): Promise<string | null>	Public method; parameters and return type are shown in the signature.
invalidate	method	invalidate(ref: PrefixCacheRef): Promise<void>	Public method; parameters and return type are shown in the signature.
put	method	put(ref: PrefixCacheRef, content: string): Promise<void>	Public method; parameters and return type are shown in the signature.
stats	method	stats(): { entries: number; totalBytes: number; }	Public method; parameters and return type are shown in the signature.

InMemoryKvCacheProviderOptions

In Memory Kv Cache Provider Options interface with 3 public fields or methods.

• Kind: interface
• Import: import type { InMemoryKvCacheProviderOptions } from '@codesoul-co/hypha-inference';
• Source module: manager

Declaration

export interface InMemoryKvCacheProviderOptions {
    maxEntries?: number;
    maxEntryBytes?: number;
    maxTotalBytes?: number;
}

Contract members

Member	Kind	Signature	Description
maxEntries	property	maxEntries?: number	Public property; its type, readonly modifier and optionality are shown in the signature.
maxEntryBytes	property	maxEntryBytes?: number	Public property; its type, readonly modifier and optionality are shown in the signature.
maxTotalBytes	property	maxTotalBytes?: number	Public property; its type, readonly modifier and optionality are shown in the signature.

InMemoryPrefixCacheProviderOptions

In Memory Prefix Cache Provider Options interface with 3 public fields or methods.

• Kind: interface
• Import: import type { InMemoryPrefixCacheProviderOptions } from '@codesoul-co/hypha-inference';
• Source module: manager

Declaration

export interface InMemoryPrefixCacheProviderOptions {
    maxEntries?: number;
    maxEntryBytes?: number;
    maxTotalBytes?: number;
}

Contract members

Member	Kind	Signature	Description
maxEntries	property	maxEntries?: number	Public property; its type, readonly modifier and optionality are shown in the signature.
maxEntryBytes	property	maxEntryBytes?: number	Public property; its type, readonly modifier and optionality are shown in the signature.
maxTotalBytes	property	maxTotalBytes?: number	Public property; its type, readonly modifier and optionality are shown in the signature.