Voxta docs
DevelopersModules

Chats and game integrations

Run Voxta chats from a module: act as an account with IUserContexts, start a headless chat, let the session report playback and transcripts, steer turns, and serve a game mod on Voxta's own port.

A chat bridge or a game integration runs chats with nobody connected: the module is the client. This page covers the pieces such a module uses: who the chat runs as, starting it, what the session can do for you, steering a turn, and how a game mod reaches the module.

The Nudge Bridge and Game Bridge examples are quoted whole in the voxta-module-chat-bridge and voxta-module-game-integration skills of Module Development.

Acting as an account

Everything a module does outside a chat it does as an account: that account's characters, services and data. IUserContexts settles which account once and returns an IUserContext to do it through. Inject it like any Voxta service.

CallWhoFor
For(auth)The signed-in caller.An HTTP endpoint answering a signed-in request.
ForUserAsync(userId, ct)An account, with nobody signed in.A bridge running one chat per account, background work.
ForModuleAsync(serviceName, userId, ct)That account's install of the module.A game mod's request (its API key's account), or whoever pressed Start on the Chat Bridges page.

ForModuleAsync returns an IModuleUserContext, which also carries that install's Module and its Settings, read fresh on every call. It returns null when the account has not enabled the module, or cannot be read now. An encrypted account is locked until its owner signs in (or one of their API keys is used), so ask again later instead of once at startup. The account always comes from the caller. Never pick one yourself.

A context with nobody signed in gets the account's content and none of the authority of someone at the keyboard: no prompts, and tools that need approval are refused.

var user = await userContexts.ForModuleAsync(VoxtaModule.ServiceName, request.UserId, cancellationToken);
if (user == null) return; // Not enabled on that account, or the account is locked.
var characterId = user.Settings.GetOptional(ModuleConfigurationProvider.CharacterId);

// The account's own text generation, as its default configuration picks it; null when it has none.
await using var textGen = await user.CreateServiceAsync<ITextGenService>(ServiceTypes.TextGen, freeOthers: false, cancellationToken);

freeOthers: true unloads the local models of the same type this one does not need, to make room. Use it for a request that is the generation. Never use it beside a running chat, whose models it would unload.

Do not build an IAuthenticationContext yourself, resolve a service through profiles and configuration sets, or copy settings to a file of your own to read them later. IUserContext does each of these.

Files the module keeps belong under Voxta's data folder: ICommonFolders.GetDataFolder("MyModule", "songs"). AppContext.BaseDirectory is Voxta's program folder, not a place for data.

Starting a chat

IUserContext.StartChatAsync starts a chat as that account and returns an ILocalChatSession. It is an ordinary chat: it shows in the chats list, the user can watch it and close it. Its client lives in your module instead of behind a websocket.

var session = await user.StartChatAsync(new LocalChatSessionRequest
{
    // How it shows in the chats list.
    Client = VoxtaModule.ServiceName,
    // Nothing plays the replies here: each one counts as played when it ends.
    AcknowledgePlayback = true,
    Start = new ClientStartChatMessage
    {
        CharacterId = characterId,
        DisableServices = [ServiceTypes.TextToSpeech, ServiceTypes.SpeechToText],
    },
}, cancellationToken);
  • Messages is everything the chat produces, in order. Read it to the end: it is unbounded, and a reader that stops leaks.
  • SendAsync sends a client message. SessionId = Guid.Empty is filled in for you.
  • DisposeAsync ends the chat.

The token bounds the start only. The session lives until you dispose it.

What the session can do for you

A websocket client owes the chat two things, and a local one does too. The request can ask the session to do them:

FlagWhat it doesLeave it off when
AcknowledgePlaybackReports each reply played the moment it ends.You play the audio and want the next reply to wait for it: send ClientSpeechPlaybackCompleteMessage yourself once it has played.
SubmitTranscriptsSends what Voxta's microphone heard as the user's message.You read the transcript first (a command, a name) and send it yourself.
HoldAudioKeeps each line's audio file (AudioOutputClientCapabilities.LocalFile) until ReleaseAudio(audioUrl) or the session ends.The audio is played straight away. Without it, Voxta deletes a file a couple of minutes after writing it.

The chat holds its next reply until the last one is reported played. With no audio, nothing else reports it: without AcknowledgePlayback or your own ClientSpeechPlaybackCompleteMessage, the chat replies once and then never again.

Audio for a game

With AudioOutput = AudioOutputClientCapabilities.LocalFile, set PcmSampleRate in the request's Capabilities to the rate the game mixes at (for example 24000). Every line's file is then a 16-bit mono PCM WAV at that rate, whatever the voice service produces, so the game plays it as is. Give each chat its own AudioFolder, and delete it after disposing the session. A chat clears the voxta_* files in its folder when it starts.

Starting with several contexts

KeyedContexts on the start message (and on ClientUpdateContextMessage) sets several contexts, each under its own key, in one message. The first reply already knows all of them, and each key can still be changed or cleared on its own later. ContextKey and Contexts work as before beside it.

Start = new ClientStartChatMessage
{
    CharacterId = characterId,
    KeyedContexts = new()
    {
        ["scene"] = [ContextDefinition.Single("A tavern in Whiterun, late at night.")],
        ["quests"] = [ContextDefinition.Single("The player is looking for the Golden Claw.")],
    },
},

Waiting for a reply

A message sent while a reply is still being set up cancels it. When your module asks for a line and then wants another (characters talking among themselves), wait for the first to start:

var asked = session.Replies;
await session.SendAsync(new ClientSendMessage { SessionId = Guid.Empty, Role = ChatMessageRole.Event, Text = direction, NextSpeakerId = guardId });
// The guard's line, not the narrator's; null when nothing started within 15 seconds.
var started = await session.WaitForReplyAsync(asked, start => start.SenderId == guardId, TimeSpan.FromSeconds(15), cancellationToken);

WaitUntilWrittenAsync waits until every line of what is being written has arrived in Messages. session.Replies.Writing > 0 says whether anything is being written now. Both waits return when the session ends.

Steering a turn

ClientSendMessage carries what used to take prompt text and retries:

  • NextSpeakerId: a character's published id, who takes the turn this message starts. Read on a user message, an event (Role = ChatMessageRole.Event, that character reacts) and a reply request. It is what a scenario script's chat.setNextSpeaker does.
  • Interrupt: false queues the message behind the reply being written instead of stopping it. For something that happened around the conversation. Null follows the chat's setting.
  • ExpiresAfterTurns: an event or note that leaves the prompt after that many turns. 1 for a direction meant for one line.
// The guard reacts to what the player just did, after the line being said, and the direction is gone once she has.
await session.SendAsync(new ClientSendMessage
{
    SessionId = Guid.Empty,
    Role = ChatMessageRole.Event,
    Text = "The player draws a sword.",
    NextSpeakerId = guardId,
    Interrupt = false,
    ExpiresAfterTurns = 1,
});

Tools that end the turn

Tools declared in ClientStartChatMessage.Tools arrive as ServerToolCallRequestMessage, and the model waits for your ClientToolCallResultMessage. After the result, the model is asked to go on. For a tool that only acts in the game (follow, wave, open a door), the line said with the call already was the reply, and asking again gets a second one. Set EndsTurn = true:

await session.SendAsync(new ClientToolCallResultMessage
{
    SessionId = Guid.Empty,
    RequestId = call.RequestId,
    Result = "Done. It is happening in the game.",
    EndsTurn = true,
});

It applies only when every call of the round says so, and only once the reply has said something. A call made before any words still gets its line. It is ignored with IsError, since the model has to read a failure to work around it. Tools a chat augmentation serves have the same switch on ChatToolCallResult.EndsTurn. Voxta sets it itself for a scenario action called as a tool that reports nothing back, and for the Lovense control tool when every toy took the command.

Interruptions

A module that plays the lines itself decides when the player cut in. Inject IInterruptionPolicy and IBackchannelWords to use Voxta's own judgement. On each partial transcript while a character speaks, call Decide(text, backchannelWords.ForCulture(culture)). It returns Interrupt when the player is taking the turn, and Backchannel for "mm" or "right" said while listening. On Interrupt, stop the line in the game and send ClientInterruptMessage.

Endpoints for a game mod

A game mod reaches the module over HTTP. Contribute IModuleHttpEndpoints and it is served on Voxta's own port under /api/extensions/{RoutePrefix}/. There is no port of your own to open, configure or find free.

public class GameEndpoints(IUserContexts userContexts) : IModuleHttpEndpoints
{
    // http://localhost:5384/api/extensions/mygame/...
    public string RoutePrefix => "mygame";

    // The mod has nowhere to sign in: it calls with one of the player's API keys.
    public ModuleHttpAccess Access => ModuleHttpAccess.ApiKey;

    public async Task<ModuleHttpResponse?> HandleAsync(ModuleHttpRequest request, CancellationToken cancellationToken)
    {
        // Path is what follows the prefix, without a leading slash. Null is a 404.
        if (request is not { Method: "POST", Path: "talk" }) return null;
        // The key's account, with its settings for the module.
        var user = await userContexts.ForModuleAsync(VoxtaModule.ServiceName, request.UserId, cancellationToken);
        if (user == null) return ModuleHttpResponse.Json(new { error = "Enable the module on the account of this API key" });
        // Start or continue the conversation as user.
        return ModuleHttpResponse.Json(new { ok = true });
    }
}
// VoxtaModule.Configure
builder.Services.AddSingleton<GameEndpoints>();
builder.Contribute<IModuleHttpEndpoints>(sp => sp.GetRequiredService<GameEndpoints>());

Access decides who may call:

  • SignedIn (the default): a signed-in Voxta account. request.UserId is theirs, and IUserContexts.For acts as them.
  • ApiKey: also a program that cannot sign in, sending one of the account's API keys with the Chats scope (Authorization: Bearer <key>). request.UserId is the key's account, which the key also unlocks when it is encrypted. Anything without a key is refused, wherever it comes from.

Getting the mod a key

The mod should take a key both ways:

  • Pasted into its config, from Voxta's API keys page.
  • Device authorization, when it has none or Voxta refuses the one it has (401 or 403):
    1. POST /api/device/code with {"client_id":"voxta","scope":"chat","label":"My Game","prompt":true} returns user_code, verification_url and device_code. Show the player "enter user_code at verification_url". When prompted is true, Voxta also asks its signed-in users to approve: show "or pick binding_message in Voxta" too.
    2. POST /api/device/poll with {"device_code":"..."} every few seconds: 204 while waiting, 200 with token once approved (save it to the config), 400 once the code expired, was used or was denied (access_denied: don't ask again on your own).

The player carries the code: the mod only shows it. Do not open the device page with the code already filled in. Anything on the machine could then have its own code approved by someone clicking through.

A request can wait: a mod polling for the next line can be held until one is ready, as long as it answers before the mod's own timeout.

What your module keeps

  • Memory books it makes. Set MemoryBook.ModuleKey when creating one (prefixed with your module's name, for example "mygame:{save}|{npc}") and find it again with unitOfWork.MemoryBooks.GetByModuleKeyAsync(key, cancellationToken). No list of ids to keep in a file, and a book the user deleted is not found.
  • Characters by name. A game knows its characters by name: unitOfWork.Characters.GetItemsByUserAsync(new GetCharactersQueryParameters { Name = npcName }, cancellationToken) returns those with that name, ignoring case, the most recently edited first.
  • Files. ICommonFolders.GetDataFolder("MyGame", ...), inside Voxta's data folder.

On this page