Chats and game integrations
Run Voxta chats from a module: act as an account with IUserContexts, start a headless chat, let the session report playback and transcripts, steer turns, and serve a game mod on Voxta's own port.
A chat bridge or a game integration runs chats with nobody connected: the module is the client. This page covers the pieces such a module uses: who the chat runs as, starting it, what the session can do for you, steering a turn, and how a game mod reaches the module.
The Nudge Bridge and Game Bridge examples are quoted whole in the voxta-module-chat-bridge and voxta-module-game-integration skills of Module Development.
Acting as an account
Everything a module does outside a chat it does as an account: that account's characters, services and data. IUserContexts settles which account once and returns an IUserContext to do it through. Inject it like any Voxta service.
| Call | Who | For |
|---|---|---|
For(auth) | The signed-in caller. | An HTTP endpoint answering a signed-in request. |
ForUserAsync(userId, ct) | An account, with nobody signed in. | A bridge running one chat per account, background work. |
ForModuleAsync(serviceName, userId, ct) | That account's install of the module. | A game mod's request (its API key's account), or whoever pressed Start on the Chat Bridges page. |
ForModuleAsync returns an IModuleUserContext, which also carries that install's Module and its Settings, read fresh on every call. It returns null when the account has not enabled the module, or cannot be read now. An encrypted account is locked until its owner signs in (or one of their API keys is used), so ask again later instead of once at startup. The account always comes from the caller. Never pick one yourself.
A context with nobody signed in gets the account's content and none of the authority of someone at the keyboard: no prompts, and tools that need approval are refused.
var user = await userContexts.ForModuleAsync(VoxtaModule.ServiceName, request.UserId, cancellationToken);
if (user == null) return; // Not enabled on that account, or the account is locked.
var characterId = user.Settings.GetOptional(ModuleConfigurationProvider.CharacterId);
// The account's own text generation, as its default configuration picks it; null when it has none.
await using var textGen = await user.CreateServiceAsync<ITextGenService>(ServiceTypes.TextGen, freeOthers: false, cancellationToken);freeOthers: true unloads the local models of the same type this one does not need, to make room. Use it for a request that is the generation. Never use it beside a running chat, whose models it would unload.
Do not build an IAuthenticationContext yourself, resolve a service through profiles and configuration sets, or copy settings to a file of your own to read them later. IUserContext does each of these.
Files the module keeps belong under Voxta's data folder: ICommonFolders.GetDataFolder("MyModule", "songs"). AppContext.BaseDirectory is Voxta's program folder, not a place for data.
Starting a chat
IUserContext.StartChatAsync starts a chat as that account and returns an ILocalChatSession. It is an ordinary chat: it shows in the chats list, the user can watch it and close it. Its client lives in your module instead of behind a websocket.
var session = await user.StartChatAsync(new LocalChatSessionRequest
{
// How it shows in the chats list.
Client = VoxtaModule.ServiceName,
// Nothing plays the replies here: each one counts as played when it ends.
AcknowledgePlayback = true,
Start = new ClientStartChatMessage
{
CharacterId = characterId,
DisableServices = [ServiceTypes.TextToSpeech, ServiceTypes.SpeechToText],
},
}, cancellationToken);Messagesis everything the chat produces, in order. Read it to the end: it is unbounded, and a reader that stops leaks.SendAsyncsends a client message.SessionId = Guid.Emptyis filled in for you.DisposeAsyncends the chat.
The token bounds the start only. The session lives until you dispose it.
What the session can do for you
A websocket client owes the chat two things, and a local one does too. The request can ask the session to do them:
| Flag | What it does | Leave it off when |
|---|---|---|
AcknowledgePlayback | Reports each reply played the moment it ends. | You play the audio and want the next reply to wait for it: send ClientSpeechPlaybackCompleteMessage yourself once it has played. |
SubmitTranscripts | Sends what Voxta's microphone heard as the user's message. | You read the transcript first (a command, a name) and send it yourself. |
HoldAudio | Keeps each line's audio file (AudioOutputClientCapabilities.LocalFile) until ReleaseAudio(audioUrl) or the session ends. | The audio is played straight away. Without it, Voxta deletes a file a couple of minutes after writing it. |
The chat holds its next reply until the last one is reported played. With no audio, nothing else reports it: without AcknowledgePlayback or your own ClientSpeechPlaybackCompleteMessage, the chat replies once and then never again.
Audio for a game
With AudioOutput = AudioOutputClientCapabilities.LocalFile, set PcmSampleRate in the request's Capabilities to the rate the game mixes at (for example 24000). Every line's file is then a 16-bit mono PCM WAV at that rate, whatever the voice service produces, so the game plays it as is. Give each chat its own AudioFolder, and delete it after disposing the session. A chat clears the voxta_* files in its folder when it starts.
Starting with several contexts
KeyedContexts on the start message (and on ClientUpdateContextMessage) sets several contexts, each under its own key, in one message. The first reply already knows all of them, and each key can still be changed or cleared on its own later. ContextKey and Contexts work as before beside it.
Start = new ClientStartChatMessage
{
CharacterId = characterId,
KeyedContexts = new()
{
["scene"] = [ContextDefinition.Single("A tavern in Whiterun, late at night.")],
["quests"] = [ContextDefinition.Single("The player is looking for the Golden Claw.")],
},
},Waiting for a reply
A message sent while a reply is still being set up cancels it. When your module asks for a line and then wants another (characters talking among themselves), wait for the first to start:
var asked = session.Replies;
await session.SendAsync(new ClientSendMessage { SessionId = Guid.Empty, Role = ChatMessageRole.Event, Text = direction, NextSpeakerId = guardId });
// The guard's line, not the narrator's; null when nothing started within 15 seconds.
var started = await session.WaitForReplyAsync(asked, start => start.SenderId == guardId, TimeSpan.FromSeconds(15), cancellationToken);WaitUntilWrittenAsync waits until every line of what is being written has arrived in Messages. session.Replies.Writing > 0 says whether anything is being written now. Both waits return when the session ends.
Steering a turn
ClientSendMessage carries what used to take prompt text and retries:
NextSpeakerId: a character's published id, who takes the turn this message starts. Read on a user message, an event (Role = ChatMessageRole.Event, that character reacts) and a reply request. It is what a scenario script'schat.setNextSpeakerdoes.Interrupt:falsequeues the message behind the reply being written instead of stopping it. For something that happened around the conversation. Null follows the chat's setting.ExpiresAfterTurns: an event or note that leaves the prompt after that many turns.1for a direction meant for one line.
// The guard reacts to what the player just did, after the line being said, and the direction is gone once she has.
await session.SendAsync(new ClientSendMessage
{
SessionId = Guid.Empty,
Role = ChatMessageRole.Event,
Text = "The player draws a sword.",
NextSpeakerId = guardId,
Interrupt = false,
ExpiresAfterTurns = 1,
});Tools that end the turn
Tools declared in ClientStartChatMessage.Tools arrive as ServerToolCallRequestMessage, and the model waits for your ClientToolCallResultMessage. After the result, the model is asked to go on. For a tool that only acts in the game (follow, wave, open a door), the line said with the call already was the reply, and asking again gets a second one. Set EndsTurn = true:
await session.SendAsync(new ClientToolCallResultMessage
{
SessionId = Guid.Empty,
RequestId = call.RequestId,
Result = "Done. It is happening in the game.",
EndsTurn = true,
});It applies only when every call of the round says so, and only once the reply has said something. A call made before any words still gets its line. It is ignored with IsError, since the model has to read a failure to work around it. Tools a chat augmentation serves have the same switch on ChatToolCallResult.EndsTurn. Voxta sets it itself for a scenario action called as a tool that reports nothing back, and for the Lovense control tool when every toy took the command.
Interruptions
A module that plays the lines itself decides when the player cut in. Inject IInterruptionPolicy and IBackchannelWords to use Voxta's own judgement. On each partial transcript while a character speaks, call Decide(text, backchannelWords.ForCulture(culture)). It returns Interrupt when the player is taking the turn, and Backchannel for "mm" or "right" said while listening. On Interrupt, stop the line in the game and send ClientInterruptMessage.
Endpoints for a game mod
A game mod reaches the module over HTTP. Contribute IModuleHttpEndpoints and it is served on Voxta's own port under /api/extensions/{RoutePrefix}/. There is no port of your own to open, configure or find free.
public class GameEndpoints(IUserContexts userContexts) : IModuleHttpEndpoints
{
// http://localhost:5384/api/extensions/mygame/...
public string RoutePrefix => "mygame";
// The mod has nowhere to sign in: it calls with one of the player's API keys.
public ModuleHttpAccess Access => ModuleHttpAccess.ApiKey;
public async Task<ModuleHttpResponse?> HandleAsync(ModuleHttpRequest request, CancellationToken cancellationToken)
{
// Path is what follows the prefix, without a leading slash. Null is a 404.
if (request is not { Method: "POST", Path: "talk" }) return null;
// The key's account, with its settings for the module.
var user = await userContexts.ForModuleAsync(VoxtaModule.ServiceName, request.UserId, cancellationToken);
if (user == null) return ModuleHttpResponse.Json(new { error = "Enable the module on the account of this API key" });
// Start or continue the conversation as user.
return ModuleHttpResponse.Json(new { ok = true });
}
}// VoxtaModule.Configure
builder.Services.AddSingleton<GameEndpoints>();
builder.Contribute<IModuleHttpEndpoints>(sp => sp.GetRequiredService<GameEndpoints>());Access decides who may call:
SignedIn(the default): a signed-in Voxta account.request.UserIdis theirs, andIUserContexts.Foracts as them.ApiKey: also a program that cannot sign in, sending one of the account's API keys with the Chats scope (Authorization: Bearer <key>).request.UserIdis the key's account, which the key also unlocks when it is encrypted. Anything without a key is refused, wherever it comes from.
Getting the mod a key
The mod should take a key both ways:
- Pasted into its config, from Voxta's API keys page.
- Device authorization, when it has none or Voxta refuses the one it has (401 or 403):
POST /api/device/codewith{"client_id":"voxta","scope":"chat","label":"My Game","prompt":true}returnsuser_code,verification_urlanddevice_code. Show the player "enteruser_codeatverification_url". Whenpromptedistrue, Voxta also asks its signed-in users to approve: show "or pickbinding_messagein Voxta" too.POST /api/device/pollwith{"device_code":"..."}every few seconds: 204 while waiting, 200 withtokenonce approved (save it to the config), 400 once the code expired, was used or was denied (access_denied: don't ask again on your own).
The player carries the code: the mod only shows it. Do not open the device page with the code already filled in. Anything on the machine could then have its own code approved by someone clicking through.
A request can wait: a mod polling for the next line can be held until one is ready, as long as it answers before the mod's own timeout.
What your module keeps
- Memory books it makes. Set
MemoryBook.ModuleKeywhen creating one (prefixed with your module's name, for example"mygame:{save}|{npc}") and find it again withunitOfWork.MemoryBooks.GetByModuleKeyAsync(key, cancellationToken). No list of ids to keep in a file, and a book the user deleted is not found. - Characters by name. A game knows its characters by name:
unitOfWork.Characters.GetItemsByUserAsync(new GetCharactersQueryParameters { Name = npcName }, cancellationToken)returns those with that name, ignoring case, the most recently edited first. - Files.
ICommonFolders.GetDataFolder("MyGame", ...), inside Voxta's data folder.
Service types
Every service type a Voxta module can implement, from TextGen and TextToSpeech to music generation and chat bridges, and how the score ranks them.
Configuration UI
Declare a Voxta module's settings page: a typed settings record, a ModuleConfigurationProvider for the form fields, and the registration.