Adding Voxta to a Virt-A-Mate scene

This guide will help you add Voxta on a Virt-a-Mate scene (when not provided by Voxta).

Preliminaries

Make sure you have followed steps below:

Adding Voxta to a blank scene

Step by step process:

Start Virt-a-Mate
Load the desired scene
Once the scene is loaded
Bring up the Menu (U)
Activate Edit mode (E)
Go to the Person atom (N)
Go to Plugins tab
Click Add Plugin
Search for Voxta
Folder should look like “AcidBubbles.Voxta.#Version#
Open folder and click on Voxta.cslist
Now Voxta Virt-a-Mate Plugin is added
Open Custom UI in Plugins tab to access settings
Check that status is “Connected”
In Chat Parameters, select Scenario and/or Character for the right Person Atom
Activate Voxta

Basic additions

This will make your Person atom talk. Cool!
But it’s not very interactive. Let’s add some more.

You may want to enable lip sync in the Person atom, in Auto Behaviors, Lip Sync, Enabled. You could also add a few plugins to make the character more alive, like Glance for eye movements for example.

Advanced additions

Note that no animations will be played; to play animations, you can use something like Scripter and use On state changed and On animation changed to run animations in Timeline. You can also use the demo scenes as a starting point.

Using Scripter to start Timeline animations

The easiest way to integrate Voxta with your scene logic is by using Scripter.

The idea is simple; when there’s a state change, for example Idle, Thinking, Listening or Speaking, you can play an animation in Timeline.

You can also play animations when the character hears something and “infers actions”, we’ll do that later. For now, let’s add Scripter on your person atom and add those two scripts:

index.js (your scene logic):

import { scripter, scene } from "vam-scripter";

import { initVoxta } from "./lib1.js";

const timeline = scripter.containingAtom.getStorable("VamTimeline.AtomPlugin");

const voxta = initVoxta({
    atom: scripter.containingAtom.getStorable("Voxta")
});

voxta.onStateChanged = state => {
    timeline.invokeAction("Play voxta_state_" + state);
};
    
voxta.onAction = action => {
    timeline.invokeAction("Play voxta_action_" + action);
};

lib1.js (generic voxta integration script):

import { scene, scripter } from "vam-scripter";

const that = {};

let voxtaState;
let voxtaUserMessage;
let voxtaCharacterMessage;
let voxtaCurrentAction;

export function initVoxta(params) {
  const voxta = params.atom;
  voxtaState = voxta.getStringChooserParam("State");
  voxtaUserMessage = voxta.getStringParam("LastUserMessage");
  voxtaCharacterMessage = voxta.getStringParam("LastCharacterMessage");
  voxtaCurrentAction = voxta.getStringParam("CurrentAction");

  scripter.declareAction("OnVoxtaStateChanged", () => {
    try {
      if(that.onStateChanged != undefined) that.onStateChanged(voxtaState.val);
    } catch (e) {
      console.log(e);
    }
  });

  scripter.declareAction("OnVoxtaCharSpeak", () => {
    try {
      if(that.onVoxtaCharSpeak != undefined) that.onVoxtaCharSpeak(voxtaCharacterMessage.val);
    } catch (e) {
      console.log(e);
    }
  });

  scripter.declareAction("OnVoxtaAction", () => {
    try {
      if(that.onAction != undefined) that.onAction(voxtaCurrentAction.val);
    } catch (e) {
      console.log(e);
    }
  });
  
  that.getState = () => {
    return voxtaState.val;
  };

  that.getLastUserMessage = () => {
    return voxtaUserMessage.val;
  };

  that.getLastCharacterMessage = () => {
    return voxtaCharacterMessage.val;
  };

  return that;
}

You can now set the three triggers in Voxta to OnVoxtaStateChanged, OnVoxtaCharSpeak and OnVoxtaAction respectively and call their Scripter counterparts.

Once you have this, you can use the following animations in a voxta_states layer in Timeline:

Play voxta_state_idle
Play voxta_state_thinking
Play voxta_state_listening
Play voxta_state_speaking

And for actions, see action inference below.

Keep in mind, this is code so you can make any condition you want.

A small tip: you could call Play voxta_state_thinking/* instead and randomize multiple animations in Timeline!

Context

The context key is very important to tell the AI what is happening. Include things like their position, what they see, even how they feel. For example:

{{char}} is standing in the middle of the room. {{char}} is looking at {{user}}. {{char}} is hesitant about what to say next.

Action Inference

Action Inference is a way to make the Virt-A-Mate character react to things the AI says by playing matching animations. You can use it for simple things, like smiling or waving, or for more complex things like moving around the room or even loading another scene.

In the Actions field, you can list “functions” the AI can call. Those functions will be called after the AI has finished generating the response (but before it finishes speaking it).

And in a layer called voxta_actions, add an animation for each of the actions you have declared, like for action jump you would have voxta_action_jump.

The syntax for action inference is:

# Comment
action: action_name
when: Explain to the UI when to use the action
effect: Text to inject to the AI response so it understands what it's doing

action: ...

Only use effect if it will improve the understanding of the AI. For example, smile may not need an effect at all. However sit_down may need an effect to help the AI understand that they themselves made this. Along with the context field, of course.

You will typically want to update the context and actions when using a Timeline animation. For example, if the model sits down, Timeline would run a trigger that updates the context to the new context (the character is sitting down on a chair) and to the new actions (stand up, cross legs, etc.)

Here is a good example of how to write actions:

action: hand_clap
when: When {{char}} is happy, proud or excited or when {{user}} asks {{char}} to clap
effect: I clap my hands happily

action: smile
when: Whenever {{user}} says something nice

action: leave_room
when: Whenever {{user}} rejects {{char}}'s invitation at least twice