Scripter

The recommended way to integrate Voxta is using Scenarios or Action List Manager, but you can also use Scripter to add Voxta to your scene and add advanced logic as needed.

Using Scripter to start Timeline animations

The idea is simple; when there’s a state change, for example Idle, Thinking, Listening or Speaking, you can play an animation in Timeline.

You can also play animations when the character hears something and “infers actions”, we’ll do that later. For now, let’s add Scripter on your person atom and add those two scripts:

index.js (your scene logic):

import { scripter, scene } from "vam-scripter";

import { initVoxta } from "./lib1.js";

const timeline = scripter.containingAtom.getStorable("VamTimeline.AtomPlugin");

const voxta = initVoxta({
    atom: scripter.containingAtom.getStorable("Voxta")
});

voxta.onStateChanged = state => {
    timeline.invokeAction("Play voxta_state_" + state);
};
    
voxta.onAction = action => {
    timeline.invokeAction("Play voxta_action_" + action);
};

lib1.js (generic voxta integration script):

import { scene, scripter } from "vam-scripter";

const that = {};

let voxtaState;
let voxtaUserMessage;
let voxtaCharacterMessage;
let voxtaCurrentAction;

export function initVoxta(params) {
  const voxta = params.atom;
  voxtaState = voxta.getStringChooserParam("State");
  voxtaUserMessage = voxta.getStringParam("LastUserMessage");
  voxtaCharacterMessage = voxta.getStringParam("LastCharacterMessage");
  voxtaCurrentAction = voxta.getStringParam("CurrentAction");

  scripter.declareAction("OnVoxtaStateChanged", () => {
    try {
      if(that.onStateChanged != undefined) that.onStateChanged(voxtaState.val);
    } catch (e) {
      console.log(e);
    }
  });

  scripter.declareAction("OnVoxtaCharSpeak", () => {
    try {
      if(that.onVoxtaCharSpeak != undefined) that.onVoxtaCharSpeak(voxtaCharacterMessage.val);
    } catch (e) {
      console.log(e);
    }
  });

  scripter.declareAction("OnVoxtaAction", () => {
    try {
      if(that.onAction != undefined) that.onAction(voxtaCurrentAction.val);
    } catch (e) {
      console.log(e);
    }
  });
  
  that.getState = () => {
    return voxtaState.val;
  };

  that.getLastUserMessage = () => {
    return voxtaUserMessage.val;
  };

  that.getLastCharacterMessage = () => {
    return voxtaCharacterMessage.val;
  };

  return that;
}

You can now set the three triggers in Voxta to OnVoxtaStateChanged, OnVoxtaCharSpeak and OnVoxtaAction respectively and call their Scripter counterparts.

Once you have this, you can use the following animations in a voxta_states layer in Timeline:

  • Play voxta_state_idle
  • Play voxta_state_thinking
  • Play voxta_state_listening
  • Play voxta_state_speaking

And for actions, see action inference below.

Keep in mind, this is code so you can make any condition you want.

A small tip: you could call Play voxta_state_thinking/* instead and randomize multiple animations in Timeline!

Context

The context key is very important to tell the AI what is happening. Include things like their position, what they see, even how they feel. For example:

{{char}} is standing in the middle of the room. {{char}} is looking at {{user}}. {{char}} is hesitant about what to say next.

Action Inference

Action Inference is a way to make the Virt-A-Mate character react to things the AI says by playing matching animations. You can use it for simple things, like smiling or waving, or for more complex things like moving around the room or even loading another scene.

In the Actions field, you can list “functions” the AI can call. Those functions will be called after the AI has finished generating the response (but before it finishes speaking it).

And in a layer called voxta_actions, add an animation for each of the actions you have declared, like for action jump you would have voxta_action_jump.

The syntax for action inference is:

# Comment
action: action_name
when: Explain to the UI when to use the action
effect: Text to inject to the AI response so it understands what it's doing

action: ...

Only use effect if it will improve the understanding of the AI. For example, smile may not need an effect at all. However sit_down may need an effect to help the AI understand that they themselves made this. Along with the context field, of course.

You will typically want to update the context and actions when using a Timeline animation. For example, if the model sits down, Timeline would run a trigger that updates the context to the new context (the character is sitting down on a chair) and to the new actions (stand up, cross legs, etc.)

Here is a good example of how to write actions:

action: hand_clap
when: When {{char}} is happy, proud or excited or when {{user}} asks {{char}} to clap
effect: I clap my hands happily

action: smile
when: Whenever {{user}} says something nice

action: leave_room
when: Whenever {{user}} rejects {{char}}'s invitation at least twice