Discussions

I’m experiencing persistent issues running a HeyGen interactive avatar that uses WebSockets for audio and LiveKit for video lip sync. The LiveKit video connection behaves inconsistently — sometimes it connects and works correctly, sometimes it starts and then freezes, and most of the time the video loads but does not respond to the text prompts I send.

⚙️ Suspected Cause

It appears the issue may be related to how the WebSocket and LiveKit connections are being handled in parallel.

All debug and console logs indicate the connections themselves are successful — room events flow normally, and the logs confirm the LiveKit room connects properly. However, the video stream intermittently stalls without warning. It looks as though the connection initializes correctly but then stops updating or syncing after startup.

🧠 Investigation Details

I’m building a lightweight prototype using JavaScript embedded directly in HTML.
I’ve carefully reviewed the HeyGen documentation, but there’s no information addressing this issue. Additionally, the sample code provided appears to be outdated — the avatar "Wayne" used in the example is no longer available, and the code no longer runs successfully.

I suspect the problem occurs around the following lines:
javascript: await room.connect(sessionInfo.url, sessionInfo.access_token);

LiveKit Test Output

Can publish video
Received video frames
Published 490 video packets
Resuming connection after interruption
Error: could not establish signal connection: requested room does not exist
Fail

The connection itself logs as successful, but the video animation becomes unstable shortly thereafter.

💻 Code Snippet (Simplified)
javascript

// Create + Start a new streaming session
async function createNewSession() {
try {
updateStatus("🔄 Creating streaming session (v2)…");

const res = await fetch("/api/session/create", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({
    avatar_name: avatarID.value,
    voice_id: voiceID.value,
  }),
});

const data = await res.json();
sessionInfo = {
  session_id: data.session_id,
  url: data.url,
  access_token: data.access_token,
  session_token: data.session_token,
  livekit_agent_token: data.livekit_agent_token,
  realtime_endpoint: data.realtime_endpoint,
};

// LiveKit room setup
room = new LivekitClient.Room({
  adaptiveStream: true,
  dynacast: true,
  videoCaptureDefaults: {
    resolution: LivekitClient.VideoPresets.h720.resolution,
  },
});

room
  .on(LivekitClient.RoomEvent.Connected, () =>
    updateStatus("✅ [LiveKit] Connected")
  )
  .on(LivekitClient.RoomEvent.Disconnected, (reason) =>
    updateStatus("❌ [LiveKit] Disconnected: " + reason)
  );

// Connect to LiveKit room
updateStatus("🔗 Connecting to LiveKit room…");
await room.connect(sessionInfo.url, sessionInfo.access_token);
updateStatus("✅ Connected to LiveKit room");

// Connect WebSocket for audio
await connectWebSocket(sessionInfo.session_id, sessionInfo.session_token);

updateStatus("🚀 Session ready — send text to talk!");

} catch (err) {
console.error(err);
updateStatus(❌ Error: ${err.message});
}
}

🔍 LiveKit Test Output
subunit

Connected to server, version 1.9.1.
LiveKit Cloud: Singapore
Establishing WebRTC connection...
Can connect via TURN
Can publish audio
Error: unable to detect audio from microphone
Can publish video
Received video frames
Published 490 video packets
Resuming connection after interruption
Error: could not establish signal connection: requested room does not exist
Fail

🧩 Current Connection Setup
json

{
"version": "v2",
"avatar_id": "Tyler-insuit-20220721",
"voice": {
"voice_id": "f5a3cb4edbfc4d37b5614ce118be7bc8",
"rate": 1.0
},
"quality": "high",
"video_encoding": "H264"
}

🧪 Troubleshooting Steps Tried

Changed avatars (several tested)
Changed voices and playback rates
Verified credentials and tokens
Confirmed no firewall restrictions
Tested with and without adaptive streaming
Used different browsers and network connections

Results: The audio stream via WebSocket works consistently, but the video stream via LiveKit remains unstable — freezing or failing to sync after initialization.

💬 Environment

HeyGen API Plan: Pro (≈ 90 credits available)
Platform: Browser-based prototype (JavaScript + HTML)
Deployment Goal: Public access via mobile and desktop web
Network: No corporate firewall; tested on multiple connections

❓Question

Has anyone else experienced similar LiveKit video stalling issues when using WebSocket audio in parallel with LiveKit video?
Any specific configuration adjustments or workarounds you recommend would be greatly appreciated.

my account is HeyGen Pro with over 90 credits available