Discussions

Hi,

I’m implementing canvas-based chroma key processing on top of a Heygen streaming avatar in an Angular 14 kiosk application.

While the experience is mostly smooth on macOS, we are seeing frequent stuttering and frame drops on Windows, especially in a kiosk environment.

Key details about the setup:

-Video element uses native autoplay and loop

-Chroma key is applied via 2D canvas

-Canvas size is set only once (not inside the render loop)

-FPS is limited (around 8–10 FPS)

-Different rendering strategies have been tested

-Despite this, on Windows kiosk we observe:

--Visible stuttering

--Occasional moments where the video appears to “restart”,

On the Heygen Tools page, the chroma key demo does not show this behavior, so I suspect the difference might be related to:

Stream frame timing

Video decoding behavior on Windows

Canvas + chroma key processing under kiosk constraints

I’m sharing a sample of the chroma key and rendering code below for reference.
I’d really appreciate your guidance on whether there are Heygen specific optimizations or a recommended approach for handling chroma key on streaming avatars in Windows kiosk environments.

Thank you for your time and support.

public applyChromaKey(  
      sourceVideo: HTMLVideoElement,  
      targetCanvas: HTMLCanvasElement,  
      options: {  
        minHue: number; // 60 - minimum hue value (0-360)  
        maxHue: number; // 180 - maximum hue value (0-360)  
        minSaturation: number; // 0.10 - minimum saturation (0-1)  
        threshold: number; // 1.00 - threshold for green detection  
      }): void {  
    // Get canvas context  
    const ctx = targetCanvas.getContext("2d", {  
      willReadFrequently: true,  
      alpha: true,  
});

if (!ctx || sourceVideo.readyState < 2) return;

// Clear canvas
ctx.clearRect(0, 0, targetCanvas.width, targetCanvas.height);

// Draw video frame to canvas
ctx.drawImage(sourceVideo, 0, 0, targetCanvas.width, targetCanvas.height);

// Get image data for processing
const imageData = ctx.getImageData(
    0,
    0,
    targetCanvas.width,
    targetCanvas.height
);
const data = imageData.data;

// Process each pixel
for (let i = 0; i < data.length; i += 4) {
  const r = data[i];
  const g = data[i + 1];
  const b = data[i + 2];

  // Convert RGB to HSV
  const max = Math.max(r, g, b);
  const min = Math.min(r, g, b);
  const delta = max - min;

  // Calculate hue
  let h = 0;
  if (delta === 0) {
    h = 0;
  } else if (max === r) {
    h = ((g - b) / delta) % 6;
  } else if (max === g) {
    h = (b - r) / delta + 2;
  } else {
    h = (r - g) / delta + 4;
  }

  h = Math.round(h * 60);
  if (h < 0) h += 360;

  // Calculate saturation and value
  const s = max === 0 ? 0 : delta / max;
  const v = max / 255;

  // Check if pixel is in the green screen range
  const isGreen =
      h >= options.minHue &&
      h <= options.maxHue &&
      s > options.minSaturation &&
      v > 0.15 &&
      g > r * options.threshold &&
      g > b * options.threshold;

  // Apply transparency for green pixels
  if (isGreen) {
    const greenness = (g - Math.max(r, b)) / (g || 1);
    const alphaValue = Math.max(0, 1 - greenness * 4);
    data[i + 3] = alphaValue < 0.2 ? 0 : Math.round(alphaValue * 255);
  }
}

// Put processed image data back to canvas
ctx.putImageData(imageData, 0, 0);
}

public setupChromaKey(
    sourceVideo: HTMLVideoElement,
    targetCanvas: HTMLCanvasElement,
    options: {
        minHue: number;
        maxHue: number;
        minSaturation: number;
        threshold: number;
    }
): () => void {
    let animationFrameId: number | null = null;
    let lastFrameTime = 0;
    let fps: number = 10;
    let frameInterval = 1000 / fps;
    let scale: number = 0.5
    let initialized = false;
    const render = (time: number) => {
        if (!initialized && sourceVideo.readyState >= 2) {
            targetCanvas.width = sourceVideo.videoWidth;
            targetCanvas.height = sourceVideo.videoHeight;
            initialized = true;
        }

        if (initialized && time - lastFrameTime >= frameInterval) {
            this.applyChromaKey(sourceVideo, targetCanvas, options);
            lastFrameTime = time;
        }
        animationFrameId = requestAnimationFrame(render);
    };

    render(0);

    return () => {
        if (animationFrameId !== null) cancelAnimationFrame(animationFrameId);
    };
}