Struggling to find out the best option for the streaming avatar without lag time for a custom LLM setup

Hello, I am trying to use a custom LLM set up with my interactive streaming avatar and I'm going in circles trying to get it to work properly without extensive lag. I had it working great but the problem was there was a 7 to 8 second lag time with the avatar.

I am currently doing a live stream through an edge function for the transcript in the hope that it will speed up the box responses, but it's causing infinite loops and issues with the VAD detection. I had it working great originally, but the problem was there was the long lag time so this was my new attempt. It's created a very difficult set up now and I'm not sure if I'm going in the right direction.

I wanted to have options for people to select the different avatars and then pass through the API to change what avatar is being used but now I'm wondering with how difficult of a set up this has become should I just be embedding the avatar and have a different page for each avatar?

I'm not sure on what to do. Should i go back to the longer lag time because that's the setup it's meant to use? This new process has become way out of my league.