Add voice id to video agent API

Currently in the video agent API config object, there is no way to set the voice id (only avatar id). I have tried specifying the voice id in the prompt, as well as the voice name, but neither gives deterministic results. Is there a way to specify this?