💡 The paper introduces Fish Audio S2, an open source text to speech system that features multi speaker capabilities, multi turn generation, and instruction following control through natural language descriptions. The system utilizes a multi stage training approach, which includes a staged data pipeline covering video captioning, speech captioning, voice quality assessment, and reward modeling. This approach allows for scalable training and improves the overall performance of the system. The authors also release their model weights, fine tuning code, and an inference engine, making it production ready for streaming. The inference engine achieves a real time factor of 0.195 and a time to first audio of below 100 milliseconds, indicating its efficiency and speed. The code and weights are made available on GitHub and Hugging Face, and users are encouraged to try custom voices on the website. Overall, the paper contributes to the advancement of open source text to speech systems, providing a robust and efficient solution for generating high quality speech.

💡 The paper introduces IndexTTS, an enhanced text-to-speech system that combines the XTTS and Tortoise models to achieve improved naturalness, voice cloning, and controllable usage. The system addresses the limitations of existing text-to-speech systems, particularly in Chinese scenarios where polyphonic characters and long-tail characters can be challenging to pronounce. To overcome this, the authors propose a hybrid character-pinyin modeling approach that allows for more controllable pronunciations.

The authors also compare Vector Quantization with Finite-Scalar Quantization for codebook utilization of acoustic speech tokens, and introduce a conformer-based speech conditional encoder and BigVGAN2 to enhance voice cloning. The results show that IndexTTS achieves significant improvements in naturalness, content consistency, and zero-shot voice cloning compared to the XTTS model.

In comparison to other popular open-source text-to-speech systems, IndexTTS has a relatively simple training process, more controllable usage, and faster inference speed, while also surpassing their performance. The system is designed to be efficient and controllable, making it suitable for industrial-level applications. The authors provide demos of the system, which are available for evaluation. Overall, the paper presents a novel approach to text-to-speech synthesis that achieves state-of-the-art results and has the potential for practical applications.