Scaling Speech Tokenizers with Diffusion Autoencoders

Abstract Speech tokenizers are foundational to speech language models, yet existing approaches face two major challenges: (1) balancing trade-offs between encoding semantics for understanding and acoustics for reconstruction, and (2) achieving low bit rates and low token rates. We propose Speech Diffusion Tokenizer (SiTok), a diffusion autoencoder that jointly learns semantic-rich representations through supervised learning and enables high-fidelity audio reconstruction with diffusion. We scale SiTok to 1.6B parameters and train it on 2 million hours of speech. Experiments show that SiTok outperforms strong baselines on both reconstruction and understanding tasks, at an extremely low token rate of 12.5 Hz and a bit-rate of 200 bits-per-second.

This page is for research demonstration purposes only.

Speech Reconstruction Comparison with Other Systems

This section compares the speech reconstruction quality of various systems with different bitrates. We choose some in-the-wild and expressive samples to better demonstrate the reconstruction quality of SiTok.

Ground Truth Mimi (1.1 kbps) WavTokenizer (0.9 kbps) BiCodec (0.65 kbps) SiTok (Ours) (0.2 kbps)

Zero-Shot TTS

This section demonstrates the zero-shot TTS capabilities built on SiTok. We present diverse test cases with in-the-wild and expressive prompts and long-form target texts, highlighting SiTok's ability to be effectively and efficiently used for zero-shot TTS.

Target Text Prompt Speech Generated Speech
  • After that point, new developments and insights in various fields, such as technology, science, and culture, continue to emerge. It is essential to stay updated with the latest information to keep our understanding current and relevant. Engaging with recent publications, discussions, and updates will enhance our knowledge and perspective on evolving topics.
  • You think you can just waltz in here and cause chaos? Well, I've got news for you. This time, there's no escaping the consequences. So, one by one, step forward, and let's see who's bold enough to face the music. It's time for a little dose of reality—prepare to be dealt with!
  • Let it envelop you like a cozy blanket, reminding you of the moments shared and the lessons learned. Embrace the wisdom that flows from those experiences, letting it guide you through challenges and inspire your dreams. Each memory is a treasure, a flicker of light that can illuminate the darkest paths. Let the love of your elder resonate within.
  • He greeted everyone with genuine warmth, making each person feel special and valued. As he engaged in conversation, his laughter was infectious, creating an atmosphere of joy that drew people in. Whether it was a simple hello or an encouraging word, his presence had the power to uplift even the heaviest of hearts, leaving a lasting impression on all who crossed his path.
  • However, I believe that justice has a way of revealing itself in unexpected manners, and while the odds may be stacked against me, I am not one to back down easily. The law can often be a labyrinth, and within its twists and turns, there may still be a glimmer of hope. I have gathered a team of legal minds who are willing to explore every avenue, challenge every assumption, and shed light on the truths that others may wish to keep hidden. With tenacity and resilience, we will forge ahead, for sometimes the most remarkable victories come from the most unlikely circumstances.
  • It might give you the space to collect your thoughts and emotions, to reflect on everything we've been through. After all, sometimes a little distance can provide clarity, and I want you to feel comfortable.
  • You'll see a large oak tree nearby, and just beyond that, there's a small playground with swings and slides. It's a great spot for relaxing or letting the kids have some fun. Just follow the path, and you'll find plenty of space to enjoy the fresh air.
  • I can't shake this feeling of guilt, this notion that happiness is something I have to earn, not something I can simply take. Every laugh feels forced, every smile feels like a mask, hiding the turmoil within. I keep reminding myself of the struggles around me, the battles that so many face daily. How could I possibly revel in joy when there's so much sorrow in the world? It's a constant tug-of-war between wanting to feel free and the weight of responsibility pressing down on my heart.