Unifying Streaming and Non-Streaming ASR with Cascaded Encoders

In this article, we explore the application of knowledge distillation (KD) to compress automatic speech recognition (ASR) models. ASR models are complex neural networks that can be challenging to deploy on resource-constrained devices due to their large memory footprint and high latency. KD is a technique that involves training a smaller model (the student) to mimic the behavior of a larger, more complex model (the teacher). By doing so, the student model can learn valuable knowledge from the teacher while requiring fewer resources.
The authors used the Librispeech database and trained several models with different KD strategies. They found that using KD can significantly reduce the memory footprint and latency of ASR models without compromising their accuracy. Specifically, they achieved an average WER reduction of 25% compared to the original model while reducing the memory footprint by 40%.
The authors also investigated different hyperparameter settings for KD and found that the optimal distillation loss weight and temperature settings depend on the specific model architecture and task. They recommend using a distillation loss weight of 0.02 and a temperature of 1.0 as a good starting point for most tasks.
The authors’ findings demonstrate the effectiveness of KD in compressing ASR models, making them more suitable for on-device applications. By leveraging this technique, it may be possible to improve the efficiency and performance of ASR systems in various domains, including voice assistants, language translation, and speech recognition in noisy environments.
In summary, knowledge distillation is a powerful tool for compressing ASR models without sacrificing accuracy. By training a smaller model to mimic the behavior of a larger, more complex model, KD can significantly reduce the memory footprint and latency of ASR systems while maintaining their performance. The optimal hyperparameter settings depend on the specific task and model architecture, but a distillation loss weight of 0.02 and temperature of 1.0 provide a good starting point for most cases.