Whisper (OpenAI)

OpenAI's Whisper represents a cutting-edge neural network designed to match human-level robustness and precision in recognizing English speech. It was trained on an extensive collection of 680,000 hours of multilingual and multitask supervised data, allowing it to effectively manage accents, background noise, and specialized terminology. The system offers flexibility in transcribing various languages and translating them to English, built on an encoder-decoder Transformer architecture. Comparison to Existing Approaches: In contrast to conventional models using limited paired audio-text datasets, Whisper's use of a broad and varied dataset delivers exceptional robustness. While it might not dominate particular benchmarks such as LibriSpeech, it achieves 50% fewer errors in zero-shot evaluations over diverse datasets. Its strength in speech-to-text translation, notably exceeding state-of-the-art results on CoVoST2 for English translation, distinguishes it. Impact and Availability: Whisper has the potential to transform application development via the incorporation of reliable voice interfaces. OpenAI has released its paper, model card, and code for public access, promoting continued research and advancement in the area.

OpenAI's Whisper is an automatic speech recognition (ASR) system trained on 680,000 hours of multilingual and multitask supervised data collected from the web. Its large and diverse training set gives it exceptional robustness to accents, background noise, and technical language. Whisper supports transcription in multiple languages and translation from those languages into English, all within a single end-to-end encoder-decoder Transformer architecture. Input audio is split into 30-second chunks, converted into log-Mel spectrograms, and passed through an encoder. A decoder predicts text captions intermixed with special tokens for language identification, timestamps, and translation tasks. Whisper is open-source, with models and inference code publicly available, and achieves 50% fewer errors in zero-shot evaluations across diverse datasets compared to prior models, though it does not surpass models specialized for benchmarks like LibriSpeech.