Neura Market
Whisper logo

Whisper

Free

Robust speech recognition via large-scale weak supervision. [#opensource](https://github.com/openai/whisper)

FreeFree tier
Inputs: audioOutputs: text
Type
Open Source
Company
OpenAI
Visit Website GitHub (40 stars)

About Whisper

Whisper is an automatic speech recognition (ASR) system developed by OpenAI, trained on 680,000 hours of multilingual and multitask supervised data collected from the web. It uses a simple end-to-end encoder-decoder Transformer architecture, processing audio in 30-second chunks converted to log-Mel spectrograms. Whisper demonstrates improved robustness to accents, background noise, and technical language, and can transcribe speech in multiple languages as well as translate non-English speech into English. The model achieves 50% fewer errors in zero-shot performance across diverse datasets compared to specialized models. OpenAI has open-sourced Whisper's models and inference code to serve as a foundation for building speech applications and further research.

Key Features

Trained on 680,000 hours of multilingual and multitask supervised data
Encoder-decoder Transformer architecture processing 30-second audio chunks
Robust to accents, background noise, and technical language
Multilingual speech transcription and to-English translation
Zero-shot performance with 50% fewer errors than specialized models on diverse datasets
Open-sourced models and inference code

Pros & Cons

Pros
  • Excellent zero-shot performance across diverse datasets and conditions
  • Supports multiple languages and translation to English in one model
  • Open-source with available code and pre-trained models
  • Robust to accents, background noise, and technical jargon
Cons
  • Does not beat specialized models on specific benchmarks like LibriSpeech
  • Relies on 30-second audio chunk processing which may not suit very long recordings natively

Best For

Speech transcription in multiple languagesSpeech-to-text translation from non-English to EnglishVoice interfaces for applicationsResearch on robust speech processing

FAQ

What is Whisper?
Whisper is an automatic speech recognition (ASR) system by OpenAI, trained on 680,000 hours of diverse multilingual data and open-sourced for research and application development.
What languages does Whisper support?
Whisper supports transcription in multiple languages and can translate non-English speech into English. About a third of its training data is non-English.
How does Whisper differ from other ASR systems?
Whisper uses a large-scale weak supervision approach with an encoder-decoder Transformer, making it more robust to accents, noise, and diverse conditions without fine-tuning to a specific dataset.