How AI is Learning to Create Music and Sounds from Scratch

- Home
- Case studies
- How AI is Learning to Create Music and Sounds from Scratch
Problem
Sound is one of the hardest things for AI to generate. A single second of high-quality audio contains thousands of individual samples, and what makes it sound right depends on patterns at very different time scales at once: the shape of a single wave, the pitch of a note, the rhythm of a phrase, the structure of a whole piece.
Older approaches avoided this difficulty. They worked on simplified, compressed descriptions of audio, or assembled sound from pre-recorded fragments. That limits what a system can produce and how natural it sounds. Companies that need large amounts of fresh audio, such as game studios, media producers, app developers and telecom providers, have long depended on manual recording and sound design, which is slow and expensive to scale.
Also applicable to
- Gaming: generating sound effects and ambient audio for large game worlds.
- Media and entertainment: music, soundscapes and audio for video and podcasts.
- Speech and voice products: natural-sounding audio for assistants and text-to-speech.
- Telecommunications and signal processing: models that understand and reconstruct audio signals.
- Creative tools: software that helps musicians and sound designers explore new material.
Solution / Approach
The Elementera team's research on neural audio generation is best known through SampleRNN, published at ICLR 2017 with Elementera co-founder Soroush Mehri as first author, together with researchers at Mila (Université de Montréal), including Aaron Courville and Yoshua Bengio.
SampleRNN generates audio one sample at a time, directly as a raw waveform, with no hand-designed intermediate format. The challenge is keeping track of structure over very long stretches of samples. The paper's answer is a hierarchical design that combines two kinds of building blocks:
- Stateful recurrent neural networks that run at slower time scales and carry memory of what has happened over long spans.
- Memory-less autoregressive multilayer perceptrons that work at the finest scale, predicting each next sample from the ones just before it.
Higher levels summarise the past and guide lower levels, much as a musician keeps the overall phrase in mind while playing each note. According to the paper, this structure lets the model capture underlying sources of variation in audio over very long time spans, and it was demonstrated on three datasets of different nature.
Why this matters for a company using AI: a model that learns sound directly from recordings can, in principle, produce new audio in the style of whatever it was trained on, without a team building every sound by hand. The same idea was later extended as the neural vocoder in Char2Wav, an end-to-end speech synthesis system Soroush Mehri also co-authored.
Impact
The paper reports that in human evaluation, listeners preferred SampleRNN's generated samples over those of competing models, and it shows how each component of the hierarchy contributes to that performance. SampleRNN became one of the reference points in early neural audio generation research, and its open-source code is publicly available on GitHub.
For businesses, this line of research means:
- Scalable audio creation: generating new sound from data instead of recording every variation.
- Natural-sounding output: working at the level of raw waveforms avoids the artefacts of older, simplified formats.
- One approach, many uses: the same modelling ideas apply to music, speech and sound effects.
- Deep in-house experience: the Elementera team built one of the early end-to-end audio generation models and knows its design choices first-hand.
Technologies
- Recurrent neural networks: modelling long-range structure in audio.
- Autoregressive models: predicting each audio sample from the samples before it.
- Hierarchical sequence modelling: combining several time scales in one model.
- Raw waveform generation: producing audio directly, with no hand-built features.
Read the paper: SampleRNN: An Unconditional End-to-End Neural Audio Generation Model (ICLR 2017). Code: github.com/soroushmehr/sampleRNN_ICLR2017.
Soroush Mehri’s other papers cover fast adaptation, reinforcement learning and the reliability of language models; they are listed with the rest of our founders’ work on Peer-reviewed research by Elementera’s founders.

Ex-Microsoft. 10+ years of experience in AI/ML, MSc from Université de Montréal (Mila). US patent holder.

AI/ML scientist with 20+ years of research & industry experience. Ex-Google DeepMind, MIT, Harvard, Mila. McGill-Mila PhD.
