Futureverse CEO Aaron McDonald presents JEN-1 at Alibaba's 2023 Apsara Conference, Hangzhou. Photo credit: Futureverse
“Music is the universal language of mankind,” wrote Henry Wadsworth Longfellow. Now, artificial intelligence is making it easier to put words into song.
Riding the wave of technological advances in recent years is New Zealand-based AI and metaverse startup Futureverse, which has launched a text-to-music AI generator called JEN after training it on Alibaba Cloud’s AI platform.
While artificial intelligence has long augmented the creative process in the music industry, technology took a leap forward with the introduction of machine-learning models that generate music based on text prompts.
Generating high-fidelity and realistic music is still challenging due to the complexity of musical structures and the high sampling rate required. With humans being sensitive to musical dissonance, music generation allows little room for imperfection.
Music utilizes the full frequency spectrum, requiring high sampling rates of around 44.1KHz; in contrast, speech focuses on linguistic content and uses lower sampling rates of about 16kHz. Futureverse’s JEN-1 is capable of generating high-quality music at a 48kHz sampling rate.
High Fidelity
Founded in 2022, Futureverse trained JEN-1 on Alibaba Cloud’s machine learning platform for AI. The platform has over 140 built-in optimization algorithms for enterprises and developers, supporting complicated computing tasks, such as foundation model training and inference.
“A real challenge is getting access to high-quality infrastructure to train these big models,” said Futureverse‘s co-founder and CEO Aaron McDonald in an interview with Alizila.
Futureverse is particularly proud of JEN-1’s high fidelity after using 5,000 hours of high-quality private music data to train her. In a paper published by Futureverse, the company claimed that JEN-1 had demonstrated higher-quality output than state-of-the-art baselines, such as Google’s MusicLM and Meta’s MusicGen.
Jamming With Jen
While music-making tools are speeding up the creative process by sparking ideas and variations for artists in seconds, AI can also be disconcerting for artists concerned about copyright.
“We worked with the music industry to help us understand how to build a product that both respects creators and their creativity, and helps enable them to become better creators,” said Futureverse‘s McDonald.
Futureverse said music industry veteran Mike Caren is joining the company as a founding partner of JEN. It also committed to pursuing fair and equitable AI in its Jen Doctrine.
Backed by growth equity fund 10T Holdings and Ripple Labs, Futureverse has inked tech partnerships with Reebok, Wimbledon, FIFA and Mastercard among others. It has rolled up 11 metaverse infrastructure and content companies into one ecosystem.
Below is a transcript of Alizila’s interview with McDonald on how generative AI could reshape how people compose, share, and appreciate music. The interview has been edited for brevity and clarity
Q: How do you leverage AI in the music industry?
A: One of the really exciting things that we’ve just launched at Futureverse is JEN, a generative AI model for creating music. We worked with the music industry to help us understand how to build a product that both respects creators and their creativity and helps enable them to become better creators. With Jen, you can take what’s in your mind and make music.
Some of the challenges in developing Jen are on the industry side. We want to really make sure that we respect copyright. And so, the way that we developed Jen comes with a doctrine that we built with the industry, and that doctrine tells us how we should respect the data and reward the creators who created the data that help models become good at what they do.
The other thing that’s been a real challenge is getting access to high-quality infrastructure to train these big models. They’re in big demand around the world. There’s lots of activity in generative AI, on both the research and development front.
Q: Why did you decide to join forces with Alibaba Cloud to train this generative AI music model?
A: When we were building JEN, one of the things we needed was access to a lot of training infrastructure, but not all the time. So, JEN has to be able to access cloud infrastructure when the research team has new things to train it on. When we were rolling out the next version in production, we didn’t need as much infrastructure.
So, being able to access lots of training capability in the cloud and having that on a flexible, on-demand basis was really helpful for developing Jen.
When we were looking at potential partners for building JEN, we actually tested a lot of cloud infrastructure around the world and benchmarked them to see which ones met our expectations for performance, who had capacity available, who had capacity that could be elastic, and the cost of that infrastructure. Across those benchmarks, the Alibaba Cloud solution came out on top.
We partnered with Alibaba Cloud, particularly its PAI Platform (Platform for Generative AI) to enable us to take the software we’d built around it and teach it how to make really high-quality music and build products that could work alongside creators.
Q: How’s your collaboration with Alibaba Cloud going so far?
A: What we’ve developed is a really strong relationship with our partner managers, with the technical teams, with the executive teams inside of Alibaba and Alibaba Cloud, to help us not only develop these models technically, but also extend them into communities, help onboard new users, help enter new markets and help find new use cases.
Alibaba is not just somewhere we go for product, but they’re a partner in this journey with us from a business side too.
Q: How do you see generative AI shaping the future of the music industry?
A: With the launch of JEN, this is just the start of our journey and what we want to do is call out to all the creators out there, musicians, artists, producers, rights owners, come and talk to us about how we can take what you’ve got and help you become better creators with these powerful AI tools.
