He built TikTok’s music AI tech. But he says robots will never replace human songwriters.

Welcome to the latest episode of the Music Business Worldwide Podcast. The MBW Podcast is supported by Voly Music.

You might have noticed via MBW’s recent headlines that, in 2023’s music business, there are no two hotter subjects than (i) Artificial Intelligence and (ii) TikTok.

Ed Newton-Rex, then, is perhaps the single most topical interviewee we could speak to right now.

That’s because London-based Newton-Rex founded and built the astonishing, pioneering music-making AI platform, Jukedeck.

And then, in 2019, he sold it… to TikTok.

As a result, Jukedeck’s wizardry is now firmly embedded in ByteDance‘s tech stack – which, from an AI standpoint, was overseen by Newton-Rex for two years subsequent to that 2019 sale.

During that period, he was Product Director of TikTok’s in-house AI Lab, gaining a unique perspective on what the world’s biggest short-form video platform plans to do with AI-generated audio.

(Some alarmed members of the modern music rights business have been doing a bit of fretting over that subject of late… especially since TikTok began experimenting with removing major label music from its platform for certain users in Australia.)

Ed Newton-Rex’s credentials in the world of music tech and generative AI go far beyond TikTok, though.

He left ByteDance two years ago to join music tech startup Voisey – an impressive music-making platform that was acquired (and then bizarrely shut down) by Snap Inc.

These days, Newton-Rex is once again looking to revolutionize music’s relationship with AI, at yet another cutting-edge tech business.

As VP of Audio at Stability AI, he oversees Harmonai, which bills itself as “a community-driven organization releasing open-source generative audio tools to make music production more accessible and fun for everyone”.

Stability AI is best known for being the home of Stable Diffusion, an AI-powered text-to-image generator that’s been at the forefront of the “oh wow, AI really is about to change the world” chatter pinging around the tech-o-sphere in the past six months.

On top of all of that, Newton-Rex is a professional (human!) composer himself, with his choral work published by the esteemed Boosey & Hawkes company (owned by Concord).

On this MBW Podcast, we ask Newton-Rex about the future of AI in music, whether robot music-making can ever take over from human composers, and what the future might hold for TikTok.

Listen to MBW’s Podcast interview with Newton-Rex above, and/or read an abridged version of our discussion below…

You were building Jukedeck since 2010, so you’ve been in the world of generative AI for over a decade. Is the sudden widespread interest in the field great news for you? Or are you a little bit, ‘Hi guys. I’ve actually been here a while…’?

I’ve definitely been here a while! It does feel like it’s been a long time coming.

I’m not sure I know of anyone who’s been building AI music products for longer than me; I’m sure they’re out there. But I’m really pleased that the things that I and others have been talking about for a while are coming to pass.

I’ve always believed, and I’ve said many times, that generative AI will be a force for good – so I was always going to be happy when it did finally hit the mainstream.

Let’s start with the positive: what in your mind is the greatest thing that generative AI could bring to the music industry?

Democratizing creativity has always struck me as a very powerful idea. Even back at Jukedeck, we had people reaching out to say how thrilled they were that they could finally make music. They were generating backing tracks using our tech and then writing a melody over the top, singing and recording themselves.

I think you have this opportunity to find great singers, great artists who wouldn’t otherwise have access to producers and [wouldn’t have been able to] make that kind of music [without AI].

I’ve always also been excited about this idea of giving people personalized music. I started Jukedeck in 2010, and about four years later, when I’d been head-down working on the algorithm, no-one was talking about generative music; it was not a great time. I was thinking, “God I’ve been doing this for years and it’s really hard.” I was sitting on a plane, and I watched the film Her with Joaquin Phoenix, where he sort of falls in love with this AI.

“I was sitting on a plane, and I watched the film Her with Joaquin Phoenix, where he sort of falls in love with this AI.”

There’s a bit in it where the AI writes him a piece of music, and it’s actually by Arcade Fire in real life, a beautiful piano piece. That really invigorated me, because I thought: Yes, this is where we can go with this idea of personalized music. It’s going to be incredibly powerful.

The main benefit [of AI] for the music industry is increasing value for rights holders. That may sound counterintuitive [in the context of debates around AI making music] but when you have AI, the music that you write, or that you own, can become so much more valuable, because it’s no longer just one static thing. It can be modified.

So maybe a track you’ve written or that you’ve gotten in your library is lengthened to fit a different TV ad, maybe the instrumentation is changed to get the right mood in a video, maybe you change the entire style to fit something totally new.

What starts out as one piece of music that [was] set in stone can become this living thing that can be adapted, endlessly. That’s very exciting.

Do you think that generative AI is a legitimate threat to elements of today’s music business? especially when you consider what’s recently been called ‘functional music’? Universal‘s Lucian Grange USED THAT PHRASE; he means ‘music for running’ or ‘music for sleeping’ or ‘music for relaxation’ etc? Do you think that that’s maybe an area where AI-made music will take over? Should incumbent music rights holders be worried?

The first sign I got of this [worry about AI music] in my career was at a dinner party.

It was the early 2010s, I’d been doing Jukedeck for a couple of years, I’d been super-secretive about it. I was sat next to a friend’s mom and I told her what I was doing [for a living]. She sort of nodded politely and listened. And then, just at the end of the conversation, she said: ‘I’ve got to be honest, I hope you fail in this endeavor.’ And I was like, ‘Oh, wow. Like, why?’ And she said: ‘Because I’d like to think there’s something in music that is innately human.’

That’s definitely a take I’ve heard from some people in the music industry. But I also know a whole bunch of people in the music industry who take a very different view. Those are the people, I think, who realize the opportunity that generative AI can bring the music business. I think rightsholders are in a really good position.

“she said: ‘I’ve got to be honest, I hope you fail in this endeavor.’ And I was like, ‘Oh, wow. Like, why?’ And she said: ‘Because I’d like to think there’s something in music that is innately human.'”

You talked about functional music. What if you want is music that reacts to your run [as you exercise]? Rightsholders who own the songs that people love to listen to are in a perfect position [for that]. Because AI isn’t just generative – it’s also adaptive.

I’ve never met anyone who works in this field who wants [AI-generated] music to be a threat to the music industry. Almost without exception, the people working in this field are, like me, musicians. And our friends are musicians!

I recently composed a piece using ChatGPT to write the [lyrics]. That’s exciting. But does that mean that I’m only going to [use] AI tech from now on, putting human [lyricists] out of a job? Absolutely not.

AI will be at its most powerful [as] a tool used by musicians in countless different ways. Frankly, [that includes] ways that even people like me working in the industry today can’t yet predict.

I suspect we’ll look back in 10 years and think this was actually a very good thing for the music business.

As somebody who has seen the increase in sophistication in generative AI music is: do you think it will ever be able to elicit a human emotional response in the same way that – and I’m picking this song out the air – Nothing Compares 2 U does? The lyrics from Prince, the heartbreaking vocals from Sinead O’Connor… Could AI learn to game my emotional matrix like that?

In the last wave of AI [hitting the mainstream] in 2016, 2017, I did a sort of Turing Test, where I played [Jukedeck’s] music and then played a real track, and I asked the audience to guess which one was composed by AI.

Generally, it was 50/50 – people didn’t really know. But I gamed it [a little]: I chose a particularly good AI track, and for that, to actually pass the Turing Test, you [still] need to be interacting with the AI: ‘Let’s make the tune a bit sadder, and [change] the piano…’

“I’ve come to think that maybe we shouldn’t be thinking about AI music in terms of reaching parity with human music in its effect [on listeners].”

I’ve come to think that maybe we shouldn’t be thinking about AI music in terms of reaching parity with human music in its effect [on listeners]. I just don’t kind of think that’s the aim for most people in the field.

The aim is to bring value, and to make things possible that aren’t possible without AI.

I’ve been listening to some of the pieces of music that Jukedeck created right up until the sale to TikTok in 2019, and it gets increasingly sophisticated, and increasingly impressive. By the time you joined TikTok in 2019, and during the time you spent there, how good in your view was music that Jukedeck was producing?

By the time we joined TikTok, I think we’d created one and a half million or so tracks. And [during that time] we kept climbing that quality ladder [of sophistication in AI]; we actually had our music performed in a pretty big concert in Seoul, by some pretty good bands.

We got to the point where [Jukedeck’s music] was very usable in social media videos. YouTube creators were among the biggest users of Jukedeck, and it was [making] some pretty good stuff. Pro-YouTubers started using it; then we had [Jukedeck’s music] featured in ads by some decent brands.

“By the time we joined TikTok, I think we’d created 1.5 million or so tracks.”

But compared to [what’s possible] today, it was doing nothing. So much has changed. One of the things that was hardest for [Jukedeck] was melody generation. That’s tough, but there are ways around it.

It’s funny; there are definitely music startups around AI music [today] who are taking the same approach as Jukedeck [and the] output achieved by those kinds of companies hasn’t improved that much [on what Jukedeck did]. But then there’s [startups based on this other] whole new field of how to do this that are achieving crazy good results.

I’m stunned by that number: 1.5 million tracks! There are companies with billion-dollar valuations today who effectively sell Production Music for creators on social media platforms – on YouTube on TikTok, for podcasts, for brands. I just don’t see a way that this isn’t a huge market opportunity for those building AI companies in the wake of Jukedeck.

I mean, you cannot really predict what’s going to happen now. If you look at the world of [AI-generated] images, things are moving so fast and changing so quickly, it’s a fool’s game to predict [what] the state of that tech will be even in a year’s time.

What I’m most certain of is that the music does really well on social platforms, which is largely [human-created] popular music, just will not be affected by this [AI] movement. What people really care about is a connection to the artist.

“I mean, sure, there’ll be a robot popstar at some point. Of course there will – everything that can happen will happen.”

I mean, sure, there’ll be a robot popstar at some point. Of course there will – everything that can happen will happen. But that’s not going to be the way the [majority of the] industry goes. It’ll be a one-off.

That connection with [human artists] has got to be at least half of the reason that we love music. It’s not just about what we hear.

Do you think it’s possible that music streaming services and social platforms like TikTok are going to become increasingly less reliant on licensed music as they lean more towards AI music? This is a big topic for debate right now…

I can’t see how that happens realistically. Again, it’s the human connection – people are just always going to want licensed music. And by licensed music I mean the music that’s huge on Spotify, Apple Music, all of these platforms. It’s huge for a reason.

“We care about the artists, we care about the people, we care about the story, we care about all of this stuff. I just can’t see how you step away from that as a social media platform.”

We care about the artists, we care about the people, we care about the story, we care about all of this stuff. I just can’t see how you step away from that as a social media platform. And I think if you do step away from that, you’re losing out on a huge amount – on the ability for your users, the people using your platform, to tell stories that really resound with what’s going on in culture today.

Ultimately, social media is about the culture of the day. So I don’t think [AI music replacing human-made hits on social platforms] is going to happen.

Peer into the future for us; what is the most exciting AI-related tech or project – outside music, or inside music – that you’ve seen recently? Something that hints at the possibilities of the future.

There’s a lot going on. Even just working at Stability, working with people across all these different domains, the number of academic papers and projects that get shared – there are 10 a day springing up.

Outside of what I’ve seen at work, I was reading the FT over the weekend. There was a story titled something like, ‘Good news for humanity, as human beats machine at board game Go.’  It was celebrating the fact that a human had actually beaten a machine at this incredibly complex board game.

“Basically, it shows us that these Go [AI] systems – and you can extrapolate that to other [AI] systems as well – might be impressive, but ultimately, they’re just party tricks.”

I was bowled over by this, because it shows how far we’ve come since 2016. [Back then] everyone was amazed that a machine could beat a human at Go. But this time, seven years later, it’s the other way around!

Basically, it shows us that these Go [AI] systems – and you can extrapolate that to other [AI] systems as well – might be impressive, but ultimately, they’re just party tricks. They are really useful party tricks, and they can have huge value. But if you try hard enough, you can find the bits they don’t know about [because] they’re not human.

I love that these systems have limitations. They shouldn’t be replacing us; they [should be] useful. We can design them to accomplish certain things, and they can have great value, but we shouldn’t fall into the trap of thinking that they’ve ultimately got anything ‘up’ on humans.