Last week, MBW published a widely shared report about TikTok and its parent ByteDance‘s ambitions in the field of AI-generated music.
Within that article, we told you about the extensive research conducted by ByteDance’s Speech, Audio & Music Intelligence (SAMI) team in the field of AI music.
One of those research projects describes a generative model called MeLoDy that was trained, according to ByteDance’s SAMI researchers, on 257,000 hours of music (read the paper here) that came from 6.4 million audio files.
We also told you about two US patents, secured by ByteDance in recent months, that provide IP protection in the US for the Chinese company’s AI music-related technologies.
Now, we’ve uncovered an additional batch of AI music-related patent and trademark filings from ByteDance that will raise more than a few eyebrows in the music industry – covering topics including “data mining” and “music generation”.
Interestingly, these US patents and trademark applications were filed by a mysterious ByteDance affiliate called Lemon Inc.
There’s not a lot of information about Lemon Inc. online. We do know, however, that (a) it’s incorporated in the Cayman Islands, and (b) It’s been very busy over the past couple of years submitting applications to the United States Patent and Trademark Office. (It even applied for a book publishing-related trademark at the start of 2023).
Lemon Inc. is not mentioned specifically on ByteDance’s corporate structure diagram (see below, and here).
However, the World Intellectual Property Organization, via its Arbitration reports, recently described Lemon Inc. as being “affiliated with Bytedance Ltd (“Bytedance”), an Internet technology company that provides mobile applications (“apps”) such as TikTok”.
Lemon Inc. is not to be confused with another ByteDance product called Lemon8, the Instagram/Pinterest rival launched in the US last year, which some speculate is being pushed in the market as a “back-up” in the event of a potential TikTok ban.
As reported by Axios last year, Lemon8 was developed by Singapore-based Heliophilia Pte., Ltd., which is registered at the same address as TikTok in Singapore.
Why Lemon Inc. is being used as a vehicle to apply for trademarks and patents in the United States on behalf of ByteDance is a question for another day.
For now, feast your eyes on what else ByteDance has been up to…
Lemon Inc. PAtent 1: ‘Automatic and fast generation of music audio content for videos’
In what offers the clearest glimpse at what ByteDance might be planning to use its AI music-related tech for, its affiliate Lemon Inc owns a patent in the United States for a “method of automatically and efficiently generating music audio content for videos”.
The patent application was filed in July 2022, granted in September 2023, and is assigned to both Lemon Inc. and ByteDance (see below).
According to the filing, which you can read for yourself here, this patent covers “techniques that enable efficient smart music generation and professional music editing”.
“After a content creator creates a video, music or audio that matches the transitions, intensity, and motion in the video will be automatically generated. Such music or audio may be royalty-free and can be of any length.”
Excerpt from patent
The filing adds: “After a content creator creates a video, music or audio that matches the transitions, intensity, and motion in the video will be automatically generated. Such music or audio may be royalty-free and can be of any length.”
The filing explains further that the tech can extract “information from the video, wherein the extracted information comprises information indicating motion speed in the video, information indicating motion saliency in the video, information indicating scene transition in the video, and timing information associated with the video”.
It can then generate a “plurality of sets of music notes matching the plurality of segments of frames based at least in part on the extracted information using a model, wherein the model is pre-trained and learns to correlate video motion speed with music note density, correlate video motion saliency with music note strength, correlate video scene transition with music structure, and correlate video timing with music beat”.
Just to further highlight the close links between Lemon Inc, TikTok and ByteDance, here:
Amongst the named inventors of this tech in the patent filing are Los Angeles-based researchers named Zhihao Ouyang and Daiyu Zhang, who according to LinkedIn are a Senior Research Engineer at TikTok, and ML Research Engineer at ByteDance, respectively.
Lemon Inc Patent 2: Computing system and method for music generation
Lemon Inc also has another patent pending for a “Computing system and method for music generation”.
This patent was filed by Lemon Inc USA in June 2022 and assigned to both Lemon Inc and ByteDance in September 2023 (see below). It was published in December.
The filing explains that “Programs have been developed that can generate music based on a lyric inputted by a user” but that the music generated by such programs “often lacks musical qualities that many people appreciate, and thus isn’t very song-like”.
It continues: “For example, auto-generated music from such programs can suffer from misalignments in lyrics and melody notes, scattered or disjointed organization and song structure, mismatched rhythm tracks, and lack of a catchy repeating melody.
“As a result, such programs have not achieved widespread use. As a result, a barrier presently exists to rapid song development using such programs.”
The patent application, which you can see here, describes “a music generation system” that features “a processor and a memory operatively coupled to the processor and storing a rhythm template database comprising a plurality of rhythm templates”.
The invention also comprises “a music generation program stored in the memory and executed by the processor to be configured to receive a user input of lyrics, identify a plurality of syllables in the lyrics, determine a syllable pattern in the identified plurality of syllables, match the syllable pattern to a selected rhythm template of the plurality of rhythm templates”.
The invention is then proposed to “generate a melody based on the selected rhythm template, generate a music file encoding the melody and the lyrics, and output the music file encoding the melody and the lyrics”.
The diagram below sets out how it works:
Lemon Inc.’s Trademarks
In addition to its active and pending patents in the field of AI Music, Lemon Inc. has also filed for multiple trademarks in the United States that refer to AI, music generation and music tech in general.
One of those trademarks listed on the United States Patent and Trademark Office (USPTO) is for a service called ‘BytePlus‘ which appears to have been officially registered in May 2023.
TikTok and Lemon Inc parent company ByteDance launched BytePlus in 2021 reportedly as a new division to sell its AI technology. ‘BytePlus’ is listed in ByteDance’s corporate structure diagram.
The trademark listed on the USPTO describes a “Platform as a Service (PaaS)” and covers what the filing states are “computer software platforms for generating and delivering personalized recommendations to customers in the fields of artificial intelligence, data mining, data analytics, and machine learning covers a range of AI-related activities.
The trademark also covers “machine-learning based music generation, and machine language translation; software consultancy services relating to artificial intelligence, data mining, data analytics, and machine learning”.
Elsewhere on the USPTO database, a trademark is registered to Lemon Inc. for Mawf, ByteDance’s machine-learning-driven music-making app that analyzes incoming audio signals and then “re-renders” those signals using what it says is machine-learned models of musical instruments.
The Mawf trademark was applied for in July 2022 and officially registered by the USPTO in September.
The filing covers “Computer software, namely, downloadable software for creating music videos, recorded music composition software”.
Lemon Inc. has also filed (in May 2023) two applications to trademark logos for what the filling appears to describe as a music-making app (see below). These logos resemble the logos of ByteDance’s Ripple app.
The application described in the pair of filings, which appear to be under review currently, cover “Downloadable software for processing music and audio files; Downloadable music-composition software; Downloadable software for creating, recording, and editing audio to create music and audio; downloadable software for creating music and audio; Downloadable mobile application software enabling users to share music and audio they created for other users to view, access, and remix within the application; Downloadable software for use in music and audio production“.
Both trademarks for different color variations of the logo appear to have been filed in April and May 2023 and have been active as of January this year.
Lemon Inc. also filed a trademark application for ‘Ripple‘ in February 2023 and the mark was only just published on the USPO’s Trademark Official Gazette last month.
Ripple is the AI-powered music-making app launched by ByteDance in June last year that can turn a hummed melody into a song.
Interestingly, Lemon Inc also filed a trademark application back in 2021 for SoundOn, which is the DIY music distribution service launched by TikTok in March 2022.
SoundOn lets artists upload their music directly to TikTok and Resso, and it can also distribute artists’ music to platforms like Spotify and Apple Music.
And just in case you’re still not convinced about Lemon Inc’s association with TikTok parent ByteDance, Lemon Inc. applied for a trademark for “ByteDance” in January.
MBW’s reporting on ByteDance’s and TikTok’s AI-related music activities in the United States arrives against a backdrop of two massive ongoing stories involving the company and its subsidiary TikTok.
The first story is a looming TikTok ban in the United States, due to the potential forced sale of its US operations via a bill passed through the House of Representatives last week, which is now one large step closer to becoming law.
The other story that we’ve been following closely is TikTok’s falling out with Universal Music Group, which refuses to renew its licensing deal with TikTok for a number of reasons, including the app’s compensation for artists and songwriters, and UMG’s concerns about TikTok’s use of AI.
On March 1, Universal Music Publishing’s catalog of ~4 million songs became unlicensed for use on TikTok, joining UMG’s portfolio of ~3 million recordings, whose license on TikTok expired (so far without renewal) on February 1.
In a statement issued to UMPG’s songwriters on February 29, the company turned much of its attention to the role AI-generated audio is playing on TikTok.
UMPG claimed that, so far, TikTok has not provided Universal with any assurances that the platform won’t train its AI models on the music company’s songs.
MBW has been writing for some time now about the hypothetical potential for TikTok and other services to fill their catalogs with music generated by AI, which would, in the process, dilute the market share of traditional rightsholders.
In February last year, we published an ‘MBW Reacts’ article asking if TikTok could pull off a “heist” on the music industry in this regard, following its aggressive investment in generative AI technology.
The “heist” we were referring to: Using licensed music as a cornerstone in the rise of TikTok to well over a billion users globally, before using first-party, AI-created songs to crowd out music owned by traditional music rightsholders on the platform.
We wrote: “With music playing such a key role in TikTok’s rise, if major label content does disappear from the platform – and the gap is somehow successfully filled by indie and AI-driven creations – TikTok could be said to have pulled off one of the biggest heists in music business history. A bait and switch for a billion users.”Music Business Worldwide
