Introduction

With the rise of generative AI in recent times, its applications have expanded far beyond generating simple text or solving complex mathematical problems. Today, generative AI is emerging as a powerful creative force, especially in the film and music industries. In recent years, AI’s ability to generate entirely new content — whether it’s visuals, music, or even full films — has accelerated, sparking new levels of creativity. From crafting hyper-realistic visual effects in blockbuster movies to composing original musical scores, generative AI is reshaping our understanding of creativity and redefining artistic expression. In this article, we will explore some recent examples of how people are creatively harnessing generative AI.

Generative AI in the Film Industry

1. Generative AI in Visual Effects: Avatar: The Way of Water

James Cameron’s Avatar: The Way of Water (2022) is a landmark example of how generative AI can transform the visual effects (VFX) process. For the film’s underwater sequences, generative AI was used to simulate realistic environments and characters’ fluid movements. Instead of manually animating every detail, AI was trained on real-world water physics and marine life, allowing it to generate highly realistic underwater scenes autonomously. This significantly reduced the time required for VFX production while ensuring a level of visual complexity that was previously impossible.

2. AI-Generated Storyboarding

A more recent innovation is AI-assisted storyboarding with tools like StoryboardHero AI and StoryboardThat. These tools allow filmmakers to input text-based scripts, which the AI interprets to generate visual storyboards. By leveraging generative models trained on extensive datasets of films and scenes, these tools create visuals for each part of the script, giving directors a quick and efficient way to conceptualise their films. This technology has proven particularly beneficial for independent filmmakers, enabling them to visualise their projects without the need for large teams.

3. GANs for Background Generation in The Mandalorian

In 2023, The Mandalorian leveraged generative adversarial networks (GANs) to generate realistic environments. Using GANs, filmmakers could create entire alien landscapes and intricate backgrounds on demand. These AI systems were trained to produce photorealistic visuals that blend seamlessly with live-action footage. This technique allowed the production team to cut costs while still maintaining high-quality visual effects, showcasing the cost-saving potential of generative AI in major film productions.

Generative AI in the Music Industry

1. Holly+ : AI-Generated Vocal Performances

In 2022, experimental musician Holly Herndon introduced Holly+, an AI-powered vocal clone. Holly+ is a generative AI that can take input melodies or lyrics and generate completely new vocal performances in Herndon’s signature style. This project opened up collaborative possibilities between AI and musicians, allowing other creators to experiment with her AI-generated voice. Holly+ represents a shift toward generative AI enabling artists to extend their creative reach — Herndon can “perform” music without physically being involved in every recording.

2. AI-Generated Music: OpenAI’s Jukebox

Jukebox, OpenAI’s generative AI model, took centre stage in 2023 for generating complete songs in various musical genres and styles. This deep-learning model was trained on thousands of tracks, enabling it to generate entirely new compositions with lyrics, melodies, and harmonies. In 2023, Jukebox was used in several independent and experimental music projects to produce AI-generated tracks that mimic the styles of famous artists. Unlike traditional music tools, Jukebox doesn’t just produce loops or fragments — it generates full-length, coherent songs, showcasing the evolving power of generative AI in music creation.

3. Endlesss for Real-Time Generative Music

Endlesss, a real-time generative music app, gained traction in 2022 as a platform that allows users to collaborate with AI in generating music on the fly. By utilising generative AI, the app can automatically generate beats, melodies, and loops that musicians can then manipulate or combine with their own input. This interactive use of AI helps musicians break through creative blocks and experiment with sounds they may not have thought to create themselves. The integration of AI-generated ideas in real-time collaboration has shifted how artists approach music production, blending machine learning and human creativity seamlessly.

4. AI-Generated Music Videos with GANs: Piknic Electronik

In 2022, the Piknic Electronik Festival collaborated with AI platforms like Runway to create entirely AI-generated music videos using GANs (Generative Adversarial Networks). These GANs were trained to transform sound waves into dynamic and visually engaging digital art that matched the electronic music played at the festival. The videos, generated on the fly, showcased how AI can autonomously create not just music but also synchronised visual elements, bringing a new dimension to the festival experience.

AI-Generated Pop Stars and Virtual Musicians

1. AI-Generated Virtual Pop Star: FN Meka

FN Meka, the first fully AI-generated virtual pop star, made waves in 2022 for his AI-generated music and online presence. His songs, lyrics, and even social media posts were produced using generative models. FN Meka operates as a digital persona, with his creators using AI to generate music based on audience feedback and preferences, reflecting a new direction in music marketing. Though he sparked controversy regarding authenticity and cultural appropriation, FN Meka represents how AI is being used to generate entire musical careers without a human performer.

2. AI-Generated Collaboration Albums: Aiva in Classical Music

Aiva (Artificial Intelligence Virtual Artist) is a generative AI that specialises in composing classical music. In 2022, Aiva was used to co-create albums with human artists by generating classical compositions based on specific emotional themes. These collaborations between human musicians and Aiva blurred the lines between AI-generated and human-created art, opening new doors for symphonic and orchestral music. As the AI learned from thousands of classical pieces, its ability to generate novel yet familiar music highlighted the potential for AI to play a substantial role in the future of composition.

Generative AI in Film Scoring and Sound Design

In The Batman (2022), generative AI played a role in creating parts of the film’s atmospheric and experimental soundtrack. AI-driven tools like Amper Music were used to generate musical compositions that blended dark, orchestral tones with eerie, electronic sounds. While human composers still oversaw the final arrangement, the generative AI provided quick iterations of thematic music, allowing for rapid experimentation. This use of generative AI sped up the composition process and enabled composers to try more variations than traditional workflows would allow.

The Downsides of Generative AI in the Film and Music Industries

While generative AI offers immense creative possibilities, it also presents several challenges and downsides in the film and music industries. One significant concern is the potential for job displacement, particularly for artists, animators, and musicians whose skills might be replaced or diminished by AI-driven tools. The use of AI in creative decision-making raises questions about authenticity and originality, as AI-generated content may lack the emotional depth and nuance typically brought by human creators. Moreover, ethical issues around intellectual property arise when AI models are trained on vast datasets without consent from the original creators, potentially leading to copyright disputes. As generative AI continues to evolve, striking a balance between its creative benefits and the preservation of human artistry will be essential.

Conclusion

Generative AI is undeniably transforming the film and music industries, offering groundbreaking tools that enhance creativity, streamline production processes, and open up new possibilities for artists and filmmakers. From generating hyper-realistic visual effects to composing entire musical pieces, AI is redefining the boundaries of artistic expression. However, with these advancements come challenges, including concerns over job displacement, authenticity, and intellectual property rights arises. As generative AI continues to evolve, the industry must navigate these complexities carefully, ensuring that human creativity and emotional depth remain at the heart of artistic endeavours. Balancing the immense potential of AI with ethical considerations will be key to shaping a future where technology and human artistry coexist and thrive.

References

• Marr, Bernard. “Generative AI Is Revolutionizing Music: The Vision for Democratizing Creation.” Forbes, 5 Oct. 2023, www.forbes.com/sites/bernardmarr/2023/10/05/generative-ai-is-revolutionizing-music-loudlys-vision-for-democratizing-creation/.
• Davenport, Thomas, and Randy Bean. “The Impact of Generative AI on Hollywood and Entertainment.” MIT Sloan Management Review, 19 June 2023, sloanreview.mit.edu/article/the-impact-of-generative-ai-on-hollywood-and-entertainment/.