Over the past years, we have tried our best to keep you updated on AI-enhanced tools and developments relevant to our industry. And boy, did we receive backlash and get criticized for it! However, we still believe that filmmakers should keep up with technology updates and at least become familiar with what it is capable of nowadays. So, here it is, your short recap “Previously on AI!”

The era of AI video generators

The beginning of the year was marked with one announcement after another. Following Runway’s lead, companies (from small startups to tech giants) started to present their own text-to-video generators. The first “big news” came from Midjourney, an already well-known AI for image creation (however, their promised video generation tool still hasn’t been released). Then, there was Google with Lumiere (which became Google’s Veo later in the year), OpenAI with Sora, which caused a lot of buzz in February (and was released to the general public only a few weeks ago), Luma AI’s Dream Machine, and the list can go on and on.

It seems no company with its own AI research can stand aside and restrain from participating in this generative AI race. Thus, even Amazon recently presented the so-called Nova, a variety of generation models, including Nova Canvas for images and Nova Reels for videos.

With the ability to process text, image, and video as prompts, customers can use Amazon Nova-powered generative AI applications to understand videos, charts, and documents, or generate videos and other multimedia content.

A quote from the anouncement

According to Amazon’s announcement, you can see a “Pasta City” above. This is a short video advertisement created by Amazon Ads using the freshly launched text-to-video generator. With this example, the company aims to showcase innovative approaches for advertisers. To be honest, it seems scary. I can vividly imagine how product pages on Amazon will soon be drowned in artificial content (we will talk about it below).

AI-enhanced software for film production

However controversial the topic of generative AI may be, its development does bring some useful AI tools for filmmakers. Last year, we highly praised AI language understanding for bringing automated transcripts and captions into the editing software. This year, the developers have taken it a step further. They started to augment applications, tailored specifically for filmmakers, with AI-enhanced functions.

One such highlight of 2024 has been PRODUCER – Maker Machina, an all-in-one software that tackles shoot production topics with automated solutions. For instance, once you have your storyboard filled in with reference pictures and important notes on shots, the application will automatically create a shot list for each day based on the information that you provided. After you connect the actors and add your crew members for this project, it will generate a decent call sheet. The main idea behind such tools is the following: if we use AI to take over mundane, repetitive tasks, we gain more time for creative processes, which is not a bad idea per se.

Another example for digging deeper into the AI nerve is the shift by companies previously focused on a single niche to now incorporating more automated tools and features for filmmakers. For instance, Artlist. We all know it as a stock library with music and video clips. In 2024, they announced an AI-powered search (which was somewhat to be expected). But then, Artlist also presented an AI text-to-speech generator, expanding their offer. My assumption is that stock companies try to keep up and rapidly think of new tools because they are also afraid of generative AI’s capabilities and quick development. After all, this past year literally buzzed with new music generators that can create coherent songs (even with lyrics!) that won’t replace original composers (in my opinion) but can surely replace stock melodies over time.

Multi-modal emerging AI tools for filmmakers

A further trend we’ve observed developing in the generative AI tools for filmmakers in 2024 is multi-modality. Nowadays, it’s not only about which software can generate a better image, a more realistic voiceover, or a less troubled video clip. It’s also about combining those for the filmmaker’s convenience to achieve a faster result. The most prominent example from the past year is LTX Studio. This generative AI software is designed to handle everything from storyboarding and visualizing your concept to generating music and voice-overs. Our Johnnie met with the Lightricks company’s CEO and Co-Founder, Zeev Farbman. They discussed not only the software’s potential but also the negative impact it may have on video creators:

Announced play’s Holly is also an example of multi-modality but aimed at video editors. Their software, which is still in the testing phase, can create a rough cut based on your text prompt. For that, the tool uses machine learning to analyze delivered video clips. Afterward, the neural network provides them with detailed metadata and transcribes speech (understanding almost 100 languages). Sure, many editors don’t want anyone (or anything, for that matter) to take over their rough cut. Yet these functions alone in one software are pretty useful, aren’t they?

Photorealism and abundance

A trend that we can’t look past is that companies developing and training generative AI try to achieve results that are as close to reality as possible. Meaning: images that look like photos, video clips that could have been shot on a smartphone, and voices that can trick you into thinking they are real people. All that brings along a troublesome tendency that, more often than not, makes it hard to distinguish real content from generated. That’s another reason why we should educate ourselves and the people around us on AI. Knowing their capabilities can amplify critical thinking and remind us to surf more cautiously. Don’t believe everything you see, read, or hear. This has always been an important tip for Internet users, but now more than ever.

At the same time, we notice (as you probably do as well) that social media is flooded with AI-generated content. There is so much of it that it no longer inspires or excites. People are already fed up with neural network creations, and this trend will worsen. Thus, human-made art with unique ideas and insights starts to become scarce. That’s why I believe it will stay valuable, even though it is often more expensive and time-consuming.

Massive backlash towards AI tools for filmmakers

Last but not least, this year, we’ve seen how filmmakers react to AI and its development. People are scared of losing their jobs to the machines. They are furious that deep-learning models scrap the Internet and use their art for training without any residuals or attributions. On top of that, they are angry at huge corporations and companies that prefer neural networks to humans and at governments for the lack of restrictions and regulations. There are a lot of unresolved ethical questions and, above all – a massive fear behind all the criticism and hate that we get from our articles explaining AI tools.

We don’t say that this fear is unreasonable. Neither do we say that there are no consequences of generative AI development for filmmakers (Johnnie published a comprehensive guide on the options that we have nowadays and which may help to overcome the general panic –please read it). What we are saying is that the point of no return has already been reached. We can’t stick our heads in the sand, pretending that the generative AI revolution hasn’t happened. Instead, we should gain knowledge of it and implement the tools that will help us augment and speed up our processes but not replace our creativity. And develop our craft even more. Now, anyone can tip a few words into a text field and generate a video clip. But how many of these people understand visual storytelling, camera language, the power of rhythm in editing, the added level of narration in the score, and the importance of the smallest nuances in the voice and acting? A good question, indeed.

What about you? What trends in AI tools for filmmakers have you noticed over the past year? How do you react to this development? Share your thoughts with us in the comments below!

Feature image: generated with Midjourney for CineD.

The rapid development of artificial intelligence for video continues to keep us busy. On the one hand, it has the potential to make our professional lives a bit easier, but at the same time, we are forced to deal with the “unknown.” Questions like: Will some of our colleagues lose their jobs? Or will we see business opportunities thanks to such a development?

Given that we are in the early stages of this next revolution, we caught up with Zeev Farbman, the Co-founder and CEO of Lightricks, the company behind some popular apps like Facetune and Videoleap.

In the above video, we had the opportunity to conduct a friendly conversation with Zeev, ask about his career, and learn more about the development of LTX Studio, a product that has the potential to become a leader in the field of AI for video applications. This Generative AI software is designed to handle everything from storyboarding and visualizing your concept to generating music and voice-overs. But maybe two of the most significant features are “continuity,” meaning the ability to create repetitive visual results in order to build a sequence, next to mimic facial expressions! Some implemented AI models are licensed, while others were developed in-house.

Zeev was transparent in answering many of our questions, from the potential of losing jobs to how they train their application.

If you find the subject of AI interesting to explore, I recommend that you watch this interview. It will give you a unique opportunity to understand where we are headed and, not less importantly, to understand the point of view of a leading figure in the AI industry.

Are you planning to explore tools like LTX Studio and see what they can do? Does AI for video, in general, interest you? Please share your thoughts with us in the comment section below.

The justified buzz around OpenAI’s SORA still lingers as Lightricks ups the game with their revolutionary new tool – LTX Studio. While SORA seems to be more realistic and visually refined at first glance, LTX Studio possesses a much more significant trait. It gives the user unprecedented control over the final video.

LTX Studio solves one of the more prominent problems of AI-based content creation – selective control. At its core, the creative process follows the basic rule of AI generators: Text to video. You simply type a text specifying your creative vision and the machine spews out a video. Most generators won’t go much further. You’ll be able to repost the prompt and the AI will generate a new video, but precise or selective control is a pretty rare feature. At this point LTX Studio parts away from their peers, letting you specifically select, edit, or add multiple items.

Firestarter for each visual process

So you have your general idea for your next blockbuster. Once you type in some relevant keywords, the LTX Studio will turn them into a rudimentary script, and once this script is fed into the AI, it will generate a video based on it. This part alone can speed up every ideation or inspirational process, but it’s what comes next that makes this system unique because, unlike most current AI generators, you’ll get more than a video. You’ll get footage to work with.

As you can see in this early workspace screenshot, the video is split into cuts with a specific synopsis for each. These cuts may be switched, replaced, cut, or expanded (by generating continuous “footage”).

Complete editing suite

LTX Studio provides you with a complete editing suite. Music, VFX, voice-over, cuts, and transitions will all be available to fine-tune in the final video. This feature (and others – so keep reading) makes the LTX Studio potentially revolutionary, and sets it aside from the AI crowd. But there’s more.

Character consistency

Character consistency stands as one of the major hurdles in the Turing tests of our time. AI generators seem to struggle with this kind of continuity, creating ever so slightly different character renderings that are highly apparent to the human eye. LTX Studio claims to have this issue covered. The system allows for the generation of distinct characters that encompass both appearance and personality traits, aiming to emulate characters that feel more relatable and “authentic.”

3D perception

The LTX Studio cam also “perceives” three-dimensional objects. While not a 3D application per se, this perception allows for some interesting features. The user will be able to control camera movement and angles, characters or object movement, and lighting positions. This also allows for different styling, atmosphere, ambient lighting conditions, and other modifications while keeping the main characters unchanged.

Casting

Just as we can change the ambient atmosphere, every character can be changed, edited, and fine-tuned to perfectly fit the creative vision. You can have the exact same scene and then switch the character’s hairdo, clothes, or makeup. But why stop there? Ethnicities, gender, height, or species – all are subject to your will.

It seems like the possibilities are limitless, which they probably are. We are, of course, far from covering every one of them in this article, and as always with AI, it’s just the beginning. However, as powerful as this new tool is (and IT IS), there still are some shortcomings.

Still missing

As of now, the video renders coming out of the LTX Studio bear the visual signature associated with AI-generated content. They don’t look as (scaringly) refined as SORA’s output, but will definitely hold in various use cases. Furthermore, it seems the AI can’t quite generate speech. It does away with voice-over, but I haven’t seen a character talk with lip-sync and vocal audio to match. There are also some issues with hands and with “unsynced” motion (when, for example, a man moves faster than his feet, “sliding” on the pavement, etc.). As testing ensues, we’ll probably spot more shortcomings, but those are vastly outweighed by the disruptive nature of this product.

What does the future hold?

At this point, I wouldn’t dare to prophesize. Last month brought such exorbitant advancement to this field, and the same goes for the last couple of years. Perplexing times indeed. I’ll wait to see if and how Lightricks incorporates this powerful tool into other existing (and future) tools. Also – we’ll see if the company chooses to work with authentication initiatives such as C2PA. Imagine editing your first feature film or independent documentary with these kinds of tools. Imagine readjusting the camera or lighting in post, or testing different styles and vibes on the same scene. Imagine adding those few extra frames based on your own footage so it perfectly aligns with the music. Imagine adapting the same commercial video to different audiences at the click of a button. What a time to be alive. LTX Studio has a waiting list open for requests.

Do you believe tools like this will take over visual motion content and filmmaking in years to come? How are you preparing yourself for the foreseeable future in this regard? Let us know in the comments.