Lumiere is a cutting-edge AI-based text-to-video generator created by a group of Google Research AI experts. An article detailing the group’s efforts can be found on the arXiv preprint server.
Recent years have seen a shift in the deployment of artificial intelligence applications from research labs to the general public. LLMs like ChatGPT, for instance, have been connected with browsers to let people create text in previously unheard-of ways.
More recently, users have been able to produce surreal pictures with text-to-image generators. Additionally, people can now create brief video clips with just a few lines by employing text-to-video generators. With the launch of Lumiere, a text-to-video generator, the Google team has elevated this final category to new heights in their latest endeavour.
Lumiere, which takes its name from the Lumiere brothers, who invented the first photographic apparatus, lets users enter a text like “two raccoons reading books together” and receive a finished film in incredibly high definition that shows the two raccoons doing just that. With significantly better-looking outcomes, the new generator marks a step forward in the development of text-to-video generators.
The new generator’s technology is characterized by Google as a “groundbreaking Space-Time U-Net architecture.” Its goal was to produce animated videos using just one model pass.
In the demo video, it is demonstrated that Google has introduced more functionality. For example, users can alter an existing film by marking a section and inputting commands, like “change dress colour to red.” Additionally, the generator generates a variety of outputs, including stylizations, which create a subject’s style rather than a full-color portrayal. Substyles, such as various style references, are also permitted. It also does cinemagraphics, which allow the user to animate a still image by highlighting a portion or all of it.
Google did not say in its announcement if Lumiere would be released or distributed to the general public, most likely because of the apparent legal issues that could result from the possible production of videos.