A Groundbreaking Advancement in Video Technology, LUMIERE

   On January 23, Google unveiled ‘LUMIERE,’ a video-generated AI model on GitHub. LUMIERE is a space-time diffusion model for video generation. It was designed as a comprehensive video that describes practical, various, and consistent motion. As LUMIERE evaluated advanced performance by comparison with public video-generated AI, it attracts academia and industry.

▲ Examples of CinemaGraphs (Photo from arXiv)

   LUMIERE utilizes the 'Space-Time U-Net architecture,' distinguishing itself from existing video-generated AI. While traditional methods pre-create frames for the beginning and ending scenes, filling in the middle scenes afterward, LUMIERE generates the entire video simultaneously by processing both temporal and spatial aspects of objects. As a result, it can reduce video-generated errors and create natural video.

▲ Video Inpainting Function (Photo from GitHub LUMIERE)

   LUMIERE provides various functions such as CinemaGraphs, Inpainting, Text-to-Video, and Image-to-Video. CinemaGraphs is a function that converts only the user-provided part of the image into a video. It animates elements within the image, such as making a butterfly sitting on a flower flip its wings or creating choppy waves on a calm lake. Inpainting is a function that reconstructs a damaged part in a video or edits a localized part according to the text prompt. LUMIERE can seamlessly insert the missing half of the pizza to complete a perfect circle in a video where half of the pizza is obscured. Also, it can edit the owl’s head to wear a crown or sunglasses with just a single prompt. Text-to-Video generates videos from the input text, while Image-to-Video generates videos from both images and text prompts. Based on a study conducted by Google Research, users exhibit a preference for these two functions over other AI video models such as Pika, Stable Video Diffusion (SVD), and AnimateDiff due to their superior quality.

▲ Users Preference for Text-to-Video and Image-to-Video (Photo from arXiv)

   While LUMIERE presents a limitation in its ability to edit or generate complex videos involving scene conversions, its potential for major advancements in AI video models is undeniable. Nonetheless, caution must be exercised in the development of this technology to mitigate the risk of misuse, particularly in the creation of deceptive or harmful content.