GazeGen: Gaze-Driven User Interaction for Visual Content Generation
He-Yen
Abstract
We present GazeGen, a user interaction system that generates visual content (images and videos) for locations indicated by the user's eye gaze. GazeGen allows intuitive manipulation of visual content by targeting regions of interest with gaze. Using advanced techniques in object detection and generative AI, GazeGen performs gaze-controlled image adding/deleting, repositioning, and surface material changes of image objects, and converts static images into videos. Central to GazeGen is the DFT Gaze (Distilled and Fine-Tuned Gaze) agent, an ultra-lightweight model with only 281K parameters, performing accurate real-time gaze predictions tailored to individual users' eyes on small edge devices. GazeGen is the first system to combine visual content generation with real-time gaze estimation, made possible exclusively by DFT Gaze. This real-time gaze estimation enables various visual content generation tasks, all controlled by the user's gaze. The input for DFT Gaze is the user's eye images, while the inputs for visual content generation are the user's view and the predicted gaze point from DFT Gaze. To achieve efficient gaze predictions, we derive the small model from a large model (10x larger) via novel knowledge distillation and personal adaptation techniques. We integrate knowledge distillation with a masked autoencoder, developing a compact yet powerful gaze estimation model. This model is further fine-tuned with Adapters, enabling highly accurate and personalized gaze predictions with minimal user input. DFT Gaze ensures low-latency and precise gaze tracking, supporting a wide range of gaze-driven tasks. We validate the performance of DFT Gaze on AEA and OpenEDS2020 benchmarks, demonstrating low angular gaze error and low latency on the edge device (Raspberry Pi 4). Furthermore, we describe applications of GazeGen, illustrating its versatility and effectiveness in various usage scenarios.
Community
We propose GazeGen, a system that lets users generate visual content simply by looking, using gaze as a natural, hands-free control method.
- Gaze-Based Image Editing: With GazeGen, users can edit images by focusing on specific parts of the image, allowing:
- Adding objects.
- Replacing or repositioning objects.
- Applying an object’s material to another object the user focuses on.
- Creating Videos with Gaze: Users can create videos by looking at specific areas, allowing actions such as:
- Adding animated objects where they’re looking.
- Replacing static objects with animated ones.
Real-Time Gaze Prediction: Our lightweight gaze estimation model, DFT Gaze (around 300KB), provides real-time gaze estimation with just 360ms latency on edge devices, like a Raspberry Pi 4.
Object Recognition with Gaze: GazeGen recognizes object categories based on where the user looks, allowing object-specific interactions.
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- DreamVideo-2: Zero-Shot Subject-Driven Video Customization with Precise Motion Control (2024)
- VideoGLaMM: A Large Multimodal Model for Pixel-Level Visual Grounding in Videos (2024)
- Upper-Body Pose-based Gaze Estimation for Privacy-Preserving 3D Gaze Target Detection (2024)
- MotionCom: Automatic and Motion-Aware Image Composition with LLM and Video Diffusion Prior (2024)
- Zero-Shot Pupil Segmentation with SAM 2: A Case Study of Over 14 Million Images (2024)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment:
@librarian-bot
recommend
Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper