Technical assignment (TA) for the development of a 3D avatar of a real person for real-time events
1. Project goal
Creation of a high-quality 3D avatar of a real person with the most realistic facial expressions and animations for use in real-time at events such as interviews, online broadcasts, webinars, and virtual conferences.
2. Main requirements
2.1. General characteristics
Avatar type: Fully digital 3D avatar based on the appearance of a specific person (photos, videos, 3D scanning).
Realism: High level of detail for the face, skin textures, hair, eyes, teeth, and clothing.
Usage format: Real-time rendering for streaming broadcasts and interactive events.
Platforms: Compatibility with popular streaming services (OBS, Zoom, Twitch, YouTube) and game engines (Unreal Engine, Unity).
2.2. Facial expression requirements
Emotion accuracy: The avatar must display the person's facial expressions in real-time using face tracking technology.
Technologies: Use of camera-based systems (e.g., webcam-based tracking) or specialized devices (e.g., iPhone TrueDepth, VR headsets with tracking).
Detailing: Realistic movement of lips, eyes, eyebrows, wrinkles, and other facial muscles according to emotions (joy, surprise, sadness, etc.).
Synchronization: Delay of no more than 50 ms between the user's facial movement and the avatar's reaction.
2.3. Animation requirements
Body movements: Natural animation of the upper body (arms, shoulders, head) with the ability to customize gestures.
Physics: Realistic behavior of hair, clothing, and accessories (e.g., earrings) during movement.
Voice synchronization: Lip sync in real-time based on audio input from the microphone.
2.4. Technical requirements
Resolution: Minimum 1080p at 60 FPS for avatar rendering.
Optimization: Operation on mid-high range PCs (e.g., with a graphics card level of NVIDIA RTX 3060 and above).
Integration: API or plugin for connection to streaming software.
Output formats: Ability to export to FBX, VRM, or other popular formats for further refinement.
2.5. Data source
Source materials: High-quality photos (at least 50 shots from different angles), videos (4K preferred), optionally — 3D scanning of the face and body.
Tracking: Use of available user devices (webcam, smartphone with ARKit, or specialized trackers).
3. Development stages
Data collection (1-2 weeks):
Photoshoot/video recording/3D scanning of the person.
Agreement on appearance (clothing, hairstyle, accessories).
Modeling and texturing (3-4 weeks):
Creation of a 3D model of the face and body in programs (Blender, ZBrush, Substance Painter).
Application of realistic textures (skin, hair, clothing).
Rigging and animation (2-3 weeks):
Setting up the skeleton and control points for facial expressions and movements.
Integration with the face and body capture system.
Integration with real-time engine (2-3 weeks):
Transfer of the model to Unreal Engine/Unity.
Setting up shaders, lighting, and physics.
Testing and optimization (1-2 weeks):
Checking performance in real conditions (streams, interviews).
Bug fixing, performance improvement.
Final assembly and documentation (1 week):
Delivery of the finished solution to the client with usage instructions.
4. Tools and technologies used
Modeling: Blender, ZBrush, Maya.
Texturing: Substance Painter, Photoshop.
Engine: Unreal Engine 5 or Unity (with ray tracing support for realism).
Face tracking: ARKit (iPhone), VSeeFace, OpenFace, or custom solutions based on ML.
Physics: NVIDIA PhysX or built-in engine tools.
Programming: C++, Blueprints (Unreal), C# (Unity).
5. Additional wishes
Ability to change clothing and background in real-time.
Support for multiple languages for lip sync (optional).
Easy interface for controlling the avatar (e.g., manual selection of emotions).
6. Result
Ready 3D avatar with real-time functionality.
Software for management and integration.
Documentation and user instructions.
Support and updates (by agreement).
If you have any additional wishes or clarifications (e.g., specific avatar style or budget), let me know, and I will adjust the TA!
