Video Avatar based on RAG + D-ID + Chat GPT

X Avatar — a site where the user interacts with video avatar X (its digital copy). The avatar responds with voice and video (via D-ID), uses AI for RAG responses (via ChatGPT API), and connects to a knowledge base compiled from client materials + some external sources.

It is extremely important to understand your budget range and timelines: from – to + how you envision the project implementation.

Main areas of work:

VIDEOPLAYER – For our task, we need to develop a full-fledged custom player that can combine everything together. All the functions that we will need.

RAG – We need to train the avatar from two sources. The first source is our public source via API. The second source is some websites from which we need to extract information using Perplexity.

D-ID - This tool gives us a ready-made avatar. That is, we will already have a ready avatar, where a separate task is performed, and this avatar can be connected to. The creation of the Avatar itself, lip synchronization during responses, and so on, will all be implemented and provided via API.

Experience is highly desirable:

Building RAG on some RAG frameworks

Working with video players

THE AVATAR ITSELF WILL BE BUILT ON THE BASIS OF D–ID

Main components of the project X Avatar

1️⃣ Project CRM system

It is necessary to develop a simple but functional CRM through which all aspects of the project will be managed:

• connection settings (API, databases, external services),

• training parameters,

• role management,

• content moderation and manual data addition.

The CRM will become the central control panel of the system — the point of control and configuration of all X Avatar modules.

2️⃣ Avatar intelligence — RAG system (Retrieval-Augmented Generation)

The main function of the project — the intelligent core of the avatar.
The RAG system, built on Python using the LangChain framework, combines the client's personal knowledge base with the ability for dynamic search and response generation.

It makes the avatar's intelligence "alive": the system not only stores texts but also understands the context and forms personalized responses in the style of Neil's thinking and speech.

3️⃣ Custom video player and media widgets

Our own video player is a key element of the project's realism.
It provides:

• video synchronization with voice and emotions;

• background and reaction changes;

• adding jokes, photos, and videos next to responses.

The player is created specifically for the project so that the "live" avatar looks natural and can dynamically respond during the conversation.

4️⃣ Basic (fundamental) RAG training

At this level, the foundation of intelligence is formed:

• instructions,

• behavior rules,

• response templates,

• basic structure of the knowledge base.

This is the foundation for all subsequent stages of personalization and self-learning.

5️⃣ Self-learning RAG from Malini CMS materials

The system automatically receives new texts and articles from Malini CMS.
The materials undergo conversion and vectorization, after which they are added to the knowledge base.
Thus, the avatar is constantly updated without manual intervention, maintaining the relevance and coherence of the data.

6️⃣ Advanced RAG training from external sources (Google Gemini + Perplexity)

The RAG system is supplemented by connections to external sources.
Through Perplexity, the avatar gains access to fresh information — news, weather, sports results, publications from NASA, etc.
This makes the avatar's intelligence not only personalized but also relevant in real-time.

7️⃣ Manual training through CMS

The user can manually train the avatar by adding new texts, notes, or articles directly from the CMS interface.
The system includes a GPT assistant that helps structure and improve the material before training, preserving the author's style.