Need a specialist in Big Data, ML, NLP, RAG

1. Goal

Create a tool for analyzing large arrays of text data (correspondence on a dating site) with the aim of identifying key psychological characteristics of clients.

2. Brief description of the task

• There are a couple of hundred large files (several hundred pages in each file, possibly up to tens of thousands of messages per page), containing client correspondence.

• It is necessary to analyze this data in terms of a number of psychological indicators.

• The RAG (Retrieval-Augmented Generation) approach and vector databases are planned to be used for effective search and analysis.

3. Requirements for the specialist

1. Big Data: understanding of working with large volumes of data (Spark/Hadoop or analogs).

2. NLP / ML:

   • Experience in text processing (preprocessing, tokenization, cleaning) and applying ML models.

   • Knowledge of modern text analysis methods (sentiment, topic modeling, classification).

3. Vector Databases / RAG:

   • Practical experience with Pinecone, Milvus, Weaviate (or analogs) and embedding models.

   • Ability to build a Retrieval-Augmented Generation pipeline (embedding generation, similarity search, LLM integration).

4. Additionally:

   • Ability to document solutions, explain tool choices.

   • DevOps skills (Docker/Kubernetes) are welcome.

4. Expected tasks

1. Data preparation: reading, cleaning, structuring correspondence.

2. Embedding generation: model tuning (BERT, Sentence-BERT, OpenAI Embeddings, etc.).

3. Using vector DB: loading embeddings, searching for relevant fragments.

4. RAG analytics: integration with a language model to extract key characteristics of client behavior.

5. Assessment of psychological criteria: tuning/creating models that allow highlighting the necessary aspects. (there is a ready-made list of criteria by which clients will need to be assessed)

5. Key skills and technologies

• Language: Python (pandas, scikit-learn, PySpark, HuggingFace Transformers or others).

• Infrastructure: knowledge of cloud services (AWS, GCP, Azure or others) or local Big Data clusters.

• Vector databases: Pinecone, Milvus, Weaviate or FAISS or others.

• NLP libraries: spaCy, NLTK, as well as tools for lemmatization and cleaning.

• ML pipeline: knowledge of MLOps tools (Airflow, MLflow, Docker or others).