Automation in N8N
110 USD Hello!
I am working on the app. The backend already has a built flow in N8N. This flow is responsible for parsing data from several government websites with open information for the public. Parsing is done through APIs that these sites officially provide. The automation task is to upload the entire database to Supabase and then only update it. The problem with the flow is pagination. The site returns 500 records via API. It stops there. Iteration tools did not help. We need to solve the problem of uploading the database from the sites.
This flow is only part of the chat agent of this application. The second part is the actual Agent that will communicate with the user. It will initially analyze the request, search for answers in the database (downloaded in advance), and if the user's request goes beyond the context, it will call the Gemini API. This agent model is designed to save on requests to Gemini.
It is necessary to help solve the problem of uploading data from both sites with all the information.
If this problem is solved, it will be possible to discuss building a chat agent in N8N.
The information is located on Polish websites. Translation is provided. There is a technical analysis of the API documentation of these sites.
If the backend work is done excellently, it will be possible to discuss the frontend.
What I tried to do to solve the "500 ceiling" issue of records:
Added pagination
→ Passedpageandlimit=500parameters in the URL to navigate through pages.Created a loop that automatically increments the page (
page + 1)
→ Used a Code node + IF to check if there are still new records (items.length > 0).Merged data from multiple pages
→ To combine everything into a single list for further processing.Split records into batches (
Split in Batches) of 25 items
→ To avoid overloading Supabase during mass uploads.Added a pause (
Wait) between requests
→ To prevent blocking or timeout issues.Attempted to send data to Supabase via HTTP POST
→ But often received empty objects or data simply did not appear in the Success branch, although a test row appeared in Supabase.
Що я пробував зробити, щоб вирішити проблему "потолка 500" записів:
Додали пагінацію
→ Передавали в URL параметриpageіlimit=500, щоб переходити по сторінках.Зробили цикл, який сам збільшує сторінку (
page + 1)
→ Використали вузол Code + IF щоб перевіряти, чи ще є нові записи (items.length > 0).Merge даних з кількох сторінок
→ Щоб об'єднати все в один список для подальшої обробки.Розбили записи на порції (
Split in Batches) по 25 штук
→ Щоб не перевантажити Supabase при масовому завантаженні.Додавали паузу (
Wait) між запитами
→ Щоб не зловити блокування або таймаут.Пробували відправити дані в Supabase через HTTP POST
→ Але часто приходили пусті об'єкти (empty item) або дані просто не з'являлись у Success-гілці, хоча в Supabase тестовий рядок з'явився.
Client's review of cooperation with Dmitro V.
Automation in N8NExcellent work. Very satisfied. Roman - a professional in his field
Freelancer's review of cooperation with Maximilian D
Automation in N8NEasy and comfortable work. Everything is clear and understandable. Understanding and clear task setting.
-
7 days209 USD
226 7 days209 USDGood day. I work a lot with n8n and often have to solve similar tasks. Including data collection for creating an RAG system. I have also encountered pagination issues, so I can help with both solving it and further developing the assistant. I can also do the front end. Feel free to contact me, I will be glad to cooperate!
-
7 days148 USD
95670 1272 1 10 7 days148 USDHello. I have experience with various APIs. Ready to help. Contact me.
-
1 day135 USD
2848 17 0 1 1 day135 USDHello!
I can help figure out the problem.
When is convenient to look?
-
7 days135 USD
1322 13 1 7 days135 USDHello, ready to help with your problem.
If needed, I will write the necessary code for optimizing the existing one.
Also, I can help with the next stages without any problem and can immediately suggest abandoning the scpabeis and have proposals for a better vector database.
-
10 days484 USD
1315 7 0 10 days484 USDGood day.
Ready to take on your project.
I can create or improve your automations on n8n.
Write in private, we will discuss all the details and can start implementation.
-
10 days157 USD
421 10 days157 USDHi ,
I can resolve your pagination issue in N8N to fully extract data from the Polish government APIs and streamline the Supabase upload process. Additionally, I’d be happy to collaborate on the chat agent integration (Gemini API fallback) once the backend is robust. Here’s my proposed plan:
Solution Breakdown
1. Pagination Fix for Government APIs
Problem: Current flow stops at 500 records due to pagination limits.
Fix:
…
Implement recursive API calls in N8N using loops or custom JavaScript.
Handle rate limits/retries to ensure complete data extraction.
Optimize Supabase batch uploads to avoid timeouts.
2. Chat Agent MVP in N8N
Workflow:
User Request Analysis: Pre-process queries to search local Supabase DB.
Gemini API Fallback: Trigger only for out-of-context questions (cost-saving).
Response Formatting: Clean output for frontend integration.
Features:
Context-aware routing (DB vs. Gemini).
Error handling for API failures.
3. Future Frontend Prep
Ensure backend outputs structured data (JSON) for easy frontend consumption.
-
Ніхто не дає скачувати свої бази даних,для того встановлюють ліміти на сторінки
-
Я розумію. Для того можна скачувати поступово - лімітами
-
Ну да, для цього пишуть подібні алгоритми,але n8n кроме того що там є чат чпт, нічим особо від зено постера не відрізняється
-
Current freelance projects in the category Bot Development
Software maintenance
111 USD
It is necessary to make adjustments to the software and maintain it in the future; the software places orders based on specified parameters and sends them to a Telegram bot. Python, Bot Development ∙ 17 hours 35 minutes back ∙ 50 proposals |
Telegram bot
269 USD
It is necessary to completely replicate the interface and functionality, but without the payment platform. It needs to be done in a short time. Bot Development ∙ 1 day 15 hours back ∙ 107 proposals |
Automatic video posting on social media according to a scheduleTechnical Assignment (TA) Automatic video publication on social media according to a scheduleProject Description A script/bot (Python preferred, or any other solution of your choice — the main thing is stability and ease of maintenance) is needed, which automatically publishes… Python, Bot Development ∙ 1 day 17 hours back ∙ 55 proposals |
Bot for receiving/searching applications
28 USD
In short: Create a bot where users can create and find profiles, the bot should have referral links, etc. Everything is detailed below. Client-side bot: Before the /start button, text in the center of the chat: What can this bot do? Welcome to Sugar Secret Agency! -Premium… Python, Bot Development ∙ 1 day 19 hours back ∙ 60 proposals |
Signal bot with automation in Google SheetsA bot is needed that will accept an application and automatically place it in Google Sheets, for example: store1, 26.06, 500kg or 200kg, and it will automatically mark 500 or 200 in the cell of the date on the store's line in the table. Bot Development ∙ 1 day 19 hours back ∙ 62 proposals |
Warszawa