I am working on the app. The backend already has a built flow in N8N. This flow is responsible for parsing data from several government websites with open information for the public. Parsing is done through APIs that these sites officially provide. The automation task is to upload the entire database to Supabase and then only update it. The problem with the flow is pagination. The site returns 500 records via API. It stops there. Iteration tools did not help. We need to solve the problem of uploading the database from the sites.

This flow is only part of the chat agent of this application. The second part is the actual Agent that will communicate with the user. It will initially analyze the request, search for answers in the database (downloaded in advance), and if the user's request goes beyond the context, it will call the Gemini API. This agent model is designed to save on requests to Gemini.

It is necessary to help solve the problem of uploading data from both sites with all the information.

If this problem is solved, it will be possible to discuss building a chat agent in N8N.

The information is located on Polish websites. Translation is provided. There is a technical analysis of the API documentation of these sites.

If the backend work is done excellently, it will be possible to discuss the frontend.

Update #1 from 22 July 2025

What I tried to do to solve the "500 ceiling" issue of records:

Added pagination
→ Passed page and limit=500 parameters in the URL to navigate through pages.
Created a loop that automatically increments the page (page + 1)
→ Used a Code node + IF to check if there are still new records (items.length > 0).
Merged data from multiple pages
→ To combine everything into a single list for further processing.
Split records into batches (Split in Batches) of 25 items
→ To avoid overloading Supabase during mass uploads.
Added a pause (Wait) between requests
→ To prevent blocking or timeout issues.
Attempted to send data to Supabase via HTTP POST
→ But often received empty objects or data simply did not appear in the Success branch, although a test row appeared in Supabase.

Proposals 7 Rejected 2 Discussions 3

Andrey K.

1 298 1

Budget: 550 PLN Deadline: 7 days

Hello. I have experience with various APIs. Ready to help. Contact me.

Dmitro Nesterov

0 0

Projects -
Rating -
Rating 274

Budget: 777 PLN Deadline: 7 days

Good day. I work a lot with n8n and often have to solve similar tasks. Including data collection for creating an RAG system. I have also encountered pagination issues, so I can help with both solving it and further developing the assistant. I can also do the front end. Feel free to contact me, I will be glad to cooperate!

Andrii Prokopiev

18 0

Budget: 500 PLN Deadline: 1 day

Hello!

I can help figure out the problem.
When is convenient to look?

Zahar H.

13 1

Budget: 501 PLN Deadline: 7 days

Hello, ready to help with your problem.

If needed, I will write the necessary code for optimizing the existing one.

Also, I can help with the next stages without any problem and can immediately suggest abandoning the scpabeis and have proposals for a better vector database.

The list does not show proposals concealed by the client or freelancer with a Plus profile, as well as proposals violating rules

Yaroslav Stopin

7 0

Budget: 1800 PLN Deadline: 10 days

Good day.
Ready to take on your project.
I can create or improve your automations on n8n.
Write in private, we will discuss all the details and can start implementation.

Joseph Collins Mahihu

0 0

Projects -
Rating -
Rating 421

Budget: 585 PLN Deadline: 10 days

Hi ,

I can resolve your pagination issue in N8N to fully extract data from the Polish government APIs and streamline the Supabase upload process. Additionally, I’d be happy to collaborate on the chat agent integration (Gemini API fallback) once the backend is robust. Here’s my proposed plan:

Solution Breakdown
1. Pagination Fix for Government APIs
Problem: Current flow stops at 500 records due to pagination limits.

Fix:

Implement recursive API calls in N8N using loops or custom JavaScript.

Handle rate limits/retries to ensure complete data extraction.

Optimize Supabase batch uploads to avoid timeouts.

2. Chat Agent MVP in N8N
Workflow:

User Request Analysis: Pre-process queries to search local Supabase DB.

Gemini API Fallback: Trigger only for out-of-context questions (cost-saving).

Response Formatting: Clean output for frontend integration.

Features:

Context-aware routing (DB vs. Gemini).

Error handling for API failures.

3. Future Frontend Prep
Ensure backend outputs structured data (JSON) for easy frontend consumption.

Current freelance projects in the category Bot Development

Signal bot for trading

Python 67 proposals 31 July

67 USD
Creation of an OLX bot

58 proposals 30 July

Not specified
Create an SEO system based on n8n

AI & Machine Learning 58 proposals 30 July

Not specified
Парсинг маркетплейсу

Data Parsing 37 proposals 30 July

38 USD
Post reposting bot for Telegram

90 proposals 28 July

69 USD

Maximilian D
Warszawa, Poland

Projects 31
Contests 1
Rating 4.9
Rating 1 098

Maximilian D

Dmitro V.

Andrey K.

Dmitro Nesterov

Andrii Prokopiev

Zahar H.

Proposals are currently absent

Yaroslav Stopin

Joseph Collins Mahihu

Current freelance projects in the category Bot Development

Signal bot for trading

Creation of an OLX bot

Create an SEO system based on n8n

Парсинг маркетплейсу

Post reposting bot for Telegram