Create a program or script for Python / Notepad++ or another application for data filtering.
There are many csv files, each containing about 10-20 million lines.
Sublime Text cannot handle opening such a large number.
However, Notepad++ opens quickly.
It should also be able to open txt files.
I will consider your proposals, possibly a script for a similar program using Python.
Or a separate application.
The main task is for the application to be able to open such large files and work without hanging.
After the file is opened, I need the ability to filter data.
By city, age, date of birth, or other data that I specify.
Also with the indication of how many lines I need to collect, for example, 500.
The software should collect these 500 lines (there should be an option to specify collection in order or randomly) based on the necessary filters, and I should have the ability to save it in Excel or csv.
After I have filtered the lines, those that have been filtered should be marked, and not used in the next filtering.
However, if after filtering the software has collected an insufficient number of entries with the specified filters, it can offer me to use the previous lines that are already marked as used, with the option to specify how many to add to the file.
After that, it creates an Excel or csv file with the specified number of entries.
-
3 days90 USD
3 days90 USD
I have experience in similar Data Science projects where it is necessary to read and process millions of rows. I specialize in Python development and am ready to develop a program for you for reading + a GUI interface for filtering and displaying data. I can work with various file formats such as txt, csv, xlsx, json, or others.
I am ready to offer the following solution:
- a custom Python program for quickly reading csv or txt files using pandas/polars;
- we can consider options for changing file formats before processing to speed up reading or processing (to csv, or importing into a database);
- a convenient GUI interface through PyQT/tkinter with filtering and visualization of data;
- export to any format upon completion of processing;
Write to me, we can discuss any unclear points and start development!
-
3 days112 USD
3 days112 USD
Hello! This is a very interesting and classic task in the field of data processing. You are absolutely right: text editors, even powerful ones like Notepad++, are not designed for processing and filtering data. They can display a file by quickly reading it in parts, but any "Find" or "Filter" operation will cause it to freeze.
The problem you are facing is a lack of RAM (Random Access Memory). 20 million lines is too much to load them into memory all at once for analysis.
The solution I propose will not freeze and will handle this task.
-
1 day112 USD
1 day112 USD
I carefully read your technical specification and recommend that you transfer all data from the CSV files to the database because it will be the easiest way to do everything you need. Alternatively, you need to write Python scripts for this task.
It is also very difficult to assess the work without seeing the structure of the files. Is the structure the same for all or different? Can you send a couple of files for review?
I am ready to perform everything with quality.
Feel free to contact me.
-
3 days101 USD
1345 4 0 3 days101 USD
Great request — I have a ready technical approach that can handle 10–20 million rows without lag and provides fast filters + export.
What I will do
I will open large CSV/TXT files without lag: line-by-line/columnar loading and lazy reading (without fully reading into memory).
Instant filtering by: city, age, date of birth, any fields (multi-filters, AND/OR, contains/equals/ranges).
Selection of N rows (in order or randomly) — for example, exactly 500.
Export to CSV or Excel (XLSX).
Marking "used" rows: we do not take them again; if there are not enough, I will suggest "adding" from already used ones with an indication of how much to add.
Very fast viewing of large files with pagination (virtual table, non-blocking UI).
…
How it will work (under the hood)
Python + Polars/DuckDB for processing 10–20 million rows with streaming and predicate pushdown.
Desktop application (PySide6/Qt): table with virtual page loading, instant filters, saved presets.
Index of "used" rows — a separate lightweight database (SQLite) with keys by file and offsets/hashes: we do not rewrite the huge file, but remember what we have already given out.
Random selection — deterministic/truly random sampling (of choice).
TXT — parsed by delimiter or by pattern (I will set a default, can be changed in settings).
Advantages of my approach
Does not crash on files with 10–20 million rows (streaming/lazy loading).
Filters are very fast (reading only the necessary columns).
"Used" rows are marked without rewriting the source.
Ready for growth: later, indexes can be added, conversion to Parquet for ultra-speed.
What you will get
.exe/.app (portable build) + source files.
Simple interface:
1) Open file - 2) Select filters - 3) Specify "500, order/random" - 4) Save CSV/XLSX.
Action log and repeat last scenario launch.
A little about me
Python developer, I create fast data tools and bots. I have experience with Flask/integrations and processing large data.
Ready to start immediately. If you send a sample of a large CSV/TXT (or a column schema) — I will assemble a test demo build for your cases.
-
3 days101 USD
2264 18 3 3 days101 USD
Good day. I am engaged in development in Python. I can develop a program for you for convenient and fast processing of large CSV files, stack: Python + Pandas + SQLite + graphical interface through Streamlit (perfectly suited for data processing). Write to me - we will discuss the details and I will immediately start working.
-
3 days112 USD
316 1 0 3 days112 USD
Hello. I am ready to complete your project. Write to me and we will discuss the details.
-
12 days112 USD
82 12 days112 USD
Good day, it will be possible to create a small program with a graphical interface and the ability to select any file and filter it, I can arrange this for you in Python+PyQt5.
-
7 days112 USD
1337 25 0 7 days112 USD
I will do it in C++, it will work independently of Sublime, Notepad, or anything else. Contact me to discuss the details of the task.
-
3 days112 USD
3355 11 0 3 days112 USD
Hello 👋🏻
I am ready to develop a program in Python for processing large CSV and TXT files (up to tens of millions of rows) without freezing.
The script will open files line by line, filter data based on necessary criteria (city, age, date of birth, etc.), consider row limits (for example, 500), mark already used records, and suggest additional entries if there is a shortage.
Results are saved in Excel or CSV, with the possibility of re-filtering without overlaps.
I use streaming processing and memory optimization (pandas/polars, chunksize) to ensure everything works quickly even on a weak PC.
The deadline is 3–5 days, and the cost is 5,000 UAH.
I am ready to discuss the details and show a demo example. 💻
-
5 days112 USD
357 5 days112 USD
Hello! I am ready to implement your project in Python.
I can create an application that opens large CSV/TXT files (10–20 million rows) without freezing, supports filtering by city, age, date of birth, etc., with the option to select the number of rows (in order or randomly).
I will also implement marking of used rows, reusing when data is insufficient, and saving in CSV or Excel.
I would like to clarify the details — file format, data examples, and desired interface.
-
2 days27 USD
690 5 0 2 days27 USD
Hello, Igor!
I would like to ask you a couple of questions that interest me for working with such a project brief. It would also be good to see an approximate structure of the files, to hear what types of files you need, and so on.
I will be waiting!
-
1 day45 USD
656 9 0 1 day45 USD
Good evening, Igor!
In general, the task is clear, but for an accurate response regarding deadlines and price, I would like to clarify some questions that arose after analyzing your task.
Please write in private messages — we will discuss the details and your wishes.
-
1 day90 USD
172 1 1 1 day90 USD
Hello! I am ready to complete this project and have extensive experience in developing various applications.
-
30 days67 USD
691 8 0 30 days67 USD
I can write in Python or Bash, possibly with a web interface,
many years of experience in processing text files and others,
-
2 days17 USD
5911 345 0 2 days17 USD
I will consider an example of the file, I will propose a specific software implementation.
-
2 days34 USD
6366 74 1 2 days34 USD
Good day. I have extensive experience with such tasks. I will create filters and save in any format. I need a more detailed technical specification and a sample of the data. I will be happy to help.
-
1 day22 USD
573 5 0 1 day22 USD
Hello, I worked on the "Folder" Project: processed CSV files with over 15 million rows, optimized filtering and export to XLSX.
How do you want to prioritize processing the rows: in order or selectively, to ensure speed and convenience?
I suggest we get in touch, I will provide you with a free consultation on the technical side and we will create a development plan + I will tell you about my team!
-
1 day22 USD
1595 7 0 1 day22 USD
My name is Roman, and I am in the top 5 developers in the category of "Artificial Intelligence and Machine Learning" among ~1600 specialists on the platform. I guarantee: - Fast and quality execution of the task - Clear adherence to deadlines - Regular communication throughout the entire process I would be happy to discuss the details of your project in private messages.
-
Добрый день, а взять что-то готовое? Может гит уже может что-то предложить?
-
Доброго дня!
Якщо я правильно зрозумів, то Вам необхідна утиліта для швидкого опрацювання великих файлів таблиць csv з функціонаом фільтрації, редагування та експорту в Excel/csv?
Задачка цікава, але хотілося б поглянути на файл прикладу, щоб зрозуміти деякі технічні нюанси... -
Можно увидеть образец файла для понимания? Выполнить не проблема, нужно более детальное описание задачи.
-
Current freelance projects in the category Databases & SQL
Accounting, planning, and sales system for a mushroom farm
607 USD
Here is the complete, final text of the Technical Assignment (TA). It combines all your requirements: 16 chambers, 20 contractors, a schedule by days, accounting for containers, profitability calculation, and a mandatory division into three grades of mushrooms. You can fully… Databases & SQL, Client Management & CRM ∙ 22 hours 51 minutes back ∙ 45 proposals |
External report 1C 8.3 — forecast of goods balances
22 USD
An external report (.erf) is needed for 1C:Enterprise 8.3 (configuration to be specified). What it should do: Extract product balances from the database Analyze sales history for the last 30 days Calculate the average sales rate for each product Determine how many days until the… Databases & SQL, Client Management & CRM ∙ 23 hours 27 minutes back ∙ 10 proposals |
Web Application & Database Security Audit for Custom CRM — BaaS / Database-as-API Specialist (Penetr
Project Overview We operate a custom-built customer relationship management (CRM) platform that runs two service businesses on a single system. It is a modern JavaScript web application backed by a backend-as-a-service (BaaS) database and deployed on a serverless hosting… Databases & SQL, Testing & QA ∙ 1 day 11 hours back ∙ 9 proposals |
Database synchronization
Synchronization of Microsoft Access programs and CRM SalesDrive. Data transfer from CRM to Microsoft Access in the first stage (changing the funnel status). Data transfer from Microsoft Access to CRM in the second stage (changing the status in the program). Databases & SQL ∙ 1 day 17 hours back ∙ 10 proposals |
Setting up a backup system and optimizing server infrastructure
Objective of the work: Ensure reliable data storage for the CRM system and application by implementing an automated backup system, as well as carry out a series of server improvements to enhance the stability, security, and performance of the infrastructure. DevOps, Databases & SQL ∙ 2 days 16 hours back ∙ 23 proposals |
Kyiv