it is necessary to organize a way to collect newly created individual entrepreneurs and, if possible, limited liability companies. whether it's parsing or something else - you tell me what this method should be
Proposals concealed
Proposals are currently absent
Proposals concealed
-
Igor Pavlov
18 September 2021
1. У инстаграмм стоит защита на парсинг веб-версии.
Ее нужно обходить через кучу аккаунтов + прокси.
2. Объем данных очень большой, а записывать и отдавать их надо быстро.
-
Igor Pavlov
18 September 2021
Добрый день, Сергей.
Конечная цель - сделать апи, с помощью которого можно будет поднимать десятки вот таких проектов как этот:
Он должен уметь налету вытягивать данные из instagram и представлять у себя на сайте
Бюджет аргументировано обсуждается
Для скрипта подойдет фришное решение, главное его настроить и поднять для работы
-
Nikita Korkin 18 September 2021Вам стоит значительно пересмотреть свой бюджет, ваш проект стоит на несколько порядков дороже
-
Igor Pavlov
18 September 2021
Бюджет не проблема, но хотел бы понять или, хотя-бы, увидеть аргументы в его пользу.
-
Oleg B. 18 September 2021100TB даних яб сказав що тут не одна база потрібна, а баз так з 40,на різних серверах.
на php можна тільки візуальну частину(взаємодію з користувачем) написати, не більше не менше.
Писати щось подібне тільки на php це ....
Тай сама інфраструктура(проксі,сервера) для цього всьо буде коштувати вдень наверно більше вашого бюджету.
-
Igor Pavlov
18 September 2021
40 баз зачем? Есть десяток сущностей, под каждую из них отдельная таблица. Под все одна база. Сперва ограничимся 1TB. Когда начнём превышать - задумаемся о кластерах. Даже с кластерами если не MySQL, то PostgreSQL потянет. Кластеризацию можно заложить, но думать о ней пока не стоит.
А вот по поводу «писать на php это…» - совсем не понял аргумент.
инфраструктура в 1к проксей и 1к аккаунтов в МЕСЯЦ обойдётся в 100 баксов, согласен, - это выше текущего бюджета, но это приемлемо. -
Oleg B. 18 September 2021Я раз робив виборку з бази яка займала 50 гігабайт, ця виборка йшла один час, і це був простий select запрос
-
Igor Pavlov
18 September 2021
Надо смотреть что это была за база и что за запрос. Но 50 ГБ для MySQL базы - это чепуха.
Current freelance projects in the category Data Parsing
Good morning,I will commission the creation of a stable and reliable solution for automatically monitoring prices and availability of products from a specific large Polish online store in the RTV/AGD sector.Project goal: The system must regularly check selected product links and, in case of a price change or availability status, send a notification (preferably via Telegram, Discord, or email – to be determined).Challenge: The store's website has very aggressive anti-bot systems (Akamai/Cloudflare). Ordinary scripts that fetch the page code are immediately detected and blocked. The solution must effectively bypass these protections, mask bot traffic, and operate stably over the long term, without constantly throwing "Access Denied" errors.What I expect from the contractor: Proposing a complete architecture and technology (I do not impose a programming language or specific tools). Delivering a ready-made solution that I can easily launch and manage. Suggestions regarding infrastructure (e.g., what proxies will be necessary for the system to operate smoothly).In response to the announcement, please briefly describe: What approach/technology do you propose to effectively and sustainably bypass the store's protections? How do you see the issue of sending notifications? What is the estimated cost of execution and time of completion?I invite you to submit your offers.
About the Company Trading company. We work with a product group of more than 2000 items across different categories.Current Situation Currently, the nomenclature is maintained in Google Sheets — data is consolidated by tabs (categories). Tab Structure: Product name Price groups: cost price, wholesale, retail Characteristics: weight, quantity per package, etc. Important: the number of columns varies for different product categories, as they have different characteristics.Why the Current Solution is Inadequate Google Sheets does not allow setting access rights at the level of individual columns. We need to: Grant users rights to view certain columns (for example, only cost prices) Grant rights to edit certain columns (for example, retail prices) While restricting access to other columns in the same tabWhat Needs to Be DoneMain Requirements Flexible Rights Management System Access at the level of individual columns (read/write) Assignment of rights by roles or users Management of rights without the involvement of programmers Support for Different Data Structures Different product categories have different sets of characteristics Adding new columns/characteristics without programming Independence from Developers Administration by internal staff Adding categories, columns, users — through the interface Integration with ERP Exporting current prices to our ERP system Export or automatic integration via API Data Analysis Using AI (preferably) Ability to analyze the entire nomenclature list Enrichment, verification, recommendations — if you have ideas, please describeExpected Result A working solution in which: The nomenclature is structured by categories with different sets of characteristics Column rights are flexibly configured (view/edit) Data is exported to ERP The team can manage the system independentlyWhat We Need from You When Responding Describe in general terms how you envision the solution: What tool/platform do you propose
Good evening, there is a task: you need to check the availability and relevance of the prices of products added to the website (the website is on the prom.ua platform) from the supplier's site 1-2 times a day. Can you calculate the price and how it will look?
Set up automatic daily updates of product availability on our website on prom.ua. We have a supplier who sends a price list of products in Excel format to our email every day. The items on our website and in the supplier's price list are the same. The values in the "stock" column are either out of stock, a number, or more than a box - these need to be updated on the site to either Ready for shipment or Out of stock. Items that are not in the supplier's price list should remain unchanged. Please propose a solution, timeline, and budget. Thank you in advance for your response, I look forward to collaborating with a specialist.