Budget: 800 UAH Deadline: 3 days
need to try. It should work out,
my linux notes
http://blog.wispenergo.pp.ua/linux/
Now I am running this command manually, it needs to be automated so that downloading 100 domains simultaneously can be started with one script.
wget --mirror --page-requisites --no-check-certificate --backup-converted --execute robots=off -w 5 --random-wait --compression=auto --reject-regex "/search|/rss" --no-if-modified-since --user-agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36" https://strapi.io/
Write 3 bash scripts. The first script creates the second script. The second script is run with the payload.
First script:
1) There is a txt file containing a list of domains.
2) You need to take the domains from the txt file and create a second script that will run the payload – simultaneous downloading of 100 domains from the txt file.
3) There is a folder /xxx where we will save the local copy of the downloaded sites.
4) The second script must create a folder for each domain in the /xxx folder with the name of the site, for example, /xxx/site1.com /xxx/site2.com and upload a copy of the corresponding site into this folder.
5) In the /xxx/site1.com folder there can be several copies of the downloaded site. When the script is run, a folder is created with the start time of the download and everything is poured into it.
/xxx/site1.com / 01-08-2024-19-15
/xxx/site1.com / 01-09-2024-19-20
In the /xxx/site1.com / 01-08-2024-19-15 folder we save the log of the download execution and after the download, we close the terminal window.
I assume that a separate terminal window will be opened for each domain (this is negotiable) and after the download is completed, the window closes. The main thing is to be able to see in real time that the download is in progress and everything is okay.
It is also assumed that a third script will be created, which will run the second (the second scripts with the payload) script in turn through cron. For example, there will be 10 scripts with payloads, each of which starts downloading 100 sites, so the third script starts downloading the first 100 sites, then the next 100 sites (the second script) and so on, so that the server constantly updates copies of 1,000 sites at a leisurely pace. The third script controls that there are three backups in the folder; when a fourth backup is created, the oldest one is deleted and the three most recent backups remain.
Please suggest deadlines and price.
Thanks
Budget: 800 UAH Deadline: 3 days
need to try. It should work out,
my linux notes
http://blog.wispenergo.pp.ua/linux/
Budget: 1200 UAH Deadline: 1 day
Good day! I have some experience in this field, so I can complete the task you set quickly, and most importantly - with quality! Feel free to reach out!
Budget: 1000 UAH Deadline: 1 day
Good day!
I am ready to discuss the execution of your contract.
Write in private messages.
а можно одним скриптом, которому отдаётся параметрами файл с сайтами и "параллельность" ?
тогда скрипт просто читает файл и скачивает все сайты в нужные папки но но более заданого часла сайтов одновременно.