Write a Bash script
Now I am running this command manually, it needs to be automated so that downloading 100 domains simultaneously can be started with one script.
wget --mirror --page-requisites --no-check-certificate --backup-converted --execute robots=off -w 5 --random-wait --compression=auto --reject-regex "/search|/rss" --no-if-modified-since --user-agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36" https://strapi.io/
Write 3 bash scripts. The first script creates the second script. The second script is run with the payload.
First script:
1) There is a txt file containing a list of domains.
2) You need to take the domains from the txt file and create a second script that will run the payload – simultaneous downloading of 100 domains from the txt file.
3) There is a folder /xxx where we will save the local copy of the downloaded sites.
4) The second script must create a folder for each domain in the /xxx folder with the name of the site, for example, /xxx/site1.com /xxx/site2.com and upload a copy of the corresponding site into this folder.
5) In the /xxx/site1.com folder there can be several copies of the downloaded site. When the script is run, a folder is created with the start time of the download and everything is poured into it.
/xxx/site1.com / 01-08-2024-19-15
/xxx/site1.com / 01-09-2024-19-20
In the /xxx/site1.com / 01-08-2024-19-15 folder we save the log of the download execution and after the download, we close the terminal window.
I assume that a separate terminal window will be opened for each domain (this is negotiable) and after the download is completed, the window closes. The main thing is to be able to see in real time that the download is in progress and everything is okay.
It is also assumed that a third script will be created, which will run the second (the second scripts with the payload) script in turn through cron. For example, there will be 10 scripts with payloads, each of which starts downloading 100 sites, so the third script starts downloading the first 100 sites, then the next 100 sites (the second script) and so on, so that the server constantly updates copies of 1,000 sites at a leisurely pace. The third script controls that there are three backups in the folder; when a fourth backup is created, the oldest one is deleted and the three most recent backups remain.
Please suggest deadlines and price.
Thanks
-
691 8 0 need to try. It should work out,
my linux notes
http://blog.wispenergo.pp.ua/linux/
-
3012 73 4 2 Good day! I have some experience in this field, so I can complete the task you set quickly, and most importantly - with quality! Feel free to reach out!
-
455 4 0 Good day!
I am ready to discuss the execution of your contract.
Write in private messages.
-
а можно одним скриптом, которому отдаётся параметрами файл с сайтами и "параллельность" ?
тогда скрипт просто читает файл и скачивает все сайты в нужные папки но но более заданого часла сайтов одновременно. -
Current freelance projects in the category Linux & Unix
Deployment of a Ruby on Rails project on VPS + Inbound Email RoutingNeed help deploying a platform for automating requests for public information (based on the open-source engine Alaveteli). The project works as a mail router: it generates unique email addresses for each request, sends them to government agencies, and receives responses back to… DevOps, Linux & Unix ∙ 1 day 18 hours back ∙ 10 proposals |
Pentest, penetration testing
25 USD
Project Description: Our company is looking for an experienced freelance pentester to participate in information security projects for our clients. What needs to be done: • conduct external/internal penetration testing (VPN, firewall, web, API, DNS) • web application & API… Linux & Unix, Testing & QA ∙ 13 days 9 hours back ∙ 9 proposals |