How to recursively Download A Large Quantity of Videos From YouTube

Generic Terminal Commands Across Multiple Linux Distributions
Post Reply
User avatar
Administrator
Site Admin
Posts: 1623
Joined: Thu Mar 22, 2018 10:19 am

How to recursively Download A Large Quantity of Videos From YouTube

Post by Administrator » Tue Sep 18, 2018 5:05 pm

Ok you need to download a ridiculously large amount of videos from YouTube

First make sure you have the latest version of youtube-dl, currently package managers are out of date.
also install aria2 to speed things up a little

Make a sub folder named
dictionaries
to hold your dictionaries and create a process shell script for each using the following code as a template

process_1.sh

Code: Select all

#!/bin/sh

filename='dictionaries/1.txt'
echo Start
while read p; do 
    youtube-dl --all-subs --write-all-thumbnails --write-info-json --write-annotations --convert-subs vtt --yes-playlist -w -i -f "(webm)[height>2000]"+251 --external-downloader aria2c --external-downloader-args "-x 5 -j 5 --user-agent='Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:61.0) Gecko/20100101 Firefox/61.0' " -o '%(playlist)s/%(id)s/%(id)s.%(ext)s' https://www.youtube.com/results?search_query="$p"+creative+commons+4k+uhd
done < $filename
This code will loop through all the single words in the dictionaries file and prefix it to a search query on youtube.com, youtube-dl will skip any video that don't match the settings and continue until it finds a match.

If you don't want to download with aria remove

Code: Select all

--external-downloader aria2c --external-downloader-args "-x 5 -j 5 --user-agent='Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:61.0) Gecko/20100101 Firefox/61.0' "
Note this will download 8k as-well depending on the available formats


Make sure your in the correct directory and type

Code: Select all

bash process_1.sh

User avatar
Administrator
Site Admin
Posts: 1623
Joined: Thu Mar 22, 2018 10:19 am

Re: How to recursively Download A Large Quantity of Videos From YouTube

Post by Administrator » Tue May 26, 2020 2:17 am

Only download Creative Common Videos from youtube using the best compression format of av1 and opus audio combined.

Code: Select all

#!/bin/sh

filename='dictionaries/1.txt'
echo Start
while read p; do 
    sudo ./youtube-dl --all-subs --write-all-thumbnails --write-info-json --write-annotations --match-filter "license='Creative Commons Attribution license (reuse allowed)'" --convert-subs vtt --yes-playlist -w -i -f 399+251 --external-downloader aria2c --external-downloader-args "-x 5 -j 5 --user-agent='Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:61.0) Gecko/20100101 Firefox/61.0' " -o '%(id)s/%(id)s.%(ext)s' https://www.youtube.com/results?search_query="$p"
done < $filename

User avatar
Administrator
Site Admin
Posts: 1623
Joined: Thu Mar 22, 2018 10:19 am

Re: How to recursively Download A Large Quantity of Videos From YouTube

Post by Administrator » Sat Jul 18, 2020 10:35 pm

Administrator wrote:
Tue May 26, 2020 2:17 am
Only download Creative Common Videos from youtube using the best compression format of av1 and opus audio combined.

Code: Select all

#!/bin/sh

filename='dictionaries/1.txt'
echo Start
while read p; do 
    sudo ./youtube-dl --all-subs --write-all-thumbnails --write-info-json --write-annotations --match-filter "license='Creative Commons Attribution license (reuse allowed)'" --convert-subs vtt --yes-playlist -w -i -f 399+251 --external-downloader aria2c --external-downloader-args "-x 5 -j 5 --user-agent='Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:61.0) Gecko/20100101 Firefox/61.0' " -o '%(id)s/%(id)s.%(ext)s' https://www.youtube.com/results?search_query="$p"
done < $filename
just tried this, and i does not currently work on youtube-dl

User avatar
Administrator
Site Admin
Posts: 1623
Joined: Thu Mar 22, 2018 10:19 am

Re: How to recursively Download A Large Quantity of Videos From YouTube

Post by Administrator » Sat Jul 18, 2020 11:24 pm

New code for av1 search

Code: Select all

#!/bin/sh

filename='dictionaries/1.txt'
echo Start
while read p; do 
    sudo ./youtube-dl --all-subs --write-all-thumbnails --write-info-json --write-annotations --match-filter "license='Creative Commons Attribution license (reuse allowed)'" --convert-subs vtt --yes-playlist -w -i -f 399+251 --external-downloader aria2c --external-downloader-args "-x 5 -j 5 --user-agent='Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:78.0) Gecko/20100101 Firefox/78.0' " -o '%(id)s/%(id)s.%(ext)s' --default-search="ytsearch" "$p"
done < $filename

Post Reply