How to recursively Download A Large Quantity of Videos From YouTube

[Administrator]

Ok you need to download a ridiculously large amount of videos from YouTube

First make sure you have the latest version of youtube-dl, currently package managers are out of date.
also install aria2 to speed things up a little

Make a sub folder named

dictionaries

to hold your dictionaries and create a process shell script for each using the following code as a template

process_1.sh

Code: Select all

#!/bin/sh

filename='dictionaries/1.txt'
echo Start
while read p; do 
    youtube-dl --all-subs --write-all-thumbnails --write-info-json --write-annotations --convert-subs vtt --yes-playlist -w -i -f "(webm)[height>2000]"+251 --external-downloader aria2c --external-downloader-args "-x 5 -j 5 --user-agent='Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:61.0) Gecko/20100101 Firefox/61.0' " -o '%(playlist)s/%(id)s/%(id)s.%(ext)s' https://www.youtube.com/results?search_query="$p"+creative+commons+4k+uhd
done < $filename

This code will loop through all the single words in the dictionaries file and prefix it to a search query on youtube.com, youtube-dl will skip any video that don't match the settings and continue until it finds a match.

If you don't want to download with aria remove

Code: Select all

--external-downloader aria2c --external-downloader-args "-x 5 -j 5 --user-agent='Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:61.0) Gecko/20100101 Firefox/61.0' "

Note this will download 8k as-well depending on the available formats


Make sure your in the correct directory and type

Code: Select all

bash process_1.sh

[Administrator]

Only download Creative Common Videos from youtube using the best compression format of av1 and opus audio combined.

Code: Select all

#!/bin/sh

filename='dictionaries/1.txt'
echo Start
while read p; do 
    sudo ./youtube-dl --all-subs --write-all-thumbnails --write-info-json --write-annotations --match-filter "license='Creative Commons Attribution license (reuse allowed)'" --convert-subs vtt --yes-playlist -w -i -f 399+251 --external-downloader aria2c --external-downloader-args "-x 5 -j 5 --user-agent='Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:61.0) Gecko/20100101 Firefox/61.0' " -o '%(id)s/%(id)s.%(ext)s' https://www.youtube.com/results?search_query="$p"
done < $filename

[Administrator]

Only download Creative Common Videos from youtube using the best compression format of av1 and opus audio combined.

Code: Select all

#!/bin/sh

filename='dictionaries/1.txt'
echo Start
while read p; do 
    sudo ./youtube-dl --all-subs --write-all-thumbnails --write-info-json --write-annotations --match-filter "license='Creative Commons Attribution license (reuse allowed)'" --convert-subs vtt --yes-playlist -w -i -f 399+251 --external-downloader aria2c --external-downloader-args "-x 5 -j 5 --user-agent='Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:61.0) Gecko/20100101 Firefox/61.0' " -o '%(id)s/%(id)s.%(ext)s' https://www.youtube.com/results?search_query="$p"
done < $filename

just tried this, and i does not currently work on youtube-dl

[Administrator]

New code for av1 search

Code: Select all

#!/bin/sh

filename='dictionaries/1.txt'
echo Start
while read p; do 
    sudo ./youtube-dl --all-subs --write-all-thumbnails --write-info-json --write-annotations --match-filter "license='Creative Commons Attribution license (reuse allowed)'" --convert-subs vtt --yes-playlist -w -i -f 399+251 --external-downloader aria2c --external-downloader-args "-x 5 -j 5 --user-agent='Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:78.0) Gecko/20100101 Firefox/78.0' " -o '%(id)s/%(id)s.%(ext)s' --default-search="ytsearch" "$p"
done < $filename