The modern world is saturated with all kinds of information, and in our difficult times, it's important to be able to not only find it but also to save it. Many have probably noticed that on YouTube, besides the junk, cats, and other useless things (which we sometimes don't mind watching), there is a lot of useful material on a wide variety of topics. And sometimes it would be nice to save this material for the future, so as not to depend on the changing moods in the world.
In this article, I want to explain how you can download videos, audio (Part 1 of the article), playlists, and entire channels from YouTube (Part 2 of the article) without using a VPN and in pure Python. A quick disclaimer: we won't need a VPN, but we will create our own tool that will solve the "problem with outdated and worn-out equipment Google Global Cache" (you know what I mean). I think this tool will be especially relevant today, when for many Russians, YouTube barely works or doesn't work at all.
Why reinvent the wheel, and why Python?
I program in Python, so what else would I talk about?
Python has extensive capabilities for creating such tools and ready-made libraries with great functionality.
There are excellent utilities like yt-dlp and youtube-dl, but they are suitable for when you need to quickly download something. Fine-tuning and downloading, for example, an entire channel, is quite a hassle (if not, correct me), as you constantly need to google things and know numerous flags and parameters.
There are also other desktop programs, mobile apps, and websites, but as a rule, they are either paid, packed with ads, limited in functionality, or all of the above.
And in my opinion, a custom script is always more convenient than any fancy utility, and the beautiful lines of code in the editor are a pleasure to look at.)
Step 1. Bypassing the block... oops!.. Solving the problem with outdated and worn-out equipment.
Why doesn't YouTube work for us as it should? Of course, everyone knows that it's not about the equipment, but about the fact that providers use such a malicious (in this case) thing as DPI (Deep Packet Inspection)
How it works
Most DPIs work like this. When you try to access a blocked website, the DPI sends you an HTTP 302 Redirect, and it does so faster than the website you are trying to reach. Unlike firewalls, Deep Packet Inspection analyzes not only packet headers but also the content of requests, which allows ISPs and government agencies to restrict access to prohibited resources, detect network intrusions, and stop the spread of computer viruses.
This is a rather simplified model of a passive DPI, but since the purpose of this article is not to provide a detailed introduction to this topic, and I must admit I'm not a great expert in it, I think this is enough to understand what is happening. If you are interested in this topic, you can read here and here.
What to do
But, for every problem, there's a solution, and there are ways to get around this thing (although they don't always work). Many have probably heard of GoodbyeDPI or zapret. These are great tools, but since we're talking about Python, let's make our own equivalent in Python. After a bit of searching, I found nodpi, which uses only Python.
How it works
nodpi creates a proxy server (socket) through which we route traffic. It randomly fragments all outgoing connections, thereby confusing the DPI.
Pros
It's written in Python!
Simple, cheap, effective, and works for most providers
Doesn't require administrator privileges, unlike GoodbyeDPI, for example
The proxy server can be configured only in certain applications, so you don't have to route all system traffic through it.
Cons
Simplicity doesn't always mean quality, so there are no guarantees
Works only for TCP
Won't help if the site is banned by IP
Writing... no - rewriting the code
The program is written quite carelessly, as if the developer's eyes were closing and they just uploaded the code as is, without figuring it out. Therefore, I will provide a cleaned-up version of the code and briefly explain what it does.
First, let's create a text file blacklist.txt and add the domains we want to bypass blocking for. For YouTube, these are:
youtube.com youtu.be yt.be googlevideo.com ytimg.com ggpht.com gvt1.com youtube-nocookie.com youtube-ui.l.google.com youtubeembeddedplayer.googleapis.com youtube.googleapis.com youtubei.googleapis.com yt-video-upload.l.google.com wide-youtube.l.google.com
Now let's create a file nodpi.py and add the following code to it
import random import asyncio BLOCKED = [line.rstrip().encode() for line in open('blacklist.txt', 'r', encoding='utf-8')] TASKS = []
What we are doing here:
Import the necessary libraries
Create a list of blocked domains
Create a list where we will store data transfer tasks between the client and server
async def main(host, port): server = await asyncio.start_server(new_conn, host, port) await server.serve_forever()
Create an entry point - an asynchronous function that starts a socket server on the specified host and port and use new_conn (below) to handle new connections.
async def pipe(reader, writer): while not reader.at_eof() and not writer.is_closing(): try: writer.write(await reader.read(1500)) await writer.drain() except: break writer.close()
Create an asynchronous function that reads data from the reader and writes it to the writer until the end is reached or the connection is closed.
async def new_conn(local_reader, local_writer): http_data = await local_reader.read(1500) try: type, target = http_data.split(b"\r\n")[0].split(b" ")[0:2] host, port = target.split(b":") except: local_writer.close() return if type != b"CONNECT": local_writer.close() return local_writer.write(b'HTTP/1.1 200 OK\n\n') await local_writer.drain() try: remote_reader, remote_writer = await asyncio.open_connection(host, port) except: local_writer.close() return if port == b'443': await fragemtn_data(local_reader, remote_writer) TASKS.append(asyncio.create_task(pipe(local_reader, remote_writer))) TASKS.append(asyncio.create_task(pipe(remote_reader, local_writer)))
Create an asynchronous function that handles a new connection.
Read the HTTP headers and extract the request type and target address.
If the request type is not CONNECT, close the connection.
Send an HTTP/1.1 200 OK response to the client.
Open a connection to the target server.
If the port is 443 (HTTPS port), call the function
fragemtn_data(below)Create tasks for transferring data between the client and the server.
async def fragemtn_data(local_reader, remote_writer): head = await local_reader.read(5) data = await local_reader.read(1500) parts = [] if all([data.find(site) == -1 for site in BLOCKED]): remote_writer.write(head + data) await remote_writer.drain() return while data: part_len = random.randint(1, len(data)) parts.append(bytes.fromhex("1603") + bytes([random.randint(0, 255)]) + int( part_len).to_bytes(2, byteorder='big') + data[0:part_len]) data = data[part_len:] remote_writer.write(b''.join(parts)) await remote_writer.drain()
Create an asynchronous function that fragments data before sending it to the server.
Read the header and data.
If the data is not from blocked domains, send it without modification.
Otherwise, split the data into random parts and add headers before sending. This helps bypass DPI blocking, as the fragmented data looks like random packets.
asyncio.run(main(host='127.0.0.1', port=8881))
Start the server on the local 127.0.0.1 and listen on port 8881
That's it, you can use it!
Let's check. If you have Firefox, go to Settings → Network Settings and enter the IP and port:
Open YouTube and make sure everything is working or not working :)
Step 2. Downloading!
Installing the necessary tools
First, let's install the pytubefix library with the command:
pip install pytubefix
A little about pytubefix
pytubefix is a powerful Python library for downloading videos, audio, playlists, and channels from YouTube with its own CLI. It is essentially a fork of pytube, but the last changes to pytube were made over a year ago, and in August of this year, after changes to the YouTube API, it stopped working. Also, pytubefix has fixed many of its predecessor's shortcomings and added new features. pytubefix, like pytube, works with the YouTube API, and in some places, it also parses its HTML/JSON.
We will also need ffmpeg to merge audio and video (if, of course, you want to download in high quality). Why? The fact is that YouTube provides video with audio in a single stream only at 360p resolution (it used to be 720p), and to download a video in, say, 1080p, you first need to download the 1080p video, then download the audio, and then merge it all with a converter. ffmpeg is also available for Windows. You can download a verified build from my Google Drive, but you can also find it yourself on the internet.
Simple video download
Let's try to download a video. Create a file yt_downloader.py and paste the code into it:
from pytubefix import YouTube from pytubefix.cli import on_progress url = "https://www.youtube.com/watch?v=xxxxxxxxxxx" video = YouTube( proxies={"http": "http://127.0.0.1:8881", "https": "http://127.0.0.1:8881"}, url=url, on_progress_callback=on_progress, ) print('Title:', video.title) stream = video.streams.get_highest_resolution() stream.download()
Let's break down what this code does. First, we import the YouTube class, which we use to download videos, and the function on_progress. It is needed to display a progress bar during the download. You'd agree it's much more convenient this way, right? If you want, you can write your own callback function. It must have the arguments (stream: Stream, chunk: bytes, bytes_remaining: int)
Into the variable url paste your download link, as I have replaced the video ID with X's. Next, we create an instance of the YouTube class and pass it a dictionary with our proxy server address, the URL, and the callback. All arguments except the URL are optional. It's also worth noting that the callback is called each time a new chunk is downloaded (pytubefix downloads videos in pieces - chunks).
Next, we print the video title. You can also find out the publication date, video length (in seconds), description, number of views, and author. The file size can be found using stream.filesize, as it depends on the selected stream.
In the line stream = video.streams.get_highest_resolution() we get a list of streams and sort it by video resolution (quality). Here I want to pause and explain in a bit more detail. Every video on YouTube has several dozen streams, each of which is a single video, a single audio, or both, in different formats, encodings, and qualities. For my video, the list of streams looks something like this:
Warning, long wall of text!
[ { "itag": 18, "mime_type": "video/mp4", "resolution": "360p", "vcodec": "avc1.42001E", "acodec": "mp4a.40.2", "progressive": True, "type": "video", }, { "itag": 315, "mime_type": "video/webm", "resolution": "2160p", "vcodec": "vp9", "acodec": None, "progressive": False, "type": "video", }, { "itag": 337, "mime_type": "video/webm", "resolution": "2160p", "vcodec": "vp9.2", "acodec": None, "progressive": False, "type": "video", }, { "itag": 701, "mime_type": "video/mp4", "resolution": "2160p", "vcodec": "av01.0.13M.10.0.110.09.18.09.0", "acodec": None, "progressive": False, "type": "video", }, { "itag": 308, "mime_type": "video/webm", "resolution": "1440p", "vcodec": "vp9", "acodec": None, "progressive": False, "type": "video", }, { "itag": 336, "mime_type": "video/webm", "resolution": "1440p", "vcodec": "vp9.2", "acodec": None, "progressive": False, "type": "video", }, { "itag": 700, "mime_type": "video/mp4", "resolution": "1440p", "vcodec": "av01.0.12M.10.0.110.09.18.09.0", "acodec": None, "progressive": False, "type": "video", }, { "itag": 299, "mime_type": "video/mp4", "resolution": "1080p", "vcodec": "avc1.64002a", "acodec": None, "progressive": False, "type": "video", }, { "itag": 303, "mime_type": "video/webm", "resolution": "1080p", "vcodec": "vp9", "acodec": None, "progressive": False, "type": "video", }, { "itag": 335, "mime_type": "video/webm", "resolution": "1080p", "vcodec": "vp9.2", "acodec": None, "progressive": False, "type": "video", }, { "itag": 699, "mime_type": "video/mp4", "resolution": "1080p", "vcodec": "av01.0.09M.10.0.110.09.18.09.0", "acodec": None, "progressive": False, "type": "video", }, { "itag": 298, "mime_type": "video/mp4", "resolution": "720p", "vcodec": "avc1.4d4020", "acodec": None, "progressive": False, "type": "video", }, { "itag": 302, "mime_type": "video/webm", "resolution": "720p", "vcodec": "vp9", "acodec": None, "progressive": False, "type": "video", }, { "itag": 334, "mime_type": "video/webm", "resolution": "720p", "vcodec": "vp9.2", "acodec": None, "progressive": False, "type": "video", }, { "itag": 698, "mime_type": "video/mp4", "resolution": "720p", "vcodec": "av01.0.08M.10.0.110.09.18.09.0", "acodec": None, "progressive": False, "type": "video", }, { "itag": 135, "mime_type": "video/mp4", "resolution": "480p", "vcodec": "avc1.4d401f", "acodec": None, "progressive": False, "type": "video", }, { "itag": 244, "mime_type": "video/webm", "resolution": "480p", "vcodec": "vp9", "acodec": None, "progressive": False, "type": "video", }, { "itag": 333, "mime_type": "video/webm", "resolution": "480p", "vcodec": "vp9.2", "acodec": None, "progressive": False, "type": "video", }, { "itag": 697, "mime_type": "video/mp4", "resolution": "480p", "vcodec": "av01.0.05M.10.0.110.09.18.09.0", "acodec": None, "progressive": False, "type": "video", }, { "itag": 134, "mime_type": "video/mp4", "resolution": "360p", "vcodec": "avc1.4d401e", "acodec": None, "progressive": False, "type": "video", }, { "itag": 243, "mime_type": "video/webm", "resolution": "360p", "vcodec": "vp9", "acodec": None, "progressive": False, "type": "video", }, { "itag": 332, "mime_type": "video/webm", "resolution": "360p", "vcodec": "vp9.2", "acodec": None, "progressive": False, "type": "video", }, { "itag": 696, "mime_type": "video/mp4", "resolution": "360p", "vcodec": "av01.0.04M.10.0.110.09.18.09.0", "acodec": None, "progressive": False, "type": "video", }, { "itag": 133, "mime_type": "video/mp4", "resolution": "240p", "vcodec": "avc1.4d4015", "acodec": None, "progressive": False, "type": "video", }, { "itag": 242, "mime_type": "video/webm", "resolution": "240p", "vcodec": "vp9", "acodec": None, "progressive": False, "type": "video", }, { "itag": 331, "mime_type": "video/webm", "resolution": "240p", "vcodec": "vp9.2", "acodec": None, "progressive": False, "type": "video", }, { "itag": 695, "mime_type": "video/mp4", "resolution": "240p", "vcodec": "av01.0.01M.10.0.110.09.18.09.0", "acodec": None, "progressive": False, "type": "video", }, { "itag": 160, "mime_type": "video/mp4", "resolution": "144p", "vcodec": "avc1.4d400c", "acodec": None, "progressive": False, "type": "video", }, { "itag": 278, "mime_type": "video/webm", "resolution": "144p", "vcodec": "vp9", "acodec": None, "progressive": False, "type": "video", }, { "itag": 330, "mime_type": "video/webm", "resolution": "144p", "vcodec": "vp9.2", "acodec": None, "progressive": False, "type": "video", }, { "itag": 694, "mime_type": "video/mp4", "resolution": "144p", "vcodec": "av01.0.00M.10.0.110.09.18.09.0", "acodec": None, "progressive": False, "type": "video", }, { "itag": 139, "mime_type": "audio/mp4", "resolution": None, "vcodec": None, "acodec": "mp4a.40.5", "progressive": False, "type": "audio", }, { "itag": 140, "mime_type": "audio/mp4", "resolution": None, "vcodec": None, "acodec": "mp4a.40.2", "progressive": False, "type": "audio", }, { "itag": 249, "mime_type": "audio/webm", "resolution": None, "vcodec": None, "acodec": "opus", "progressive": False, "type": "audio", }, { "itag": 250, "mime_type": "audio/webm", "resolution": None, "vcodec": None, "acodec": "opus", "progressive": False, "type": "audio", }, { "itag": 251, "mime_type": "audio/webm", "resolution": None, "vcodec": None, "acodec": "opus", "progressive": False, "type": "audio", }, ]
You can see the full list of streams for your video by simply printing print(video.streams), but this is usually not necessary, as the library provides tools for sorting and filtering them. In our case, it's get_highest_resolution(). This function returns the stream with the best resolution that contains both audio and video. As I've already explained, this resolution will be 360p, because YouTube provides video with audio in a single stream only at 360p resolution (it used to be 720p). To download a video in, say, 1080p, you first need to download the 1080p video, then download the audio, and then merge it all with a converter. We will write a script that downloads video in higher quality a little later.
And finally, in the line stream.download() we download the video. A progress bar will be displayed during the download, and you will see the download process.
Run the script and see the result. Don't forget to run our DPI bypass program before this!
Downloading video in high resolution
A 360p video is, of course, not great. Let's expand our program. If you haven't downloaded ffmpeg yet, do it. Without it, downloading in a different resolution is impossible. It is assumed that ffmpeg is in the same folder as the script.
Create a file yt_downloader_2.py and add the following code to it:
import os from pytubefix import YouTube from pytubefix.cli import on_progress url = "https://www.youtube.com/watch?v=xxxxxxxxxxx" def combine(audio: str, video: str, output: str) -> None: if os.path.exists(output): os.remove(output) code = os.system( f'.\\ffmpeg.exe -i "{video}" -i "{audio}" -c copy "{output}"') if code != 0: raise SystemError(code) def download(url: str): yt = YouTube( proxies={"http": "http://127.0.0.1:8881", "https": "http://127.0.0.1:8881"}, url=url, on_progress_callback=on_progress, ) video_stream = yt.streams.\ filter(type='video').\ order_by('resolution').\ desc().first() audio_stream = yt.streams.\ filter(mime_type='audio/mp4').\ order_by('filesize').\ desc().first() print('Information:') print("\tTitle:", yt.title) print("\tAuthor:", yt.author) print("\tDate:", yt.publish_date) print("\tResolution:", video_stream.resolution) print("\tViews:", yt.views) print("\tLength:", round(yt.length/60), "minutes") print("\tFilename of the video:", video_stream.default_filename) print("\tFilesize of the video:", round( video_stream.filesize / 1000000), "MB") print('Download video...') video_stream.download() print('\nDownload audio...') audio_stream.download() combine(audio_stream.default_filename, video_stream.default_filename, f'{yt.title}.mp4') download(url)
What has changed? We added the function combine, which is responsible for merging video and audio. We do this with the command .\ffmpeg.exe -i "filename_video" -i "filename_audio" -c copy "output_filename"
In the main code, we extract the video and audio streams separately. Here, the video does not contain audio, but it is in high quality. To get it, we first filter the streams by type (video) filter(type='video'), then sort by resolution, then sort by descending resolution and take the first stream. The same goes for audio. After that, we print detailed information about the video being downloaded and download the audio and video separately, after which we merge them using ffmpeg. Nothing complicated!
Run our DPI bypass script and the download script (replace the URL with your own). Done!
How to get a list of available resolutions
To find out in which resolutions a given video can be downloaded, you can write a function like this:
def resolutions(video: YouTube): res = [] streams = self.video.streams.\ filter(type='video').\ order_by('resolution').\ desc() for stream in streams: if stream.resolution not in res: res.append(stream.resolution) return res
A stream with the selected resolution can be obtained as follows:
res = '1080p' # например stream = self.video.streams.\ filter(resolution=res, progressive=False).desc().first()
If you are not using ffmpeg, replace progressive=False with True
Downloading audio
From the previous example, you saw how we downloaded audio:
... audio_stream = yt.streams.\ filter(mime_type='audio/mp4').\ order_by('filesize').\ desc().first() audio_stream.download()
Just keep in mind that the audio format will be m4a, so if you need a different one, you'll have to use ffmpeg
I want to stop here. In the next part (if there is one), I will talk about how to download entire playlists, channels, and even subtitles from YouTube.
Known issues
DPI bypass doesn't work
Yes, that is certainly possible. But on my provider, TTK, everything works perfectly. Therefore, I can only advise using other tools, such as GoodbyeDPI.
Remote end closed connection
Now this is a real problem that I encountered. It appeared when YouTube started to slow down and I switched to nodpi. Either YouTube is banning suspicious activity, or Roskomnadzor is up to something, or maybe there's something wrong with nodpi. I never figured it out. If there are any experts here, maybe they can tell me what's wrong and how to fix it.)
Warning
If you find this article interesting, don't put it off, as some time after publication it may fall under geographical restrictions at the request of Roskomnadzor