We're in UltraHD Morty! How to watch any movie in 4K

    You’ve probably heard about Yandex’s DeepHD technology they once used to improve the quality of old Soviet cartoons. Unfortunately, it’s not public yet, and we, regular programmers, don’t have the dedication to write our own solution. But I personally really wanted to watch Rick and Morty on my 2880x1880 Retina display. And I was deeply disappointed, as even 1080p video (the highest available for this series) looks really blurry on a Retina display! Don’t get me wrong, 1080p is often good enough, but Retina is designed in such a way that an animation with its pronounced outlines in 1080p looks awfully blurry, like 480p on a FullHD monitor.

    I decided I want to see Rick and Morty in 4K, even though I can’t write neural networks. And, amazingly, I found a solution. You don’t even need to write any code: all you need is around 100GB of free space and a bit of patience. The result is a sharp 4K image that looks better than any interpolation.


    First, we need to understand that a technology for upscaling video using neural networks doesn’t exist. Or at least it’s not publicly available. And since that’s the case, we need to transform our video into a bunch of still frames!

    Adobe Premiere Pro or any other video editing suite can do it, but, since it’s probably not installed on a lot of PCs, I used the console utility ffmpeg. Took the first episode of the first series and away I went:

    $ ffmpeg -i RiM01x01_4K.mp4 -q:v 1 IM/01x01_%05d.jpg

    Why JPG and not PNG?
    A fair question. The thing is, 31 000 resulting PNGs would’ve taken insanely much space. So much that a slight sacrifice in quality is worth it. Also, the =q:v 1 parameter means we’re outputting JPGs in the highest possible quality.

    Around 10 minutes later, we get a huge folder filled with images. For me it took all of 26GB.

    Now all we need is to process every single one of those!


    I found three options that worked reasonably well — rather famous Let's Enhance, anime-oriented waifu2x, and Mail.ru Vision.

    I’ll show the examples later.

    Mail.ru Vision and Let’s Enhance do a good job, but unfortunately they aren’t open-source, meaning to process 31 000 images I’d need to write an email to their creators and probably pay quite a bit. Waifu2x is open-source, but the result was rather poor — a lot of noises and artifacts. After all, Rick and Morty is not anime.

    I almost resigned myself to digging GitHub and topical forums, but… a saviour appeared! I found a solution that worked on the machine locally, processes 1 image in less than a second and delivers in quality. You won’t believe who came to our rescue once again!

    Adobe Photoshop!

    And no, I won’t tell a story about how you could enhance a picture with a couple of filters. Adobe actually trained a proper neural network that can “complete” an image when you upscale it within the application!

    To start, we need to open the source image, go to the top menu, then Image — Image size, and choose the “Preserve Details 2.0” resampling option.

    The result was surprisingly good! Probably only trailing Let’s Enhance. Here’s the comparison (with the image zoomed in around 800%):

    And what now? Manually process every frame?
    Of course not! Photoshop has the Actions tool that allows to first record a sequence of actions, and then apply it to an entire folder of images. The process is a cursory Google search away.

    I left my laptop to process 31 000 frames through the night with a simple instruction: “upscale 2x and save”. The next morning everything was ready. I had another folder full of images, but now in 4K and taking up 82GB of disk space.

    Back to video

    Ffmpeg to the rescue again.

    First, we remember we forgot the audio track and extract it out of the original file:

    ffmpeg -i RiM01x01_1080p.mp4 -vn -ar 44100 -ac 2 -ab 320K -f mp3 sound.mp3

    Then we put the sound file into the folder with all the 4K images. We’re now ready for the final montage!

    ffmpeg -i 01x01_%05d.jpg -i sn.mp3 -vcodec libx264 -preset veryslow -crf 10 -r 23.976 RiM_01x01_4K.mp4

    Be careful: after -r specify the exact framerate of the original, or else the audio track would misalign with the video!

    It’s done!

    We now have the first episode of Rick and Morty in 4K. Here is the sample video. Of course, the whole process was a bit amateurish, but this approach has a serious advantage. While importing the images into Photoshop we can manually tinker with them to get it just right. Adjust the focus, color balance based on a couple of frames, write it into an Action, apply to the whole video and — voila! A perfect result without too much math trickery. This brings “miracle upscaling” a bit closer to an average user. The most complex of technologies that have been in development for centuries can not be used fast and without special knowledge — what is it, if not the future?
    Support the author
    Share post

    Comments 7

      So, 82 + 26 GB. But what's about the __time__? Probably this is not a quick operation.
        No, ffmpeg did each operation just in 35 minutes (MacBook Pro 2018, core i7)

          Potentially, other enhancing methods (not Photoshop, I mean) make possible to build a cluster of parallel jobs to enhance all the images really fast, so it'll be like a great system of making low-res films beautiful in real time (or not real, but very fast)

            Cool idea, but neural multithreading could turn your PC into a fiery box :)

            Although, something similar already exists: madvr.com has neural-based upscaling method, called NGU.
            The question without answer from previous post:
            Did you try to do the same by using ffmpeg only?
            ffmpeg -i input.mp4 -vf scale=3840:-2 -c:v libx264 -preset slow -crf 22 -c:a copy output.mp4
              This is a regular scale, the result will have more pixels, but the same quality as the original.
              That's definitely quite handy.
              But question, is it possible to do upscale in real time?
              The thing is I often don't have enough free space, and thus wouldn't be able to convert movies using this method. Is is interesting, but not really viable.

              And, well, ofc if i want to download some movies to my disk, I'll need even more disk storage for 4K versions, but if the real-time upscale is possible using regular video players, this problem could be easily avoided.

              Also, I would like to know how your method compares to the regular TV/TVBoxes 4K upscales?

              I assume that is is possible to use something like VLC filter to upscale video in real-time, but I have no idea how reliable it is compared to this simple, yet working method.

              Only users with full accounts can post comments. Log in, please.