Matt Jones Tech
  • Blog
  • Video Projects
  • Web Projects
  • How To Pause and Resume FFMPEG

    How to Pause and Resume FFMPEG

    This is the story of how I learned how to pause and resume ffmpeg. I’m a pretty big multitasker when it comes to computing. I usually end up with a few hundred tabs open when I’m browsing and I always have a terminal open. Last week I was doing a job using Davinci Resolve.

    In order for me to complete the job, I needed to convert about 60 or so .MTS files into Apple ProRes .MOV files. The entire project contained over 10 hours of content, and every file needed to be converted.

    So I employed my fancy batch script to loop through all the .MTS files in a given folder, transcode them one at a time, and output them to another folder. Such an aweseome script, and I use it very often. But eventually, the way I was working through the project, I ran into a problem.

    I needed to render a timeline out of Resolve, but my resources were already being used on this massive batch render. So how do I solve this problem?

    TLDR; Here’s the Answer

    It pretty much comes down to managing your Linux processes. If you’ve got a lot going on, and you need more control over which process gets the most resources, you have full control over that. And by the way, this trick can be used for ANY Linux process, not just FFMPEG!

    Typically, if you have a running process in a terminal window, you can kill it using CTRL+C. That will kill the process, and you’ll have to start the whole command over again if you want to resume the task. So if I hit CTRL+C in the middle of my ffmpeg transcode… I’d end up with some finished files, a half-encoded file, and an incomplete batch. That sucks.

    But there’s a different command that just stops the process instead of killing it. Basically pausing it in its current state, waiting for you to do something else. CTRL+Z is that magic command. This will pause a running process and send it to the background, and assigns that process a number. This will also return control of the terminal window back to you.

    If you run a different task, then CTRL+Z, you’ll have multiple processes paused in the background, each with a number.

    To see all the paused background jobs, you can simply run jobs to see a list of paused jobs.

    Now that my running ffmpeg process is paused and in the background, I can render out that Davinci Resolve sequence without overloading my system. Awesome!

    The fg Command

    Okay, now my Davinci Resolve render is complete! I’ve delivered that sequence, but I need to continue my batch encode so I can finish the rest of the project. To bring that ffmpeg process back to the foreground and pick up where I left off, all I have to do is use the fg command.

    The fg command can either take a job number as a parameter, or no parameter at all. If you don’t specify a job number, it will bring the default job [1] to the foreground and continue running that job.

    That’s it!

    The bg Command

    If you have a running process that you want to keep running, but regain access to the terminal input, you can simply hit bg. That’ll keep the process going, but send it to the background. Just remember- CTRL+C doesn’t work on background processes. In order to kill it, you’ll need to bring it to the foreground with fg.

    matt

    March 24, 2020
    FFMPEG, Linux, Video Editing
    ffmpeg, jobs, linux, processes, tasks
  • Desire: Making of

    Behind The Scenes of Desire

    Welcome to a behind the scenes look at Desire, a short video project created to tell a story using only macro shots.

    Preproduction

    A month or so ago, I was tasked with a brand new challenge: Make a video about sex… for a church… that’s not a joke, or takes the topic lightly. It took me a week or so of pondering, but the resulting concept, I think, was pretty cool. Most of the preproduction work was done for us, as the project follows very closely a video that inspired me a several years ago. So in essense, our preproduction was already done for us. The biggest challenge was reverse-engineering the shots. I’d never worked with macro shots that were close enough to make the human eye fill the entire frame.

    Production

    The entire project was a relatively quick turnaround, which actually worked out because all of it was filmed in a single location. The trick was trying to convey different locations and moods, almost entirely with lighting.

    Postproduction

    This project was one of the first (professional) projects I cut in Davinci Resolve. My experience was nearly seamless coming from Premiere Pro. Didn’t have to search for much, and instantly fell in love with the tab system. Overall, had a great experience with this project and it’s probably my favorite of 2019. Check out the final below!

    Final Result

    matt

    November 15, 2019
    3D Animation, DaVinci Resolve, Video Editing
    b3d, davinci resolve, postproduction, preproduction, resolve, studio
  • Transcode Files for Davinci Resolve 16

    Today I learned that you only have to transcode files to import them into the free version of Resolve. If you’re rocking the full version, then this shouldn’t really apply, unless you have an absolutely crazy file format on your hands.

    Didn’t We Already Cover This?

    So this stems off a previous blog post from a while back, where I was using ffmpeg to transcode some .MTS files (from a Panasonic/Sony camcorders similar to this) for use in Resolve on Linux. The code snippets from that post were okay… But not great. The biggest problem was that after transcoding, the file size often inflated… I mean DEFINITELY inflated. FFMPEG often defaults to the highest tier of quality in a given codec if you don’t take the time to specify exactly what you want.

    ProRes

    I really struggled with this because it’s easily argued that ProRes is the codec that is literally designed to be edited. So it’s great to just cut ProRes from the start, right? And I’ve been involved in a lot of professional workflows where the first step is to convert to ProRes. Sounds like a good idea. Except when you consider file size.

    Recently, I’ve been getting lots of projects involving mass quantities of footage. Often times I’ll get 3-5 hours of footage, climbing up to 60GB for a single project. Keep doing projects like that and I’ll have to start an Amazon subscription to hard drives… So I’m not in a place where I can afford to transcode 60GB of .MTS into 240GB of ProRes.

    MPEG4

    Luckily Resolve takes codecs other than ProRes! So I experimented a bit trying to figure out how to transcode a ton of .MTS files into something that Resolve can read, but without inflating the file size to something that’s nearly unusable. After a bunch of trial and mostly error, I came up with a pretty good preset. I even saved it as a text file so I can reference it later and use it in bash scripts.

    So here it is, mostly for my own reference so I can go back here and copy it again:

    ffmpeg -i "file.MTS" -c:v mpeg4 -b:v 20M -c:a pcm_s24le /output/destination.mov

    I’ve also tested this export setting with .MOV files that came from an iPhone and didn’t play natively with Resolve. I just converted them using the snippet above, and they played back perfectly in Resolve. Plus with the variable bitrate setting, the filesizes are VERY comparable to the original sizes. Often only off by a few megabytes.

    Enjoy, and happy editing on Linux!

    matt

    September 30, 2019
    FFMPEG, Linux, Video Editing
    editing, ffmpeg, linux, resolve, transcoding, video editing
  • Batch Process FFMPEG

    Batch Process FFMPEG

    An incredible use case for simple Bash scripting is the ability to batch process ffmpeg tasks. First off, I’ll assume you guys have got a basic understanding of FFMPEG, what it is, how to install and use it, that kinda thing. If not, feel free to check out the basics. FFMPEG is absolutely amazing. It can convert a chicken sandwich to ProRes. So once you’ve got the basic single-file convert down, how do you batch convert?

    Shell Scripts

    This is new for me, and I’m happy to learn new stuff. So upon a basic introduction, shell scripting looks incredibly powerful. I’m easing my way into Python because Blender is written in Python, but apparently Python is something you can integrate into shell scripts as well, so all the better!

    More good news, someone had the same issue as me, and got a pretty rad response on StackOverflow. The 2nd answer below the ‘correct’ one uses something called parameter expansion that I don’t really know a lot about. But from my limited understanding, I gather that it’s a way to take the existing filename and stick it on to the ffmpeg file export. So let’s say you have vacayFootage.avi that you want to convert to .mp4. Parameter expansion grabs the file name and sticks it onto the resulting export so you’ll get vacayFootage.mp4 without the hassle of having to name a thousand files by hand, each time you convert a file. So here’s an example of how you would batch process ffmpeg commands:

    for i in *.avi; do ffmpeg -i "$i" "${i%.*}.mp4"; done

    So like a true professional, I pasted that Stackoverflow answer into a shell script, modified the lines a bit to match my needs, and boom! It worked! I was able to convert about 500 files in just a few minutes with a single line of code… So awesome!

    So Many Possibilities…

    So with that fancy new script… and learning about how read is a cool Python command that can be used to collect user input… I’d love to write a shell script that can work for anyone. Just enter the directory where all your media is, choose your output options, select your output directory, and boom! Batch encodes for all! Gotta get on that… right after I learn Python.

    matt

    September 16, 2019
    FFMPEG, Linux, Video Editing
    bash, code, ffmpeg, file conversion, linux, python, scripting, shell
  • Switching from Windows to Manjaro Linux

    I use Adobe CC on a Windows machine for my video production pipeline at work. I do video side work on the weekends, and have used Windows and Adobe for those projects as well, but I’ve always felt trapped. Like I’m a slave to Adobe or something. I’ve spent most of my professional career learning and using software like Premiere Pro, After Effects, and Photoshop.

    Making the Switch

    I’ve used Ubuntu a TON, as it’s a super popular distro, and it was my introduction to Linux. A few months back, I was checking out https://opendata.blender.org/ and noticed that a good chunk of data was coming from a distro called Manjaro, but I’d never heard of it. Shortly after seeing all that performance data, I decided to give Manjaro a shot. After some basic research, I discovered Manjaro is actually based on Arch (unlike Ubuntu, based on Debian).

    I gotta say, installing Manjaro was a breeze (cursor theme). I will admit however, I have never known the pain of installing Arch. Like actual Arch. But I’m aware of the memes. So I’m sure it’s pretty involved. While I can’t exactly go around trolling like, “btw I use arch”, I was seriously surprised at the installation and how much was up and running right out of the box! Ubuntu is pretty straightforward, but depending on the hardware, you may need to find a few hacks or packages to get everything full up and running. For example, I couldn’t get the wifi working when I put Ubuntu on my 10 year old Macbook until I installed a special package that didn’t come with the default install.

    Everything Works.

    Immediately… Very first boot up. Everything works. Hardline network connection works. Wifi card works. GRAPHICS look great. Manjaro automatically grabs graphics drivers for you upon install! And I was actually impressed with the pre-installed software collection. Most of the time, in Ubuntu, I have to delete a bunch of crap software I’ll never use. But Manjaro… Like, I use Steam all the time. It’s pre-installed. Libre Office pre-installed. Tweaks pre-installed. Gparted pre-installed (this might be normal). But I was really surprised how little I actually had to do post-installation.

    Smooth UI Experience

    It’s a bunch of little things that add up to a massive amount of time saved. One of my pet peeves in Windows is switching apps on the task bar. If your app has multiple windows (like Blender, and a render window), you have to click once to reveal thumbnails, then click the thumbnail of the window you want. If they’re small or detailed windows like terminals, good luck. I love the single hot corner in the upper left to give you an expose-like look of all your running apps. I love the best-of-both worlds option to launch a full-screen app tray from the bottom left, or tap the drop-down xcfe-like ‘start menu’ from the upper left. It’s pretty awesome. Just gotta get used to typing pacman -S instead of apt-get

    Haven’t logged a ton of hours on Manjaro just yet, still very much in the “moving in” phase. Feeling great so far. Looking forward to diving into my brand new postproduction workflow in Manjaro using Blender, Resolve, Gimp, Inkscape, and Ardour. More on that in the coming weeks! Peace

    matt

    September 9, 2019
    Blender, Linux, Video Editing
    Blender, linux, manjaro, open source, pipeline, workflow
  • Blender, Linux, and R3D

    Convert with FFMPEG?

    If you’re wondering how to edit r3d on linux, you’ve come to the right place. This post is inspired by an official, yet very under-the-radar release of something called REDline Linux released back in mid 2018. As a fan of Linux as well as continuing my quest to discover a complete replacement for the Adobe Creative Cloud, I ran into this question: Can Linux handle R3D raw? After just a little bit of research, I discovered that ffmpeg can actually decode an early version of the R3D codec! The downside is, as of the writing of this article, ffmpeg isn’t capable of decoding the latest version of the R3D codec. However, this could change if someone comes along and reverse engineers the new R3D codec.

    Yet the question still remains. How can I work with RED footage and use Linux at the same time? Enter Resolve. (Opinion alert!) Blackmagic Design has positioned themselves in a way that could completely dethrone the current king of postproduction software, Adobe. Adobe makes a great suite of programs. One of the biggest advantages they have is that their software can not only ingest and work with project files from other software within the suite, but it also does this seamlessly across Mac and Windows. For example, I can save a Photoshop project on a Mac, and open that same Photoshop file inside of After Effects on Windows (and still have access to all of the Photoshop layers). The biggest disadvantages of Adobe CC is that it’s a fairly expensive monthly subscription model that doesn’t support Linux.

    Blackmagic Resolve

    Blackmagic Resolve has been the industry standard for color correction for a very long time. Recently, Blackmagic Design revamped Resolve to be able to handle much more than just color correction. In a lot of situations, Resolve can be a sort of one-stop-shop for all you postproduction needs, providing basic edits, basic audio workflow, and some basic VFX. The biggest news is that Resolve is free to download, and upgradable to the full version for a one-time purchase of $300. So after your first 6 months, you’re already saving money over the standard Adobe CC subscription.

    Anyway, point being, Resolve is quickly becoming the powerhouse one-stop solution for professional post video. Handling R3D files is as simple as importing them off your RED mini mag, opening up Resolve, and grabbing them from inside the Media Pool tab. Pretty much like using the Media Browser inside Premiere Pro. Except better.

    R3D in, TIFF out

    From there, you can make the edits you need and export to whatever format you want, even image sequences like TIFF and others. What’s more, is that you don’t have to convert your footage with FFMPEG or something similar before importing. You can just import raw R3D without a hitch. Happy editing and Blendering!

    matt

    May 21, 2019
    Lifestyle, Linux, Video Editing
    Blender, compatability, linux, postproduction, r3d, red digital cinema, sequences, vfx
  • I’m a Purist

    Opinion

    This idea has been with me since I was a kid learning card tricks. The purist in me never liked some special custom-made gimmick or some thing that essentially made the trick possible. Because suddenly, now I’m relying on that gimmick everywhere I go. Now I can’t perform if I don’t have that gimmick.

    The same holds true in my professional life. And instead of gimmicks for magic tricks, it’s plugins for applications. They might seem cool at first and give you some ability you didn’t have with the stock version of the software, or save you a bunch of time in creating a custom look or help you deliver an effect on a tight deadline.

    Reasoning

    However, in the end, just like with card tricks, now you’re stuck relying on this plugin that now you can’t live without. And even worse, if you’re working on a team and need to hand off a project to another team member, they can’t work on the project unless they have all the same plugins you have.

    Just a minor, technical note: for the sake of this article and the naming conventions used by different softwares and companies, I’m using the term ‘addon’ and ‘plugin’ interchangeably. There very well may be a technical difference between the two terms, but Premier and After Effects call them plugins and Blender calls the addons. They’re almost the same thing to me.

    Of course there can be exceptions with regard to my ‘purist’ view on the idea of this subject. One is the nature of the plugin, and Blender definitely falls into its own category when it comes to plugins/addons. Especially since it comes with its own addons panel in the preferences menu, addons are an integral part of Blender. Plus you can bake or otherwise export things that you make using addons into a filetype or object that anyone else can import and use. The addons are limiting in that way.

    Paid addons for Adobe products on the other hand, have the potential to cause issues. If you’re on a team and need to make some minor changes to a project, but don’t have the addons, now you’re in trouble. Plus you might even run into licensing issues, depending on the plugin.

    matt

    April 23, 2019
    Blender, Linux, Video Editing
    addon, Adobe, after effects, b3d, Blender, plugin, premiere pro
  • Hunted

    This is a VFX shot and edit sequence created for a short video in late March 2019. We couldn’t find the ideal location (an open field surrounding a lone tree) for this shot, and I needed too many specifics to match any stock footage. Short on time, I turned to Blender for the solution. All things considered, I’m pretty pleased with the shot. The entire composite took about 8 hours to design and create, plus and overnight render.

    Timelapse of the compositing process
    Capture 1
    Capture 2

    The project was filmed in less than 4 hours on an overcast morning. I tried to get as little of the sky as I possibly could, as I knew I’d be grading day-for-night. Below is the final result:

    matt

    April 7, 2019
    3D Animation, 3D Modeling, Compositing, Video Editing, Video Portfolio
  • Flowblade Review: Most Capable NLE on Linux?

    Flowblade 2.0 Released

    Janne Liljeblad and other contributors released Flowblade 2.0 recently and I thought I’d try it out. I edited a quick 1 minute video from about 10 drone clips shot in 4k. The first impressive feature was how easily I was able to render proxies. It was just as easy to replace them with the original media before the render.

    As both a Premiere user and a Linux user, I’ve been on a quest to find an NLE that is just as capable and intuitive as Premiere, but on Linux. So far, I’ve used only a few and had just ‘okay’ experiences. My first Linux based NLE that I dove headfirst into was Kdenlive. I was cutting some footage for a client and built out the complete project in Kdenlive for a few reasons. First and foremost, at the time, I no longer had access to Adobe CC through a former employer. And finally, even if I had $53/mo, I could think of 1,000 things I’d do before I got an Adobe CC subscription.

    But I digress, this is a post about Flowblade. My experience with Kdenlive was fine, the hotkeys took some getting used to. The alpha channels weren’t automatic either, but it worked. Flowblade, however gave me a much more ‘automatic’ experience. I’m on an older mid-range GTX-970 machine and 4k footage doesn’t playback smoothly. Regardless, it felt like proxies were easier to create in Flowblade than in Premiere. Just a few clicks and it was done. Updated right there in my timeline. And encoding was a breeze. Once I made my edits, which was a pleasure by the way, All I had to do was choose “replace proxies with original media” and I was all set to render.

    Intuitive Experience

    There were several hotkeys that carried over from Premiere and others that just made sense. The i and o keys set in and out points, the HOME and END key pops your playhead to the beginning and end of your timeline, stuff like that. The alpha transparency and title card system took some getting used to, but it wasn’t that bad. I feel like color correction is slightly easier in Flowblade than in Kdenlive. That’s just my experience, and the clips I happen to be working on.

    Overall, I still can’t completely commit to Flowblade, even though it’s a pretty great application for basic edits. If I were a vlogger and just needed something to spit out videos with speed, I’d definitely use Flowblade for everything. And even as a pro editor, I still may use Flowblade for some quick edits here and there, based on the situation. But as long as Resovlve remains an option for Linux users, it no doubt offers the absolute best postproduction experience so far. The grades are otherworldly. I’ve never had so much control over color. Not even in Premiere. Lumetri Color doesn’t even compete with Resolve.

    So far, my only issue with resolve is it can’t take Panasonic .MTS files by default, they have to be transcoded first. And that just may be a “free version” limitation, I’m not sure. Either way, I’m just a few freelance jobs away from picking up a full copy of Resolve for my personal Linux machine, so I’ll keep you posted if my experience changes once I get everything up and running. Until next time.

    matt

    March 19, 2019
    Linux, Ubuntu, Video Editing, Vlogging
    Adobe, creative cloud, kdenlive, linux, nle, postproduction, premiere, ubuntu, video editing
  • How to Use ffmpeg (The Basics)

    The Problem

    It all started when Premiere Pro started glitching out. When I was trying to import a simple image sequence, like I’d done a thousand times before, it only imported the first 25% of the frames. Everything else was missing. There was no inconsistency in the file naming convention. Everything was sequential. There were no missing frames. Premiere is just glitchy.

    The Workaround

    I came up with an idea for a workaround. Instead of importing an image sequence, what if I just concatenated all those image files into a ProRes MOV? That way I can just drop the whole thing into Premiere with no problems. I tried to get Adobe Media Encoder to do that, but it wasn’t the most intuitive experience of my life. Every time I tried to import a sequence, it only gave me the first frame, and ended up rendering frame 1 over the entire length of the sequence. Solution? FFMPEG.

    What is FFMPEG?

    Ffmpeg is a command line application that allows you to manipulate images, video, and audio. It does take some getting used to, but once you get the hang of it, it is WELL worth the time you put in to learn the basics of the application. I do want to emphasize just the basics, as ffmpeg and all of its options are pretty vast.

    As a 3D artist and animator, I deal a lot with image sequences. Unfortunately Premiere and other NLEs can sometimes glitch out and not give you what you want when you import a sequence. FFMPEG allows you to concatenate a series of images and encode those frames into a single video file with a codec and container of your choosing.

    If you have other open source video applications like VLC player or Handbrake, you might already have ffmpeg installed on your system. You can check by running this in the terminal:

    ffmpeg -version

    If you don’t have it, you can get it easily by running:

    sudo apt install ffmpeg

    Now that you’ve got ffmpeg, you can do TONS of stuff with it. There’s a giant list of commands or you can check out the man page:

    man ffmpeg

    You can convert video files as simply as:

    ffmpeg -i input.mp4 output.avi

    The Tricks

    ffmpeg is used to initialize the program, -i is used to set your input file(s), and you can add more options like scale, fps, codecs, bit rates, and more with a long list of options available. For my purposes, I usually want to take a folder full of .png files and convert them into an MOV file for editing. For that, I can just ‘cd’ into the folder containing all my .png frames, and run something like:

    ffmpeg -f image2 -pattern_type glob -framerate 23.976 -i cyclesShotFour_*.png shotFour.mov

    Let’s break that down. The -f tack lets you indicate a format. To see all available formats, just run:

    ffmpeg -formats

    It’s worth noting that formats are different than codecs. To view the massive list of codecs available in ffmpeg, just run:

    ffmpeg -codecs

    Easy, right? Next we told ffmpeg to grab all the .png files in the current working directory matching the pattern we defined. In my example, all my files were named cyclesShotFour_0001.png, cyclesShotFour_0002.png, etc. There’s quite a few ways to define a pattern, and you can learn more here. Once ffmpeg has all your images defined, all you need to set your framerate and other stuff you may want and you’re good to go! Tack -framerate defines your framerate. At the end, just type the name of your output file. That’s it!

    Some of the other options that wasn’t mentioned in the above example, that I use quite frequently is setting the bitrate and the codec. The tacks for those options are -b:v (bitrate for video) -b:a (bitrate for audio) -c:v (codec for video) -c:a (codec for audio). So let’s say I want to encode a bunch of .jpg files into a Apple ProRes MOV file. Clearly, the output won’t have audio, so we don’t have to specify any audio options.

    ffmpeg -f image2 -c:v prores -framerate 23.976 -b:v 6000k -pattern_type glob -i frame*.jpg output.mov

    Anway, there are tons of different things you can do with ffmpeg, and it’s an absolutely amazing tool. It saved my butt just last week! Have fun and keep learning.

    matt

    February 26, 2019
    3D Animation, Linux, Ubuntu, Video Editing
    audio, bitrates, codecs, encoding, ffmpeg, formats, handbrake, linux, media, postproduction, transcoding, video
1 2
Next Page

Prove all things; hold fast that which is good. 1 Thess 5:21