HOW TO: 10-bit x264 and 10/12-bit x265 encodes with HandBrake! (Mac OS X, Linux, Windows)

Update #2 (Jan 2018): The latest nightlies (https://handbrake.fr/nightly.php) seem to include 10/12 bit for x265. So if that’s all you need, you don’t need to read/do anything below – just grab the latest nightlies. OTOH if you’re looking for 10-bit x264, that info is still below.

Update (July 2017): This was originally written back in Feb 2016, and the libraries I supplied back then are now too old for newer versions of HandBrake and likely won’t work or will have issues. I built a new Mac version on July 3rd 2017 which is below, and fortunately there are now good alternate options for Windows/Linux. Details as follows:

  • If using Windows, under the Windows section below I have a link to the Handbrake forums where they provide updated libraries. Use that! The 10-bit-only CLI version I have linked in the Windows section should work too, but it’s old.
  • If using Linux, Stephen Smith was kind enough to mention in the comments that the x265 libraries are now available on Ubuntu ( apt-get install libx265-dev ). Use that (don’t follow the rest of the instructions in the Linux section). I suspect other distros have variants available, and there’s also the Fedora repo listed in the Linux section.
  • If using Mac OS, the original (2016) and newer (2017) 10-bit-only builds should work. I wouldn’t suggest using the .dylib option, as those .dylibs are from the original write-up and quite old now.

The original write-up is still below, but keep the above in mind.

I recently got a Tronsmart Vega S95 (media player) which claimed to support 10-bit HEVC encodes. Unfortunately, I didn’t have any 10-bit encodes on hand to test out those claims with!

Short version is that I wanted to get 10-bit encoding working with HandBrake on OS X. Long version ends with figuring it out on Linux and Windows too because I figured it’d be helpful for people who don’t have 3 OS’s to swap between. These are the results:

8/10/12-bit HandBrake on Mac OS X
8/10/12-bit HandBrake on Linux (Ubuntu)
10-bit HandBrake CLI on Windows

Before somebody decides to encode their entire library in 12-bit HEVC and ends up utterly disappointed, once you’ve grabbed whatever info/files you want for your OS, scroll down waaaaaay to the end and have a read.

Okay, so one at a time, here we go…

(jump to Linux instructions)
(jump to Windows instructions)
(jump to the note on HandBrake and 10-bit)

Mac OS X

You basically have 2 options here:

  1. A separate 10-bit version (which I’ve included in the writeup) that you can run beside your current HandBrake install.
  2. Use the “nightly” HandBrake, and manually install some library files (which I’ve included at the end).

Option 1

I compiled a 10-bit version (both x264 and x265). Just download it, unzip, throw it in your Applications folder if you want, and run it (the same way you’d run your current version). A few notes on it:

  • It’s the easiest/cleanest option. If you decide to get rid of it, uninstalling is as simple as dragging to the trash.
  • It’s ONLY 10-bit. So don’t go un-installing your current version.
  • For x264 encodes, you must change the Profile to “high10” (not auto), or the encode will instantly fail.
  • Don’t “check for updates” or anything, because if it actually finds an update, it’ll be from the HandBrake website (and for the standard 8-bit version most likely).
  • The 2016 version was built on “El Capitan” (OS X 10.11.3), and 2017 version on Sierra. I didn’t muddle through things to see what the minimum OS version was set at. So if you’re using an older version of OS X, it may or may not work, but feel free to leave a comment if you try.

If you’re interested, you can grab it here:
handbrake-10-bit-mac-20170703.zip (July 3, 2017)
handbrake-10-bit-mac.zip (mid-2016)

Option 2

I compiled some x264 and x265 libraries for 8/10/12-bit support that will work with the latest “nightly” version of HandBrake. You can grab it from the main HandBrake website at:
https://handbrake.fr/nightly.php
(note: you are after the result that does not have “CLI” in the file name)

Get it installed (you can sit it beside your regular version if you want). Next, you have to download the libraries I’ve packaged together:
handbrake-libs-mac.zip

Unzip the handbrake-libs-mac.zip file and you should have a new folder called “handbrake-libs-mac”, with contents that look like this:

handbrake-libs-mac folder

Those are the library files that you need to install on your system. For HandBrake to see them, they have to go in the /usr/local/lib/ folder, which is hidden from the Finder by default. If you’re comfortable in the terminal, a sudo cp will likely do it.

Otherwise, you can tell Finder to manually go to the correct folder by opening a new Finder window and clicking “Go” from the top menu, then “Go To Folder“. A few images below to help (click for larger versions):

Finder: Go To Folder
Finder: /usr/share/lib/
Finder: Drag and Drop to copy to "/usr/share/lib/" folder

 

Basically, drag the libraries you downloaded into the /usr/local/lib/ folder and you should be good to go!

Note that technically you only need to copy the 3 that have a “10” or “12” in the filename if you’re using the HandBrake nightly build (since it already contains the standard 8-bit versions built-in), but it shouldn’t hurt anything to just copy them all.

Now open up HandBrake (or exit/restart if it was running). You should now have access to 8/10-bit versions of x264 and 8/10/12-bit versions of x265!

A few notes:

  • 10-bit x265 might immediately crash on the slowest settings (slow, veryslow, etc). The separate 10-bit-only version I compiled above doesn’t seem to suffer from this issue, so you may want to use that if you’re doing x265 10-bit at those speeds.
  • On my machine, the x264 10-bit version was often coming up as the default selection. If you’re in the habit of just assuming x264 8-bit is the default, make sure you double-check before you hit encode!
  • At some point, there will undoubtedly be new versions of x264 and x265. When that happens, the ones you installed will be a bit dated, so keep that in mind. They don’t auto-update or anything.

(jump to Windows instructions)
(jump to the note on HandBrake and 10-bit)

Linux

A quick note for Fedora/RHEL users… negativo17.org has a HandBrake repo and they now have the 8/10/12-bit stuff enabled there. I strongly suggest just adding their repo via their instructions if you’re on one of those distros.

These instructions are primarily for Ubuntu, though other similar (debian-based) distros should work. Non-debian-based distros should be doable if you can locate a program that converts .deb packages to whatever you need (for example, Alien is common for converting from .deb to .rpm).

Anyway, let’s get on to it.

Step 1 – Grab/Install the Latest Nightlies

If you’re using a supported version of Ubuntu, follow the instructions for installing the PPA at: https://launchpad.net/~stebbins/+archive/ubuntu/handbrake-git-snapshots . Basically, the commands (if you’re both lazy and trusting) are:

sudo add-apt-repository ppa:stebbins/handbrake-git-snapshots
sudo apt-get update
sudo apt-get install handbrake-gtk

If NOT on Ubuntu but are on another Debian-based distro, you can still manually download one of the packages in .deb format and install it. Head to: https://launchpad.net/~stebbins/+archive/ubuntu/handbrake-git-snapshots/+packages , expand one of the down arrows – generally the top arrow if you’re on a fairly recent version of your distro, and grab the .deb that includes “gtk” in the file name (the “amd64” variant for 64-bit). Incidentally, this is also what you’d do if on a non-debian-based distro and were using a package converter. Then try and install the package – if you don’t have a built-in package manager, you can usually do it from the command line with sudo dpkg -i theFileName.deb

Okay, let’s assume you got it installed, and it runs (just type “ghb” in a Terminal if it doesn’t show up with your applications).

Step 2 – Install the 10/12-bit Libraries

I’ve built the libraries for linux, and you can grab them here:
handbrake-libs-linux.zip

Extract them, and you should have 5 .so files, somewhat similar to the earlier Mac screenshot. Now you just have to copy them to your library.

  • For Ubuntu, your library location is usually: /usr/lib/x86_64-linux-gnu/
  • For Fedora, your library location is usually: /lib64/
  • For others, you’ll probably have to look around. It could be one of the 2 shown above, but you’re best to check first. Chances are if you run across a location that has “lib” in the directory structure and contains a zillion .so files, you’ve found the right place.

The file managers in Linux have a tendency to refuse access to root folders rather than asking for privilege escalation, so chance are you’ll have to copy from a terminal/shell window. Fire up “Terminal” or something similar, and navigate to the location you unzipped the 5 files. Do a quick “ls” to make sure you’re in the right place. For example:

cd ~/Downloads/handbrake-libs-linux/
ls -l

(should show the 5 .so files if you’re in the right place)

Next, you’ll have to copy them to the library location mentioned above. As an example:

# For Ubuntu:
sudo cp -n libx26* /usr/lib/x86_64-linux-gnu/

# For Fedora:
sudo cp -n libx26* /lib64/

(the “-n” means no-clobber – it won’t overwrite existing files).

Assuming that was successful, try firing up HandBrake. Hopefully you’ll have all the x264 and x265 options available now!

A few notes:

  • Similar to the Mac instructions, you don’t actually need to install all 5 .so files. You only really need the 3 with “10” and “12” in the file names since HandBrake already will have taken care of the standard 8-bit versions.
  • This is a bit of a “dirty” way to put the .so files in the library directory, so remember that you put them there. Usually, the system’s package manager has some idea about which packages are throwing what in there, but since you’re brute-force-copying, that might throw a wrench into things down the line.
  • If you’re comfortable with sym-linking, a slightly cleaner way to do this would be to add version numbers to each of the files (.so.148) and symlink to .so, which is what most packages do to help keep things at least somewhat organized.
  • I didn’t include any pre-built versions (just the libraries) – it just caused too many dependency headaches on fresh systems if HandBrake wasn’t installed at some point anyway. It also depended on the x264 library being installed anyway. If somebody really needs the binary, ask in the comments and if I haven’t wiped the VM yet, I’ll see what I can do.

(jump to the note on HandBrake and 10-bit)

Windows

Update: It looks like the Handbrake devs have Windows DLL’s available that you can simply use with the current nightlies for 10-bit support (x264 and x265) and 12-bit support (x265). The page on their forum that carries them is: https://forum.handbrake.fr/viewtopic.php?f=11&t=34165 .

I haven’t tried them myself, but would imagine they just get plopped into your Handbrake directory. You need to use the nightly version of Handbrake though ( https://handbrake.fr/nightly.php ).

In the event that the links end up broken at some point in the future (or you don’t have 7zip installed), I rezipped them (.zip format) and you can grab  local copies from here:
https://mattgadient.com/download/handbrake/libx264_main10.dll.zip (x264 10-bit)
https://mattgadient.com/download/handbrake/libx265_main10.dll.zip (x265 10-bit)
https://mattgadient.com/download/handbrake/libx265_main12.dll.zip (x265 12-bit)

Those DLL’s are probably the best solution for the vast majority of situations. Feel free to leave a comment whether it works seamlessly for you, or if you hit issues.

Note that Lazarus also came across a GUI solution in the comments below which you can jump to here: #comment-22958 .

The remainder of my original write-up is below anyway in case you’re curious, are doing something a little different, or are looking for the CLI bit…

Visual Studio hates me (and now I hate it back), so I was only able to get the CLI version done. I did build the libraries for the GUI version, but we’ll just do the CLI bit now and talk more about the GUI stuff later.

The 10-bit CLI version I built can be grabbed from here:
handbrake-cli-10bit-windows.zip

Extract it, and you should have a HandBrakeCLI-10bit.exe program that can be run from the command prompt.

It supports both 10-bit x264, and 10-bit x265. It also ONLY supports those x264/x265 variants, so if you’re using the standard 8-bit CLI, you’ll still want to keep it around.

It’s critical that when using it, you specify the encoder-profile. For example:

HandBrakeCLI-10bit.exe -i inputVideo.mp4 -o newVideo.mp4 -e x264_10bit --encoder-profile high10
HandBrakeCLI-10bit.exe -i inputVideo.mp4 -o newVideo.mp4 -e x265_10bit --encoder-profile main10

The important bits are:
-e x264-10bit –encoder-profile high10 (for x264 10-bit)
-e x265-10bit –encoder-profile main10 (for x265 10-bit)
…if you don’t include the encoder profile, it will *fail*.

As to the rest, if you’re new to using the CLI, you’re best to browse through the guide at:
https://trac.handbrake.fr/wiki/CLIGuide

Alternately you can type “HandBrakeCLI-10bit.exe –help” to see the options, but it’ll completely fill your screen. You can save all that text to a file instead to read with Notepad later with “HandBrakeCLI-10bit.exe –help > hb-help.txt

The GUI bit on Windows

As I alluded to earlier, I did get libraries made up for the Windows GUI version. Unfortunately, they didn’t seem to work with the current nightly version (it reads them, but doesn’t display the additional x264/x265 options). I’m guessing either:

  • The nightly build from Feb 13th missed the patch (from the same day) that enabled 10/12-bit DLL support in the Windows GUI (hopefully).
  • Something built wrong on my end (hopefully not).

Due to the Visual Studio hatred mentioned previously, I haven’t tried building the GUI myself from the patched source yet.

In any case, if you’re building from source yourself or are quicker at finding the new nightly than I am, the 8/10/12-bit x264/x265 libraries that I built can be downloaded from here:
handbrake-libs-windows.zip

They go in the same directory as the HandBrake Nightly. Usually this is C:\Program Files\HandBrake Nightly\ .

For reference, the nightly I tried (unsuccessfully) was HandBrake-b7cb7d6_x86_64-Win_GUI.exe . Only try these libraries on a newer version.

Keep in mind that because adding libraries to the Windows version would be remarkably clean compared to linux/mac (they could just be included with the program files on Windows!), it’s within the realm of possibility that the HandBrake devs might just add the libraries to the install itself at some point.

A Note on HandBrake and 10-bit Encodes

At the beginning I mentioned that it might be a good idea to scroll waaaaay down before someone goes off and encodes their entire library at 12-bit HEVC. There are a number of reasons NOT to encode via 10-bit and 12-bit in HandBrake.

It’s worth mentioning that the builds/libraries above should be considered somewhat “experimental”. It’s within the realm of possibility that something won’t work correctly (or at all). Don’t be quick to blame HandBrake or x264/x265 if you’re running into issues – I could have very well missed something when compiling.

If you copied the libs on Mac or Linux, be sure you’ve got some idea as to where you stuck the libraries too, just in case they conflict with something and/or you want to remove them later.

Why NOT to use 10-bit and 12-bit

First of all, HandBrake’s entire toolchain is set up for 8-bit 4:2:0. Basically that means if you have a source that’s 10-bit 4:4:4, by the time HandBrake spits it out at the end, it’s probably going to have been reduced to 8-bit 4:2:0. There’s not a whole lot you can do about that, and there aren’t any simple “switches” that the developers can really flip either. They may be able to migrate over time if they want to go that route, but it’s a *really* large code-base that relies on a lot of external tools they don’t necessarily have a lot of control over.

The next problem is that device support is minimal at best.

  • For x264, you’ll have a hard time finding a phone/tablet/device that’ll play 10-bit video. Often, devices that *try* to decode it in hardware will have the colour palette looking super-saturated. You’re generally limited to computer playback.
  • For x265 (HEVC), it wasn’t until the end of 2015 that devices started trickling out that could handle 10-bit. There’s also a wonky situation going on with some of the latest stuff where you get devices that claim 10-bit HEVC and 4K display, but in actuality can either do 10-bit at lower resolutions (1080p), or 8-bit at 4K.

Really, unless you’ve literally upgraded all the phones, tablets, and media players in your household within the last few months, chances are that moving to 10-bit encodes is going to leave some of your devices out.

There’s also the time issue. 10 and 12-bit encodes tend to take longer (in test encodes while building the stuff above, a 10-bit x264 encode also sucked up a surprising amount of RAM).

Really, you’re “paying” a lot (time, device support, etc), with not a whole lot in return.

That being said…

Pros to 10-bit? Efficiency and Banding?

10-bit x264 has been common in the anime community for quite some time due to a reduction in banding. I haven’t heard whether the same is supposed to hold true for x265 10-bit (I’m not sure why it wouldn’t), so I’ll leave you to do your own research and draw your own conclusions there.

There’s also a notion that 10-bit is more efficient (even with an 8 bit source), probably as a result of the document here: http://x264.nl/x264/10bit_02-ateme-why_does_10bit_save_bandwidth.pdf .

There may be good cases for using 10/12-bit, but at the same time, it’s not something that comes “free” and it’s a little tough to quantify. My best advice would be to run your own test encodes back-to-back and decide for yourself.

Hopefully the libraries above will help you do just that :)