Update (June 25 2012) – I have a more complete & updated version of this write-up now. It’s quite long, but you can find it here.
I wrote this up quite a while ago, and it’s been sitting as a draft since. I pulled information on each setting from a few different websites, and while I don’t have any “final results” to show you (and the write-up wasn’t quite completed), hopefully the information’s useful in helping you to better determine what each setting does (and how it will affect encode time and quality), and to come up with your own settings.
I’ve used Handbrake in the past to encode movies and various TV series from DVD source. Recently, I grabbed the latest version (0.9.3), and decided to encode a TV series from DVD.
In short, here’s what I’m looking to get from my encode:
-High quality, even if it takes a while to encode.
-File size of between 250-400mb for each 1-hour episode (about 42-44 mins).
Of course, the best way to see how well you’ve done is to find something to compare it to. Therefore, I grabbed a torrent of the same TV show, and aimed to create a better quality encode at a smaller file size.
Before dumping into the x264 settings, it’s worth noting that I used a 2-pass encode, and mixed the audio down to a 160kbps Dolby Surround encode.
Here’s a list of the x264 settings I chose in Handbrake, and why:
Reference Frames: 6
Normally, I’d go much higher here. However if you go too high, both hardware and *some* software players can choke. Unfortunately, reference frames are subject to diminishing returns. According to a document at mplayerhq.hu , while going from a setting of 1 to 2 would improve the signal-to-noise by 0.15dB, going from 6 to 12 would typically improve the signal-to-noise ratio by only 0.02dB but result in a 15-20% longer encoding time. Really high diminishing returns for something you probably won’t notice, and something that might cause issues if you ever try to play your video in a hardware player. That said, go crazy if you want and try 16. Just make sure your video plays fine afterwards.
Mixed References: Yes
You get some quality at the cost of some speed. I’ve never seen anyone give a good reason to turn it off.
There’s another option called “adaptive b-frames” which is on by default (and there’s no setting to turn it off in Handbrake), so the encoder will use however many B-Frames it sees fit to use. Typically it won’t use more than 3 or 4. Feel free to pump this up to 16 if you want – it probably won’t make a lick of difference though unless for some reason it decides that more than 3-4 was needed.
Direct Prediction: Automatic
None is bad, and if you’re intent on choosing one, Spacial should be better than Temporal according to the documentation. Everyone seems to point at leaving this at Auto so the encoder can decide when to use what, and I’d tend to stick with that.
Weighted B-Frames: Yes
A fun bit from mplayerhq.hu – there’s often not a lot to gain from this except during fades where it’ll save some bitrate… except that apparently the “adaptive b-frames” that’s on by default tends to avoid b-frames during fades. So where it should help the most, it probably doesn’t. It can save some bitrate though anyway, and some people have noticed 1-2% larger file sizes without it (which means you should get an overall quality increase with a static file size in most cases, since I haven’t heard of any negative effects). It also doesn’t add much time to have enabled, so I’d leave it on. Should have a larger effect on film than it does on animation.
Pyramidal B-Frames: Yes
More quality, or a smaller bitrate. Adds time to the encode.
Motion Estimation Method: Uneven Multi-Hexagon (umh)
UMH will typically give the best quality. Exhaustive would take much longer, but probably wouldn’t give any benefit whatsoever.
Motion Estimation Range: 64
Higher is better, but takes longer to encode. More important for high-motion stuff.
Subpixel Motion Estimation: 9
Again, higher is better. Rather than just being “levels of quality”, each number turns on different features, or applies features to different/more frame types.
About 10-15% longer encodes, about 0.1db improvement in signal to noise in high-motion stuff, and does almost nothing in low-motion stuff. Despite the rather small benefit, I use it anyway.
8×8 DCT: Yes
Simply adds another block size to the Analysis. If you have Analysis already set to “all”, you may as well choose this also.
Deblocking: 0 0
There’s no “best” setting for this, so the default is probably fine for most. The first number is “strength”, the second is “threshold”. Think of it this way – the first is how much to blur, and the second is more like “when” to blur. Negative numbers will blur less and give a sharper image. Positive numbers will give more blur. If you’ve chosen an insanely low bitrate/filesize, or if the image looks too sharp or quite blocky, you may want to increase the numbers slightly to add some blur to help cover up any artifacting/blocking. If you’ve gone with a healthy bitrate and your encode is more blurry than the source, you can try going into the minuses. It’s generally accepted that going beyond +/- 3 won’t be terribly good. Really, I’d leave it at 0/0 unless you’re unhappy with the result.
2 uses Trellis more often than 1, will take longer to encode, and may increase the quality a bit.
No Fast P-Skip: Yes
Selecting this makes it take a little longer to encode, but can reduce blockiness. Leave it checked.
No DCT-Decimate: No
DCT-Decimate is usually a good thing, so leave this box unselected. It saves space where it thinks you won’t notice, which means more space is there to increase quality elsewhere (where you will notice it).
CABAC Entropy Coding: Yes
Better compression at the same quality. No Fast P-Skip also needs this to work. Playback requires a more work though, so old computers and some hardware players may choke. Keep it enabled unless you’re going to play the video on something that can’t cope.