Update: If you’re using Handbrake 0.9.9 and would rather use the new “x264 Presets”, I have an updated guide for Handbrake 0.9.9 which focuses on that. However, if you’re sticking with the “Advanced Settings” or want more in-depth information on the Audio Settings, Picture Settings, etc (which I did not repeat in the new guide), keep reading….

 

The old best H.264 / x264 settings for Handbrake wasn’t very complete, and became outdated. Here I’ll try to take a more thorough peak at settings here to help you get the best x264 / Handbrake settings possible for your encode. Warning: the previous write-up was short. This one is insanely long and detailed. If you find yourself overwhelmed (don’t feel bad, there’s a lot to take in), muddle through in Handbrake the best you can and just use this as a reference when you get to an option you’re unsure about.

The focus here is on getting the best quality using x264 through Handbrake, even if it takes a long time to encode (within reason on a modern processor anyway).

Heaven help you if you’re using an old Athlon…

I’ll start with the advanced settings and work backwards.

 

Advanced Settings

Reference Frames: max possible

This goes up to 16. Unfortunately, 16 will break playback on a lot of hardware devices (and possibly some software players). So if you’re using say… an iPad3, WDTV, Roku, AppleTV3, etc, you’re going to have to find the max for that player. The easiest way is to see what “Profile” your device supports.

I’ll use an example. The iPad3. The specs page  lists “Profile level 4.1″ under the “TV and Video” section (right now anyway – that page might change whenever the iPad4 comes out). So we don’t want to go higher than what “Level 4.1″ supports if we want the video to play back on the iPad3.

Next we take that “Level 4.1″ to the MPEG4 page on Wikipedia where we’ll see a bunch of cryptic stuff under the “Level 4.1″ part. When it comes to Reference Frames, we need to look at the following in the last column:
1280×720@68.3 (9)
1920×1080@30.1 (4)
2048×1024@30.0 (4)
The stuff in brackets (max stored frames) means Reference Frames.

A typical DVD will fit within the 1280×720 confines and can use up to 9 reference frames*.

For a 1080p BluRay, up to 4 reference frames.

If we go above this, the iPad 3 (using Level 4.1) might choke when trying to play it back.

*For the sake of simplicity, I’m being conservative here mentioning 9 ref frames for a DVD. If you look at a lower level such as Level 3.1 you’ll notice that it actually lists a DVD’s resolution of 720×480, and will handle 13 reference frames. So you might be able to push 13 reference frames (or more) at Level 4.1 on a DVD. Suffice it to say, finding the true max ref frames at your particular resolution and framerate can get very tricky – the wikipedia page just lists a few examples for each Level. If you’re very hard-core about finding the true max ref frames for your source, you can try to math-it-out or simply do some trial-and-error.

So look at your device’s specs to find the “Profile Level” and then match it up with the Wikipedia link above to find out the max reference frames you can use. Remember, this depends on the resolution and as we saw in the above example, a DVD can get away with 9 reference frames on the iPad 3, while a BluRay can only get away with 4 ref frames.

If you’re just playing your content on a decent computer (through VLC for example), feel free to crank it all the way up to 16. Just remember that it probably won’t play on many (if any) current hardware devices. And a really slow HTPC might even choke.

Note that high values will take noticably longer to encode, even on a fast machine. If 16 makes your encode take way too long, feel free to bump down to something like 8 (the default for the “slower” preset in x264).

 

Maximum B-Frames: 16

16 is considered a “placebo” setting in x264. That said, it doesn’t add a ridiculous amount of time to the encode.

This just tells the encoder how many consecutive bframes it can use. Chances are that the encoder isn’t going to find a situation where it makes sense to actually *use* 16 consecutive bframes. If your encode times are are really hurting, 8 might be more reasonable.

 

CABAC Entropy Encoding, 8×8 Transform, Weighted P-Frames: yes (checked)

They’re all different, but just about everything relatively-recent supports these. If you’re trying to play the video on a really old or really weak device (where many of the other settings I’m listing will probably be problematic too), you might have to turn them off.

CABAC simply uses better compression for certain data in the encode.
8×8 Tranform lets the encoder do another nifty technical thing with i-frames.
Weighted P-frames tends to improve compression when fades take place.

 

Pyramidal B-Frames: Default (Normal)

If you’re creating an actual BluRay disc from your encode, use “Strict”. Otherwise, most recent devices support it (if you’re using something old/weak, you may need to turn it off).

B-Frames are great. This makes them even better. This allows B-Frames to be used as references for other frames (like other B-Frames!).

 

Adaptive B-Frames: Optimal

We’re aiming to pump as much quality into a filesize, so set this to Optimal.

From what I gather, if you set this to “none”, B-Frames would be put into a simplistic “cookie cutter” order, which isn’t likely to be ideal. Both “fast” and “optimal” allow the encoder to use them effectively/efficiently, with optimal doing a better job.

 

Adaptive Direct Mode: Auto

This has to do with “motion vector prediction”. I assume that spacial looks at individual frames whereas temporal looks across multiple frames. Spacial is supposed to be the most useful, but because that isn’t *always* the case, using Auto is the best way to go so that the encoder can use whatever is most useful.

 

Motion Estimation Method: Uneven Multi-Hexagon (umh)

UMH is about the highest you can go without running into insane encode times.

The encoder uses certain motion estimation patterns (do some googling if you want to know what each looks like). If you go beyond this to exhaustive (or transformed exhaustive – considered the “placebo” setting), it becomes something of a dumb brute force search.

If you’re intent on going with Exhaustive or Transformed Exhaustive, your encode even on a fast machine might take days, so consider if it’s really worth it for the “placebo” effect.

 

Subpixel ME & Mode Decision: 10 QPRD in all frames

Suffice it to say, higher is better, but 11 would take a good bit longer and is considered a “placebo” setting.

 

Motion Estimation Range: 24-64

Despite 64 being considered far beyond even the “placebo” setting, my DVD encodes have had pretty reasonable encode times (under an hour for a 44-min episode), which is the only reason I’ve allowed it to stay so high. BluRay encodes on the other hand…. those approach the better part of a day at a setting of 64.

Something in the range of 16-24 is probably more sensible.

This is the maximum range in pixels that the encoder will search for motion. Higher settings should be more valuable at higher resolutions (since a hand moving on a BluRay travels more pixels than a hand moving on DVD, despite travelling the same distance on your screen). Obviously, a high-motion video will tend to be more sensitive to this setting than a low-motion video.

 

Partition Type: All

As I understand it, the “most” (default) setting enables 4 primary partition types, and “all” enables a 5th, which is supposed to have negligable usefulness and slows down the encode. The “some” setting only enables the 2 most critical/beneficial ones.

If you’re using a modern 2011/2012 processor, I’d be inclined to set it to “all”. I myself haven’t noticed adverse effects on encode times (and I used “all” 3 years ago on older systems too), so if it might help sometimes, may as well keep it.

 

Trellis: Always

It’s required by the Subpixel ME & Mode Decision setting of 10 that was used earlier. If you’re doing 2-pass encodes and are finding the first to be painfully slow, you could consider setting it to “Encode Only”

 

Deblocking:
-1, -1 for film (tv/movies) and 3d animation (Pixar stuff)
1, 1 for animation (Bambi, etc)
-2, -2 for high levels of film grain

Note that if you’re trying to tune to retain film/grain, you should read over additional manual settings at http://forum.handbrake.fr/viewtopic.php?f=6&t=19426

The above are typical guidelines from x264′s “tune” settings. Unsurprisingly, this is designed to reduce “blocking” in your final video. It comes at the expense of “smoothing” the overall image though.

Generally, go with the above suggestions. If you’re seeing “blocking” in your video that didn’t exist in the source, you’re better off either increasing the RF setting (or average bitrate) first and/or playing with the AQ/Trellis settings I mention next.

 

Adaptive Quantization Strength: (slider)
1.0 (Default) for tv/film and 3d animation (Pixar stuff)
0.6 for animation (Mikey Mouse, Bambi, etc)
0.5 for high levels of film grain (you’ll need a high bitrate or quality setting)

These are x264 “tune” defaults and are good guidelines. The easiest way to explain this is that higher values tend to take some bits away from highly-detailed areas and put them in lower-detailed (flatter) areas. I’ve seen it explained as ringing (high AQ strength) vs banding (low AQ strength), though even that depends on the source.

From what I’ve tested, it doesn’t seem to have much visual impact for low-bitrate film encodes although it can affect the file size. Unless you’re willing to do your own testing/research, I’d recommend you use the recommended settings for a “one size fits all”.

 

Psychovisual Rate Distortion: (slider)
1.0 (Default) for tv/film and 3d animation (Pixar Stuff)
0.4 for animation (Mikey Mouse, Bambi, etc)

Again, x264 “tune” defaults, and good guidelines. This bears similarities to the above “AQ” setting in that it moves around bits. Higher values try to maintain “detail” the way you’d see it across the scene, at the expense of overall quality.

 

Psychovisual Trellis: (slider)
0.15 for tv/film and 3d animation (Pixar Stuff)
0 (default) for animation (Mikey Mouse, Bambi, etc)
0.25 for high levels of film grain

Tied into the Trellis setting above. The defaults are good guidelines.

 

Other goodies:

There’s a slew of custom settings at http://mewiki.project357.com/wiki/X264_Settings you can add in the text box.
You may have to modify some of them slightly to suit Handbrake. Settings in the text-box tend to be in the format settinga=1:settingb=5:settingc=32:etc with a colon (:) separating settings. By glancing at the text box in handbrake, you can probably figure out the format.

rc-lookahead – If you click the “High Profile” preset in Handbrake, you’ll see this pop up in the text box (there’s no option box or slider). There are a couple systems in the encoder that make use of this, and higher is generally better (max is 250). That said, rc-lookahead=60 is about the max recommended (even the “placebo” setting doesn’t go higher).

A couple reasons for this… First, it gobbles up RAM. If you’ve got 8GB+ of RAM and find yourself wishing that Handbrake would utilize more, this will do it. By the same token, if you’re on a 32-bit system or are on 2GB or 4GB of RAM, I’d imagine increasing this too much will probably crash the encoder.

The next reason is that huge values don’t really have much benefit – large values take quite a bit longer for the encode, and file size / quality don’t seem to change. I tend to set it to around 120, but I can’t say I’ve noticed any benefit.

nr – this isn’t touched/referenced by handbrake, but you can use it by manually adding nr=100 to the text box. It’s a denoiser built into x264. The good news is that it’s built in, motion-compensated, is minimal in the way of negative side-effects, will reduce your filesize slightly, and it’s fast. The bad news is that it’s not terribly good (the x264 devs are open about this) – from what I gather they only added it because they already had motion-compensated data in the encoder so it was easy to implement.

I used nr=100 and values up to nr=1000 in an encode with a very noisy/grainy source. There wasn’t a massive difference between 100 and 1000 (though quality didn’t really suffer either, and filesize went down).

Surprisingly, it’s alright at the 1 thing the denoiser in Handbrake is bad at – noise in the source during motion. See, the denoiser included with handbrake (hqdn3d) is good at temporal denoising during static scenes, but rather poor at addressing noise during motion – normally you end up having to bump up the spacial denoiser to clean that up, but at huge expense to your image.

However, the x264 denoiser does alright when catching noise during motion, which complements the hqdn3d denoiser in Handbrake quite well. I’d imagine it still pales in comparison to other motion-compensated denoisers, but it’s better than nothing, and as I said, doesn’t seem to hurt the rest of the image much (if it does at all… I really couldn’t tell – I’d have to try it on a clean source to see if I notice anything).

There are a number of other x264 settings you can manually plug into Handbrake too, but the 2 above are the only I’ve played with.

 

Chapters:

By default, if you load a source that includes chapters, they’ll be included here, often named Chapter 1, Chapter 2, etc. Not very descriptive.

Here, you can edit the chapter names.

In DVD/BluRay TV shows and movies, some have a “select scene” feature in the menu that lists all the scene names and lets you jump to a scene (which is actually jumping to a chapter). If you’d like, you can manually enter these chapter names in Handbrake. Alternately, you can visit a site like http://chapterdb.org/ and see if your movie is listed (and has actual names instead of chapters), and manually copy/paste that way. It’s also possible to import a CSV with Chapter names if you happen to obtain one.

Handbrake does NOT support editing anything other than the names though. So if your video has no chapters (or you want to add/remove some), you have to use a different program to do this.

That said, in the upper-section of Handbrake, you can choose which Chapters to encode. Sometimes a DVD/BluRay will have the last chapter be where the credits start. If you don’t want to encode the credits, check the DVD/BluRay in a video player (jump to the last chapter), and if it’s all credits, feel free to leave them out of your encode.

Note that credits don’t take a *lot* of bitrate, but in the case of something like a “Lord of the Rings” encode, they do take a long time (and cutting them out can save a good bit of filesize). In other cases, cutting them is just convenient – I’m not a fan of seeing constant credits when I watch 2-3 episodes of a show back-to-back.

 

Subtitles:

If Handbrake found subtitle tracks in your source, you can add them here (as many as you want). The “default” selection will make it the default track that plays when the video does (if there’s no default, none will play automatically unless you turn them on).

However, not all devices will play subtitle tracks. This can be particularly frustrating when you have “foreign audio” or “forced” subtitles. For example, when Jabba the Hutt speaks in Star Wars or somebody speaks Russian in a TV show. In these cases, you can “burn in” a track so that it’s now part of the picture.

But how do you know which track has the “foreign audio” subtitles? Handbrake makes it easy – select “Foreign Audio Search” along with “Forced Only” and “Burned In”. This will search subtitle tracks for anything that plays 10% or less of the time (which is usually the forced subtitles). If it finds anything meeting this criteria, it’ll burn it right into the picture so that it’s always there no matter what device it’s playing on. The only downside is that if you add regular subtitles too, they’ll overlap while the video plays. So if you’re burning in “forced subs”, you may want to consider leaving out any others.

The benefit to “Foreign Audio Search” is that if you’re not sure whether your video had a scene or two with some foreign audio, you can turn this on anyway – if there was, handbrake will find it. If there wasn’t any, nothing will be added.

Note that in some sources, subtitles are already burned-in to the picture. A good example is the US-region BluRay for Lord of the Rings (Extended Edition). So don’t panic if you don’t see a separate subtitle track in your video… it might already be burned in.

 

Audio:

Handbrake will list the audio tracks from your source and automatically add the one it believes is the main one (usually English with the most channels). You can add more tracks if you’d like – often there will be a Main 5.1ch of sorts, a Main 2.0 channel, perhaps a few 2-channel with director’s commentary, other languages, etc.

For the audio codec, most devices will play either AAC or MP3. AAC is generally considered to be superior in quality nowadays, although most people won’t notice the difference.

For AAC, the “CoreAudio AAC” codec is generally considered to be the best. However, it’s limited to those encoding on the Mac (it will play on anything that supports AAC though, including Windows machines). If you’re stuck on Windows, the “faac AAC” is the best option here, unless you’re willing to use something like MeGUI instead of Handbrake (which supports a Nero AAC encoder which is considered as good as CoreAudio)

Dolby Pro Logic II is a common mixdown option – it’ll play on anything in 2-channel, and is alright when it comes to “faking it” if you play it on say.. a home theater system with multiple channels, though you want to read further for “passthru” details which you may prefer.

Samplerate – you can generally leave it at “Auto”. If you have a reason for specifying something specific (like 44.1), go ahead and do so, but note that by manually setting it, you risk low-samplerate audio being needlessly upsampled, and vice-versa.

Bitrate – 160 is common. 128 is often considered too low (a number of people can tell the difference between 128 and 160). You could push higher, but some devices aren’t rated for it. It’s very tough to distinguish bitrates above 160, and insanely difficult to distinguish anything above 192. Note that if you’re adding “commentary” tracks, it can be a good idea to drop the bitrate of those simply so you don’t waste space – the quality of a couple people talking isn’t as critical as the main movie track, you don’t need a high bitrate to get good quality of simple voice anyway, and they often aren’t listened to frequently.

There are also “Passthru” options – these are a little riskier to use since it will depend on device support to play them. For example, you can passthrough AC3, or DTS, but if your player doesn’t support it, you might be out of luck. In addition, these don’t compress the audio stream, so passthrough is a terrible idea if you’re aiming for low-filesize encodes. If you play your content extensively through a 5.1 channel Home Theater system though, passthrough may be your best option.

 

Video:

For the codec, x264 (hopefully you didn’t read this far without committing to it!).

For the framerate, “Same As Source” (Variable Framerate) is generally best unless you have a very good reason to target a specific rate. This goes double if you do anything like detelecing the source. Really, “same as source” is awesome. Setting something manually just runs the risk of creating issues.

Constant Quality vs Avg Bitrate - Both work in a similar fashion. They both vary the bitrate throughout the encode – using more bits when necessary, and less bits when they’re not needed.

The difference is that Constant Quality targets a certain level of quality whereas Average Bitrate essentially targets a file-size.

Note that “compression-type” improvements in the advanced settings tend to affect these in different ways. With Constant Quality, higher settings (like going from 1 to 5 reference frames) will tend to maintain the quality level you set, but decrease the file size. With an Average Bitrate, an improvement (like going from 1 to 5 reference frames) will tend to maintain the file size you chose, but increase the quality.

A great example of a benefit of CQ here is if you’re encoding a TV series with Constant Quality. Episode 10 will look as good as Episode 1, even if Episode 10 had a zillion explosions, lots of movement, etc. Sure, Episode 10 might end up being a bit larger, but Episode 15 which had people standing around talking for most of the episode was probably a bit smaller.

On the other hand, had you used an Avg Bitrate instead, Episodes 1, 10, and 15 would probably be roughly the same file size. Episode 1 might look good, but Episode 10 might look terrible. Episode 15 might look the same as Episode 1, but could be larger than it otherwise would have been if you’d used Constant Quality instead.

Really, the only tangible benefit to Avg Bitrate is if you’re targetting a specific filesize. If you absolutely require all your shows to be 350mb so that you can fit 2 per CD, pulling out a calculator and calculating the Average Bitrate is the only way you’re going to be able to do it easily. And because Average Bitrate needs 2-pass to perform best (since it has to figure out how exactly it should pack all the bits optimally into the file-size bottle), you’re also spending more time during the encode – had you gone with Constant Quality (which doesn’t need a 2nd pass), you would have saved time – time you could have put into a higher quality advanced setting if nothing else.

Constant Quality does look more confusing on the surface at first. You have an RF setting, and if you’re new to this, you probably have no idea what it means. To start, lower settings here mean higher quality. RF:16 will be much better quality (and larger in size) than RF:24 for example.

As a basic guideline, RF:20 is a common starting point for DVD encodes and RF:22 for BluRay encodes. If you’re a real stickler for quality, 16-18 for a DVD might be more ideal (or even lower numbers if you’re determined to make it indistinguishable). On the other hand, if you’re looking for really low filesizes and are willing to accept that it won’t look as good as the source, 22-25 can work for DVD’s.

After a few encodes with Constant Quality, you should have a good idea as to what setting you prefer, whether you’re looking for high-quality, or are looking for low filesizes.

If you do need Avg Bitrate though, go with 2-pass. The encoder really needs that 1st pass to learn which scenes are complex (and will need more bits), and which scenes are simple (and will need fewer bits) before it actually goes and starts allocating things.

Think of it this way. If you’re hosting a party and have 350mb of pizza to go around (i know pizza isn’t measured in MB – bear with me), it’s good to know how many people are arriving beforehand, and ideally how many are kids (small eaters) and how many are adults (big eaters). 2-pass is like that – it does some research beforehand (during the 1st pass) so that it can cut up the pizza properly so that everybody is fed (during the 2nd pass). Without that research, you might start handing out too many portions to people at the beginning and run out when the adults arrive later.

Ok, not the greatest analogy (better than the “milk” one I was going to use mind you), but you get the idea. 2-pass is good if you’re going Average Bitrate. If encode time is an issue, you can use “Turbo first pass” – it does less thorough research beforehand, but is still much better than none.

 

Video Filters

(Recommended)
Detelecine: Default
Decomb: Default
Deinterlace: Off
Deniose: depends
Deblock: Off (usually)

Handbrake is beautiful in this area.

For both Detelecine and Decomb, at the default settings Handbrake looks for telecined and interlaced content, and then detelecines/deinterlaces as necessary (and only IF it detects these things). That means you can set-and-forget. If you have a really weird issue with an encode you can go and turn these on manually, but 99% of the time there’s no need to. Set them to default and let Handbrake do the work of determining whether they’re needed.

 

Denoise…. I’ve gone into details in a long post here. UPDATE: A new, better post with images/video here. The short version is that If you have noise (or film grain) in your SOURCE that you want to get rid of, start with the “Weak” setting. It will “smooth” your image some, but will usually at least reduce the noise/film. If “Weak” isn’t enough, I strongly suggest reading the write-up and using custom settings because “medium” and “strong” tend to soften the image too much, making people look like Ken & Barbie dolls.

It’s worth noting that because denoise smooths out detail, it also reduces the filesize (or increases quality) of your encode. If you’re ok with the results of the “Weak” filter, it’s a great trick to bump down the file size of your encode rather substantially. However, you do run the risk of over-smoothing or banding so don’t be overzealous.

 

Deblock…. this is supposed to remove “blocking” that exists in your SOURCE. It really smoothes out video to the point where it probably causes more damage than it fixes (I can’t stand it). If your source has heavy blocking for some reason, it may be worth the trade-off to use it.

Note that the Deblocker in handbrake has a similar effect to high de-noise settings. Because it drastically smoothes, it tends to destroy grain and other noise. That means you could effectively use it as a denoiser. In some cases it will do a better job denoising than the denoiser.

 

Picture

I’ll assume you’re not trying to resize the video (if you are, I’m sure you’re capable of figuring out how).

Anamorphic is almost always used, since it acts the way a DVD would, and plays well with both computer software players and various devices. Reading up on the Handbrake site is probably the easiest way to go here for further details on what it means.

It used to be that Anamorphic LOOSE was recommended because it makes everything divisible by 16 for the encoder. However, from what I gather the encoder has improved, and Anamorphic STRICT is generally the preferred way to go now (since loose might do some slight resizing which we usually don’t want).

 

Cropping - Automatic is usually fine. Handbrake basically looks for (and then chops off) any black bars in the source. Nearly every player will add it’s own black bars (since the video is set to Anamorphic), so encoding them is senseless.

You can of course manually set these if you have a good reason to.

 

 

Well, that’s that. All on one (really long) page, so you’re not exposed to a zillion ads in a multi-page article.

I should end by noting that this may not all be 100% correct. I’ve gathered the information from as many sources as possible, but chances are I read something I shouldn’t have taken at face value.

So take it with a grain of salt, and do your own tests before you spend a week encoding a TV series only to find that it all went horribly wrong because I made a mistake somewhere :p

 

Good luck!