Update: I kinda felt like a jerk for telling everyone “oh, just do all these tests” to see various settings in action further down. So I put together a new write-up with a video and pictures which you can click here to visit. Hopefully you guys find it to be a little more helpful.

If you’ve ever tried ripping a DVD with noisy source material to small (low-bitrate) files in Handbrake, you probably realized quickly that you just opened up a big bag of hurt.

If you used constant quality and went with higher RF values, you probably ended up with a larger-than-expected file that looked ugly anyway. If you went with an average bitrate instead, your file size was probably fine, but it may have looked so bad that you wondered if maybe you forgot to enter a digit.

In any case, noise/grain is the bane of small x264 encodes. As far as the encoder can tell, it’s extra “detail” that it should try hard to keep – and it takes a sick amount of bitrate to try and do it. Not as big a problem if you’re fine with good quality settings and the huge file sizes necessary to store the grain/noise, but a massive problem the rest of the time – grain becomes small ugly blocks, and the rest of the image suffers.

The good news is that handbrake includes a denoiser. The bad news is that the custom settings aren’t well documented.

If you’re ready to delve into the custom settings and are here simply to find out how, here are the details for each of the built-in presets to get you started:

Weak –  2:1:2:3
Medium –  3:2:2:3
Strong –  7:7:5:5

You can use those as starting points for some experimentation. Before you go entering something like 10:10:10:10 into the custom field though, it might be helpful to know what each value actually means. So here we go.

The format for the denoiser in Handbrake is actually ls : cs : lt : ct

We’ll look at the first 2 values first (ls:cs:__:__):

These first 2 values affect the denoising when looking at a single frame (“spacial”). So if you watch a video and pause it, the frame you’re looking at is essentially an example of what the denoiser looks at. Try this now. Pause your noisy video source (presumably a DVD – open it up in the player).

With the video paused, it might be tough to see the noise/grain. If that’s the case (if it looks pretty good), high values for LS/CS probably aren’t going to do you much good. The denoiser is essentially going to find all the bits of outstanding detail and blur/blend them – sure, it’ll probably get the noise/grain but it’ll grab detail from other areas in the process.

On the other hand, if noise/grain are very evident with the video paused, LS/CS have a good chance of cleaning it up. You have to keep in mind that the higher the values are, the more detail you will lose in the image. A thin strand of hair, a freckle, a drop of rain, and a piece of film grain can all look pretty similar to the de-noiser. It’s going to try to soften these out.

To give you and idea as to the function of the last 2 values (__:__:lt:ct)…

This is for the “temporal” part of the denoiser, which means it works across frames. It works very well for noise/grain that’s very evident while the video is actually playing (grain that you might not notice when paused but is evident during playback).

A simple way of thinking of it (not necessarily accurate, mind you) is that if you have say… a black surface during playback and a white “dot” shows up in 1 frame (then disappears again), this is something the denoiser is going to notice – when you’re looking at multiple frames, it stands out as something that shouldn’t be there. At the same time, if there’s actually *supposed* to be a white dot on that black surface, it’ll be there across all the frames. So by looking at multiple frames, the denoiser should have a good idea as to what’s supposed to be there, and what sticks out (and should be denoised).

Another (still not accurate) way of thinking about it is that it averages out the pixels between frames. So if you’re looking at a face where the skin color seems pixellated and seems to be moving around like crazy, it should even this out.

Temporal is particularly good for videos with what I like to call “dancing dots” or “dancing pixels”. For example, when a surface in the video like a wall (which should look pretty static) is grainy and that grain continually moves/dances around during playback (despite looking ok when paused). The temporal denoiser does a pretty good job of noticing this and denoising these areas while leaving the other stuff that looks fine alone.

As you can probably tell, temporal has the potential to be pretty fantastic, particularly when parts of the image look great while other areas look grainy during playback. Because it’s a little more “targetted” than the single-frame settings, you can usually get away with bumping this up a fair bit without as much consequence to the overall scene. That said, if you go too high you’ll start to see bad things like ghosting.

“l” vs “c”

This is where things can get a little more complicated. In ls/cs/lt/ct, the L means “luma” (brightness/darkness), and C means “chroma” (color). This applies both to the single-frame and temporal settings.

Figuring out which to turn up more is nothing short of a nightmare, will depend on your source, and will depend on the type of grain/noise/etc. The only sure way is to experiment with your source (which I’ll go into later).

It’s worth noting that if you go overboard on the luma, you’ll probably notice a loss of detail. If you go overboard on the chroma, you’ll probably end up with color banding.

I tend to keep LS:CS similar, and LT:CT similar. So something like 2:1:9:8 might be reasonable if I were focusing on temporal denoising (applying it to both the luma/chroma somewhat equally).

The Need to Test

The above should give you a rough idea as to what each setting does. However, the best way to learn is to do multiple test encodes and compare it to the source to get a feel for what everything does. You can certainly browse the web for examples of denoising, but your time is honestly better spent creating your own test encodes so that you can see first-hand all the effects and truly understand the effects (both good and bad). So do tests!

Since the denoiser in handbrake (hqdn3d) is hard to find in standalone DVD player software, the only way for you to see the effects of each setting is probably going to be running multiple test encodes in handbrake at crazy high settings (so that x264 preserves all the grain/noise left over by the filter for comparison instead of hiding/masking it during compression).

In Handbrake, go for constant quality and pump the RF to 10-18 for a DVD. Feel free to hit lower values if you want the quality even higher (though it shouldn’t be needed and your bitrate/filesize might get too big for your player). Don’t use Trellis or any other fancy settings (we don’t want x264 doing anything extra which might affect the noise/grain), and go with most of the defaults. Remember – high quality & fast – you don’t care about file size since these are tests you’ll eventually delete.

I’d recommend the following denoise settings for testing – it’s a lot to do, but remember that you’re doing it to get a feel for what each setting does:

(note: put your settings in the filename)

none – just to have something close to the source (hopefully indistinguishable) to make sure all is well. If it doesn’t look the same as your source, double check that you have the minimums on in advanced settings, and crank up the quality further if need be

handbrake presets – weak, medium, strong – just to see how a small change can make a big difference. Don’t be surprised if you don’t notice a difference at “weak”, but you should by the time you hit “strong”. Note that the temporal setting doesn’t actually change that much (in fact, medium/weak use the same temporal setting). The increased blurriness you see is largely from the spatial (single-frame) setting.

2:1:10:10 / 2:1:15:15 / 2:1:25:25 – do this to get a feel for the effects of “temporal”. By the time you hit 2:1:25:25, you’ll probably notice massive ghosting on any movement (and a lingering ghost in any fades to black). Compare these to the “weak” handbrake preset.

 15:15:2:3 – you’ll hopefully notice 2 things – first, it will be blurry as heck. Second, sections that had “dancing pixels” still dance. Compare these to the “weak” handbrake preset.

You could try values of 0 instead of 2:1 / 2:3 (and then compare to your “none” encode) – I wasn’t sure if 0 would work when I tested myself, so I used those values and compared against the “weak” settings.

If you have a lot of time, try cranking up *just* luminance or *just* chroma to see the effect each has on your source by itself.

The reason for using really high settings in some of the tests is that they’re pretty much guaranteed to have negative consequenes, and seeing these effects will help you know what negative-consequences to watch for when you’re tweaking. At the same time, you will see the limits of the denoiser.

If you’re lucky (well, technically unlucky….) your source had some noise/grain that was affected by temporal denoising, and noise that was affected by single-frame denoising. Open 2+ copies of your video player (or 2+ video players) to make it easier to compare side-by-side. Screenshots can help if grain is visible in a paused frame. Make sure to look at a few parts of the video – not just the most grainy parts. You want to make sure killing the noise/grain isn’t also killing other sections of your video.

Look through them all carefully and look for the differences between your test encodes – not only will you hopefully get a good understanding as to what type of noise/grain each is good at dealing with (and not good at dealing with) by seeing where it reduced the grain/noise, but you will also be able to start narrowing down the values you will want to use for your “real” encode.

Incidentally, these are the settings I tried when fighting noise/grain in the TV Show Alias (Season 3, Episode 7). If you happen to own the DVD’s, check out that episode (“Prelude”), paying particular attention to the scenes with Sydney/Sloane (both in the Chinese establishment and in the car later), Sydney fighting the Chinese guards, and Lauren’s car in the parking lot (and a section of pavement when she gets out of it). All have ridiculously grainy sections (I’m pretty sure they had 1 bad camera – you’ll notice this during camera swaps when Sydney is fighting the guards). Once you’ve checked those, look at most of the other scenes (which are generally good) to see what effect each setting had.

How far can you go?

According to http://www.aquilinestudios.org/avsfilters/spatiotemp.html#hqdn3d , the spacial values (ls & cs) shouldn’t go above 10, and the temporal values (lt & ct) shouldn’t go above 13. The settings should technically cap out at 255:255:255:255 at which point you’ll have no picture.

I myself tend to think that even 10 on the spacial is too high for TV/film – even the “strong” setting in Handbrake is only 7 and is quite blurry at that. It’s easy to start making people look like they’re made of plastic. Maybe higher is fine for animation, mind you.

13 seems like a realistic max for the temporal. I didn’t notice ghosting at 15, but something just didn’t “feel” right when watching. I can tell you with certainty that at 25 you’ll have massive ghosting that’ll make you want to puke.

Effects on Compression – my observations

Spacial (ls/cs – first 2 values) has a massive benefit to compression. Even if it doesn’t fix the grain (like “dancing grain” which it won’t fix), it blurs all the detail enough for pretty significant gains. Then again, you could run this filter on a perfect no-noise/no-grain source and you’d see compression gains due to the loss of detail. I was seeing 10-30% lower filesizes depending on the settings I used, but what you see will depend entirely on your source – if there’s lots of detail you’ll save more when it blurs it out. If there isn’t a lot of detail (ie animation), you probably won’t see a whole lot.

Temporal (lt/ct – last 2 values) varies. If you don’t have much grain/noise, it doesn’t do a lot, though you can get substantial benefits if there *is* a lot. Once the setting is high enough to remove the bulk of grain/noise, you won’t see much benefit to filesize by going further.

Combining them has the highest potential benefit to compression (unsurprisingly).

A not-so-simple Conclusion

When it comes to a TV show (like Alias), the single frame settings (first 2) really “blur” everything, and small changes here really have a large impact. It should be good for static noise & animation, but it has no benefit when it comes to “dancing noise” – you still get the “dancing”, it’s just more blurry.

The temporal settings (last 2) are fantastic for noise that wouldn’t necessarily look bad in a screenshot, but that you see when the video’s actually playing. It works incredibly well on grainy material and “dancing noise”. You can turn this up quite a bit too without as much consequence to the overall picture. It doesn’t work as well on motion though – from what I can tell, gradual motion like someone turning their head is affected (though not too badly). If the seting is turned up enough, fast motion is affected enough to notice it when you freeze the frame (though during playback fast motion tends to be forgiving. You also will inevitably hit a limit here too with ghosting.

I’ve found that for a TV-series, working on the temporal settings first is ideal. Check anything that may be inadvertently affected (rain, a cloud of dust, or other things that might look like artifacts to the denoiser), and make sure it’s having a positive effect on the noise/grain as you increase it. If you get to the point where you see ghosting, back off a good bit. Once your temporal’s at a point where you’re pleased, look at the remaining noise/grain and decide if it’s worth turning up the spacial (single frame) settings.

My final settings for Alias ended up sitting around the 2:1:7:7 mark, though I’m sure I’ll watch through a season and decide to tweak further. A high Temporal value does the bulk of the work cleaning up all the “dancing” and grain quite well, and a low Spacial value (equivalent to handbrake’s “weak” setting) rounds the rest out alright without losing enough detail for me to notice.

Other Options

Finally, it’s worth noting that there are better denoisers out there (though there are certainly worse ones too!). This one (hqdn3d) simply happens to be the one tied to Handbrake.

The advantage to using hqdn3d is that it’s easy, it doesn’t add much to the encode time, and it’s built right into Handbrake. As soon as you start delving into other denoisers, you start having to worry about using another program (often a full-blown editor), adjusting plugins, recoding/transcoding, worrying about subtitles, etc. It’s a lot of extra work for something like a TV-series.

However, if you’re working on something that’s rather important to you (like a home video or your favorite all time movie), you might want to consider looking at other denoisers if you’re willing to put in time in an editor – some are very capable & have very complex algorithms.

For most typical video though, simply tweaking the custom denoise settings in Handbrake is sufficient.