In-depth look at de-noising in Handbrake 0.9.9 (with image/video examples)

NOTE: This write-up was originally written for version 0.9.9. While it still applies to newer versions, formatting of custom settings has changed. The easiest thing to do is follow the guide as-is and read the updated section at the end to “convert” your desired settings to the format that newer Handbrake versions require.

 

When encoding your favourite DVD’s or BluRays, one of the issues you’re bound to run into at some point is a noisy source.

Now I’m using the term “noise” very loosely here – when I say “noise”, I’m basically referring to little dots on the screen that aren’t part of the actual picture. This can be the result of film grain, compression artifacts, a poor camera, etc. Before I start to sound like I’m trying to get everyone on the “noise is evil” bandwagon, note that sometimes film grain is digitally added to a movie during production to go for a certain look. In less common cases, digital film grain can be added to cover up other issues such as color banding in the master. Whether noise is good or bad often depends on it’s intent, and also depends on your own preferences – some people like it, others loath it.

 

That said, when it comes to the video encoder, noise is extra detail – detail that the encoder is going to try and keep, at least to some extent. And this is expensive detail to keep. Encoding a noisy source can easily double the bit-rate/file-size necessary at a given quality. And if you don’t give a noisy source enough bitrate, the overall quality of the picture tends to suffer heavily.

If you’re looking for small file sizes, or you’re not a fan of noise on a visual level, removing some of this noise can be helpful. Handbrake includes a denoiser, so here I’ll show you how to use it.

 

THE DENOISER

Handbrake uses a denoiser called “HQDN3D”. It’s not the best or most powerful denoiser out there, but it’s fairly quick and does a reasonable job as long as your source wasn’t a disaster and/or your expectations aren’t crazy.

It’s accessed in the “Picture Settings” section of Handbrake.

The default settings are:

  • Off
  • Weak
  • Medium
  • Strong
  • Custom

 

At some point you probably want to see what denoising actually does, so in the video below, I start by showing the effects of Weak/Medium/Strong before delving into the custom settings. I suggest you tinker with the buttons in the player to set this to 1080p and fullscreen (if your connection can’t handle 1080p, go as high as you can – otherwise you’ll have to rely on the pictures below, as lower resolutions on YouTube kill too much detail).

If you’re planning to de-noise, but are not interested in the Custom stuff and just want to use the presets, my suggestion is to start at “Weak”, because you start to get significant blur as you move towards “Medium” and “Strong”.

CUSTOM DENOISE SETTINGS

If you’re remotely serious about denoising, custom settings are the way to go.

 

Custom values are entered in the form of 0:0:0:0 . Using all zeroes would not do any denoising, so realistically you’d put some actual numbers in there. But before you start plunking in random numbers, to give you a little bit of a reference, here are what those presets above actually use:

WEAK: 2:1:2:3

MEDIUM: 3:2:2:3

STRONG: 7:7:5:5

Handbrake’s denoiser does 2 types of denoising. One is called “spacial” which is represented by the first 2 numbers. The other is called “temporal” which is represented by the last 2 numbers. We’ll look at both.

 

 

SPACIAL

 

The first 2 numbers have to do with “spacial” denoising. For example, 2:2:0:0 would only do spacial denoising.

The technical jargon has to do with looking at a single frame. To put it in an easy-to-understand way, the end effect is that the picture is softened, smoothed, or blurred (pick your favourite term). The noise is caught in the process, and softened/smoothed/blurred away.

I’ve used high spacial denoising below (higher than a rational person would normally use) to show what happens as this setting is pushed too far. Click for a larger image.

Screenshot that was not denoised
 
Example of high spacial denoising (15:15:0:0)

You can easily see the effects of spacial denoising when a resulting video is paused. Tiny changes in these numbers can have a drastic effect.

 

Spacial denoising is very powerful, but:

  1. it reduces the sharpness of your picture (unavoidable)
  2. it can make everyone look like Ken & Barbie dolls (worse at higher settings).
  3. a high enough setting will kill all the noise, but will kill all the other detail in the process.

Unless you really love a soft picture, be very conservative with the spacial settings.

 

 

TEMPORAL

The last 2 numbers have to do with “temporal” denoising. 0:0:4:4 would only do temporal denoising.

 

The technical jargon has to do with comparing sequential frames. To put it in an easy-to-understand way, the end effect is that all those little “dots” that keep bouncing around the screen are averaged out and tend to disappear (mostly).

Below, an example of the “original” compared to somewhat rational noise-reduction settings – the same that were included in the video (0:0:10:10). The 2nd has had the grain “calmed” somewhat, and there isn’t much loss of detail. Since these are 1080p images (around 1MB each) and won’t fit in a pop-up window, you may want to right-click and open-in-a-new-window.

Example without noise reduction
 
Example of medium temporal denoising.

You generally see the effects of temporal denoising when the resulting video is played back – those little dots that bounced around flat surfaces don’t bounce around as much anymore. The effects of temporal are usually hard to see when paused.

Below is an example of what happens when this is pushed too far (“original” vs custom settings of 0:0:25:25). It’s called “ghosting” and if you look at the larger version, you’ll see a “ghost” image. Obviously, settings of “25” are considered unreasonably high. Normally you don’t ever want to go higher than “10”. Click for a larger image.

Example of a motion scene (no denoising)
 
Example of motion scene with high temporal settings (0:0:25:25)

Temporal denoising is much more forgiving than spacial denoising, and you generally don’t lose any “sharpness” in the image, but it:

  1. is not guaranteed to get rid of all the noise, even at high settings
  2. will do almost nothing for you if your source is already “clean” (no dancing-dots/grain/noise/etc)
  3. can cause ghosting at high settings
  4. is not motion compensated – your lower motion scenes may look clean, but during high-motion scenes, all those little dots that went away tend to reappear in areas of the screen that contain motion.

While high settings inevitably cause ghosting, you can usually be fairly liberal with this setting up until the point of ghosting (which tends to start at around 0:0:15:15). Personally I wouldn’t usually go higher than 0:0:10:10 as things start to look “off” before you actually see the ghosting.

 

COMBINING SPACIAL AND TEMPORAL (2:2:6:6, 1:1:8:8, etc)

You may have noticed that none of the presets use “0’s”. They all do some degree of spacial AND temporal denoising. It’s fairly common to combine them to some degree.

Due to the type of noise I usually try to reduce (“dancing dots”), I typically push up the temporal denoising first, and then add a slight bit of spacial denoising if necessary. Depending on the noise *you* are targeting, your methodology may be a little different.

Note that these have various effects on file-sizes, and now’s as good a time as any to take a look at an example….

 

FILE SIZE COMPARISONS

I encoded the full 43m21s episode of “Alias” Season 1, Episode 7 at RF18, using the “veryslow” preset and various denoise settings. Here were the results:

 

Standard (no denoising) – 806.8 MB

Spacial (5:5:0:0) – 474.8 MB
Spacial (10:10:0:0) – 373.6 MB
Spacial (15:15:0:0) – 332.8 MB

Temporal (0:0:10:10) – 482.0 MB
Temporal (0:0:15:15) – 451.2 MB
Temporal (0:0:25:25) – 419.1 MB

Combined (3:2:9:7) – 422.3 MB

 

You’ll notice that:

  1. Nearly any denoising was helpful with this source. I didn’t use lower (sane) values, but even small settings here do help (trust me!).
  2. I used fairly high values but got pretty high gains too.
  3. Numbers used for the “combined” setting were smaller than any of the individual tests, yet the file size came out quite small. That’s where the value really is in combining spacial and temporal. If I needed the file to be about 420 MB for example, trying to do that with only spacial would result in a very blurry video. Trying to do that with only temporal would result in a very ghosty unwatchable video. On the other hand, by having *some* spacial and *some* temporal, I’d get a very watchable video.

As a reminder, if your source is already “clean”, temporal will do zilch for you. That includes file-size. If you try using temporal denoising on a “clean” source you might not even save 5%. It’s one of those things that works wonders when there’s noise, and seems useless when there isn’t.

Spatial on the other hand will *always* reduce the file size (though it will *always* blur to some extent too).

 

USING SIMILAR NUMBER PAIRS (2:2:0:0, 0:0:6:6, 1:1:4:4, etc):

 

This part gets slightly technical. If you don’t really care “why” we use similar number pairs, just use them and you’ll be fine (skip to the next section).

Each number pair has to do with luma and chroma. The format of the custom setting is actually broken down like this:

SpacialLuma : SpacialChroma : TemporalLuma : TemporalChroma

  • “Luma” is basically a black and white scale – how dark or how bright the pixel is. Back in the days of B&W TV’s, the television signal that contained the picture was all Luma.
  • “Chroma” is the color information, which is applied to the Luma to give you the picture.

 

Most of the time (with most types of noise), “Luma” does the brunt of the work when denoising. That’s because most types of noise affect at least the luma channel (sometimes only the Luma channel).

Chroma noise is more hit-and-miss. It’s rare to find noise that only affects chroma. Chroma noise is usually a little harder to see visually too.

 

So while using 2:0:7:0 wouldn’t be a crazy thing to do (and would probably clean up most sources nearly as well at 2:2:7:7), you shouldn’t ever find a reason to use the opposite like 0:2:0:7 unless something wonky happened to your source.

For a bit of a balance, it’s not uncommon to use similar values but with the luma slightly higher. So 2:1:7:6 might be used instead of 1:1:6:6 or 2:2:7:7. That said, minor tweaks like that aren’t likely to be very noticeable. So don’t spend too much time trying to figure out whether it’s advantageous to bump the luma higher or not for your current source.

CASUAL DENOISING – RECOMMENDATIONS

If the noise in the source doesn’t particularly *bother* you, and your goal is simply to clean up a little excess noise, or slightly bump down the file size with little-to-no degradation of the actual picture (a “freebie” size reduction), something like:

0:0:3:3

..is a nice safe custom setting. Your file size will drop slightly, you’ll have slightly less noise, but you probably won’t notice a lick of difference when watching.

————-

If file-size reduction is what you’re aiming for (or you’re trying to pack more quality in a fixed file size), try:

1:1:4:4

…your picture will be softened a little, but it’s so slight that you probably aren’t going to notice with most sources unless you’re doing a side-by-side comparison.

————-

If the source is visibly noisy or you’re hard-core about dropping down the file size, but are still hoping you won’t notice the difference try:

2:2:9:9 for a DVD

4:4:9:9 for a BluRay

…BluRays are a little more forgiving when it comes to softening via the spacial setting due to the increased resolution. That said, whether you notice the difference or not is going to depend on your source and…. well… you.

 

 

PITFALLS TO WATCH FOR:

The more aggressive you are with denoise settings, the more unintended consequences you’re going to run into. Every source is different, and thus, the pitfalls will be different.

 

Here are a few in particular to watch for:

  • The “Ken and Barbie doll” look – people look like plastic, or like they’re smothered in make-up (spacial settings too high)
  • Color banding – instead of say… a sky gradually going from light-blue to dark-blue, you have “bands” of colors” (the actual cause can be complicated, but high spacial OR temporal settings tend to exasperate this). Note that sometimes noise actually covers up *existing* banding in the source, in which case you’ll have to make a the choice between seeing noise or seeing banding. Other tools can “fix” banding via dithering, but Handbrake is not meant for that.
  • Ghosting – at the low end, tends to start looking like weird motion-blur. At the high end, you get an after-image during scene changes. If you inexplicably start to feel motion-sick while watching your encoded video, chances are this is why. Caused by high temporal settings.
  • Noise during motion – since the temporal denoising isn’t motion compensated, if the source was very noisy and temporal did a great job of cleaning it up, you’ll probably run into an issue where as soon as there’s motion, you get a few frames where there’s massive noise where the motion just took place. This sudden change can draw visual attention to the noise which is arguably worse than just having noise there all the time.

When experimenting with denoise settings to try and “fix” a problem area of the video, BE SURE TO CHECK OTHER PARTS OF THE VIDEO for unintended consequences. Making a noisy scene look perfect almost always comes at the expense of other scenes. Denoising is a balancing act. You usually have to accept that you’ll never remove all the noise without severe detriment to the rest of the video.

 

THE X264 NR SETTING

Finally, x264 includes a built-in denoiser which you can access with some custom settings in Handbrake.

The run-down:

  • it’s motion-compensated
  • negative side-effects are minimal
  • it can bump down file size pretty well
  • it’s usually not very good/helpful for most noise

To use it, you’ll have to add something like this in the “custom settings” box:

nr=400

(this would be for a setting of 400).

Manually setting x264's NR setting

Typically, values in the 100-1000 range would be used, with higher being stronger denoising. You could try higher/lower, but YMMV. If I understood it correctly back when I read up on it, it drops certain detail which is-probably-noise. Warning: I may not have understood correctly.

Personally, I’ve found it fairly useless on it’s own for most sources (aside from the file size benefit). However, combined with a custom denoise setting, I’ve found that it seems to slightly help some of the noise that remains during motion. That said, I haven’t done extensive testing with it and found it pretty hard to come to solid conclusions because the end result isn’t nearly as obvious as with the spacial/temporal stuff, so you might want to do your own testing and draw your own conclusions here. Again, your mileage may very.

UPDATE FOR NEWER VERSIONS (v0.10 or greater)

A setting such as 1:1:7:7 became 1:1:1:7:7:7 in v0.10. I’ll use some color to help point to where that extra digit comes from:

1:1:7:7 becomes 1:1:1:7:7:7

…or using different numbers…

1:2:8:9 becomes 1:2:2:8:9:9

If you recall the “luma” and “chroma” sections above, what has actually happened is the chroma has been split into chroma-blue and chroma-red. So where it used to be:
LUMA:CHROMA:LUMA:CHROMA
…it’s now:
LUMA:BCHROMA:RCHROMA:LUMA:BCHROMA:RCHROMA .

For 99% of us, fine tuning the red vs blue chroma channels isn’t going to be an endeavor worth pursuing, so the easiest way to convert old to new is by changing 1:2:3:4 to 1:2:2:3:4:4.

FURTHER UPDATE FOR VERSIONS NEWER THAN 0.10

Handbrake made yet another change so that you have to add text with your numbers.

1:1:1:4:4:4 becomes y-spatial=1:cb-spatial=1:cr-spatial=1:y-temporal=4:cb-temporal=4:cr-temporal=4

…or using different numbers…

1:2:3:4:5:6 becomes y-spatial=1:cb-spatial=2:cr-spatial=3:y-temporal=4:cb-temporal=5:cr-temporal=6

You *must* add this text now. To ensure it’s working, open the Activity Log in Handbrake during your encode and check the line called “Denoise (hqdn3d)”.

If it looks like this:

Denoise (hqdn3d) ()

… it did not work.

If it looks like this:

Denoise (hqdn3d) (y-spatial=1:cb-spatial=1:cr-spatial=1:y-temporal=4:cb-temporal=4:cr-temporal=4)

… it’s working.

Note that I never bother to remember the actual text – I copy/paste and fill in the numbers I want.

UPDATE 2 FOR NEWER VERSIONS: NLMeans Denoiser

A new de-noiser called NLMeans was added to Handbrake. Short version:

  • Presets in NLMeans are generally considered better than presets in HQDN3D. It’s also more capable in general
  • Many more preset combinations to choose from in NLMeans (4 presets with 6 tune options each)
  • Custom settings in NLMeans are considerably more complex.

NLMeans custom settings are complex enough that I won’t be putting together a specific guide for it here. If you want to get a sample, BradleyS put up a guide on the Handbrake forums at https://forum.handbrake.fr/viewtopic.php?f=4&t=30135&p=139077 which goes into all the details.

For those who don’t visit that link, to give you a brief notion of the complexity, custom settings in NLMeans look like this:

y-strength=y:y-origin-tune=y:y-patch-size=y:y-range=y:y-frame-count=y:y-prefilter=y:cb-strength=c:cb-origin-tune=c:cb-patch-size=c:cb-range=c:cb-frame-count=c:cb-prefilter=c:cr-strength=c:cr-origin-tune=c:cr-patch-size=c:cr-range=c:cr-frame-count=c:cr-prefilter=c

…some values have to be odd numbers, some between 0.00 and 1.00, others can trigger/combine between 1-8 different prefilters… you get the idea. Most people will be best off just using the presets.

So, hopefully this write-up is more helpful than the previous one that lacked images/video. If not, or if you have something to add, or if you notice that I’ve said something horribly wrong, feel free to leave a comment below!