mattgadient.com

Fighting noise/grain in handbrake – custom denoise settings

Update: I kinda felt like a jerk for telling everyone “oh, just do all these tests” to see various settings in action further down. So I put together a new write-up with a video and pictures which you can click here to visit. Hopefully you guys find it to be a little more helpful.

 

If you’ve ever tried ripping a DVD with noisy source material to small (low-bitrate) files in Handbrake, you probably realized quickly that you just opened up a big bag of hurt.

If you used constant quality and went with higher RF values, you probably ended up with a larger-than-expected file that looked ugly anyway. If you went with an average bitrate instead, your file size was probably fine, but it may have looked so bad that you wondered if maybe you forgot to enter a digit.

 

In any case, noise/grain is the bane of small x264 encodes. As far as the encoder can tell, it’s extra “detail” that it should try hard to keep – and it takes a sick amount of bitrate to try and do it. Not as big a problem if you’re fine with good quality settings and the huge file sizes necessary to store the grain/noise, but a massive problem the rest of the time – grain becomes small ugly blocks, and the rest of the image suffers.

 

The good news is that handbrake includes a denoiser. The bad news is that the custom settings aren’t well documented.

If you’re ready to delve into the custom settings and are here simply to find out how, here are the details for each of the built-in presets to get you started:

Weak –  2:1:2:3
Medium –  3:2:2:3
Strong –  7:7:5:5

You can use those as starting points for some experimentation. Before you go entering something like 10:10:10:10 into the custom field though, it might be helpful to know what each value actually means. So here we go.

The format for the denoiser in Handbrake is actually ls : cs : lt : ct

We’ll look at the first 2 values first (ls:cs:__:__):

These first 2 values affect the denoising when looking at a single frame (“spacial”). So if you watch a video and pause it, the frame you’re looking at is essentially an example of what the denoiser looks at. Try this now. Pause your noisy video source (presumably a DVD – open it up in the player).

With the video paused, it might be tough to see the noise/grain. If that’s the case (if it looks pretty good), high values for LS/CS probably aren’t going to do you much good. The denoiser is essentially going to find all the bits of outstanding detail and blur/blend them – sure, it’ll probably get the noise/grain but it’ll grab detail from other areas in the process.

On the other hand, if noise/grain are very evident with the video paused, LS/CS have a good chance of cleaning it up. You have to keep in mind that the higher the values are, the more detail you will lose in the image. A thin strand of hair, a freckle, a drop of rain, and a piece of film grain can all look pretty similar to the de-noiser. It’s going to try to soften these out.

To give you and idea as to the function of the last 2 values (__:__:lt:ct)…

This is for the “temporal” part of the denoiser, which means it works across frames. It works very well for noise/grain that’s very evident while the video is actually playing (grain that you might not notice when paused but is evident during playback).

A simple way of thinking of it (not necessarily accurate, mind you) is that if you have say… a black surface during playback and a white “dot” shows up in 1 frame (then disappears again), this is something the denoiser is going to notice – when you’re looking at multiple frames, it stands out as something that shouldn’t be there. At the same time, if there’s actually *supposed* to be a white dot on that black surface, it’ll be there across all the frames. So by looking at multiple frames, the denoiser should have a good idea as to what’s supposed to be there, and what sticks out (and should be denoised).

Another (still not accurate) way of thinking about it is that it averages out the pixels between frames. So if you’re looking at a face where the skin color seems pixellated and seems to be moving around like crazy, it should even this out.

Temporal is particularly good for videos with what I like to call “dancing dots” or “dancing pixels”. For example, when a surface in the video like a wall (which should look pretty static) is grainy and that grain continually moves/dances around during playback (despite looking ok when paused). The temporal denoiser does a pretty good job of noticing this and denoising these areas while leaving the other stuff that looks fine alone.

As you can probably tell, temporal has the potential to be pretty fantastic, particularly when parts of the image look great while other areas look grainy during playback. Because it’s a little more “targetted” than the single-frame settings, you can usually get away with bumping this up a fair bit without as much consequence to the overall scene. That said, if you go too high you’ll start to see bad things like ghosting.

“l” vs “c”

This is where things can get a little more complicated. In ls/cs/lt/ct, the L means “luma” (brightness/darkness), and C means “chroma” (color). This applies both to the single-frame and temporal settings.

Figuring out which to turn up more is nothing short of a nightmare, will depend on your source, and will depend on the type of grain/noise/etc. The only sure way is to experiment with your source (which I’ll go into later).

It’s worth noting that if you go overboard on the luma, you’ll probably notice a loss of detail. If you go overboard on the chroma, you’ll probably end up with color banding.

I tend to keep LS:CS similar, and LT:CT similar. So something like 2:1:9:8 might be reasonable if I were focusing on temporal denoising (applying it to both the luma/chroma somewhat equally).

The Need to Test

The above should give you a rough idea as to what each setting does. However, the best way to learn is to do multiple test encodes and compare it to the source to get a feel for what everything does. You can certainly browse the web for examples of denoising, but your time is honestly better spent creating your own test encodes so that you can see first-hand all the effects and truly understand the effects (both good and bad). So do tests!

Since the denoiser in handbrake (hqdn3d) is hard to find in standalone DVD player software, the only way for you to see the effects of each setting is probably going to be running multiple test encodes in handbrake at crazy high settings (so that x264 preserves all the grain/noise left over by the filter for comparison instead of hiding/masking it during compression).

In Handbrake, go for constant quality and pump the RF to 10-18 for a DVD. Feel free to hit lower values if you want the quality even higher (though it shouldn’t be needed and your bitrate/filesize might get too big for your player). Don’t use Trellis or any other fancy settings (we don’t want x264 doing anything extra which might affect the noise/grain), and go with most of the defaults. Remember – high quality & fast – you don’t care about file size since these are tests you’ll eventually delete.

I’d recommend the following denoise settings for testing – it’s a lot to do, but remember that you’re doing it to get a feel for what each setting does:

(note: put your settings in the filename)

none – just to have something close to the source (hopefully indistinguishable) to make sure all is well. If it doesn’t look the same as your source, double check that you have the minimums on in advanced settings, and crank up the quality further if need be

handbrake presets – weak, medium, strong – just to see how a small change can make a big difference. Don’t be surprised if you don’t notice a difference at “weak”, but you should by the time you hit “strong”. Note that the temporal setting doesn’t actually change that much (in fact, medium/weak use the same temporal setting). The increased blurriness you see is largely from the spatial (single-frame) setting.

2:1:10:10 / 2:1:15:15 / 2:1:25:25 – do this to get a feel for the effects of “temporal”. By the time you hit 2:1:25:25, you’ll probably notice massive ghosting on any movement (and a lingering ghost in any fades to black). Compare these to the “weak” handbrake preset.

 15:15:2:3 – you’ll hopefully notice 2 things – first, it will be blurry as heck. Second, sections that had “dancing pixels” still dance. Compare these to the “weak” handbrake preset.

You could try values of 0 instead of 2:1 / 2:3 (and then compare to your “none” encode) – I wasn’t sure if 0 would work when I tested myself, so I used those values and compared against the “weak” settings.

If you have a lot of time, try cranking up *just* luminance or *just* chroma to see the effect each has on your source by itself.

The reason for using really high settings in some of the tests is that they’re pretty much guaranteed to have negative consequenes, and seeing these effects will help you know what negative-consequences to watch for when you’re tweaking. At the same time, you will see the limits of the denoiser.

If you’re lucky (well, technically unlucky….) your source had some noise/grain that was affected by temporal denoising, and noise that was affected by single-frame denoising. Open 2+ copies of your video player (or 2+ video players) to make it easier to compare side-by-side. Screenshots can help if grain is visible in a paused frame. Make sure to look at a few parts of the video – not just the most grainy parts. You want to make sure killing the noise/grain isn’t also killing other sections of your video.

Look through them all carefully and look for the differences between your test encodes – not only will you hopefully get a good understanding as to what type of noise/grain each is good at dealing with (and not good at dealing with) by seeing where it reduced the grain/noise, but you will also be able to start narrowing down the values you will want to use for your “real” encode.

Incidentally, these are the settings I tried when fighting noise/grain in the TV Show Alias (Season 3, Episode 7). If you happen to own the DVD’s, check out that episode (“Prelude”), paying particular attention to the scenes with Sydney/Sloane (both in the Chinese establishment and in the car later), Sydney fighting the Chinese guards, and Lauren’s car in the parking lot (and a section of pavement when she gets out of it). All have ridiculously grainy sections (I’m pretty sure they had 1 bad camera – you’ll notice this during camera swaps when Sydney is fighting the guards). Once you’ve checked those, look at most of the other scenes (which are generally good) to see what effect each setting had.

 

How far can you go?

According to http://www.aquilinestudios.org/avsfilters/spatiotemp.html#hqdn3d , the spacial values (ls & cs) shouldn’t go above 10, and the temporal values (lt & ct) shouldn’t go above 13. The settings should technically cap out at 255:255:255:255 at which point you’ll have no picture.

I myself tend to think that even 10 on the spacial is too high for TV/film – even the “strong” setting in Handbrake is only 7 and is quite blurry at that. It’s easy to start making people look like they’re made of plastic. Maybe higher is fine for animation, mind you.

13 seems like a realistic max for the temporal. I didn’t notice ghosting at 15, but something just didn’t “feel” right when watching. I can tell you with certainty that at 25 you’ll have massive ghosting that’ll make you want to puke.

Effects on Compression – my observations

Spacial (ls/cs – first 2 values) has a massive benefit to compression. Even if it doesn’t fix the grain (like “dancing grain” which it won’t fix), it blurs all the detail enough for pretty significant gains. Then again, you could run this filter on a perfect no-noise/no-grain source and you’d see compression gains due to the loss of detail. I was seeing 10-30% lower filesizes depending on the settings I used, but what you see will depend entirely on your source – if there’s lots of detail you’ll save more when it blurs it out. If there isn’t a lot of detail (ie animation), you probably won’t see a whole lot.

Temporal (lt/ct – last 2 values) varies. If you don’t have much grain/noise, it doesn’t do a lot, though you can get substantial benefits if there *is* a lot. Once the setting is high enough to remove the bulk of grain/noise, you won’t see much benefit to filesize by going further.

Combining them has the highest potential benefit to compression (unsurprisingly).

 

A not-so-simple Conclusion

When it comes to a TV show (like Alias), the single frame settings (first 2) really “blur” everything, and small changes here really have a large impact. It should be good for static noise & animation, but it has no benefit when it comes to “dancing noise” – you still get the “dancing”, it’s just more blurry.

The temporal settings (last 2) are fantastic for noise that wouldn’t necessarily look bad in a screenshot, but that you see when the video’s actually playing. It works incredibly well on grainy material and “dancing noise”. You can turn this up quite a bit too without as much consequence to the overall picture. It doesn’t work as well on motion though – from what I can tell, gradual motion like someone turning their head is affected (though not too badly). If the seting is turned up enough, fast motion is affected enough to notice it when you freeze the frame (though during playback fast motion tends to be forgiving. You also will inevitably hit a limit here too with ghosting.

I’ve found that for a TV-series, working on the temporal settings first is ideal. Check anything that may be inadvertently affected (rain, a cloud of dust, or other things that might look like artifacts to the denoiser), and make sure it’s having a positive effect on the noise/grain as you increase it. If you get to the point where you see ghosting, back off a good bit. Once your temporal’s at a point where you’re pleased, look at the remaining noise/grain and decide if it’s worth turning up the spacial (single frame) settings.

My final settings for Alias ended up sitting around the 2:1:7:7 mark, though I’m sure I’ll watch through a season and decide to tweak further. A high Temporal value does the bulk of the work cleaning up all the “dancing” and grain quite well, and a low Spacial value (equivilant to handbrake’s “weak” setting) rounds the rest out alright without losing enough detail for me to notice.

 

Other Options

Finally, it’s worth noting that there are better denoisers out there (though there are certainly worse ones too!). This one (hqdn3d) simply happens to be the one tied to Handbrake.

The advantage to using hqdn3d is that it’s easy, it doesn’t add much to the encode time, and it’s built right into Handbrake. As soon as you start delving into other denoisers, you start having to worry about using another program (often a full-blown editor), adjusting plugins, recoding/transcoding, worrying about subtitles, etc. It’s a lot of extra work for something like a TV-series.

However, if you’re working on something that’s rather important to you (like a home video or your favorite all time movie), you might want to consider looking at other denoisers if you’re willing to put in time in an editor – some are very capable & have very complex algorithms.

For most typical video though, simply tweaking the custom denoise settings in Handbrake is sufficient.

  • Bender

    This is fantastic info
    Thanks so much for the great explanation and tips!

  • Russ

    What about a black and white noise source, say Young Frankenstein. It looks like it was up sampled from a regular dvd. I understand a bit of grain would be an enticement for the “film” look, but geeze there are dancing ants everywhere. Would the C setting have any effects on black and white? since it is associated with “color”?

    • Russ: I wouldn’t expect C to have any effect. A black & white video should just contain luma information (I mean, it could have a chroma channel contained in the source, but I’d expect it to be rather empty/static). So just playing with L should hopefully do it.

      That said, the best way to find out for sure is to try. Start with a test encode with just luma denoising (I myself would probably go with 0:0:5:0 as a start point), and another with both luma & chroma denoising (0:0:5:5 as an example). See how they look, and go from there 🙂

  • Russ

    Hej Matt, Thanks for the info, I thought the same too, but this BR disc was soooo noisy, almost the worst I have ever seen. And such a classic movie. I found the parts that were the worst were with a lighter background, just horrible noise, but I found a good medium on temporal at around 13, it seems like a lot but I set the spacial at 3 ~4. I often wondered what other filters might be out there, that I could use with Avisynth on the PC side, I wish they could have a time line in Handbrake to set noise filtering at different settings during the encode. I’d pay for a commercial version of that. All in All Handbrake is a great tool for the experienced user.

  • Will

    Can you enter a custom value to ONLY denoise temporally? E.g., 0:0:7:7? I tried this once and my imac crashed so I am guessing zero is not a valid option, but, maybe it was a fluke. Not wanting to risk another crash, thought I’d ask you instead.

    • Will: I’ve done a few encodes with 0:0:x:x and they went fine. In rare cases I’ve had certain sources crash at a consistent location (just crash handbrake, never crashed the entire machine), but denoising hasn’t ever been the cause so far as I’ve been able to tell.

      • Will

        Thanks. Gave it another try and it worked with zeroes. Though I ultimately found 1:1:6:6 worked best for what I was converting (a grainy vhs source).

        I’ve been finding that 1:1:3:3 seems to be a good idea for pretty much every ts from digital tv. Live concerts and such – they all have a bit of compression artifacts, like the squiggles you see around edges of objects in JPGs, that go away with just a light touch. Thanks for this guide, it was very helpful since imo even the Weak setting is too strong.

  • girly7383

    Very helpful information! That saved me a lot of trial and error and precious disk space. Thank you!

  • Steve

    So helpful. Thank you very much. I’m beginning to get so anal with the quality of my DVD rips and this has given me some useful hints. For a lot of TV shows I found the weak de-noise way too strong and so settled on just using ‘deblock 5’ for the majority of my 80/90’s TV show rip’s – which is actually quite good. It’s less lossy than weak denoise – but still not perfect.

    I’ve found dark scenes to be the most troublesome with h.264. Do you have any advice for this?

    • Steve, if you’re referring to the dark “blotchiness” that seems to plague everyone from time to time, there’s rarely a simple one-size-fits-all solution. Sometimes it exists when watching the source too, but is covered up by grain in the source & thus isn’t apparent. A few things you can try:

      -Sometimes monitors/TV’s simply aren’t color-calibrated very well so you end up with a huge contrast between “shades of black”. Calibration can help if that’s the issue.
      -Selecting “no dct decimate” in the options sometimes helps.
      -Increasing the value of Psychodelic Rate Distortion can help too.
      -Retaining grain will often cover it up (though obviously that’s the opposite of what you usually want to happen when denoising).

  • Steve

    Thanks Matt. That is exactly what I was referring to. I’ll try out your suggestions.

    I’ve noticed even with high bit-rate rips, the darkest episodes will still come out tiny in comparison to more bright episodes. For example, I’ve been ripping my X-Files DVD’s and the darker episodes can come out at 300MB, and the normal ones around 800MB. This is with the same settings and the constant quality set to 17. The source material is certainly not great for the darker episodes, and you are right – the grain helps, but I am wondering whether it’s a limitation with h.264 because it seems such a big difference to me.

    Thanks again for your help. It’s very much appreciated.

  • Steve

    No dct decimate is like a magic switch. Thank you! When combined with de-block set to 5 it works great (at least on my trial rip’s!). I didn’t have any joy with de-noise – it just seems too strong. I tried 1:1:3:3 and 1:1:6:6 as suggested above, but the digital artefacts and loss of detail were not very bearable for me.

  • tom

    I’ve done tests on all three settings and I can say it’s better to leave DNR off completely. It creates an unnatural order to grain (blocking) and looks bad, even on weak setting. Not to mention it takes away detail. Grain shouldn’t be your enemy as long as it retains the original look of the film.

  • greg

    in my usage of ‘handbrake’ i routinely see ‘white flecks’ strewn across the image of movies that have been transcoded from DVD’s. this happens on literally each and every movie ive transcoded from DVD. it has NOTHING to do with the original camera, nor with the ‘grainyness’ of the scene (as the problem is also seen in ‘title sequences’ – which are typically NOT even shot on cameras, and which have no conventional ‘grainyness’ issues as would be found if a true camera were filming in low light). essentially these are individual pixels at random locations that are ‘white’ in color value.

    am i the only person experiencing this? is there a suggested fix for this specific situation?

    would be grateful for any suggestions that remove the ‘white random pixels’ that dont otherwise soften or blur the overall image.

    im open to ditching ‘handbrake’ if there is a more robust solution

    • Hey Greg,

      Sounds like some artifacting. I’d start by ruling out the video player – try playing the video on VLC, then in your OS’s standard player (Quicktime or Windows Media Player), and on another device (tablet/phone or bluray player connected to a TV). I’ve had artifacting that only presented itself in certain players, and that can help narrow down the cause.

      If it happens across all players/devices, try a couple quick encodes using the built-in Handbrake presets (even if they end up spitting out smaller files for an ipod touch, that’s fine). See if you get the same thing happening there – if not, you’ll have to do a little detective work to figure out which setting is causing it.

      If you *still* get the issue happening there, I’d try doing an encode on another computer, using the same settings you’ve been using – if the encode done on another computer looks fine, you may have some sort of hardware issue going on. Memtest86+ is probably the first thing I’d run in that case, possibly followed by the testing mode in Prime95. One symptom that would lead me towards testing for a hardware issue first would be if you get programs (or the computer) crashing on a somewhat regular basis – really odd things can happen when a bit randomly flips.

      Hopefully at least something there helps. Good luck.