How To Roto Brush In After Effects. Part 2: Advanced Settings (Illustrated Guide)

In this second part of how to use the roto brush tool to do rotoscoping in After Effects, we’ll dive deeper into the advanced settings of Roto Brush 2.0.

In Part 1 of the Roto Brush tutorial, we covered all the basics of how to roto basic shapes, refining the edge, and freezing the propagation. If you’re new to rotoscoping in After Effects, I suggest you read part one first.

Also, did you know that you can easily use the data from the Roto Brush to create a keyframed mask in After Effects? Here’s a breakdown.

Now let’s focus on the advanced features of Roto Brush 2. Here’s a breakdown of all the settings in 7 easy steps.

1. Version and Quality Settings

The first parameters you’ll see are Version and Quality.

Version:

If you’re using After Effects v17.5 and later, you’ll get the option to switch between these two versions:

• Roto Brush 1.0 (Classic)
• Roto Brush 2

Roto Brush 1.0 uses the older and lesser effective algorithm. And Roto Brush 2.0 uses the improved algorithm of v2.

Quality:

In Roto Brush 2.0 you can switch between:

Standard is faster and does a great job if there’s a lot of contrast between the foreground and the background and not a lot of fast movement or motion blur.

Best is slower but handles low contrast footage and movement and motion blur better. In general, it handles edges better.

2. The Propagation Settings

Changes you do under the propagation settings will require After Effects to recalculate all propagated frames. Your base frames are not affected.

Search Radius

The Search Radius is the size of the area Roto Brush searches for matching pixels between two adjacent frames.

If you have a lot of motion, try starting with a search radius of 50 or more

View Search Region

Enable the View Search Region to display the area as Roto Brush looks for your subject as a green box.

When you increase or decrease the Search Radius the box changes size. Anything outside the green box will not be included in the Roto Brush.

So make sure to make the box big enough to contain the range of the subject’s movements in the footage. But don’t make it so big so that it includes separate moving subjects/objects you don’t want to be included in the roto.

Enable Classic Controls

If you want to enable the classic propagation controls from Roto Brush 1.0 I’ve covered all the settings in this article.

Invert Foreground/Background

Invert Foreground/Background changes which strokes Roto Brush considers foreground strokes, and which are considered background strokes (i.e. the green and the red strokes you create on your base frames).

It’s a quick way to invert the roto brush matte and remove a moving subject while keeping the background.

Fine-tune Roto Brush Matte

Enable Fine-tune Roto Brush Matte for further options for fine-tuning the edges you’ve created with the Roto Brush Tool (as opposed to the matte you’ve created with the Refine Edge Tool, which we’ll address later)

If there are small areas of the background that have been included in your roto that you can’t get with strokes, you can try fine-tuning the roto brush matte instead.

Enabling Fine-tune Roto Brush Matte gives you more control over the Feather, Contrast, Shift Edge, and Reduce Chatter.

• Feather is the softness of the segmentation boundary.
• Contrast increases the contrast (useful for tightening up the boundary further).
• Shift Edge either expands (positive value) or decreases (negative value) the segmentation boundary of your roto.
• Reduce Chatter reduces visible noise along the edges of your roto. Chatter is seen when the edges of your roto move erratically. Reduce chatter reduces that noise and gives you cleaner edges.

What exactly is chatter in After Effects?

Reducing chatter is more memory intensive and is not always the best choice as you might miss important information near fingers or hair.

Fx, in the case of fingers, it’s like the roto outline becomes more “stiff” and prefers to treat all the fingers as one shape/hand.

If that is the case – dial back the Reduce Chatter and go in with fine background strokes roto brush and manually paint the space between each finger.

Also, if your roto subject isn’t moving, but the edges are jumping around erractically between frames, increase the Reduce Chatter Value. Likewise, if the edges aren’t keeping up with the movement of your subject, decrease the Reduce Chatter value.

The process is the same for hair – but here it’s sometimes more useful to draw background strokes with the Refine Edge Brush (not always though).

3. Render Refine Edge

Render Refine Edge makes it possible to see the refined edge of your roto brush. Render Refine Edge works for both base frames and propagated frames.

Use it to see if you’ve included all the details you want with the Refine Edge Tool.

4. Base Refine Edge Radius

Base Refine Edge Radius lets you expand the area of the refined edges you’ve already created with the Refine Edge Roto Brush tool.

Use it to expand the area of the refined edges when you’ve used a brush that is too small to help After Effects calculate the refined edges between frames.

5. Fine-tune Refine Edge Matte

Enable Fine-tune Refine Edge Matte for options for fine-tuning the edges you’ve created with the Refine Edge Tool.

The options are:

• Smoothness reduces the sharpness of the curves of the roto edges. Set it to low for things such as hair or fringes on clothes.
• Feather is the softness of the refined edge segmentation boundary.
• Contrast increases the contrast of the refine edge (useful for tightening up the boundary further).
• Shift Edge either expands (positive value) or decreases (negative value) the segmentation boundary of your roto.
• Chatter Reduction reduces visible noise along the edges of your roto. Choose between…
  • Off: no chatter reduction.
  • More detailed: the “standard” chatter reduction. This sometimes gives the best results for hair or fringes on clothes as the smoother option below misses things like stray hairs.
  • Smoother (Slower): smoothes out the edges further (compared to the more detailed option above). Use it to clamp down on jumping edges on things such as clothes or fingers you’ve created with the refine edge tool.

6. Use Motion Blur

If your subject has a lot of movement, you should enable Use Motion Blur. That way Roto Brush can account for blur around the edges due to the movement of your subject, the frame rate and shutter angle/speed.

To fine tune the motion blur the options are:

Samples Per Frame determines the number of samples the roto brush uses to simulate the motion blur. Increase the value for a more smooth-looking motion blur. Decrease for faster rendering.

Ideally, if your footage is shot with 25 frames per second with a shutter angle of 180 (1/50 shutter speed), you’d need 50 samples per frame for a naturally-looking motion blur. However, this would tremendously slow down the rendering process.

Shutter Angle simulates the shutter angle (or shutter speed) options found in cameras.

The option uses the frame rate of your footage as a basis for calculation. A shutter angle of 180 degrees is default and usually does a good job if you’ve used the same in your camera (or a shutter speed of double the frame rate).

A low parameter accounts for almost no motion blur, and a high parameter accounts for a lot of motion blur.

Higher Quality generates a cleaner edge (at the cost of processing speed).

7. Decontaminate Edge Colors

Enable Decontaminate Edge Colors to remove the background color from foreground edge pixels.

Use it to fix halos around your subject or background color from motion blur.

Fine tune it with these options:

Amount is the amount of decontamination is applied.

Extend Where Smoothed decontaminates edges that are moved due to the Reduce Chatter option.

Increase Radius increases the radius of decontamination including feather, motion blur, and extended (where smoothed) decontaminated pixels.

View Map shows the pixels cleaned by Decontamination as a white outline.

Got any questions?

Even though Roto Brush 2.0 is REALLY good, it can still only do so much and I still find myself having to draw masks manually from time to time.

Still it’s an amazing time saver, and I find myself only having to use masks to clean up small bits and pieces.

It’ll be exciting to see which improvements Adobe will come up with for the roto brush tool in the future – I expect AI to be implemented to some extend in a near future iteration.

Did I miss anything important? Or do you have any questions? Please let me know in the comments below.

---