How To Roto Brush In After Effects. Part 2: Advanced Settings (Illustrated Guide)

Published:
Updated:

In this second part of our tutorial on using the roto brush tool to do rotoscoping in After Effects, we’ll dive deeper into the advanced settings of Roto Brush 2.0.

In Part 1 of the Roto Brush tutorial, we covered all the basics of rotoscoping basic shapes, refining the edge, and freezing the propagation. If you’re new to rotoscoping in After Effects.

Now, let’s focus on the advanced features of Roto Brush 2. Here’s a breakdown of all the settings in 7 easy steps.

1. Version and Quality Settings

image 22

The first parameters you’ll see are Version and Quality.

Version:

If you’re using After Effects v17.5 and later, you’ll get the option to switch between these two versions:

  • Roto Brush 1.0 (Classic)
  • Roto Brush 2

Roto Brush 1.0 uses the older and less effective algorithm, while Roto Brush 2.0 uses the improved algorithm of v2.

Read more on How And Why To Enable Classic Controls In Roto Brush 2 In After Effects

Quality:

In Roto Brush 2.0, you can switch between:

Standard is faster and does a great job if there’s a lot of contrast between the foreground and the background and not a lot of fast movement or motion blur.

Best is slower but handles low-contrast footage, movement, and motion blur better. In general, it handles edges better.

2. The Propagation Settings

image 24

Changes you do under the propagation settings will require After Effects to recalculate all propagated frames. Your base frames are not affected.

Search Radius

The Search Radius is the size of the area Roto Brush searches for matching pixels between two adjacent frames.

If you have a lot of motion, try starting with a search radius of 50 or more.

View Search Region

View Search Region v2
When you change the size of the Search Radius, you can see the Search Region Expanding.

Enable the View Search Region to display the area as Roto Brush looks for your subject as a green box.

When you increase or decrease the Search Radius, the box changes size. Anything outside the green box will not be included in the Roto Brush.

So make sure to make the box big enough to contain the range of the subject’s movements in the footage. But don’t make it so big that it includes separate moving subjects/objects you don’t want to be included in the roto.

Enable Classic Controls

If you want to enable the classic propagation controls from Roto Brush 1.0, I’ve covered all the settings in this article.

Invert Foreground/Background

Invert Foreground Background Matte Roto Brush

Inverting Foreground/Background changes which stroke Roto Brush considers foreground strokes and which are considered background strokes (e.g., the green and red strokes you create on your base frames).

It’s a quick way to invert the roto brush matte and remove a moving subject while keeping the background.

Fine-tune Roto Brush Matte

image 25

Enable Fine-tune Roto Brush Matte for further options for fine-tuning the edges you’ve created with the Roto Brush Tool (as opposed to the matte you’ve created with the Refine Edge Tool, which we’ll address later)

If there are small areas of the background included in your roto that you can’t get with strokes, you can try fine-tuning the roto brush matte instead.

Enabling Fine-tune Roto Brush Matte gives you more control over the Feather, Contrast, Shift Edge, and Reduce Chatter.

  • Feather is the softness of the segmentation boundary.
  • Contrast increases the contrast (useful for tightening up the boundary further).
  • Shift Edge either expands (positive value) or decreases (negative value) the segmentation boundary of your roto.
  • Reduce Chatter reduces visible noise along the edges of your roto. Chatter is seen when the edges move erratically. Reduce chatter reduces that noise and gives you cleaner edges.

Did you know that you can easily use the data from the Roto Brush to create a keyframed mask in After Effects? Here’s a breakdown.

What exactly is chatter in After Effects?

Reducing chatter is more memory intensive and is not always the best choice, as you might miss important information near your fingers or hair.

Fx, in the case of fingers, it’s like the roto outline becomes more “stiff,” and it prefers to treat all the fingers as one shape/hand.

If that is the case, dial back the Reduce Chatter, use fine background strokes and a roto brush, and manually paint the space between each finger.

Also, if your roto subject isn’t moving but the edges are jumping around erratically between frames, increase the Reduce Chatter Value. Likewise, if the edges aren’t keeping up with your subject’s movement, decrease the Reduce Chatter value.

The process is the same for hair, but it’s sometimes more useful to draw background strokes with the Refine Edge Brush (not always, though).

3. Render Refine Edge

Render Refine Edge Roto Brush 1

Render Refine Edge allows you to see the refined edge of your roto brush. It works for both base frames and propagated frames.

Use it to see if you’ve included all the details you want with the Refine Edge Tool.

4. Base Refine Edge Radius

image 28

Base Refine Edge Radius lets you expand the area of the refined edges you’ve already created with the Refine Edge Roto Brush tool.

Use it to expand the area of the refined edges when you’ve used a brush that is too small to help After Effects calculate the refined edges between frames.

5. Fine-tune Refine Edge Matte

image 29

Enable Fine-tune Refine Edge Matte for options for fine-tuning the edges you’ve created with the Refine Edge Tool.

The options are:

  • Smoothness reduces the sharpness of the curves of the roto edges. Set it to low for things such as hair or fringes on clothes.
  • Feather is the softness of the refined edge segmentation boundary.
  • Contrast increases the contrast of the refined edge (useful for tightening up the boundary further).
  • Shift Edge either expands (positive value) or decreases (negative value) the segmentation boundary of your roto.
  • Chatter Reduction reduces visible noise along the edges of your roto. Choose between…
    • Off: no chatter reduction.
    • More detailed: the “standard” chatter reduction. This sometimes gives the best results for hair or fringes on clothes, as the smoother option below misses things like stray hairs.
    • Smoother (Slower): smoothes out the edges further (compared to the more detailed option above). Use it to clamp down on jumping edges on things such as clothes or fingers you’ve created with the refine edge tool.

6. Use Motion Blur

image 31

If your subject has a lot of movement, you should enable Use Motion Blur. That way, Roto Brush can account for blur around the edges due to the movement of your subject, the frame rate, and shutter angle/speed.

To fine-tune the motion blur, the options are:

Samples Per Frame determine the number of samples the roto brush uses to simulate the motion blur. Increase the value for a smoother-looking motion blur or decrease it for faster rendering.

Ideally, if your footage is shot with 25 frames per second with a shutter angle of 180 (1/50 shutter speed), you’d need 50 samples per frame for a naturally-looking motion blur. However, this would tremendously slow down the rendering process.

Shutter Angle simulates the camera shutter angle (or shutter speed) options found in cameras.

The option uses the frame rate of your footage as a basis for calculation. A shutter angle of 180 degrees is the default and usually does a good job if you’ve used the same in your camera (or a shutter speed of double the frame rate).

A low parameter accounts for almost no motion blur, and a high parameter accounts for much motion blur.

Higher Quality generates a cleaner edge (at the cost of processing speed).

7. Decontaminate Edge Colors

Decontaminate Edge Colors

Enable Decontaminate Edge Colors to remove the background color from foreground edge pixels.

Use it to fix halos around your subject or background color from motion blur.

Fine-tune it with these options:

Amount is the amount of decontamination is applied.

Extend Where Smoothed decontaminates edges that are moved due to the Reduce Chatter option.

Increase Radius increases the radius of decontamination, including feather, motion blur, and extended (where smoothed) decontaminated pixels.

View Map shows the pixels cleaned by Decontamination as a white outline.

image 33

Got any questions?

Even though Roto Brush 2.0 is REALLY good, it can only do so much, and I still find myself having to draw masks manually occasionally.

Still, it’s an amazing time saver, and I only have to use masks to clean up small bits and pieces.

Did I miss anything important? Or do you have any questions? Please let me know in the comments below.

Up Next: How to create the Speedramping effect in video editing.

Author

    by
  • Jan Sørup

    Jan Sørup is a indie filmmaker, videographer and photographer from Denmark. He owns filmdaft.com and the Danish company Apertura, which produces video content for big companies in Denmark and Scandinavia. Jan has a background in music, has drawn webcomics, and is a former lecturer at the University of Copenhagen.

Leave a Comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.