How To Convert A RotoBrush Selection To A Keyframed Mask In After Effects

DISCLOSURE: AS AN AMAZON ASSOCIATE I EARN FROM QUALIFYING PURCHASES.
THIS POST MAY CONTAIN AFFILIATE LINKS, MEANING, AT NO ADDITIONAL COST TO YOU, I EARN FROM QUALIFYING PURCHASES. AFFILIATE LINKS ARE MARKED WITH #ad. "I" IN THIS CASE MEANS THE OWNER OF FILMDAFT.COM. PLEASE READ THE FULL DISCLOSURE FOR MORE INFO.

Did you know that you don’t have to manually create and track masks with the pen tool in Adobe After Effects if you’ve already created a RotoBrush selection?

Instead, you can quickly and easily create keyframed masks from a subject/object you’ve already rotoscoped. The trick is to use the auto-trace function.

In this illustrated guide, you’ll see the process broken down into eight easy steps. If you already know how to use the rotoscope tool, you can jump directly to step 5.

I’ve also created this video, that explains the process after the rotoscoping part:

And here’s the breakdown in more detail:

1. Duplicate the layer you want to rotoscope. Call the new layer something with ‘roto.’

It’s good practice to first create a duplicated layer for your Rotobrush work.

2. Double-click on the new roto layer to work directly on the layer (as the rotoscope tool doesn’t work in the composition panel)

Make sure to work in the layer panel as the Rotobrush tool won’t work in the composition panel.

3. Pick the RotoScope brush tool in After Effects and paint over the subject you want to rotoscope out.

Select the RotoBrush tool and start selecting the part of your subject, you want to roto. I find I get the best results with version 2.0.

4. Track your subject (by pressing ‘space’ on your keyboard) with the rotoscope tool. Refine to taste.

Press ‘Space’ on your keyboard and start tracking your subject.

5. Go to Layer -> Auto-trace.

When you’re satisfied with the rotoscope mask you’ve created, it’s time to use the auto-trace feature in After Effects to create a keyframed mask from the rotoscoped selection.

6. The auto trace panel opens up. Start with the default settings and hit OK.

I usually get the desired result by only tweaking the tolerance between 1 and 5 and leaving the rest of the settings on default. Choose ‘work area’ and remember to set the in and out points to fit the length of the layer you want to trace, so it doesn’t trace your whole project composition.

7. A new auto-traced layer is created with a keyframed mask using the data from the RotoBrush track you created.

If you don’t like the result, delete the auto traced track, and open the auto-trace panel again. Then start tweaking the settings one at a time.

8. You can now copy the keyframed mask-data to all the layers you want.

Here I have copy-pasted the mask data to the lightsaber layer so that my son will block out the saber every time he swings it in front of him.

That’s it really. I hope you found this tip helpful. If you got any questions let me know in the comment section below.


About the author:

Jan Sørup is a videographer and photographer from Denmark. He’s the owner of filmdaft.com and of the Danish company Apertura, which produces video content for big companies in Denmark and Scandinavia. Jan has a background in music, has drawn webcomics, and is a former lecturer at the University of Copenhagen.

Leave a Comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.