Rotoscope to keyframed mask after effects

How To Convert A RotoBrush Selection To A Keyframed Mask In After Effects

by

DISCLOSURE: AS AN AMAZON ASSOCIATE I EARN FROM QUALIFYING PURCHASES.
THIS POST MAY CONTAIN AFFILIATE LINKS, MEANING, AT NO ADDITIONAL COST TO YOU, I EARN FROM QUALIFYING PURCHASES. AFFILIATE LINKS ARE MARKED WITH #ad. "I" IN THIS CASE MEANS THE OWNER OF FILMDAFT.COM. PLEASE READ THE FULL DISCLOSURE FOR MORE INFO.

Did you know that you don’t have to manually create and track masks with the pen tool in Adobe After Effects if you’ve already created a RotoBrush selection?

Instead, you can quickly and easily create keyframed masks from a subject/object you’ve already rotoscoped. The trick is to use the auto-trace function.

In this illustrated guide, you’ll see the process broken down into eight easy steps. If you already know how to use the rotoscope tool, you can jump directly to step 5.

I’ve also created this video, that explains the process after the rotoscoping part:

And here’s the breakdown in more detail:

1. Duplicate the layer you want to rotoscope. Call the new layer something with ‘roto.’

doublicate layer
It’s good practice to first create a duplicated layer for your Rotobrush work.

2. Double-click on the new roto layer to work directly on the layer (as the rotoscope tool doesn’t work in the composition panel)

double click on layer
Make sure to work in the layer panel as the Rotobrush tool won’t work in the composition panel.

3. Pick the RotoScope brush tool in After Effects and paint over the subject you want to rotoscope out.

Rotobrush tool
Select the RotoBrush tool and start selecting the part of your subject, you want to roto. I find I get the best results with version 2.0.

4. Track your subject (by pressing ‘space’ on your keyboard) with the rotoscope tool. Refine to taste.

track roto brush
Press ‘Space’ on your keyboard and start tracking your subject.

5. Go to Layer -> Auto-trace.

Auto trace
When you’re satisfied with the rotoscope mask you’ve created, it’s time to use the auto-trace feature in After Effects to create a keyframed mask from the rotoscoped selection.

6. The auto trace panel opens up. Start with the default settings and hit OK.

Auto trace Adobe after effects
I usually get the desired result by only tweaking the tolerance between 1 and 5 and leaving the rest of the settings on default. Choose ‘work area’ and remember to set the in and out points to fit the length of the layer you want to trace, so it doesn’t trace your whole project composition.

7. A new auto-traced layer is created with a keyframed mask using the data from the RotoBrush track you created.

auto traced layer
If you don’t like the result, delete the auto traced track, and open the auto-trace panel again. Then start tweaking the settings one at a time.

8. You can now copy the keyframed mask-data to all the layers you want.

Keyframed mask from roto scope after effects
Here I have copy-pasted the mask data to the lightsaber layer so that my son will block out the saber every time he swings it in front of him.

That’s it really. I hope you found this tip helpful. If you got any questions let me know in the comment section below.

Me myself and I profile

About the author:

Jan Sørup is a videographer and photographer from Denmark. He owns filmdaft.com and the Danish company Apertura, which produces video content for big companies in Denmark and Scandinavia. Jan has a background in music, has drawn webcomics, and is a former lecturer at the University of Copenhagen.

2 thoughts on “How To Convert A RotoBrush Selection To A Keyframed Mask In After Effects”

  1. Thank you!!!
    I googled and watched various Youtube videos trying to find a solution to convert a psd layer shape mask in ae cs but to no avail, but you answered this in just a few lines of explanation.

    You should make a short video and monetize your lessons–because it’s short and sweet–to the point!

    Reply

Leave a Comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.