How To Convert A RotoBrush Selection To A Keyframed Mask In After Effects

Published:
Updated:

Did you know you don’t have to manually create and track masks with the pen tool in Adobe After Effects if you’ve already created a RotoBrush selection?

Instead, you can quickly and easily create keyframed masks from a subject or object you’ve already rotoscoped. The trick is to use the auto-trace function.

In this illustrated guide, the process is broken down into eight easy steps. If you already know how to use the rotoscope tool, you can jump directly to step 5.

I’ve also created this video that explains the process after the rotoscoping part:

And here’s the breakdown in more detail:

1. Duplicate the layer you want to rotoscope. Call the new layer something with ‘roto.’

doublicate layer
It’s good practice first to create a duplicated layer for your Rotobrush work.

2. Double-click on the new roto layer to work directly on the layer (as the rotoscope tool doesn’t work in the composition panel)

double click on layer
Make sure to work in the layer panel as the Rotobrush tool won’t work in the composition panel.

3. Pick the RotoScope brush tool in After Effects and paint over the subject you want to rotoscope out.

Rotobrush tool
Select the RotoBrush tool and start selecting the part of your subject, you want to roto. I find I get the best results with version 2.0.

4. Track your subject (by pressing ‘space’ on your keyboard) with the rotoscope tool. Refine to taste.

track roto brush
Press ‘Space’ on your keyboard and start tracking your subject.

5. Go to Layer -> Auto-trace.

Auto trace
When you’re satisfied with the rotoscope mask you’ve created, use the auto-trace feature in After Effects to create a keyframed mask from the rotoscoped selection.

6. The auto trace panel opens up. Start with the default settings and hit OK.

Auto trace Adobe after effects
I usually get the desired result by only tweaking the tolerance between 1 and 5 and leaving the rest of the settings on default. Choose ‘work area’ and remember to set the in and out points to fit the layer length you want to trace so it doesn’t trace your whole project composition.

7. A new auto-traced layer is created with a keyframed mask using the data from the RotoBrush track you created.

auto traced layer
If you don’t like the result, delete the auto-traced track and open the auto-trace panel again. Then, start tweaking the settings one at a time.

8. You can now copy the keyframed mask-data to all the layers you want.

Keyframed mask from roto scope after effects
Here, I have copied and pasted the mask data to the lightsaber layer so that my son can block out the saber every time he swings it in front of him.

That’s it, really. I hope you found this tip helpful. If you have any questions, let me know in the comment section below.

Up Next: How to create a Star Wars scene in After Effects

Author

    by
  • Jan Sørup

    Jan Sørup is a indie filmmaker, videographer and photographer from Denmark. He owns filmdaft.com and the Danish company Apertura, which produces video content for big companies in Denmark and Scandinavia. Jan has a background in music, has drawn webcomics, and is a former lecturer at the University of Copenhagen.

    View all posts

6 thoughts on “How To Convert A RotoBrush Selection To A Keyframed Mask In After Effects”

  1. Thank you!!!
    I googled and watched various Youtube videos trying to find a solution to convert a psd layer shape mask in ae cs but to no avail, but you answered this in just a few lines of explanation.

    You should make a short video and monetize your lessons–because it’s short and sweet–to the point!

    Reply

    • Hi Ramon.

      Thank you for your nice comment. I’m glad you found it useful 🙂

      Best of luck with your video project.

      Best, Jan

      Reply

Leave a Comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.