Did you know that you don’t have to manually create and track masks with the pen tool in Adobe After Effects if you’ve already created a RotoBrush selection?
Instead, you can quickly and easily create keyframed masks from a subject/object you’ve already rotoscoped. The trick is to use the auto-trace function.
In this illustrated guide, you’ll see the process broken down into eight easy steps. If you already know how to use the rotoscope tool, you can jump directly to step 5.
I’ve also created this video, that explains the process after the rotoscoping part:
And here’s the breakdown in more detail:
1. Duplicate the layer you want to rotoscope. Call the new layer something with ‘roto.’
It’s good practice to first create a duplicated layer for your Rotobrush work.
2. Double-click on the new roto layer to work directly on the layer (as the rotoscope tool doesn’t work in the composition panel)
Make sure to work in the layer panel as the Rotobrush tool won’t work in the composition panel.
3. Pick the RotoScope brush tool in After Effects and paint over the subject you want to rotoscope out.
Select the RotoBrush tool and start selecting the part of your subject, you want to roto. I find I get the best results with version 2.0.
4. Track your subject (by pressing ‘space’ on your keyboard) with the rotoscope tool. Refine to taste.
Press ‘Space’ on your keyboard and start tracking your subject.
5. Go to Layer -> Auto-trace.
When you’re satisfied with the rotoscope mask you’ve created, it’s time to use the auto-trace feature in After Effects to create a keyframed mask from the rotoscoped selection.
6. The auto trace panel opens up. Start with the default settings and hit OK.
I usually get the desired result by only tweaking the tolerance between 1 and 5 and leaving the rest of the settings on default. Choose ‘work area’ and remember to set the in and out points to fit the length of the layer you want to trace, so it doesn’t trace your whole project composition.
7. A new auto-traced layer is created with a keyframed mask using the data from the RotoBrush track you created.
If you don’t like the result, delete the auto traced track, and open the auto-trace panel again. Then start tweaking the settings one at a time.
8. You can now copy the keyframed mask-data to all the layers you want.
Here I have copy-pasted the mask data to the lightsaber layer so that my son will block out the saber every time he swings it in front of him.
That’s it really. I hope you found this tip helpful. If you got any questions let me know in the comment section below.
Jan Sørup is a videographer and photographer from Denmark. He owns filmdaft.com and the Danish company Apertura, which produces video content for big companies in Denmark and Scandinavia. Jan has a background in music, has drawn webcomics, and is a former lecturer at the University of Copenhagen.
6 thoughts on “How To Convert A RotoBrush Selection To A Keyframed Mask In After Effects”
Thank you!!!
I googled and watched various Youtube videos trying to find a solution to convert a psd layer shape mask in ae cs but to no avail, but you answered this in just a few lines of explanation.
You should make a short video and monetize your lessons–because it’s short and sweet–to the point!
Thank you!!!
I googled and watched various Youtube videos trying to find a solution to convert a psd layer shape mask in ae cs but to no avail, but you answered this in just a few lines of explanation.
You should make a short video and monetize your lessons–because it’s short and sweet–to the point!
Thank you. I’m glad you found it useful. I actually did create a video: https://www.youtube.com/watch?v=pCFwTq1P34c
Unfortunately, I have not nearly enough views or subscribers to monetize anything 😀
omg u r the best thank u so much
Hi Ramon.
Thank you for your nice comment. I’m glad you found it useful 🙂
Best of luck with your video project.
Best, Jan
What if i have a lot of clips in my project and I only want to mask one layer
The mask work one the layer you assign it to. It doesn’t affect other layers.