Rotoscoping with the roto brush tool is great for separating a person or a single object from the background in After Effects.
But when separating trees, leaves, or clouds, or for sky replacement, I often find both of these methods lacking.
Instead, I resort to using the Extract effect tool instead.
First, I’ll go through you can “roto” out clouds or leaves/trees with the Extract effect, i.e., key out the background. Second, I’ll show how you can use the same process for sky replacement within After Effects.
Here is a video that shows some of the steps mentioned in this article:
So let’s dive in…
1. Choose Your Footage Carefully – good and bad scenarios for using the Extract tool for separation
Now, this method isn’t perfect either, and it works best if there’s a clear contrast between the foreground object you’re trying to separate from the background.
Good scenarios are white clouds on a blue sky, green leaves on bright background – fx a bright concrete wall or a blue sky, or naked branches against a blown out or light grey sky.
If you’ve got green leaves on a tree or bush in the foreground, you want to separate from a green field or the edge of a wood in the background, you’re going to have a hard time using this method.
2. Duplicate the footage layer
Duplicate the layer (Ctrl+D/Cmd+D) and rename the new layer to “bright” or “clouds”.
3. Apply the Extract Effect To Your New Layer
Apply the Extract effect to your new layer (in this case the layer I name ‘Base Clouds’).
The Extract effect shows your footage as a histogram with the darkest parts on the left, and the brightest parts on the right.
Below the histogram is a thin rectangular box that fades from black to white.
In each corner, there are small squares which you can drag. The squares represents the values you can see below:
- Top left square = Black Point (the threshold for cutting out blacks)
- Bottom left square = Black Softness (the “feather” of the Black Point)
- Top right square = White Point (the threshold for cutting out whites)
- Bottom right square = White Softness (the “feather” of the White Point)
You can fine-tune your selection individually for the Red, Green, and Blue channel as well as checking the alpha by clicking on the drop-down menu under Luminance:
You an also invert your selection by enabling invert:
4. Drag the Black Point square so that only the clouds are visible
Now drag the Black Point square to key out all dark areas so that only the clouds are visible.
5. Insert the object you want to appear behind the clouds (but in front of the blue sky) by placing the layer between your base layer and the clouds layer
Now insert/create the object you want to appear behind the clouds but in front of the blue sky by placing the layer between your background footage (here: Base Footage) and the clouds layer (here: Base Clouds).
For this example, I’ve chosen a UFO, but you can put in anything you want – for example, some simple text.
Track your footage with the 3D-camera tracker and connect your object to a null if necessary.
6. Set the White Softness to make a nice feathered edge on the clouds
Now set the White Softness to make a nice feathered edge on the clouds.
In this case it gives the nice effect that it looks like the UFO is emerging from the clouds.
And that’s it, basically. The process is the same for sky replacements and dark areas of the image; except for dark areas, you set the White Point and White Softness instead. Here’s a quick rundown of the two:
A note on extracting dark areas
Keying out and extracting dark areas such as branches or leaves are hard. And you shouldn’t expect it to always work.
The problem is that you often get a fringe or halo around each branch or leaf. This is because the light from the original surroundings and background of the video spills onto the edges of the branches and leaves.
To avoid this, you’ll get the best result, if you can find a replacement sky, that is somewhat similar in color, lighting, and saturation to the one, you’re trying to replace. Or you at least try to color match the two backgrounds using color correction tools.
It’s either that – or going back to the woods to place a big blue screen behind the trees in your scene and then chroma key it out it post.
If that’s not possible, there are other tricks you can use to varying degree of success.
Here I’ve extracted the dark areas and put some smoke behind the branches and leaves of the plant. The reason I chose smoke is that it isn’t a solid color layer, i.e., the particles make the fringes seem less prominent. Even so, you can still see them.
You can try choking it with the Matte Choker, Refine Hard Matte, or Refine Soft Matte effects in After Effects (found under Effects and Presets -> Matte).
Using the Refine Soft Matte tool can make dark trees semi-transparent for a cool eerie, and foggy look.
Other things, I recommend trying is using the different color blending modes.
Sometimes I’ve had luck with using Multiply on the trees and then duplicating the layer. Set the duplicated tree layer to the transparency of around 20-35% like so:
You can also try placing the sky layer between your footage and a copy of your footage and then set the sky layer to Linear Burn and the track matte (TrkMat) to Luma Matte.
Then place another layer with your foreground trees on top and set it to Multiply like so:
The two tricks above using different blend modes are great for sky replacements, speaking of which…
Sky Replacement Using The Extract Effect
The process is the same for sky replacements. Above is a shot, I did with my Mavic Pro drone. Boring and blown-out sky.
1. Use the Extract effect tool to key out the boring sky like so:
2. Add a mask to feather the transition between your footage and the new sky
I then added a mask around the bottom half of my drone footage and added some feathering for a smoother transition between my footage and the new sky.
3. Track your footage with the 3D-camera tracker if necessary
Unless your camera sat perfectly still on a tripod while capturing your footage, you should track your footage with the 3D camera tracker.
Then create a camera and a null, so that you can link the position of your new sky with the movement in your footage.
In this case, I tracked my drone footage using the 3D camera tracker and created a camera and a null.
4. Pick your sky – video or photo
You can insert videos or still photos for your sky.
Since there’s not a lot of movement going on in this clip, I chose a still photo.
Insert the sky layer below your footage.
I chose not to go overboard in this example but find a more interesting sky and match the colors of the rest of the footage.
The Mavic Pro footage isn’t good for color grading, as it quickly falls apart.
Had the footage been raw or Pro-Res, I could have changed the colors much more, which would have opened up more possibilities. Fx, I might have picked a warmer sky or a clear blue sky. Keep in mind, though, I would have had to account for reflections in the water as well.
But you can insert whatever you like. It doesn’t even have to be a sky.
5. Turn you photo into a 3D layer, link it to the null, and adjust to taste
I then turned the still photo into a 3D layer as well and pick-whipped the position to match the position of the null:
Now adjust the placement of the sky using Anchor Point (shortcut: a) and scale it taste. Here’s the end result:
Conclusion
Using the Extract effect tool in After Effects is a great way to do things such as sky replacements creating masks for clouds or trees. Rotoscoping out these things with the Roto Brush tool or creating masks manually is next to impossible.
The technique can’t solve all problems though, so make sure to think about things like contrast when you shoot your footage.
I hope this helps. If you’ve got any questions or feedback, let me know in the comment section below.
