I

n this lesson, we will be focusing on different ways

to create transparency. One of the keys to creating
an interesting composite of multiple images is to
make portions of those images transparent, so that
you can see other images behind or through them.
This is one of the main techniques that set motion
graphic design and visual effects compositing apart
from video editing.
Masks, Mattes, and Stencils
In previous lessons, we introduced two basic forms of man-
aging transparency: altering the Opacity property of a layer,
and taking advantage of a layers built-in alpha channel that
defines which parts of it are supposed to be transparent. In
this lesson, you will go beyond these by adding your own
transparent areas to an image using masks, track mattes,
and stencils.
Masking is a way to cut out sections of a specific layer. At
its default, a mask shape says I want to see only the area
inside this shape; make the area outside transparent. You
can draw your own shapes and paths directly on the layer, or
copy paths from the Adobe companion programs Photoshop
and Illustrator and paste them onto an After Effects layer to
create a mask shape. You may have multiple masks per
layer, and combine them in a variety of ways such as
adding together their shapes or using only the area where
they overlap. You can also control the opacity of a mask
(making its cutout semitransparent), define its feather (how
soft its edges are), and invert it so areas inside of its shape
rather than outside are transparent. As an added bonus,
some effects can use mask shapes to draw lines, and you
Lesson 4
82
Excerpt from After Effects Apprentice. Copyright 2007 Trish and Chris Meyer.
No part of this publication may be reproduced or distributed in any form or by any means.
Focal Press, ISBN 978-0-240-80938-0
Creating Transparency
Using masks, mattes, and stencils to cut out portions of a layer.
In This Lesson
84 Rectangular Mask tool
84 Free Transform Points
85 masking in the Layer panel
85 Mask Feather
86 animating a Mask Shape
87 creating a vignette; Mask Expansion
88 masking with the Pen tool
88 creating and editing Bezier masks
89 effecting a masked area
90 controlling Mask Shape interpolation
91 using effects with the mask path
92 Mask Modes and multiple masks
93 Mask Opacity
94 creating and editing RotoBezier masks
95 Audio Spectrum effect
96 Alpha Track Matte
97 nesting a track matte composite
99 effects and track mattes
100 Luma Track Matte
101 animating matte layers
102 Stencil Alpha and Stencil Luma
Getting Started
Copy the Lesson 04-Transparency folder from this books disc
onto your hard drive. Open the file Lesson_04.aep inside of it
to work through the exercises in this lesson.
can have text follow a mask (which well show in the next lesson). You can even
create a mask shape that is just a line or path rather than an enclosed area; these
are particularly handy when used in conjunction with effects or text.
Track mattes, by contrast, involve the combination of two layers. One layer
the matte is used just to define transparency; you dont directly see the image
it contains. This matte is then used to decide what portions of the layer immedi-
ately below it are visible. There are two types of mattes: alpha mattes, which use
the alpha channel of the matte layer to define the transparency of the second
layer, and luminance (or luma) mattes, which use the luminance grayscale
values, or brightness of the matte layer to define the transparency of the
second layer.
Stencils take the concept of track mattes further: Rather than define the trans-
parency of the next layer below, a stencil layer defines the transparency of all
the layers below, cutting a hole through the entire layer stack. Just like mattes,
stencils can also be based around alpha channels or luminance.
Its easy to confuse these methods of creating transparency, so just remember:
Masks involve one layer, mattes involve two layers (the fill and the matte), and
stencils are basically mattes that can affect multiple layers below them. In this
lesson, youll learn how to put all three to work.
Masks take a layer (A), and allow you to cut
out portions to make them transparent or
opaque (B). Mattes use either the alpha
channel or luminance of one layer (C) to
define the transparency of a second layer
(D) to create a final composite image (E).
Images courtesy Wildscaping.com,
Artbeats/Virtual Insanity.
Lesson 4
83
Excerpt from After Effects Apprentice. Copyright 2007 Trish and Chris Meyer.
No part of this publication may be reproduced or distributed in any form or by any means.
Focal Press, ISBN 978-0-240-80938-0
future vision
Keying
Keying the art of making certain colors
in a layer transparent is another way of
creating transparency. Unlike the techniques
demonstrated in this lesson, keying relies on
an effect to create transparency. Keying is
covered in Lesson 10.
A B
C D E
tip
Help with Shortcuts
For a full list of pen tool and mask editing
shortcuts, see Help > Keyboard Shortcuts
(or Help > After Effects Help, and search
for keys for using masks).
Masking
There are several ways to create masks, and several ways to use them. In this first
exercise, we will focus on using the Mask tool to create windows on footage, to
reveal a title, and to create a vignette.
1 We assume youve already opened the project Lesson_04.aep. If youd like to
preview what youll be creating in this exercise, open its Comps_Finished folder
and RAM Preview the comp 01-Masking_final. Close it when youre done by
pressing Cw on Mac (Lwon Windows).
2 In the Project panel, open the Comps folder and double-click the comp
01-Masking*starter to open it.
Select the foreground layer Cityscape.mov. Then select the Rectangular Mask
tool from the toolbar along the top of the application window; its shortcut is q.
Click and drag in the Comp panel to surround a few of the major buildings on
the right, plus some of the freeway. As you draw this outline, the areas of
Cityscape.mov beyond the outline will disappear, revealing the background.
After you release the mouse, with
Cityscape.mov still selected, press m
to reveal the new Mask and its Mask
Shape property in the Timeline panel
(see right).
In the Comp panel, you should see
a yellow outline around the mask
shape. If you dont, click the Toggle
View Masks button along the bottom
of the Comp panel to turn it on.
3 Press vfor the Selection tool. Then double-click on the yellow mask outline;
this will enable its Free Transform Points, which means you can edit its shape.
Note the eight small boxes (handles) around its shape: As you hover your
cursor over these, the cursor will change to icons that indicate you can click and
drag to resize or rotate the shape. Dragging anywhere else moves the entire shape.
Tweak the size and position of the mask to taste, then hit Eto accept your
edits and turn Free Transform off.
4 Next, you want to reveal a different section of Cityscape.mov but you cant
see the rest of the image as its already been masked out. No problem: You can
also create masks in the Layer panel where you can view the entire image.
Double-click Cityscape.mov to open its Layer panel. If it opens docked into
the same frame as the Comp panel, right-click on its tab, select Undock Panel,
then drag it side by side with the Comp panel.
Lesson 4
84
Excerpt from After Effects Apprentice. Copyright 2007 Trish and Chris Meyer.
No part of this publication may be reproduced or distributed in any form or by any means.
Focal Press, ISBN 978-0-240-80938-0
tip
Constrained Masks
Press Swhile dragging a mask shape
to create a perfect square or circle; add
Con Mac (Lon Windows) after you
start dragging to draw the mask outward
from a center point.
3 Double-click the mask outline to enable
its Free Transform Points; the shortcut is
Ct(Lt). Here we are grabbing the
top point to change the height of the shape.
2 Select the layer to mask, then select
the Mask tool (above). Click and drag in
the Comp panel to define the mask shape
you want (below). Foreground courtesy
Artbeats/Timelapse Cityscapes; background
12 Inch Design ProductionBlox Unit 02.
Along the bottom of the Layer panel is a View
popup; check that it is set to Masks. Uncheck the
Render button (to the right of View): This will reveal
the entire layer without the effect of masking.
Select the Rectangular Mask tool again. This time,
hold down the Skey while dragging out a new
mask shape along the left side of the layer; this will
constrain it to draw a perfect square. (Actually, it will
look slightly wide, as this layer has non-square pixels
refer to the Tech Corner at the end of the previous
lesson.) The result of your masking will instantly
appear in the Comp panel as you draw; when youre
done, Mask 2 will appear in the Timeline panel.
5 After you are done drawing a mask shape in the
Layer panel, you will see the individual mask points
or vertices that define it; by default, all vertices are
selected. Press vto return to the Selection tool.
While all points are selected, you can drag any one
point to move the entire mask, or use the arrow keys
to nudge it by one screen pixel. You can S+click or
drag a marquee around individual vertices to select or
deselect them, and edit the position of these directly.
You can also double-click the mask to use the Free
Transform Points as you did in the Comp panel.
Tweak your second mask shape to taste, and close
the Layer panel when youre done.
6 By default, masks have sharp, crisp edges. However,
it is possible to soften these. With Cityscape.mov still
selected, type Sfto reveal the Mask Feather para-
meter in addition to Mask Shape. Turn the Toggle View
Masks button (at the bottom of the Comp panel) off to
more clearly see the outlines, and scrub Mask Feather
to soften the masks. Set Feather for both masks to taste.
7 After Effects calculates effects for a layer after
masks have been taken into account. Apply Effect >
Perspective > Drop Shadow to Cityscape.mov; notice
that both masks get the same shadow. Slightly increase
the Distance and Softness of the shadow to taste, and
save your project.
7 Effects are applied
after mask shapes.
That means the single
Drop Shadow effect
applied to this layer
affects both masks.
Lesson 4
85
Excerpt from After Effects Apprentice. Copyright 2007 Trish and Chris Meyer.
No part of this publication may be reproduced or distributed in any form or by any means.
Focal Press, ISBN 978-0-240-80938-0
4 Double-click a layer to open its Layer panel. Set the View popup to
Masks and turn off the Render switch. You can then draw and edit masks
undistracted by other layers or the results of previous mask shapes.
6 Scrub the Mask
Feather parameter to
soften the edges.
Animating a Mask Shape (Mask Path)
Masks do not need to be fixed windows onto another layer
their shapes can also animate. This is particularly useful
for revealing the content of layers.
8 Press Hto make sure you are at the start of the
composition. Bring the Project panel forward, and open
the Sources folder. Select the footage item Bring on the
Night.ai, and type C/(L/) to add it to your comp.
9 This black title is hard to read against your video. With
it selected, add Effect > Generate > Fill. Click on its Color
swatch, and change it to white.
Thats better, but it still looks a bit flat on top of your
composite. Add Effect > Perspective > Drop Shadow, and
adjust the Distance and Softness to taste.
10 As we mentioned in a previous lesson, sometimes its
easier to start where you want to end up, and work back-
ward. With Bring on the Night.ai selected, double-click on
the Rectangular Mask tool. This will add a mask to it that
is the same size as the layer a perfect ending point for a
reveal. Enable the Toggle View Masks switch to see its
outline in the Comp panel.
11 Type mto reveal the Mask Shape parameter in the
Timeline panel. Move the current time marker to 01:00,
and click on the stopwatch next to Mask Shape to enable
keyframing and create the first keyframe.
Hit Hto return to 00:00. Then double-click on the
mask outline in the Comp panel to bring up the Free
Transform Points; the shortcut is Ct(Lt). Drag the
handle on the right edge all the way to the left until the
title is completely hidden. A new Mask Shape keyframe
will be created automatically at 00:00.
To soften the leading edge of your wipe, increase the
Mask Feather for the titles mask. You need to increase only
the first value, which is in the X (horizontal) direction.
12 Press 0on the numeric keypad to RAM Preview your
animation. Add a dramatic pause by moving Bring on the
Night.ais layer bar to start later in the timeline, such as at
01:00 its keyframes will move with it. Save your project!
Lesson 4
86
9 Add a title, use the Fill effect to color it white, and add the
Drop Shadow effect.
Excerpt from After Effects Apprentice. Copyright 2007 Trish and Chris Meyer.
No part of this publication may be reproduced or distributed in any form or by any means.
Focal Press, ISBN 978-0-240-80938-0
11 To wipe on the title, animate a mask to reveal it. Here we are
dragging one of its Free Transform Points to edit the mask shape.
Creating a Vignette
Masks can also be used to create subtle effects, such as
adding a vignette where the edges of an image are
darkened, focusing the viewer more toward the center of the screen.
13 Press Hto return to 00:00. Create a Layer > New > Solid; the shortcut
is Cy(Ly). In the Solid Settings dialog that opens, click the Make Comp
Size button, and set the Color to be something complementary to your scene
such as a dark purple or blue. Give it a name that makes sense such as vignette
and click OK.
14 Click and hold on the Mask tool; a popup will appear. You want a rounded
mask for this task, so select the Elliptical Mask Tool option, and release the mouse.
(You can also toggle between the two by pressing q.) Then, with the vignette
layer selected, double-click this tool in the toolbar to create a full-size oval mask.
15 Now you have a big dark oval obscuring the center of your image probably
not what you had in mind. But thats easy to change.
With vignette still selected, type mm(two ms in quick succession) to reveal
all of the mask properties, and make the following edits:
Click on the checkbox next to Inverted: Now the mask solid will be transparent
inside and opaque outside.
Scrub the Mask Feather until you get a nice, soft falloff around the edges.
Scrub the Mask Expansion property: This offsets a mask to draw inside or
outside of the original Mask Shape. (To better see its effect, temporarily turn
Mask Feather down to zero.) Increase Mask Expansion to push the vignette off
into the corners.
If the corners are too dark, reduce the Mask Opacity value (or use the regular
Transform > Opacity property). Then feel free to balance the Mask Feather,
Opacity, and Expansions values off of each other to get the look you want.
14 Select the Elliptical Mask Tool (inset),
then double-click in the toolbar to create a
full-size oval mask on your new solid (above).
Lesson 4
87
15 To finish off the vignette, invert the
mask (A), increase its feather (B), increase
its expansion (C), and reduce its opacity (D).
Our version is in Comps_Finished > 01-
Masking_final.
Excerpt from After Effects Apprentice. Copyright 2007 Trish and Chris Meyer.
No part of this publication may be reproduced or distributed in any form or by any means.
Focal Press, ISBN 978-0-240-80938-0
A B C D
tip
Numeric Shapes
Clicking the word Shape in the switches
column of the Timeline panel opens the
Mask Shape dialog, where you can enter
values numerically.
Congratulations youve created a tasteful little animation, plus now have a
handle on creating masks. Go ahead and tweak your animation to taste. If you
like, compare your results with ours see comp 01-Masking_final in the Project
panels Comps_Finished folder.
Masking with the Pen Tool
Next up is practicing using the Pen tool to create more detailed mask shapes.
It has two basic modes: One that creates common Bezier curves with handles
(which well do first), and a RotoBezier mode which automatically determines
the curve of the path. As with the Mask tool, you can create shapes in either the
Comp or Layer panels, plus keyframe their shapes.
Drawing a Path
1 In the Project panel for this lessons file, locate 02a-Bezier*starter in the
Comps folder and double-click it to open it.
2 Select the Pen tool (the shortcut is g). When you do, a RotoBezier option
will appear on the right end of the Toolbar; turn it off for now.
3 Drawing with the Pen tool is similar to drawing paths in programs such as
Photoshop and Illustrator, although the shortcuts can differ slightly. To practice,
start by just making a random shape (dont worry; you can always delete any
mess you create):
To create straight line segments, click with the mouse to create a series of points.
To create curved segments, click and drag to pull out Bezier handles for the
point you are creating.
While drawing with the Pen tool, you can edit the points youve just placed:
To reposition any point or handle before youre finished drawing a mask,
temporarily toggle to the Selection tool by pressing the C(L) key.
To toggle between a sharp corner point and a smooth point, press OS
(AS) and click on a point. The cursor will automatically change to the
Convert Vertex Point cursor, which looks like an upside-down V. To toggle the
last point you drew, just click on it without holding down these extra keys.
To break the continuous handles of a smooth point, click and drag one
handle; repeat to switch back to a smooth point.
Click any point other than one you
just drew to delete it (the cursor will
automatically change to the Delete
Vertex tool). To delete the most recent
point, Undo.
To add a new point between points
youve already drawn, click on the line
joining them the cursor will auto-
matically change to the Add Vertex
tool while doing so.
3 The Convert Vertex Point tool is used to
toggle between smooth and hard corners,
and to break or unbreak the continuity of
the Bezier handles. Image courtesy
Wildscaping.com.
2 Select the Pen tool, and turn off the
RotoBezier option for now.
Lesson 4
88
Excerpt from After Effects Apprentice. Copyright 2007 Trish and Chris Meyer.
No part of this publication may be reproduced or distributed in any form or by any means.
Focal Press, ISBN 978-0-240-80938-0
tip
Select All Points
To select all the points on a mask, with the
Selection tool active hold down O(A)
and select any one of the masks points.
3 continued To delete a vertex, place the cursor over a point; it will automatically change to
the Delete Vertex tool (left). To add a vertex, place the cursor between points (right).
To pick up drawing a mask shape where
you left off, make sure only the last point
is selected, then continue creating points
with the Pen tool.
To close a mask shape, click back on
the first point. Only closed masks create
transparency. If you want to create an open
path, simply change to the Selection tool
(shortcut = v) when youre done creat-
ing the path.
Editing the Path
We prefer to switch back to the Selection
tool for editing a mask with the Pen tool
active, its all too easy to delete a point
by clicking on it! However, some of the
shortcuts will then change slightly; verify
the different modifier keys by hovering
the cursor over a point and seeing which
cursor appears. Because closing the mask
creates transparency, we suggest you edit
the mask shape in the Layer window with
the Render switch off so you can see the
entire image. Feel free to zoom in as well.
4 Press vto change to the Selection
tool, and practice the following:
To add or delete points, press C(L)
and click. The cursor will automatically
change to a Delete Vertex tool when you are near a mask point, and an Add
Vertex tool when you are between points.
To toggle a point between a smooth and sharp corner point, press CO
(LA) and click on the point.
To break or rejoin Bezier handles, press C(L) and drag on a handle.
5 After youve had some fun, delete your experimental mask, and practice
drawing a mask around the foreground tulip. It will require a combination of
smooth shapes and sudden direction changes where some of the petals overlap
or meet the stem. Its fairly challenging, but a good example of a task you will
often need to perform in the real world. Our version is in the Comps_Finished
folder (02a-Bezier_final), pictured to the right.
Lesson 4
89
Excerpt from After Effects Apprentice. Copyright 2007 Trish and Chris Meyer.
No part of this publication may be reproduced or distributed in any form or by any means.
Focal Press, ISBN 978-0-240-80938-0
Applying an Effect to a Masked Area
To apply an effect to an area of a layer in Photoshop, you would create a selection
first. In After Effects, you need to duplicate the layer, create a Mask Shape on the
top layer, then apply the effect to the top or bottom layer depending on whether
you want the area inside or outside
the mask to be effected.
For instance, lets say you
wanted the foreground flower in the
PinkTulips.psd layer to be in color,
and the rest of the image to be in
black and white. Open the comp
02b-Masked Effects*starter, and
follow these steps:
Select the PinkTulips.psd layer,
and select Edit > Duplicate the
shortcut is Cd(Ld).
Select the bottom layer, and
select Layer > Masks > Remove
All Masks.
With the bottom layer still
selected, apply Effect > Color
Correction > Tint to make it
black and white. Our version is in Comps_Finished > 02b-Masked Effects_final.
You also could have started by duplicating your original layer and masking just
the top one. The end result is the same: Two layers, the top one has the mask, and
effects are applied as needed to one or both layers.