Techniques

Mask Magic

by Tim Grey | December 22, 2011

© Tim GreyLayer masks are an incredibly powerful feature in Photoshop, enabling you to apply targeted adjustments and create impressive composite images. However, at times you may find yourself frustrated by a layer mask that just won’t come together to produce a good result.

Let’s consider a typical example. One of the most common reasons I see photographers putting a layer mask to use is to apply a targeted adjustment to improve the appearance of a sky. Very often, for example, you might want to darken down a sky.

Such was the case for a photo I captured of a lilac-breasted roller while on safari in South Africa. I was excited just to see this beautiful bird, as I had been told they generally didn’t get as far south as we would be. But the bird proved uncooperative, so I was only able to get a single photo that was worthwhile, and even that one wasn’t all that great.

Lilac-breasted roller © Tim Grey

Still, this photo represented a fun experience, and I wanted to make it look as good as possible. I felt that darkening the sky would help improve the overall appearance, so I got started. Simple enough, right? Just create a selection of the sky, and add an adjustment layer that will only affect the sky.

A Fuzzy Problem

One of the biggest challenges in terms of creating accurate layer masks is the fuzzy stuff: hair, feathers, and fur. Overall the selection wasn’t too big of a challenge when it came to my photo of the lilac-breasted roller. I simply used the Magic Wand tool with the Contiguous option turned off and a Tolerance setting of 16. I clicked in the sky to create the initial selection, then held the Shift key and clicked on additional areas of the sky to add to the selection until the entire sky was selected. I then added a Curves adjustment layer, and darkened down the sky a bit.

Mask selection in Photoshop © Tim Grey

Layer palette in Photoshop © Tim Grey

Unfortunately, the feathers at the “chin” of the bird, and along the top of the head, proved problematic. There was a somewhat clear dividing line between the area that I was adjusting (the sky) and the area I wasn’t adjusting (the bird).

Zoom of bird beak © Tim Grey

In theory, just feathering the selection (no pun intended, of course) or applying a slight blur to the layer mask would provide a good solution, softening the transition between the adjusted and non-adjusted areas. However, when you have a fuzzy subject, that often won’t work well. Simple feathering is especially problematic when the degree of fuzziness varies at various areas. For example, the breast area of the bird is much less fuzzy than the chin area and top of the head. So a variable approach to refining the mask is needed. Fortunately, Photoshop CS5 provides just such a feature.

A Refined Solution

The solution to this fuzzy problem was to refine the layer mask. The same adjustments I applied could have also been applied to the selection, but I prefer to save this work for the layer mask based on the selection, so that I can preview the actual effect in the image, rather than guess about what settings are going to produce the best final result.

To refine the layer mask, you simply click on the thumbnail for the layer mask on the Layers panel, so that the layer mask is the active item. Then go to the Masks panel (choose Window > Masks if the Masks panel isn’t currently visible) and click on the Mask Edge button. This will bring up the Refine Mask dialog, which allows you to fine-tune the edge of your layer mask. The first thing I recommend doing here is changing the View option to On Layers, so you can see the effect of your work in the image itself.

Refine Mask window in Photoshop

For just about any layer mask you create, you’ll need at least a small degree of feathering. I prefer not to feather the selection from the start, so Refine Mask is where I adjust feathering, increasing the Feather slider a little bit. Generally just one or two pixels will do the trick, but the amount will vary based on the subject in the photo.

Next, I adjust the Shift Edge slider, which allows you to move the transition between adjusted and non-adjusted areas inward or outward relative to your original selection. This can be critically important, because when you feather a layer mask you are feathering in both directions, generally blocking the adjustment in part of the area you had selected for your targeted adjustment.

Refine Mask settings in Photoshop

With the Feather and Shift Edge sliders set to an optimal value for the overall image, the chin feathers stand out as the primary problem. Fortunately, the solution is a simple paint stroke away. In theory you could simply increase the Radius slider, and perhaps turn on Smart Radius, both found in the Edge Detection section of the Refine Mask dialog. However, I find that these don’t generally achieve the intended result, and a bit of manual intervention is required.

To define areas where you need the layer mask to be cleaned up, all you have to do is move your mouse over the image, and paint in the areas that are proving to be problematic. This will define those areas as needing additional blending, and Photoshop will take care of the rest. In most cases, a simple paint stroke over difficult areas is all you need. If you get a little carried away with the painting, you can click and hold your mouse on the button to the left of the Edge Detection section to bring up a flyout menu, where you can choose between the Refine Radius tool and the Erase Refinements tool. The Refine Radius tool is what you use initially to define areas that need some more blending, and the Erase Refinements tool is used to remove the blending effect from areas you painted on inadvertently with the Refine Radius tool.

Completed image © Tim Grey

Easier Than Expected

Applying targeted adjustments is, I think, one of the most powerful ways you can exercise control over the final appearance of your photographic images. And thanks to the quite-powerful Refine Mask dialog, fine-tuning your layer masks to create high-quality results is surprisingly easy.

Tim Grey spends most of his time helping photographers make the most of their digital photos, and publishes the daily Ask Tim Grey email newsletter.

About the Author

Tim Grey is an educator in digital photography and imaging, offering clear guidance on complex subjects through his writing and speaking.

Tim has written more than a dozen books on digital imaging for photographers, has published dozens of video training courses, has had hundreds of articles published in magazines such as Digital Photo Pro and Outdoor Photographer, among others. For more than a dozen years he has been publishing the daily Ask Tim Grey eNewsletter, answering questions from photographers, and produces the related Ask Tim Grey Podcast. He also publishes the monthly Pixology electronic magazine, and publishes video training courses through GreyLearning.com. Tim teaches through workshops, seminars, and appearances at major events around the country and around the world.

Tim can be reached via email at tim@timgrey.com.

Comments are closed.