Advanced Hair Masking in Photoshop: A Comprehensive Guide

Selections in digital photography and graphic design are all about precision, especially when it involves a complicated subject such as hair.

Masking hair in Photoshop is one of the most challenging tasks within this tool and, simultaneously, one of the most rewarding. If done well, it can take your images to the next level by perfectly merging the subjects with new backgrounds or even changing hair without any hint of editing.

The guide is an attempt to go through some of the more advanced techniques in hair masking in Photoshop and learn how to achieve professional results.

Why Hair Masking is Important

Hair masking is one of the main tasks in working with high-class composites and realistic retouching. Therefore, this technique finds extensive use in portrait photography, fashion, advertising, and digital art. In these fields, the slightest outline or missed strand can compromise the image’s quality. Therefore, mastering hair masking ensures you can tackle complex projects confidently.

Essential Tools for Hair Masking in Photoshop

Before diving into the techniques, let’s go over the essential tools Photoshop offers for hair masking:

1. Quick Selection Tool – Ideal for creating rough selections of your subject, which can be refined later.
2. Refine Edge Brush Tool – A powerful tool within the Select and Mask workspace, designed specifically for complex edges like hair.
3. Layer Masks – Allows non-destructive editing, so you can refine or revert your work as needed.
4. Channels – For advanced masking, the Channels panel can help isolate and select hair based on color or contrast.
5. Brush Tool – Helpful for manual adjustments when refining hair edges or adding stray strands.

Step-by-Step Guide to Advanced Hair Masking

Follow these detailed steps to achieve a flawless hair mask in Photoshop.

Step 1: Duplicate the Original Layer

Start by duplicating your original image layer: Ctrl+J or Cmd+J. This way, you will always have the original image intact and will be able to make adjustments without harming the original.

Step 2: Make an Initial Selection with the Quick Selection Tool

Use the Quick Selection Tool to create an initial selection around your subject. Focus on selecting the main parts of the hair but don’t worry about getting every fine strand just yet. Once your selection is complete, refine the edges.

Step 3: Open the Select and Mask Workspace

With your initial selection active, go to Select > Select and Mask. This opens the Select and Mask workspace, where you can access various tools to refine your selection further.

Step 4: Use the Refine Edge Brush Tool

In the Select and Mask workspace, choose the Refine Edge Brush Tool from the toolbar. Adjust the brush size to match the width of the hair edges. Carefully brush along the edges of the hair to help Photoshop detect the fine details. Pay special attention to areas where the hair meets the background.

Tip: Use small strokes and zoom in to ensure precision. The Refine Edge Brush Tool works best in areas of contrast, so try to work along high-contrast edges to capture as much hair detail as possible.

Step 5: Adjust Edge Detection and Global Refinements

In the right panel of the Select and Mask workspace, under Edge Detection, increase the Radius slightly. This helps Photoshop detect edges more accurately. You can also check Smart Radius to allow for adaptive edge detection.

In the Global Refinements section, experiment with the following settings:

• Smooth – Helps to reduce jagged edges.
• Feather – Softens the edge, which is useful for blending.
• Contrast – Sharpens the edge, making the mask more defined.
• Shift Edge – Adjusts the selection boundary inward or outward.

Tip: Every image is different, so it’s essential to experiment with these sliders to find the right balance for each specific project.

Step 6: Output the Selection to a Layer Mask

Once you’re satisfied with the selection, go to Output Settings and choose Output to: Layer Mask. This applies the refined selection as a mask on your duplicate layer. You now have a layer with a masked subject, allowing you to adjust the background or edit the subject without affecting the entire image.

Advanced Techniques for Fine Hair Masking

Sometimes, the Select and Mask workspace isn’t enough to do that, especially when the backgrounds are complex. Here are a few more advanced techniques that will help to get even better results.

Technique 1: Using Channels for Detailed Masking

1. Go to the Channels panel and look for the channel with the most contrast between the hair and the background. This is usually the Blue or Green channel.
2. Duplicate the channel by dragging it to the New Channel icon at the bottom of the panel.
3. Use Levels (Ctrl+L or Cmd+L) to increase the contrast further, adjusting the sliders until the hair stands out against the background.
4. With the duplicated channel selected, use a Soft Brush to paint over any unwanted areas, isolating the hair.
5. Once done, load the channel as a selection by pressing Ctrl (Cmd) + click on the channel thumbnail.
6. Return to the Layers panel and apply the selection as a layer mask.

This technique is highly effective when dealing with intricate hair details, especially on high-contrast backgrounds.

Technique 2: Color Decontamination for Better Blending

In cases where the background color affects the hair edges, use color decontamination to remove color fringing:

1. After applying the mask, go to Layer > Matting > Defringe and choose a pixel amount (1-2 pixels usually works well).
2. Alternatively, in the Select and Mask workspace, use Decontaminate Colors. This will replace the color fringing with colors from within the selection.

Note: Decontaminate Colors is a destructive edit, so be sure to work on a duplicate layer.

Technique 3: Manually Painting in Stray Hairs

Sometimes, Photoshop won’t capture every hair strand. You can manually paint in these strands:

1. Create a new layer above the masked layer.
2. Select the Brush Tool and set it to a thin, soft brush.
3. Sample a color from the hair (Alt + Click) and carefully paint in individual strands, following the natural flow of the hair.

This technique takes patience but can make a significant difference in realism, especially when fine hairs are missing from the mask.

Troubleshooting Common Hair Masking Issues

Hair masking can be tricky, and you may encounter some common issues along the way. Here’s how to troubleshoot them:

• Fringing or Halos: Use the Defringe command or manually paint around the edges with a soft brush set to the hair color.
• Lost Hair Details: Return to the Select and Mask workspace and refine the edge again, or try using channels for a more detailed selection.
• Incorrect Background Colors in Hair: Use color decontamination techniques or paint over the problematic areas with a soft, low-opacity brush.

Tips for Effective Hair Masking

• Work on High-Resolution Images: The more detail your image has, the easier it will be to isolate individual hairs.
• Zoom In: Zooming in helps you achieve more precise masking, especially along tricky edges.
• Use Multiple Techniques: Combining methods (like Refine Edge with Channel masking) often yields the best results.
• Experiment with Brush Settings: Use different brush sizes and hardness settings to add missing strands or clean up edges manually.

Final Adjustments and Finishing Touches

After creating a hair mask, you may need to make some final adjustments:

1. Blend with the New Background: If you’re placing the subject on a new background, adjust the Brightness and Contrast to match the lighting conditions.
2. Apply Color Adjustments: Use Hue/Saturation or Color Balance to blend any color mismatches.
3. Add Depth: For a more realistic look, consider blurring the edges slightly with a small Gaussian Blur (1-2 pixels) to mimic natural hair texture.

Conclusion

Mastering hair masking in Photoshop requires some practice and patience, but these techniques should get you well on your way to making clean, professional masks. Practice the skill of masking on a variety of images and different backgrounds to fine-tune it.

Remember, the secret behind effective hair masking lies in bringing multiple techniques into play to tackle various complications. With these advanced methods, you’ll be able to handle even the most complex hair masking projects like a pro.

Happy editing!