If you do this around all of the edges that need to be fuzzy and soft, you’ll get a very convincing cutout of the object that will look good on all colors and backgrounds. Simply click and drag to smudge and paint a fuzzy edge, recreating the softness of the butterfly, without all the ugly white pixels. Save yourself the frustration and make sure you do this! You won’t be able to smudge the edges this way unless the Layer Mask has been applied. The mask has been “Applied” so that the layer is rendered this way, so that we can make changes to the edges. Here’s the butterfly isolated into its own layer.
You may need significantly less than 100% if you’re using the mouse–if you’re using a pressure sensitive tablet, you’ll be able to control the strength much more accurately, and will likely want a high setting. There are a lot of settings, and you may have to experiment with the ones that suit you best, but the most important is the “Strength” of the Smudge Tool.
Click and hold down the Blur Tool until you can switch to the Smudge Tool. You’ll want to find the smudge tool buried in your toolbox. There is a handy trick to take a rough selection like this one and turn it into a convincing fuzzy-edged object. For our demonstration purposes, we’re going to use it and show you how we can successfully get a good image out of this bad mask. You can safely ignore using the “Refine Mask” tool in the Mask Panel if your results are as poor as this one. While the Edge Detection in Photoshop can be very useful, it’s not a magic, one-size fits all solution. The computer struggles to mask out the proper parts of the butterfly and properly isolate the object. We can refine our results with the Mask Panel and “Mask Edge” but it’s a little bit like getting blood from a stone. This was done with the paint bucket, and simply gives horrid results. This mask obviously is doing a horrible job of actually cutting out this butterfly. Let’s take a quick look at the tool for editing masks, and see how we can improve the selection and keep those fuzzy edges. Some objects, like this butterfly, have soft edges, and are notoriously hard to cut out. One of the flagship features of Photoshop CS5 is the tools for refining masks with “Smart Radiuses,” to help mask objects accurately and quickly. Removing Fuzzy Objects From The Background Simply using the right settings will help you isolate your objects better, if not perfectly. The difference is pretty obvious the left image uses a Tolerance of 5 and no anti-aliasing, while the right uses a tolerance of 40 with anti-aliasing.