Semi-Transparent Layer Rendering in 3dsMax

In this tutorial, I’ll be sharing a quick technique for removing the glows, halos, or other anti-aliasing artifacts from your matte/shadow renders or Render Elements renders.  The key here is getting the alpha map to apply to the object before the anti-aliasing effect blends the edges of your object with the background.  See the examples below.

Notice the black halo around the green areas. This is where the transparency is failing.

These black halos are caused by anti-aliasing and attempting to render at transparent image out of 3dsMax.  When the image is smoothed during the render it blends colors together.  If our color map is anti-aliased separately from the alpha map, we get color bleeding.  In the image above, I’ve replaced the back background

Summary

I detail the following steps below, but for people looking for the quick answer, here it is.

1. Under Customize > Preferences… > Rendering tab, select “Use Environment Alpha”.
2. Render against a flat, black background – this isn’t optional.
3. Save to 32-bit .tga with “Pre-Multiplied Alpha” turned off.
4. Load the image in Photoshop and select the alpha channel via Select > “Load Selection…”
5. Create a layer mask for the background layer.

Render with a Black Background & Transparency

Now I won’t tell you how to set a black background since it’s the default and, if you’ve changed the background from the default, you know how to change it back.

Go to the Customize menu and select "Preferences…"

Select the "Use environment alpha" option.

Save the Render to .tga with Pre-Multiplied Alpha Disengaged

Disengage the pre-multiplied alpha setting

Now you’re ready to open the file in Photoshop.  Just open your new .tga file, and re-apply the alpha map that has been saved separately embedded within the image file.

Once you've opened the new .tga file, go to Select and pick "Load Selection…"

Load the image's alpha channel. It should be the default.

With your alpha channel as your selection, create a layer mask.

And here’s our final result without any halos.  I’ve added an image of clouds to the areas where the background was black.  The areas in purple were a purple background that was anti-aliased into the image.

Halos are gone- the object can be anti-aliased against any background.

That’s a wrap.  Enjoy!