Introduction: Ambient Occlusion Rendering in Maya
Ambient occlusion (aka AO) is an algorithm / technique used in 3D computer graphics for calculating how exposed each object (in a scene) to ambient light from the environment.
In other words, AO attempts to simulate how ambient light would bounce around off objects in the scene. Its a cheap, yet effective, technique for calculating soft shadows in the scene. Its cheap in the sense that its faking global illumination. Global illumination is another algorithm / technique for calculating how exposed of an object's fragments are in a scene.
You should care about ambient occlusion because as a designer / artist / architect / etc its a great way to quickly create renderings that look decent and show off the beauty of the objects / scenes you create and it takes about 5 minutes to setup in Maya.
For this instructable you'll need a copy of Maya & Mental Ray. I'll be using Maya 2015, you can follow along using older versions of Maya as well.
Step 1: Create a New Scene and Populate With Objects
Go to File and click on New Scene.
Then place a couple objects in your scene. The quickest way of doing this is to click on the Polygon Tab from the Shelf and creating a couple primitives. Here I've made a plane and a cube, centered at the origin.
You can use the Channel Box / Layer Editor to type in values so you can have precise control over how big your geometry is and where it is placed in your scene.
Step 2: Create a New Render Layer
Great! Our scene setup and ready for more awesomeness. Select everything in the scene by clicking and dragging over all your objects. You can also do this my going to Edit and clicking on Select All.
Now we're going to create a new layer. One way to do this is to click on the Channel Box / Layer Editor icon located on top right of Maya's main application window. There is a photo of it here in case you get lost.
Once you have the Channel Box / Layer Editor open, click on "Create New Layer and Assign Selected Objects" icon. This icon is the right most icon in the Layer Editor Section. Its a bit cryptic, if you mouse over the icons and pause for a second, Maya will display a description of what the button does.
If your object disappeared or are not showing up, its most likely because they weren't added to the render layer. Fear not, just select all your objects and right click on the layer and click on "Add Selected Objects".
Step 3: Create an Occlusion Pass
Now that we have all our object in a new layer, we are going to tell Maya that this layer is an occlusion pass layer, which will do a lot of magic for us. First we need to get back to the Attribute Editor. To open the Attribute Editor, click on the icon third to the left (located near the top right of Maya's main window). The image above shows where it is as well.
Once the Attribute Editor is open, click on the "layer1" tab and then click on the "Preset" button. Now maya will give you the option of many different types of layer presets. Select the "Occlusion" option.
If everything went according to plan, your entire scene should have turned black.
Step 4: Render, Tweak, Render, Tweak, Render
Woohoo! RENDER TIME! Click on the Render Icon (looks like a director's clapboard) and BAM! We have a crude render.
By default, Maya's Occlusion pass has conservative settings to keep render times down. In turn the render quality is not great. But we can tweak a couple parameters and make it better. Lets do this meow.
In the Attribute Editor, click on the arrow / box icon on the right of the "Out Color" parameter. Now the Channel Editor will show the parameters we'll be changing. We'll be tweaking the Samples, Spread and Max Distance.
Its hard to exactly say what these parameters correspond with since I don't know the inner workings on Maya's AO rendering algorithm. But I'll explain what these parameters do to the render quality.
By increasing the Samples, your render will look less noisy and grainy. By decreasing the Spread, your render will look as if the contrast was increased. By increasing the Max Distance from 0.000 to any non-zero value Maya will decrease how far the shadows extend. The value 0.000 essentially represents infinity.