A Cel-Shading Example in XNA 4.0

Recently I came across an excellent cel-shading tutorial written by digitalerr0r titled XNA Shader Programming – Tutorial 7, Toon shading for the XNA 3.0 framework.  With a few minor updates we can get this working in XNA 4.0 as well.

This post will cover the changes necessary to apply the toon and edge detection shaders presented by digitalerr0r in XNA 4.0.  For a more in depth explanation of the cel-shading techniques used here please see digitalerr0r's original article.

Complete source code can be found at the end of this post.

HLSL Shaders

While it wasn't strictly necessary to update the shaders presented by digitalerr0r I chose to modify them to follow the HLSL code style preferred (inferred) by the Visual Studio effects templates.

The Toon Shader

Most of these updates are trivial variable name changes and the use of a struct for the input to the vertex shader function.  The most significant change here was splitting the composite word, view, projection matrix into three separate components.  While this might have a small performance hit (the matrix multiplication happens for each vertex) the style and use of the effect is more consistent with default effects provided by the XNA framework.  This should make the effect more recognizable to those of you that are just starting out with XNA and you more experienced programmers out there will know how to switch it back.

The Edge Shader

Again just a few minor changes here (and a  few small tweaks for correct alpha blending and more generalized scene handling).  First even though we do not have a vertex shader I am preserving the inferred effect style of the XNA templates.  Namely the input to the pixel shader is the output of the vertex shader.  Thus we still have our VertexShaderOutput struct that is the input for the pixel shader function.

Next I have replaced the float2 QuadScreenSize in the pixel shader function with an effect parameter float2 ScreenSize.  This variable is the same thing but it is set as a parameter letting the game code scale the effect to the texture that is being processed.

Finally the last change to this shader is a very small, rather subtle update to fix a problem with alpha blending in the original shader.  The return value has been changed to

Color*float4(result.xxx,1);

instead of 

Color*result.xxxx;

When we have an edge pixel result is 0 multiplying Color by result.xxxx causes the alpha channel of the color to also be multiplied by zero.  Remember that a value of 0 is a completely transparent pixel when alpha blending is on.  If we don't do this our nice black outlines could disappear on us!

Game Code

Finally ;) This is where the bulk of the changes are to use digitalerr0r's excellent shaders in XNA 4.0.  The main differences are in the way XNA 4 implements RenderTargets and applies effects.

XNA 4.0 completely reworked render targets, they are still there but the way they are set on a GraphicsDevice is different, supposedly easier and more efficient.  The new RenderTarget2D constructor that I will use takes 5 parameters, the GraphicsDevice, a width, height, color format and depth format.

celTarget = new RenderTarget2D(GraphicsDevice,
    GraphicsDevice.Viewport.Width, GraphicsDevice.Viewport.Height,
    false, SurfaceFormat.Color, DepthFormat.Depth24);

The important part here is that we set the depth format to Depth24.  This gives us a 24 bit depth buffer, without it we will not get proper (or any for that matter) depth culling.  Because our model for this example is rather simple we could get away with Depth16 for a 16 bit depth buffer but I went ahead and used 24.  The other options Depth24Stencil8 gives us a 24 bit depth buffer and an 8 bit stencil buffer (this is what you get when rendering to the default screen render target) and None which is no depth buffer and the default for new render targets.

The next XNA 4.0 change is setting the render targets.  We do not need to worry about render target indexes any more, just set it.

GraphicsDevice.SetRenderTarget(celTarget);

With the rework of the render targets Microsoft also changed up the device render states to make them easier to use.  Creating device states is still expensive so you will want to do that in code that is not time critical or use some form of state cache.  Setting states on a device is extremely inexpensive though and can be done as often as needed.  The important thing for us to know here is that when we set the graphics device to our render target our DepthStencilState will be None and we will not get depth culling.  Fix this by adding

GraphicsDevice.DepthStencilState = DepthStencilState.Default;

after setting the render target.

The next change is with the effect.  In XNA 4.0 it is no longer necessary to begin and end an effect, just apply it.  Otherwise our drawing code is still the same:

foreach (ModelMesh mesh in model.Meshes)
{
    Matrix world = bones[mesh.ParentBone.Index];
    celShader.Parameters["World"].SetValue(world * rotation);
    celShader.Parameters["InverseWorld"].SetValue(Matrix.Invert(world * rotation));

    foreach (ModelMeshPart meshPart in mesh.MeshParts)
    {
        GraphicsDevice.SetVertexBuffer(meshPart.VertexBuffer, meshPart.VertexOffset);

        GraphicsDevice.Indices = meshPart.IndexBuffer;
        celShader.CurrentTechnique = celShader.Techniques["ToonShader"];

        foreach (EffectPass effectPass in celShader.CurrentTechnique.Passes)
        {
            effectPass.Apply();

            GraphicsDevice.DrawIndexedPrimitives(PrimitiveType.TriangleList, 0, 0,
                meshPart.NumVertices, meshPart.StartIndex, meshPart.PrimitiveCount);
        }
    }
}

Our cel shaded model is now drawn into our render target and ready for post processing edge detection.  Again the begin and end for the effect are gone, now we just pass the effect into the sprite batch when we call its begin.  (In this example the parameters for the edge effect are set in the content loading and update methods.)

spriteBatch.Begin(SpriteSortMode.Deferred, BlendState.NonPremultiplied,
    null, null, null, outlineShader);
spriteBatch.Draw(celTarget, Vector2.Zero, Color.White);
spriteBatch.End();

Here we are using an overload of SpriteBatch.Begin that takes a sort mode, blend state, sampler state, depth stencil state, rasterizer state and effect.  Because we do not need special sampling, depth treatment or rasterizing here those are all null.  We do use BlendState.NonPremultiplied to enable non-premultiplied alpha blending though and of course our edge outlining effect is passed in for the effect parameter.

That's it... with those changes digitalerr0r's great cel-shader code will be running in XNA 4.0.

Full source code for my example can be downloaded here XNA 4.0 Cel-Shader