An attempt at Cel Shaded Rendering in the Quake 2 Engine
By David Dynerman

Description

Cel shaded rendering is an artistic style most often associated with cartoons (other names: cartoon shading, 'toon shading)

Some key aspects of cartoon rendering: The goal of my project was to replace Quake 2's standard renderer with a cel-shaded version. I implemented #1 and #3, but had difficulty with #2. An explanation of the main problem encountered and my solution to that problem follows.

A key factor in achieving a cel shaded look is having large areas of constant color in your textures. The important fact I didn't realize before embarking on this project was that this feature is primarily a task for artists. When working on a cel shaded game (for example, Zelda: The Wind Waker) artists are given a limited palette and use broad areas of constant color, but still express the important game information in an aesthetically pleasing fashion. Quake 2's textures were almost the opposite - all 24 bit images designed to make the environment look as realistic as possible. I experimented with various ways to algorithmically create these large areas of constant color.

Results

Click on the thumbnails for larger versions
From an earlier version. To get large areas of constant color I use the average color of each face as a color for that face. Quake 2 tends to be a darker game texture wise, so lots of stuff ends up brown.
From the final version. Here I used a modified version of a popularity/histogram algorithm to get texture colors. The result is more pleasing and closer to the original in terms of texture color. This particular image is one of my favorites.
From the final version. This illustrates my color selection algorithm breaking down. I used some heuristics to get away from Quake 2's constant grays and browns. The bulkheads in this shot refused to give enough 'non-gray' texture information, so their colors were offset and ended up purple.
From the final version. Another one of my favorites. In this particular part of the level, my color selection algorithm worked particularly well. The colors of the stairs, floor and surroundings work well together and look like authentic artist generated cel-shaded textures.

Conclusions

Algorithms for converting full color textures into ones with fewer colors are numerous; however, algorithms which take full color high detail images and produce images with broad areas of constant color that still communicate the intent and composition of the original texture are an interesting, and to my knowledge unsolved, problem. Other Quake 2 cel-shaded renderers I looked have not solved this problem. Instead, they concentrate on sharp lighting and emphasized edges. These renderers use various ad hoc methods to solve the large areas of constant color problem - for example, one renderer replaces all the textures in the game with one cel-shaded style texture. If a solution is found, the benefits are numerous - for instance, one could produce a cel-shaded game without having to train artists in a new style. Possible solutions would include area analysis of the image, edge detection combined with color quantization, or perhaps an algorithm based on aesthetic hints placed in the image.

David Dynerman