CEL SHADING - UE4
I worked on a project in 2017 which involved re-creating a similar Cel Shaded look to my artwork to that of Studio Ghibli's Spirited Away.
Below you can view some of the paper I wrote taken and put together details the process I used to develop it.
Spirited Away Art Breakdown
- Spirited Away is a Japanese Animated film set in an abandoned theme park frequented by spirits and gods at Yubaba’s fantastical bath house. Released in 2001
In this scene from the film a clear distinction can be made from the Cel Shaded animated layer assets and the traditional painted backdrop.
The image below shows the composited scene .
This image shows the layer where the assets (Cel Layer) are shaded with banded lighting or a gradient to give the impression of three-dimensional volume.
Additionally, to the Cel shading technique the colour palette and detailing are characteristic of the ‘Cel Shaded’ Art Style.
The image below shows the flat colours used on the Cel layer relying on the shading for value control and the 3D impression.
Detailing for the flat colours is done with the black lining providing suggesting contouring and geometry displacement. This can be seen in the image below.
Colour consideration:
Ok so let's take a quick look at a case study for the colouration and how this ties into the shading.
2003: Naruto: Clash of Ninja
Naruto is a good example of games created from Cel Shaded Cinematics because of how closely the two screen shots from the Anime and the Game compare artistically.
The colours of the shading are important and must be considered when aiming for a comparable likeness to the original Anime. The colours from the Game version of Naruto matches the shaded colours from the Anime.
This is a good factor to base likeness from in any following recreation of the Cel Shader in a game engine, which can also call for an understanding on how shadows affect the colouring.
Shading consideration:
“The value of Chroma will change across the surface but the hue will hardly change at all”
- Paul Foxton, 2017 - A Realism Artist
Foxton talks about how traditional artists may be tempted to simply paint translucent shadows on their paintings using grey tones (monochrome), in which he states this will look odd unless the colour of the shadow is correct to the object being painted. An example of this can be seen in the painting of the orange slice below.
Faith Te Orange Half
Here the example is coloured shadowing to match the surface colour of the object casting shade away from a directional light source.
This can be viewed digitally in the Adobe Photoshop® by multiplying layers together.
The latter of the two below shows a colour multiply layer over an existing Orange layer.
This shows the working method behind the banded lighting in Cel shading as it matches the desired effect seen in the anime and the same effect as the shadowing in Faith Te Orange Half painting above.
Effectively the grey scale shade is multiplied with the colour of the object to achieve the desired artistic look.
Whilst it is possible to use coloured shading it would change the nature of the hue which goes against the principle for this artistic style. Grey scale shading is what is needed to make this more useful for use with Unreal 4 Engine’s Material Shaders; a brief and mind-numbingly simple mathematical equation can be included below.
In Photoshop®: Previous Layer × Current Layer = Product
In Unreal 4 Engine: Diffuse Colour × Shader Layer = Product
Controlling the shaded effect will be important, the Unreal documentation suggests a Colour Look up Table (Appendix 1) for controlling the greyscale shading.
Shading consideration:
From the examples above the following devlopment for our shader can be made can be made:
Cel Colouring Shader type 1:
Creates very dark monochrome shadows, needs to be coloured shadows to meet the criteria for Anime art style.
Cel Colouring Shader Type 2:
Creates good looking coloured shadows with very specific similar hues with different values. This is like traditional painting methods. However, this is limited to set textures painted for the material. Which is not ideal for a games shader as flexibility and multiple asset usage is desirable.
In order to control the artistic look the two equations can be applied together as is shown in The shader type below:
Cel Colouring Shader Type 3:
This could be controlled by combining the two equations, first dividing the colour from the scene’s light (desaturate) with the intended colour for the light and then multiplying the original colour (or in this case texture) with the result of the division.
With the monochrome colour controlling the shading a separate scalar parameter can be multiplied with it to adjust its intensity, however if the chosen tones are non-graduated to produce a believable shading then changing the intensity will not have much of an effect.
CEL Shading in UE4 (2017)
Epic games provide a base guide on how to create a Cel shaded look in their engine via a tutorial written by Arran Langmead (2015).
The method shown in blueprint below shows how to create the material needed for a post process Cel Shaded look.
This set up is the basic derivative of the maths talked we just about, where a division of the Diffuse Colour to the scene’s lighting is applied (desaturating the colour from the light) and plugged into a Colour Look Up Table (CLUT). This creates the banded lighting look to the scene.
In order to get the texture colour back into the scene the scene’s initial diffuse colour is then multiplied with the desaturaded lighting information. This then produces an approximation of the the desired Cel shaded look.
Limitations
The method demonstrated above works well to create the flat, banded shading which is the desired effect for this project when compared to images from Anime films and T.V shows.
This system however, is very limited to what it can be applied to. It requires a Colour Look Up Table to be created to determine the overall look for the shading. This is not very efficient and would require multiple tweaks and exports of the same image to control the colouration, size and number of bands to the shading.
No Sky Dome
In addition to the issue with adaptability this shader does not allow for the default Sky box to be rendered, because of its blueprint usage for the emissive channel and post processing.
Not having a sky dome would mean that environments using this shader would work in enclosed spaces. It would work with a custom mesh as the skydome would have to be set to not cast shadows and as emissive colours are not usable with this shader also that would again just leave the scene with a black environment surrounding it.
Further limitation to using this method is the how it will impact the artistic look for the Studio Ghibli films and other animated films that use the Cel shaded look.
As is shown in below, the scene consists of both Cel shaded assets and hand drawn/ painted elements. This means that the scene will need to look both Cel shaded and hand painted at the same time which is why this shader model is not useful specifically for this type of art style.
We need a more adaptable method for the Cel shaded workflow in Unreal. This would allow for both Cel shaded and Non-Cel shaded assets in the same scene.
Further Development: Cel Shader
To have both Cel shading and Physically Based Rendering, two types of rendering must occur at the same time.
With the current solution for Cel shading this is not possible as the Cel Shading affects the entire scene based on the Post Process volume.
Post Process effects within Unreal suggests that masking is available within the influence of a Post Process volume to select which visual is rendered per asset rather than globally applying it. This is documented by Epic as Custom Depth:
“This separate feature enables masking of certain objects by rendering them into another depth buffer (called custom depth buffer). This adds extra draw calls but no more materials. The rendering is fairly cheap as we only output depth. The feature can be activated on the mesh (e.g. Static Mesh Properties / Render Custom Depth)”
-Epic Games
This could be utilised to only apply the Cel shaded render to specific assets as is required for the Art style. The previous method’s maths can still be applied to create the banding and colouration to the shade all that would be required is to define the mask.
The documentation suggests that the node Scene Texture: Custom Depth can be plugged into a RGB mask and this will create the distinction between what should be mask and what shouldn’t be. In this instance, it would be required to input a desired render into the masked space. An IF statement would do this.
Masking with Custom Depth
This set up is the basic derivative of the maths talked we just about, where a division of the Diffuse Colour to the scene’s lighting is applied (desaturating the colour from the light) and plugged into a Colour Look Up Table (CLUT). This creates the banded lighting look to the scene.
The understanding I took from the documentation is that if the Scene Depth and the Custom Depth values are different then within the Post Process Volume, set to Unbound (Tick box Unbound) for ease, this will select which render material is displayed. In this case PBR or Cel Shading.
If the Postprocess is set to serve as the default render i.e 1 and the Depth depth set to a value of 0 then the IF statement can be used to input the Material of the Post process volume custom depth as the render method for that selected asset.
For Example:
A = Scene Tecture: Scene Depth (Masked to the Red channel) 1
B = Scene Texture: Custom Depth (Masked to the Red channel) 0
IF A<B then do not show the Cel Shaded Material when plugged into Mask RGB
Meaning the Maths from the initial research in the material could be applied when A>B. This would then show the Cel shaded render style to any asset which has been enabled to use Custom Depth offset 0 as shown below:
Masking with Custom Depth
Whilst I researching the Custom Depth usage, an article ‘tutorialising’ the desired effect for the Cel shader discovered.
An Unreal community member Thura Oo, has provided a shader which utilises the Custom Depth Offset to enable the use of the Sky box, individual assets rendered with Cel Shading, Lights and normal materials.
This documentation backs up the previous postulation that not only could Depth Offset solve the issue of rendering two different type of Art styles but also that the IF statement Maths would/ does work to differentiate the material being used in the post process volume.
The result of the work is displayed below, link to article here: 80 Level
The framework:
-
Custom Depth offset:
To control which assets are Cel shaded in the Scene
-
Sky dome rendering:
Displays the Skydome even when Cel Shading is globally rendered
-
Individual Banding control:
Controls the colour and intensity of the banding rather than relying on a CLUT.
ThuraOo Framework
The set up below is as shown in the article. The result is as expected, creating banded light and allowing for objects to be both Standard and Cel shaded whilst displaying the skydome.
Whilst this scene works to create the cell shaded look further research should be made into the considerations for the spirited Away scene. It is not enough to apply the Cell shader to the scene and expect that it will visually be like the Anime. This shader model does meet the criteria listed prior to its discovery:
-
Controllable Shading
-
Displays Skybox
-
Selective asset rendering
The Material being testing in Figure 22 is a tiling padded texture. “A” has been set to use Depth offset 0 where as “B” has been set to use Depth offset 1. In this example the Shader has been setup with only a two tone banding and a direction light has been added to the scene in order to create a shadow direction.
An additional test was done to discover what parameter for Rougness and Metalness where needed, or indeed if PBR could be used with this set up. Most authors online talk about how this is not possible, or that it yeilds poor results – Ide 2017 stats that:
“…any materials that are in this cel-shaded scene need to have
-
Roughness value of 1
-
Metalic value of 0
-
Emmisive value of 0 ”
The texture method for achieving the Cel shaded art style involves using flat colours much like an Albedo with no baked in shading like a Diffuse texture would have.
A good comparison between the two methods was broken down by Ide in his explanation of textures for Cel shading. The image above shows the Diffuse texture and the Albedo Texture for a skirt to be rendered in a Cel shader.
The image above shows the comparison between the two with the Albedo on the left and the Diffuse on the right. The Albedo method comparatively to anime art looks accurate whereas the Diffuse texture looks dark and oddly shaded. The missing element here i feel is the dark detailing lines across the model.
Texture method comparison. Ide
The method of using Laplacian filtering (Appendix 5) is detailed also in Ide’s Cel shader breakdown which attempts to replace the need for hand drawing the detail lines on, instead rendering them based on edge detection in Unreal Engine 4. This method should be tested to see if it can be implemented later into the existing shader.
Traditional methods for applying the details would be to draw the lines onto the Albedo by hand which is more time consuming. Alternatively using the Laplacian filter the edge normals of the asset would need to be adjusted to ensure much harsher edges meaning the normals need to be split as discussed by おぎまふ (Oshara) 2016.
Studio Ghibli use water colours as the base art style relying on two values to differentiate the lit and shaded areas see below.
This is discussed as:
“After the colour scheme is determined, colours are hand painted onto the cells using watercolours. By using different tones for objects in front of and behind the characters, the image is given extra depth.”
- GhibleGabble commenting on the process Studio Ghibli have presented in their documentary on behind the scenes.
Painting types
The image below shows a selected asset that has the quality for Tenebrism shading style.
Assets like this will be excluded from the Cel Shading pass and allowed to be lit be the standard light and shadow method which is default to Unreal Engine 4.
The style of painting at Studio Ghibli is water colours, as talked about by the website Ghibli Gabble whom have compiled a production process from various sources at Ghibli studios. Whilst the Cel Shading colours are flat the background or static objects are painted with the additional shading.
An example for this is shown in the behind the scenes images for Nausicaä of the Valley of the Wind where demonstrate painting the background scenery as shown below which is a similar look to the backdrop for Boh’s Room in Spirited Away and the haystack scene from となりのトトロ (My Neighbour Totoro) also shown below:
Camera considerations
Development of the room should be based around the 35mm FOV from a camera set up in Unreal 4. This will ensure the scene can be closely approximated when positioning assets and setting up lighting.
There are other camera FOVs provided in the Ghibli Sprited Away Artbook for the same scene however as the examples below show they do change the perspective quite a bit.
The first image is from the film directly as a standard 35mm lens whereas the latter is from the Art book and is mostly likely a 35mm Vista Vision or Vertical Vista lens perspective.
Ideally the intention should be to match the image from the film as this is directly from the Cinematic rather than an impression render of the same room.
ANNNND I'm still writing this up... so come back another time for the rest!
