3.6 New Features 2/3: Rendering

Preface

In the last article, I introduced you to the exciting new features of the editor in Cocos Creator 3.6. As you can see, the new version brings a nice improvement in both the appearance and the productivity of the editor.

I think many of you would be like me, will pay extra attention to the engine rendering-related features in each version update. For example, whether there are new rendering technologies, whether there are rendering quality improvements, etc.

Next, let's take a look at the rendering-related features added in Cocos Creator 3.6.

Rendering-related features you need to know

• Reflection Convolution based GGX
• CSM, Cascaded Shadow Maps
• Noise Module for Particle System
• Dynamic Mesh
• Anisotropic Lighting
• Surface Shader

Reflection Convolution based GGX

This is a great feature that can greatly enhance the realism of non-smooth surfaces, see the following figure.

From the above image, we can clearly perceive the difference between the effect with convolution (GGX Convolution) and without convolution (Mipmaps).

What is Reflection Convolution based GGX

It means that the GGX algorithm is used to generate the Mipmap levels of Cubemap which is used for Environmental Reflection Calculation.

Does it run with high performance?

Since the convolutional map is pre-generated, you don't have to worry about the performance at all. The performance is the same with using and without using the convolution map, just pre-bake it in the editor.

How do we generate and use it?

Please see the following figure.

This is the Skybox panel in Cocos Creator 3.6, and we focus on two things.

• Env Lighting Type
• Reflection Convolution、Bake（a button）

Tips：PBR materials handle ambient lighting by treating the ambient map as a giant wrap-around light source, with each pixel in the map representing a direction of light brightness. This processing is called IBL (Image Based Lighting), and IBL consists of two main calculations: diffuse and environment reflection.

Env Lighting Type There are three types to choose from.

• HEMISPHERE_DIFFUSE
• AUTOGEN_HEMISPHERE_DIFFUSE_WITH_REFLECTION
• DIFFUSEMAP_WITH_REFLECTION

Selecting XXX_WITH_REFLECTION means we want environment reflection, and vice versa, no environment reflection. If we switch the same test scene to HEMISPHERE_DIFFUSE, we only get the following result.

The difference between HEMISPHERE and AUTOGEN_HEMISPHERE is that AUTOGEN_HEMIPSHERE automatically calculates Sky Lighting Color and Ground Lighting Color based on the environment map.

How to bake reflection convolution map

The Bake button after Reflection Convolution is used to generate the Reflection Convolution Map. When it bakes successfully, it will be used in the IBL stage when calculating reflection color.

If you don't want to use it, just click the Remove button and the Mipmaps of the environment map will be used in the IBL stage.

Can you elaborate on the environment reflection convolution?

For those who want to understand the principles behind it, let's start with two terms.

• GGX：A micro-surface distribution function, the material rendered with GGX algorithm, the transition will be more natural and the distribution curve will be more close to the real object. Most of all, it can restore part of the real BRDF's specular smoothness. The comparison effect is shown in the following figure.

• Reflection Convolution：The reflection map corresponding to the roughness is pre-generated and stored in each Mipmap level of Cubemap by a suitable algorithm, as shown in the following figure.

It is obvious that the Cubemap generated by the GGX algorithm has Mipmaps that are blurrier at each level than the previous one, which cannot be achieved by ordinary Mipmap.

Developers interested in learning more about this can search the web for more details.

CSM, Cascaded Shadow Maps

This is a technique that renders high quality dynamic shadows.

So what is CSM？

CSM is short for Cascaded Shadow Maps

Why it is called 'Cascaded', Please see the GIF below：

平行光 means Directional Light

In CSM, the scene inside the view cone is divided into four layers from near to far (in Cocos Creator 3.6, there are 4 CSM layers by default), and each layer corresponds to a ShadowMap, which is used to calculate the most suitable ShadowMap according to the object position in the shadow sampling stage.

This operates so that the nearer ShadowMap covers less to ensure that the nearer shadows are clear, while the more distant ShadowMap needs to cover more to ensure that the more distant shadows are visible.

What about the DrawCall overhead of CSM?

Although the 4-layer CSM requires four draws of the scene, the additional DrawCall overhead is no more than 40% of the normal ShadowMap due to the engine's culling mechanism and the optimization of the shadow pass draw.

What about the memory usage of CSM? In the Scene Properties panel, there are 4 options for ShadowMap Size to choose from.

• Low_256x256
• Medium_512x512
• High_1024x1024
• Ultra_2048x2048

In fact, CSM 4 levels share a separate ShadowMap, and each level only occupies 1/4 of the ShadowMap Size, for example, if Ultra_2048x2048 is selected, the ShadowMap of each level occupies 1024x1024 size.

In short, ShadowMap memory usage is the same with CSM on and off.

CSM Layer Coloration

In the previous introduction to the editor, we mentioned the rendering debug view feature, which allows you to view the CSM layers in visual, as shown in the following figure.

This feature allows us to visually see the changes at each level when adjusting Shadow Distance and Shadow Invisble Occlusion Range, and to pinpoint the position of each level as needed.

4 layers for CSM is too much, can it be less?

Currently, it can only be set to 4 layers. Later, it may open up the layer settings and the lambda distribution settings of the layers to achieve more accurate layer distribution management and more performance savings.

Noise Module for Particle System

Cocos Creator 3.6 adds a new Noise Module to the particle system, which makes it easy to create some random particle drifting effects, as follows.

The following parameters are currently available for the particle noise module.

• Noise Preview：Preview of the noise map used for the current sampling
• Stength X：Noise intensity in X-direction
• Stength Y：Noise intensity in Y-direction
• Stength Z：Noise intensity in Z-direction
• Noise Speed X：Noise speed in the X-direction
• Noise Speed Y：Noise speed in the Y-direction
• Noise Speed Z：Noise speed in the Z-direction
• Noise Frequency：The noise frequency used to generate the noise map, the higher the value, the more dense the noise
• Octaves：The number of layers of the algorithm used to generate the noise map, the larger the value the more variation and the more calculation
• Octave Multiplier：Strength multiplier of layers
• Octave Scale：Frequency multiplier of layers

What's the Noise Preview map used for?

Andy, One of member from engine team says:There are 3 main reasons to use it.

• 1.Currently all noise control parameters are written to a Noise Map and passed to the particle system to improve performance. So I hope to show the developers what this map looks like.
• 2.It is difficult to notice when modifying some parameters, but Noise Map enables us to perceive subtle changes in parameters
• 3.Helps better communication between designers and engineers involved in particle system development

Look at this irregular, fluttering butterflies, only need one particle emitter to deal with, is not great?

In addition, we can add details to the particle effect using the Noise module, as follows.

Dynamic Mesh

If you want to achieve a small 3D snake that bends and twists and is under your control.

If you want to generate an animated curvy line in 3D space that are dynamically generated according to the indicated direction.

If you want to take advantage of your talent, write a set of 3D effects, clouds, and weather systems by yourself.

The feature you need most is to update model data dynamically.

Creating static mesh in code

Prior to v3.6, Cocos Creator provided only a function for creating static models: the utils.MeshUtils.createMesh function. However, each time it was called, the mesh-related objects were rebuilt, and the performance was only suitable for use in cases where updates were infrequent.

Creating dynamic mesh in code

In v3.6, the utils.MeshUtils.createDynamicMesh function is provided specifically for handling dynamic models, as shown in the following figure.

Three steps to create a dynamic mesh

1、Define capacity using options

When creating a dynamic mesh, 3 parameters should be passed via options.

• maxSubMeshes：the number of sub meshes.
• maxSubMeshVertiecs：the max number of vertiecs of each sub mesh.
• maxSubMeshIndices：the max number of indices of each sub mesh.

2、Call createDynamicMesh to create

In most cases, we have only one SubMesh, so it is most convenient to fill it with data when creating it. The prototype of this function should be like this: createDynamicMesh(geometry,options):Mesh.

However, the designer felt that this was not flexible enough, so she added primitiveIndex. This parameter is used to specify which SubMesh the given geometry data belongs to.

Note：don't forget to call meshRender.onGeometryChanged to update.

3、call updateSubMesh to update

Function prototype: updateSubMesh(primitiveIndex, geometry): void

The purpose is: update the SubMesh specified by primitiveIndex with the given geometry data.

As follows：

Note: Don't forget to call the meshRender.onGeometryChanged function after updating the data.

About the number of dynamic mesh vertices

Recall the two properties in options.

• maxSubMeshVertiecs：the max number of vertiecs of each sub mesh.
• maxSubMeshIndices：the max number of indices of each sub mesh.

The reason it is called "max " is that these two values mainly determine the size of the allocated mesh data, while it is the geometry data that determines how much space is actually used, and as long as the vertex and index data of the geometry data is not out of range the values specified by options, it will work fine.

For the case of fixed vertices and indexes, it is recommended that the values of maxSubMeshVertiecs and maxSubMeshIndices be consistent with the geometry data to avoid unnecessary memory wastage.

However, for the case where vertices and indices will change, you need to make sure that these two values always satisfy the needs of geometry.

A simple example.

Suppose we want to implement a mini-game. It has 10x10x10 small cubes that form a large cube, and when the user taps on a cube, the cube disappears.

We create a dynamic mesh with only one SubMesh, with maxSubMeshVertiecs of 8000 and maxSubMeshIndices of 36000.

When the cube disappears, we just need to remove the corresponding vertex and index data from geometry and call updateSubMesh to refresh the mesh.

Similar is the case with particle systems, vegetation systems, sword slash effects, etc.

DEMO

The Cocos Engine team provides an example of a loong. Clicking on the Update button will display the loong from nothing to complete, as follows.

For those who need to study, you can check the official Github repo: https://github.com/cocos/cocos-test-projects/tree/v3.6

scene: assets/cases/dynamic-mesh

Anisotropic Lighting

In nature, there are objects with a large number of small, flush grooves on their surfaces. Ordinary graphical lighting models cannot reproduce the lighting effects of such objects' surfaces well, so a technique was created to specifically simulate such lighting effects: anisotropic lighting.

Some of hard-to-express surfaces.

Although the implementation principle is complex, using anisotropic lighting in Cocos Creator 3.6 requires only the following three steps.

• 1、Create a new material in Cocos Creator
• 2、Switch Effect of material to surfaces/standard
• 3、Turn on the IS ANISOTROPY macro

Related-Parameter explanation：

• USE ANISOTROPY ROTATION MAP：Whether to enable anisotropy rotation map.
• Anisotropy Shape：0 stands for perfectly isotropy, 1 stands for perfectly anisotropy. This parameter can also be interpreted as Intensity, but since it will determine the final shape of this effect, the word Shape is used.
• Anisotropy Rotation：Control the direction of anisotropic stripes
• Anisotropy Rotation Map：Used for precise control of stripe direction, valid only when USE ANISOTROPY ROTATION MAP macro is on

The effect rendered in Cocos Creator 3.6 is as follows.

Surface Shader

What is Surface Shader？

Surface Shader is not a new shader species, it is a process and framework provided by the engine to facilitate shader writing. It encapsulates the underlying Vertex/Fragment Shader, eliminating the need to write repetitive code, thus reducing the complexity of shader writing and improving the efficiency and compatibility of shader writing.

Why Surface Shader is needed？

As the 3D features of Cocos Creator are getting better and better, the workflow is getting perfect. More and more 3D developers, art, TA, and computer graphics enthusiasts are becoming Cocos Creator users.

About Shader writing, the feedback I received focused on the following points.

• Cocos Effect content is too much, newcomers are easy to give up at the first glance
• There are some PBR operation-related parameters and procedures that are not exposed enough, so you need to change the built-in shader file in chunks to achieve the desired effect.
• The surf function exposes too many things that don't normally need to be changed, such as normals, roughness, metallic, and other basic operations.
• No modularity, and the implementation of new features requires patching on the default Shader, leading to difficulties in compatibility with new versions and reuse across projects

Surface Shader greatly improves the above situation and makes it easier to customize shaders. The simpler writing style makes it easier to write shaders and more compatible.

How simple it could be?

As shown in the figure below, a complete standard-fs has only 10 lines in the Surface Shader.

In Surface Shader, a complete Vertex Shader or Fragment Shader needs to contain six parts: Macro Definition, Include Files, UBO Definition, Lighting Model, Shader Stage, Output Target. Developers can customize each of these six parts as needed, and keep the default for the parts that don't need customization.

How to customize Surface Shader？

There is a complete explanation of how to write a Surface Shader in the official documentation, and one of the most important things is the macro definition mechanism.

In the Internal/chunks/surfaces/default-functions directory there are three files common-vs, standard-fs, and toon-fs, which define all the built-in functions that can be replaced with macros. The following figure shows.

#ifndef CC_SURFACES_XXXX means that if CC_SURFACES_XXXX is not defined, then the following function is used.

Take the simpler RimLight as an example, just define the #define CC_SURFACES_FRAGMENT_MODIFY_BASECOLOR_AND_TRANSPARENCY macro in CCProgram surface-fragment %{ ... }% section in your SurfaceShader and implement the vec4 SurfacesFragmentModifyBaseColorAndTransparency function.

The above is about Surface Shader writing, I think the most core mechanism part. Due to the limited space, the rest of the content will not be developed in detail, you can combine the official documentation and surfaces/standard.effect to study.

Additional notes

1、The Rendering Debug View feature in the editor only supports materials that use the Surface Shader.

2、Anisotropic lighting is currently only supported in the Surface Shader.

3、For migration, compatibility and readability, it is recommended that the Surface Shader be used in preference for new projects.

Summary

The rendering-related updates are mainly focused on Reflection Convolution, CSM, Noise Module, and Dynamic Mesh. There are of course many rendering-related enhancements that are too subtle to appear in this article, but that doesn't mean they're not important.

The value of these details will be left to you to explore. What are you waiting for? Open Cocos Creator 3.6 and take action!

In the next article, I'll take you through the performance improvements in version 3.6, so stay tuned!