Soft Outlines Tutorial | UE4.27 to UE5.4
Fix for getting Tom Looman's Soft Outline Tutorial working in UE4.27 to UE5.3 and an addition to increase the quality with a particular edge case.
Anti-Aliasing problems:
There is a chance that you will see a sawtooth-like line between the object being outlined and the soft outline itself. To fix this go into your "Project Settings | Rendering | Default Settings" section then cycle through the "Anti-Aliasing Method" used. Currently I'm using TemporalAA, but you might find better results with another, and then above in the "Postprocessing" section (Project Settings | Rendering | Postprocessing) try using the "Custom Depth with TemporalAA Jitter" option too. Hope this helps.
Possible errors you will see in UE5:
Enabling the "Enable Stencil Test" option under the advanced section of the Post Process Material causes the shader to break but that should be off by default, however to make sure that it is. Click on the background of the material graph (basically no nodes selected) and you should find it under the details panel by searching for "stencil". Also in the project settings make sure the Custom Depth-Stencil Pass is set to "Enabled with Stencil" located in the "Project Settings | Pendering | Postprocessing" section as it is set to "Enabled" by default which doesn't include the stencil buffer. A quick way of testing if you have access to the stencil buffer is to select an object with a static mesh then under the static mesh component set "Render CustomDepth Pass" to true (quickest way of getting all the revelant settings is to type "custom" into the static mesh component's search bar and it will appear under the Rendering section) and change it's "CustomDepth Stencil Value" to something greater than 0 (0 is not an usable value). Last under the viewport view mode drop down menu (click on the button that usually says Lit at the top left of the editor's game window, next to the button that says Show) then hover over the "Buffer Visualization" section and then select "Custom Stencil". if the screen remains black then the stencil isn't enabled in the project settings else you should see a color along with a value between 0 and 255 inclusively.
All Links:
1. soft outlines (as seen in the tutorial I don't make a Material Function and just used a custom node, if you plan to easily re-use the code then setup the Material Function): https://www.tomlooman.com/unreal-engi...
2. custom shaders: https://www.raywenderlich.com/57-unre...
3. mobile supported outlines: • How to Make Toon Outlines in UE5
4. multicolor soft Outlines that I'm not sure if I found before using the SceneTextureLookup (also haven't read more than a minute but it looks like it could be very helpful in learning more about shaders): https://ferkizue.blogspot.com/2018/10...
5. Link showing possibly still accessible variables for custom HLSL code: / unreal-engine-4-rendering-part-4-the-defer...
Timestamps:
0:00 Intro
0:45 Setup the code from Tom Looman
0:55 Code fixes and explanation
4:00 https://docs.google.com/document/d/1Q...
4:15 Additions to the shader
7:30 Con of the shader