GTAO test

[chrisj]

Corrected GTAO

The corrected implementation in shaders.ts explicitly evaluates GTAO in view-space slices.

1. Reconstruct the surface point

The depth buffer stores 1 - gl_FragCoord.z, so the shader reverses that and reconstructs view-space position:

$$ P = \mathop{\mathrm{unproject}}(uv,z) $$

float fragZ = 1.0 - depthVal;
vec3 P = viewPosFromDepth(uv, fragZ, uInvProjection);

2. Decode the view-space normal

Normals are packed from $[-1,1]$ into $[0,1]$:

$$ N=\mathop{\mathrm{normalize}}(2N_{\mathrm{packed}}-1) $$

A zero-RGB packed value is reserved as the no-AO sentinel. Background, annotations, skeletons, highlighted meshes, and other excluded fragments therefore return full visibility immediately.

3. Determine the sampling radius

uRadius is a fraction of viewport height. It controls both:

• screen-space marching distance;
• view-space falloff distance.

The projection matrix converts the viewport fraction into a view-space radius:

$$r_{\mathrm{view}} = r_{\mathrm{screen}}\frac{2w_{\mathrm{clip}}}{P_{11}}$$

For perspective projection, $w_{\mathrm{clip}}$ varies with distance. For orthographic projection it is constant.

4. Rotate the sampling pattern

A pixel-dependent hash rotates the four sampling slices:

float noiseAngle = gtaoHash(gl_FragCoord.xy) * PI;

This prevents persistent axis-aligned artifacts. Another hash jitters the sample positions along each slice.

Only a half-circle of directions is needed because every slice is sampled on both its positive and negative sides.

5. Construct a view-space slice

For each screen-space direction, the shader reconstructs a nearby point at the same depth to obtain a view-space tangent:

vec3 sliceTangent = normalize(tangentPoint - P);
vec3 sliceNormal = normalize(cross(viewDirection, sliceTangent));

The slice is the plane containing:

• the direction toward the camera;
• the current screen-space sampling direction.

For perspective projection:

$$ V=\mathop{\mathrm{normalize}}(-P) $$

For orthographic projection:

$$ V=(0,0,1) $$

6. Project the normal into the slice

Only the component of the surface normal lying in the current slice contributes to that slice’s integral:

$$ N_s=N-n_s(N\cdot n_s) $$

The shader records both:

• the normalized projected direction;
• its original length $\lVert N_s\rVert$.

vec3 projectedNormal = N - sliceNormal * dot(N, sliceNormal);
float projectedNormalLength = length(projectedNormal);

The length weights the slice’s contribution. A slice nearly perpendicular to the surface normal contributes little.

The signed angle between the projected normal and the view direction is normalAngle.

7. Find the two horizon angles

An unobstructed normal-oriented hemisphere spans:

$$ [\theta_N-\pi/2,\ \theta_N+\pi/2] $$

These are the initial negative and positive horizon limits.

As samples are examined, each side’s horizon moves inward when geometry blocks part of the visible arc:

horizonPos = min(horizonPos, ...);
horizonNeg = max(horizonNeg, ...);

Nearby samples have more influence. The distance falloff is:

$$ f(d)=\mathop{\mathrm{clamp}} \left( 1-\frac{d^2}{r_{\mathrm{view}}^2}, 0,1 \right) $$

Rather than multiplying a slope by this falloff, the corrected version interpolates the sample horizon toward the unobstructed hemisphere boundary. A sample at the radius therefore contributes no occlusion.

The minimum accepted sample distance is also relative to the view-space radius, instead of using one fixed world-space threshold.

8. Integrate the visible arc

Once the two horizons are known, integrateArc analytically integrates visibility relative to the projected normal:

float integrateArc(float horizonAngle, float normalAngle)

The positive and mirrored negative arcs are combined:

integrateArc(horizonPos, normalAngle) +
integrateArc(-horizonNeg, -normalAngle)

This is the key difference from the old implementation. The shader now measures the angular area of the visible normal-oriented hemisphere, rather than using the largest sample-normal dot product as a proxy.

Each slice contributes:

$$ V_d= \frac{\lVert N_s\rVert}{4} \left[ I(h^+,\theta_N)+I(-h^-,-\theta_N) \right] $$

The final result is the average visibility over all four slices:

$$\mathrm{AO} = \mathop{\mathrm{clamp}}\left(\frac{1}{D}\sum_{d=1}^{D}V_d, 0, 1\right)$$

---

Behavioral Difference

Consider a tilted but perfectly flat plane.

In the previous version, samples along that plane can have positive dot products with the normal because of projection, discretization, or slice orientation. The maximum-dot heuristic may interpret this as occlusion and darken the plane.

In the corrected version, the normal is first projected into each slice. The plane’s horizons coincide with the boundaries of its normal-oriented hemisphere, so the integrated visible arc remains approximately complete:

$$ \mathrm{AO}\approx1 $$

For a crevice or pit, nearby geometry moves both horizons inward. The visible angular interval shrinks, producing a lower AO value.