How Speckle Generates Outlines from Geometry Data

Speckle View Modes

Originally, inspired largely by Rhino’s view modes, we aimed to add visual variety to our viewer with just a simple click or flip of a switch—no extra loading times or additional data required. However, by the end of the process, we ended up developing an entirely new rendering pipeline that not only supports a wide range of view modes but also allows users to create their own.

Here’s the first sketch we created:

Edges Zoo

Most of the view modes we planned to implement involved displaying edges. However, the term "edges" can be quite nuanced, especially in AEC circles.

From a strictly rendering perspective, edges have a clear definition: they are the lines connecting the vertices that form rasterisable primitives. If that were all we needed, the task would be straightforward, resulting in a classic "wireframe" view mode.

Unfortunately, the reality is more complex, requiring us to first define what edges are—or, more precisely, what we consider them to be.The viewer uses triangles exclusively to render geometry. Speckle data on the other hand, does not, as it preserves the original face cardinality. This means there is a “disagreement” between the received geometry and the rendered geometry, particularly regarding edges.

Geometry Face Edges(post-conversion):

Geometry Face Edges (pre-conversion):

Strictly speaking, based on the image above, the pre-conversion edges look cleaner. However, their existence isn't guaranteed when coming from all connectors. Some connectors send already triangulated geometries, such as BReps or other parametric surfaces, which brings us back to square one.

We could use something similar to Three.js’s built-in EdgesGeometry to remove the extra edges caused by triangulation. However, the implicit computation step takes considerable time, especially for dense geometries. Additionally, the edge extraction process requires a threshold value, which doesn't work well for all types of geometry and can lead to failures or sub-optimal edge results in certain scenarios.

To complicate things further, parametric surfaces have isolines or isocurves that aren’t necessarily edges, but are important geometrically and should be displayed alongside “normal” edges. It seems like there's an entire "edge zoo" out there.

Regardless of the approach we take, we must remember that rendering edges will be proportional to scene complexity. Edges also require memory space, both in the heap and on the video card. And if the scene happens to be "on the surface of the moon," as many projects seem to be, we’ll need to apply RTE. This doubles the storage required for vertex positions and negatively impacts the overall vertex cache hit ratio.

Screen Space Outlines

We decided to simplify things. Instead of treating edges as geometry that we have to manage ourselves, we opted for screen-space generated edges. This choice comes with several implications.

For example, the edges will no longer be geometric in nature, meaning they won't exist from the application's perspective. We won’t be able to interact with them in a meaningful way, as their existence will be limited to the contents of a framebuffer in video memory. Given this, it might not be entirely accurate to call them “edges” from here on out, so we’ll refer to them as “outlines.”

There are several methods for extracting or generating outlines from an image. This resource briefly covers many of them and provides a clear, structured overview.

Outline Extraction

Typically, the input images used for edge detection algorithms are standard colour images, which aren’t specialised for this purpose. In our case, we have the advantage of generating the input images in whatever form we choose, and we’re unlikely to use the final colour image for edge detection.

When considering gradient-based edge detection, we quickly realise that depth and normal data are ideal inputs for such algorithms, as both inherently display clear slopes around various features of the image. These two types of input can generate gradients that describe the geometry’s shape more accurately than a colour image (or even a grayscale version of it).

Additionally, they’re less prone to false positives—such as when multicoloured objects meet, which could otherwise create false edges based on colour transitions.

We chose the Sobel operator for its simplicity and generate two sets of outlines: one from the depth image and another from the normal image.

To even further clarify why we are generating gradients for both depth and normals: The gradients generated by the two are different and complementary. There are a lot of scenarios where both will produce the same gradient, but that doesn’t bother us.

Improving Outline Quality

Using Sobel on depth and normals isn’t without its potential issues. Both the depth and normal gradients can introduce their own artifacts. For example, in the case of depth, the gradient values can become distorted on surfaces viewed at grazing angles, as shown in the leftmost image below.

Grazing angles with depth gradients:

Curved surfaces with normals gradients

This happens because, in screen space, the change in depth is relatively large. The issue can be minimised by increasing the bias fudge factor, but doing so causes valid gradients to disappear entirely. Ultimately, no amount of bias can help in extreme scenarios, as shown in the image above.

Normal-based edges also come with their own set of issues. The most noticeable one is the appearance of false edges on curved surfaces, especially those farther from the camera. This occurs because the normal values change significantly enough in screen space to generate a gradient. The rightmost image above illustrates this.

With these issues in place we cannot get correct outlines independent of view and geometry, which is not acceptable for our use case.

So we started looking for solutions.

In order to improve on our depth gradients, we’ve implemented an interesting take found here, and there are considerable improvements over the vanilla Sobel on depth data. Let’s look at some comparisons.

Here are some compound images that show a direct comparison:

• Red holds the classic Sobel depth gradient
• Green holds the improved depth gradient.

As the results show, not only did we eliminate the grazing angle issues, we also improved general outline extraction in some cases.

Now, we’re left with the issue of normal gradients. Similar to the depth case, we can use the bias fudge factor to mitigate the problem, but we don’t want to rely solely on that, as we can never know the "best" value that fits all input images. So, we tried a simple change: using the Roberts Cross operator to sample neighbouring pixels that determine the gradients. While this doesn’t completely eliminate the issue with curved surfaces, it makes it less noticeable.

No We Can(ny) Not

It’s worth mentioning that in an attempt to fix all issues with Sobel depth and normal based outlines we gave Canny a shot and based our implementation on this. In theory, both the curve surfaces issue as well as the grazing angle issues should be fixed out of the box with Canny.

It did work to some extent, here’s an example:

However, there are two problems with Canny:

• You’re trading one fudge factor with two. Canny needs a value for weak edge and one for strong edge in the range of [0,1]. They are less chaotic and make more sense that the bias value we’ve been mentioning, but still we’d like to avoid relying on fudge factors
• Canny is binary by nature. Weak edges are eliminated unless connected to strong edges, but the output is either 0 or 1. This increases aliasing significantly and also erases some correct edges along the way

In the end, I don’t think we didn’t go with Canny. The overall results as well as the extra GPU time required to generated are not worthwhile.

Anti-aliasing the Outlines

So far our outlines generally do seem to be surprisingly consistent with the underlying geometry in all cases by just running Sobel with both depth and normal data as input. Another advantage is that the resources required to generate the lines are invariant with the number of resulting lines.

So no matter the scene complexity, the time it take to generate the outline, excluding the time taken to generate depth and normals, is more or less constant.

There is one big problem though: Aliasing. The most naive solution you could throw at the problem is hardware multisampling (MSAA).

There are several things happening here. First and foremost, we’re using an aliased image as input for the edge detector. And it’s aliased because by default frame buffers do not get multisampled. By this I mean the hardware anti-aliasing (multi-sampling) that’s readily and automatically available for the backbuffer does not apply to framebuffers (textures we render into) by default.

Multisampled framebuffers are supported in WebGL1.0 through an extension, and in WebGL2.0 core specs. Moreover, the source for heavy aliasing is not geometric (which is best fixed by MSAA), it’s mostly shader aliasing caused by the edge detector itself. The image above shows a comparison between no multisampling and maximum multisampling on both input normals image as well as output edges images.

Multisampling has an additional drawback, especially multisampled framebuffers : they obliterate performance. No exaggeration, MSAA in the web is very very slow. And coupled with the fact that it doesn’t provide a complete and universal anti-aliasing result, we had to give it up in favour of a better and more reliable anti aliasing technique.

SMAA vs TAA

We considered two contenders for anti-aliasing the outlines:

• SMAA (Subpixel Morphological Anti-Aliasing), based on Three.js’s stock implementation
• TAA (Temporal Anti-Aliasing), based on our own sauce.

Since SMAA was the easier option, we tried it first, but we were unsatisfied with the results. As a result, we implemented our own TAA solution, inspired by Playdead’s TAA implementation in Inside. Of course, we only use the distributed supersampling part of TAA, meaning it triggers only when the camera is stationary (just like our AO).

We won’t dive deeply into how TAA works, as it’s beyond the scope here, but we will highlight the aspects of TAA that can impact the entire rendering pipeline. For those interested, here’s a step-by-step guide for a more detailed explanation.

The first thing we need to address is jittering. TAA relies on a set of offsets generated from low-discrepancy noise (Halton) applied to the scene geometry, which causes the geometry to "jitter" from one frame to the next. Over multiple frames, these jittered pixels are blended together based on computed weights to achieve the smooth, anti-aliased result. Jittering is important because it affects the rest of the pipeline. For example, the outline extraction pass, which generates normals and depth in a separate pass, also requires these values to be jittered.

Now let’s look at some comparisons:

If we zoom in on the image, we can see better what difference TAA makes:

In addition to the improved anti-aliasing, TAA is fast. Even though we’re currently using it only for static content, upgrading it to work dynamically will still maintain its speed. Here are a few more shots to showcase the results: