Woohoo

Edge-distance anti-aliasing

Martin Robinson (Martin Robinson) — Sun, 24 Feb 2013 00:00:00 EST

(You might want to go straight to the demo)

Some months ago, I noticed that the Chromium compositor, the code which powers Chromium’s accelerated compositing implementation (and also Aura!) was anti-aliasing layer edges. This was especially surprising to me since I knew for a fact that my hardware didn’t support multisample anti-aliasing yet. Some people from the Chromium graphics team, who are incredibly friendly and helpful, pointed me to a very clever bit of code they were using to do edge-distance anti-aliasing for composited layers.

Aliasing can happen when you sample a signal that has details that cannot be captured by your sample frequency. In the case of graphics, it’s what happens when features of the image exist in an area smaller than a pixel. One important feature is the edge of a piece of geometry, such as the edge of a shape drawn onto the page. When you start rotating shapes and projecting them with 3D CSS transforms, geometry that before aligned to pixel boundaries can move to the spaces between pixel boundaries.

For instance, let’s zoom into the edge of triangle that we are rendering. Maybe in this case it’s half of a rotated div.

If we naively decided the color of every pixel based on whether any of the triangle covered it at all, we’d end up with a jagged edge.

One thing we could do to make the edge smooth is to determine how much area of the pixel is covered by the triangle. We could adjust the transparency of the shape color (the alpha value) by that percentage when painting the pixel. This process is a type of anti-aliasing and the percentage that the triangle covers the pixel is called coverage.

Think of all the complicated geometric calculations we’d have to do to figure out the ratio exactly. Since we have to make these calculations for every pixel up to 60 times per second (for smooth animations), our rendering would be pretty slow if we actually did them! Instead, it’d be nice if we could somehow estimate how much of the pixel is covered and do less work. One approach is to calculate the distance from the pixel to the edge of the shape. If the pixel is more than one pixel’s distance away from the edge, we know that we should not paint the triangle color at all. If the pixel is closer than one one pixel’s distance from the edge, we should paint the triangle color, but reduced by some transparency factor.

Luckily, OpenGL can run a little program for every pixel ¹ it paints. This program is called a fragment shader. We can write up this logic in the fragment shader and change the triangle color for every pixel.

Perhaps some of those reading who have had experience with OpenGL may notice here that this isn’t going to work as I’ve described. OpenGL already does something like the naive approach I talked about first. Some pixels (the ones OpenGL decided should be painted with the triangle color), will be properly anti-aliased, but many pixels will not be painted at all. For instance, if the coverage of the pixel is only 20% and OpenGL decided not to paint it, we wouldn’t even have a chance to determine how far it was from the triangle edge, because the fragment shader wouldn’t run for those missing pixels.

Let’s “trick” OpenGL into painting more pixels than it would otherwise. A simple way to do this is to just make the triangle a little bit bigger. In fact, we can expand all the edges by less than one pixel and OpenGL will paint all those missing pixels. Now we have smooth edges!

I spent some time implementing this for the TextureMapper accelerated compositor (and thus WebKitGTK+), so it’s not just the Chromium compositor that has this feature any longer. For me it makes a huge difference in the quality of 3D CSS content. For the purposes of demonstration, I’ve also reimplemented it in WebGL, so if you have a modern browser you can see it in action.

Fragments and pixels are actually different things, but I'm glossing over this difference for the sake of simplicity.

Accelerated compositing update

Martin Robinson (Martin Robinson) — Sat, 07 Jul 2012 00:00:00 EDT

I believe it's past time to break the silence here, so what follows is a short update on the progress we've made at Igalia toward 3D CSS transforms and hardware accelerated animation in WebKitGTK+ (otherwise known as accelerated compositing). I'm happy to say that both the initial WebKit1 and WebKit2 implementation of this work have finally landed, and you can try it out now in the latest unstable releases. Epiphany in Gnome 3.6 will include accelerated compositing (smooth animations and fast WebGL), a port to the brand new WebKit2GTK+ API (stability and security), and a new interface. It's quite pleasant to watch the progress of my own rounding error as each day it continues transforming into a modern browser.

There were some interesting design choices to make when implementing accelerated compositing in WebKit2. Ports typically choose one of two possibilities for where to actually composite the content of page layers. The first and simplest path, taken by the GTK+ and Mac ports is to do all GPU composition in the WebProcess, which the same process that renders the layer content. This is by far the simplest and most straight-forward path, as WebKit does not have to send bitmap and layer data across the process boundary. The Qt and EFL ports have chosen to do composition in the UIProcess, which is the process that controls the user interface components surrounding the page content. From what I understand, this more complex design is an attempt to lower the latency of touch events and to decrease the number of OpenGL context switches. It's unclear now what the best approach is for browsers on embedded systems, but composition in the WebProcess works quite well for Epiphany at the moment.

During this development cycle we'll continue to improve the implementation of accelerated compositing, but we hope to start looking at other rendering and interactivity improvements such as accelerated canvas, a tiled backing store, touch event handling and a threaded scrolling implementation. I'm most excited about accelerated canvas, because of the work we've been doing recently on Cairo GL. The GL backend of Cairo has been experimental for some time and isn't widely distributed, but recently there's been an attempt to increase its performance and to make it more suitable for embedded platforms.

Once we integrate Cairo GL with our accelerated compositing architecture in WebKitGTk+, canvas content can be rasterized directly on the GPU instead of the CPU. For most systems accelerated canvas means better performance and lower power usage. Hopefully, I can land the patches to enable this as early as Gnome 3.6, but I'm still not sure when the OpenGL backend in Cairo will be officially supported. This means that, for the moment at least, you'll need to compile Cairo yourself to test accelerated canvas.

WebKitGTK+ hackfest wrapup: accelerated compositing

Martin Robinson (Martin Robinson) — Thu, 08 Dec 2011 00:00:00 EST

I just returned from this year's WebKitGTK+ hackfest. Not only was it the most productive hackfest to date, the diversity of the people involved was incredible. Attendees included hackers from Igalia, Collabora, RedHat and Motorola. It's great to be part of a company which can pull this kind of event together.

Planning

One of the things different about this hackfest were the BoFs, which were mainly planning sessions. The WebKitGTK+ community has shifted from a ragtag group of developers unafraid of g_object_ref and g_object_unref furiously hacking on a tiny WebKit port to an integral piece of the Gnome renaissance moving steadily toward ambitious goals. I'm obviously biased, but I believe the investment of small companies with strong ideals like Igalia and Collabora is directly responsible.

Others have already reported on some of the incredible work at the hackfest including the bold new Epiphany redesign, accessibility support for WebKit2, and JHBuild-ification. I'd like to talk a bit about something in which I'm personally involved: accelerated compositing.

Epiphany's new look

Accelerated compositing was introduced in 2009 with the bulk of the work provided by Apple and Google. GPUs are quite fast, but because of the overhead of moving bits from main main memory to GPU memory, you need to use them intelligently to see any speedup. Accelerated compositing is the smarts needed to use the GPU intelligently in WebKit. We're also trying to push these smarts down into painting libraries such as Cairo and Skia, but that's a more difficult task, with less immediate benefit. The best performance improvements come from using the GPU at the place that maintains the drawing state.

In the context of WebKit, drawing state is the rendered page itself, with all its layered divs, canvas elements, video tags and WebGL contexts. WebKit stores elements to render in a data structure called the "render tree." Members of the the render tree which are drawn in main memory must be sent to the GPU to be displayed on the screen. Furthermore, some of these elements may be rendered on the GPU itself, so ideally we want to avoid copying them to main memory only to copy them back to the GPU at a later stage.

CPUs are very fast, so it's often sufficient to rasterize things into main memory. On the other hand, when the entire page is rendered by the CPU, it spends a lot of time doing "boring" tasks that the GPU loves doing, such as compositing (blending images into each other). Accelerated compositing attacks this by slicing the page into layers of related content. The way this slicing works is a heuristic, but generally speaking divs at the same z-index, with the same CSS transform, WebGL contexts and hardware accelerated video are their own layer. It renders each of the non-GPU layers into an image and then uploads all the images to be composited by the GPU along with layers that are already there.

Collabora, which created the Clutter port of WebKit has an implementation of accelerated compositing using Clutter. At the hackfest, Joone Hurr and Gustavo Noronha Silva started integrating this version into WebKitGTK+. The nice thing about the Clutter implementation is that it's almost complete. The unfortunate thing is that Clutter uses COGL which has very poor support for using raw OpenGL in the application. Obviously this means that the WebKitGTK+ port would need to rewrite it's WebGL backend. Igalia is also working on two implementations (OpenGL and the Cairo fallback) using No'am Rosenthal's very nice TextureMapper abstraction.

During the hackfest, we decided to attempt to integrate all three approaches into the tree. The Clutter version is a nice stopgap for people who want to play with accelerated compositing or are using Clutter already. The intial parts of this implementation should be landing soon. Over the next few months we'll also be submitting OpenGL and Cairo versions of accelerated compositing. We already made great progress at the hackfest. If you're interested in following the work, you can follow the accelerated compositing metabug.

Faster Shadows

Martin Robinson (Martin Robinson) — Sun, 13 Mar 2011 00:00:00 EST

One area we paid particular attention last year at Igalia was the performance of shadow rendering in web content. While web shadows are not yet ubiquitous, a slow shadow implementation can cause poor scrolling and redraw performance, ruining your day like an afternoon full of stepping in wet cement. Until recently, the GTK+ (and all Cairo) ports had one of these slow shadow implementations. Scrolling through identi.ca posts could lock Epiphany's user interface for several painful seconds. Alex and I set out to improve this situation.

I separate shadowed web content into three categories: CSS box and text shadows, canvas shadows and SVG shadows. Even though a different standard addresses each of these categories of shadows, the process of drawing the shadow remains very similar. The web rendering engine simply draws a copy of the shadowed object underneath itself with a solid fill (the shadow fill). This "shadow image" may also be blurred with a certain radius. For each of the standards that I mentioned, it is sufficient to blur the shadow image by estimating a Gaussian blur on its pixels.

A Gaussian blur is just the application of the normal distribution to the pixel values of an image. The value of a pixel in the blurred image is calculated by combining the values of the pixels around it in the original image. The contribution of a particular surrounding pixel decreases with the distance from the target pixel. Until recently the Cairo port performed an actual Gaussian blur with an expensive two-dimensional kernel. The first speed improvement involved making Ariya Hidayat's fast blurring algorithm (for the Qt port) cross-platform and using it for our own shadows. This algorithm estimates a Gaussian blur by performing multiple one-dimension motion blurs. This decreases the number of pixels that need to be read to calculate a target pixel and also increases the number of cache hits during the blurring calculation.

In the course of our work we realized that we were not clipping the shadowed area when blurring. For instance, identi.ca has one long column of content that extends down the page with a box blur. Instead of blurring just the part of the column visible in the viewport, we were blurring the entire box. This simple fix alone was enough to make most sites usable.

Alex performed the final and most novel optimization for box shadows. Here's an image extremely typical web content with a shadowed div.

If we remove everything but the shadow we are left with what you see below. A lot of this information is repeated. In fact, we can divide the image into nine regions of unique information. Obviously we are calculating the same few values over and over again. Keep in mind that the slowest part of creating a blurry shadow is estimating the Gaussian.

Each of non-corner regions of the box are made up of the same row of pixels repeated down their length. In fact, if we had only one a single row of those blurred pixels we could just copy it into the region where the shadow belongs. This is precisely what Alex's implementation does. Instead of rendering the large shadowed area above, we can simply calculate a smaller box and copy the data into the target image.

I'm quite pleased with the performance of shadows now. There are a few more optimizations we could do, such as extending the tiling optimization to inset shadows. In fact, the Mac port has made our implementation cross-platform and has already added support for inset shadows. Hopefully more WebKit ports will soon be able to benefit from these optimizations.

Looking back at the WebKit GTK hackfest

Martin Robinson (Martin Robinson) — Sat, 23 Jan 2010 00:00:00 EST

In December, I attended the WebKit GTK hackfest which has been summed up nicely in many other places. Some of the things I worked on (apart from getting my luggage):

With the closing of 20736 WebKit GTK should now properly support windows with RGBA colormaps. This means that WebKit GTK can now be used to create nice desktop widgets or non-rectangular applications without worrying about BadMatch errors. Surprisingly non-rectangular windows is one of the most requested features for Titanium and we now support it on OS X, Linux and Windows via the transparent-background property in tiapp.xml.

The first big part of my clipboard / drag-and-drop reorganization also landed. Once I get the remaining patches together, DOM DataTransfer access to WebKit GTK clipboard and drag-and-drop data will work properly. With luck one day you'll be able to drop your photos and videos onto your web browser and have them upload seamlessly.

The hackfest itself was in A Coruña, a picturesque Galician city right on the coast of the Atlantic (which I managed to call the Pacific once, against all odds). I also had a rather abrupt introduction to the bounty of European languages — Spain has four recognized regional languages apart from Spanish (Castilian).

Xan and me, probably trying to convince him that drag-and-drop is the future.

On a personal note, the hackfest solidified my love for open source software. I want to keep doing this, however possible. Evan Martin made a really good point about open source at some point during the hackfest. He said that people sometimes think that being open source means that you can simply host an archive of your source code somewhere. There is more to it though, including having an open and active community and understanding and responding to your users. Some of this is just good software development practice, but being open source requires doing even more work to open the development process.

One important aspect of this work is having community discussions and allowing the community to help make decisions. One look at the WebKit development mailing list shows this working. There are many interests tied up with WebKit and some of them are direct competitors, yet somehow they manage to coordinate development on a humongous project. LWN.net recently posted a great article related to this topic, which I think exemplifies what not to do.