Woohoo

Edge-distance anti-aliasing

Martin Robinson (Martin Robinson) — Sun, 24 Feb 2013 00:00:00 EST

(You might want to go straight to the demo)

Some months ago, I noticed that the Chromium compositor, the code which powers Chromium’s accelerated compositing implementation (and also Aura!) was anti-aliasing layer edges. This was especially surprising to me since I knew for a fact that my hardware didn’t support multisample anti-aliasing yet. Some people from the Chromium graphics team, who are incredibly friendly and helpful, pointed me to a very clever bit of code they were using to do edge-distance anti-aliasing for composited layers.

Aliasing can happen when you sample a signal that has details that cannot be captured by your sample frequency. In the case of graphics, it’s what happens when features of the image exist in an area smaller than a pixel. One important feature is the edge of a piece of geometry, such as the edge of a shape drawn onto the page. When you start rotating shapes and projecting them with 3D CSS transforms, geometry that before aligned to pixel boundaries can move to the spaces between pixel boundaries.

For instance, let’s zoom into the edge of triangle that we are rendering. Maybe in this case it’s half of a rotated div.

If we naively decided the color of every pixel based on whether any of the triangle covered it at all, we’d end up with a jagged edge.

One thing we could do to make the edge smooth is to determine how much area of the pixel is covered by the triangle. We could adjust the transparency of the shape color (the alpha value) by that percentage when painting the pixel. This process is a type of anti-aliasing and the percentage that the triangle covers the pixel is called coverage.

Think of all the complicated geometric calculations we’d have to do to figure out the ratio exactly. Since we have to make these calculations for every pixel up to 60 times per second (for smooth animations), our rendering would be pretty slow if we actually did them! Instead, it’d be nice if we could somehow estimate how much of the pixel is covered and do less work. One approach is to calculate the distance from the pixel to the edge of the shape. If the pixel is more than one pixel’s distance away from the edge, we know that we should not paint the triangle color at all. If the pixel is closer than one one pixel’s distance from the edge, we should paint the triangle color, but reduced by some transparency factor.

Luckily, OpenGL can run a little program for every pixel ¹ it paints. This program is called a fragment shader. We can write up this logic in the fragment shader and change the triangle color for every pixel.

Perhaps some of those reading who have had experience with OpenGL may notice here that this isn’t going to work as I’ve described. OpenGL already does something like the naive approach I talked about first. Some pixels (the ones OpenGL decided should be painted with the triangle color), will be properly anti-aliased, but many pixels will not be painted at all. For instance, if the coverage of the pixel is only 20% and OpenGL decided not to paint it, we wouldn’t even have a chance to determine how far it was from the triangle edge, because the fragment shader wouldn’t run for those missing pixels.

Let’s “trick” OpenGL into painting more pixels than it would otherwise. A simple way to do this is to just make the triangle a little bit bigger. In fact, we can expand all the edges by less than one pixel and OpenGL will paint all those missing pixels. Now we have smooth edges!

I spent some time implementing this for the TextureMapper accelerated compositor (and thus WebKitGTK+), so it’s not just the Chromium compositor that has this feature any longer. For me it makes a huge difference in the quality of 3D CSS content. For the purposes of demonstration, I’ve also reimplemented it in WebGL, so if you have a modern browser you can see it in action.

Fragments and pixels are actually different things, but I'm glossing over this difference for the sake of simplicity.

Accelerated compositing update

Martin Robinson (Martin Robinson) — Sat, 07 Jul 2012 00:00:00 EDT

I believe it's past time to break the silence here, so what follows is a short update on the progress we've made at Igalia toward 3D CSS transforms and hardware accelerated animation in WebKitGTK+ (otherwise known as accelerated compositing). I'm happy to say that both the initial WebKit1 and WebKit2 implementation of this work have finally landed, and you can try it out now in the latest unstable releases. Epiphany in Gnome 3.6 will include accelerated compositing (smooth animations and fast WebGL), a port to the brand new WebKit2GTK+ API (stability and security), and a new interface. It's quite pleasant to watch the progress of my own rounding error as each day it continues transforming into a modern browser.

There were some interesting design choices to make when implementing accelerated compositing in WebKit2. Ports typically choose one of two possibilities for where to actually composite the content of page layers. The first and simplest path, taken by the GTK+ and Mac ports is to do all GPU composition in the WebProcess, which the same process that renders the layer content. This is by far the simplest and most straight-forward path, as WebKit does not have to send bitmap and layer data across the process boundary. The Qt and EFL ports have chosen to do composition in the UIProcess, which is the process that controls the user interface components surrounding the page content. From what I understand, this more complex design is an attempt to lower the latency of touch events and to decrease the number of OpenGL context switches. It's unclear now what the best approach is for browsers on embedded systems, but composition in the WebProcess works quite well for Epiphany at the moment.

During this development cycle we'll continue to improve the implementation of accelerated compositing, but we hope to start looking at other rendering and interactivity improvements such as accelerated canvas, a tiled backing store, touch event handling and a threaded scrolling implementation. I'm most excited about accelerated canvas, because of the work we've been doing recently on Cairo GL. The GL backend of Cairo has been experimental for some time and isn't widely distributed, but recently there's been an attempt to increase its performance and to make it more suitable for embedded platforms.

Once we integrate Cairo GL with our accelerated compositing architecture in WebKitGTk+, canvas content can be rasterized directly on the GPU instead of the CPU. For most systems accelerated canvas means better performance and lower power usage. Hopefully, I can land the patches to enable this as early as Gnome 3.6, but I'm still not sure when the OpenGL backend in Cairo will be officially supported. This means that, for the moment at least, you'll need to compile Cairo yourself to test accelerated canvas.

WebKitGTK+ hackfest wrapup: accelerated compositing

Martin Robinson (Martin Robinson) — Thu, 08 Dec 2011 00:00:00 EST

I just returned from this year's WebKitGTK+ hackfest. Not only was it the most productive hackfest to date, the diversity of the people involved was incredible. Attendees included hackers from Igalia, Collabora, RedHat and Motorola. It's great to be part of a company which can pull this kind of event together.

Planning

One of the things different about this hackfest were the BoFs, which were mainly planning sessions. The WebKitGTK+ community has shifted from a ragtag group of developers unafraid of g_object_ref and g_object_unref furiously hacking on a tiny WebKit port to an integral piece of the Gnome renaissance moving steadily toward ambitious goals. I'm obviously biased, but I believe the investment of small companies with strong ideals like Igalia and Collabora is directly responsible.

Others have already reported on some of the incredible work at the hackfest including the bold new Epiphany redesign, accessibility support for WebKit2, and JHBuild-ification. I'd like to talk a bit about something in which I'm personally involved: accelerated compositing.

Epiphany's new look

Accelerated compositing was introduced in 2009 with the bulk of the work provided by Apple and Google. GPUs are quite fast, but because of the overhead of moving bits from main main memory to GPU memory, you need to use them intelligently to see any speedup. Accelerated compositing is the smarts needed to use the GPU intelligently in WebKit. We're also trying to push these smarts down into painting libraries such as Cairo and Skia, but that's a more difficult task, with less immediate benefit. The best performance improvements come from using the GPU at the place that maintains the drawing state.

In the context of WebKit, drawing state is the rendered page itself, with all its layered divs, canvas elements, video tags and WebGL contexts. WebKit stores elements to render in a data structure called the "render tree." Members of the the render tree which are drawn in main memory must be sent to the GPU to be displayed on the screen. Furthermore, some of these elements may be rendered on the GPU itself, so ideally we want to avoid copying them to main memory only to copy them back to the GPU at a later stage.

CPUs are very fast, so it's often sufficient to rasterize things into main memory. On the other hand, when the entire page is rendered by the CPU, it spends a lot of time doing "boring" tasks that the GPU loves doing, such as compositing (blending images into each other). Accelerated compositing attacks this by slicing the page into layers of related content. The way this slicing works is a heuristic, but generally speaking divs at the same z-index, with the same CSS transform, WebGL contexts and hardware accelerated video are their own layer. It renders each of the non-GPU layers into an image and then uploads all the images to be composited by the GPU along with layers that are already there.

Collabora, which created the Clutter port of WebKit has an implementation of accelerated compositing using Clutter. At the hackfest, Joone Hurr and Gustavo Noronha Silva started integrating this version into WebKitGTK+. The nice thing about the Clutter implementation is that it's almost complete. The unfortunate thing is that Clutter uses COGL which has very poor support for using raw OpenGL in the application. Obviously this means that the WebKitGTK+ port would need to rewrite it's WebGL backend. Igalia is also working on two implementations (OpenGL and the Cairo fallback) using No'am Rosenthal's very nice TextureMapper abstraction.

During the hackfest, we decided to attempt to integrate all three approaches into the tree. The Clutter version is a nice stopgap for people who want to play with accelerated compositing or are using Clutter already. The intial parts of this implementation should be landing soon. Over the next few months we'll also be submitting OpenGL and Cairo versions of accelerated compositing. We already made great progress at the hackfest. If you're interested in following the work, you can follow the accelerated compositing metabug.

Building a bike

Martin Robinson (Martin Robinson) — Fri, 23 Sep 2011 00:00:00 EDT

For some years now, I've been meaning to build a bicycle. Because I balked at the idea of owning more than one, I never did. Recently my unrestrained minimalism was given a reprieve in the form of cascading, terminal bicycle theft.

For five years, I've only locked one wheel on my trusty 1983 Nishiki International (this bicycle was actually named Super Famicom and the Crane Techniques). For some reason, the corroded spokes on the 27" wheels never seemed enough of a prize for thievery. Over time I even began to leave the rear wheel unlocked. Apparently either a downturn in the economy or my own luck was sufficient to leave my bicycle without a rear wheel one night.

I locked my limping ride to the rack in my apartment's parking garage. Upon returning two weeks later from the Igalia Assembly and the lovely Berlin Desktop Summit, I found my drop handlebars, stem, brake levers and brake cables Scrounging from wounded bikes is a gray area, it seems. In any case, it was decided: time to build a bicycle.

The process took a bit more than a month, though I worked only two or three days a week at it. My schedule generally followed the hours of the San Francisco Bike Kitchen, an awesome community bike shop bursting with incredibly knowledgeable volunteer mechanics and tools. I floundered a bit at first, because it isn't clear what order you should do things in, but the entire process went really smoothly.

Frame - I had a new frame, which was a kick-ass birthday present from Lauryn. New frames aren't always ready for assembly yet, you may need to face and chase them. Because of manufacturing tolerances and excess paint, facing and chasing prepares the metal on the ends of the headtube and bottom bracket shell. Amazingly, the Bike Kitchen had the expensive, specialized tools for this task. One of their mechanics also stood over my shoulder as I shaved bits of metal off of my frame. It reminded me a lot of the first time I installed a processor on a motherboard, hand shaking with the thought of destroying it.

Fork - My frame came with a fork, but the fork itself was actually a lot longer than I needed. Cutting a fork at a bike shop typically costs about $50, but again at the Kitchen they showed me how to measure for the cut and guided me in the proper application of a pipe cutter. This was probably the second time I returned home from the Bike Kitchen absolutely astounded.

Measuring your fork can be a little tricky. At first I was under the impression that I needed the rest of the bicycle assembled to measure it. This isn't strictly true. If you know about how much of the fork you need, you can increase the measurement slightly. By moving the spacers around you can adjust the height of your handlebars up and down. It's important to remember that you can lower your handlebars on a long fork, but raising them on a short fork requires a steep stem or specialized adapters.

Headset - The headset is composed of the two little rings that sit between your fork and the top and bottom of your headtube. They have bearings in them, so that your fork can turn side to side. I made a mistake and started trying to assemble my headtube before cutting the fork. It was a lot easier once the fork was cut. There's a tool for pressing the headset cups into the headtube, it's like a very particular vice which ensures that they press in straight.

Bottom bracket - The bottom bracket is the spindle that the pedals rotate and the metal housing and bearings that the spindle rotates in. This was a fairly commodity part, though since I was reusing the crankset and pedals from my old bicyle, I had to be careful about what size I purchased. Against all odds I was able to combine information from old Siguino crankset brochures and Sheldon Brown to find the right size. This was actually a pretty simple process, but was made difficult by the font on the crankset which made "GS" look like "G5." I'm not the only one who made this mistake.

Once again, I was able to install these parts with the tools and assistance at the Bike Kitchen. A lot of the bicycle fell into place, after that. I installed the wheels next, mostly because I was tired of carrying my frame back and forth to the shop. I picked up used drop handlebars and a stem from there for eleven dollars For some reason, no one stole the seatpost and seat from my old bicycle, so I essentially plugged that in directly. At this point my work seemed deceptively like something I could ride.

Next I assembled the rest of the drive train. Installing the rear-wheel cassette was simple with the right tool, which I purchased since it was only seven dollars. The difference between nine and eight speed chains tripped me up briefly, but was not much of an issue in the end. Once the pedals seemed to turn the rear wheel and the rear derailleur was adjusted properly, I installed the brakes, brake levers, and handle bar tape. Since that time, I have ridden it with only minor adjustments.

Observations:

The Bicycle Kitchen is awesome. The first time I attempted to go, I showed up an hour after they opened. I was disheartened to realize that the waiting list was so long that I wouldn't be able to get in that night. Everything in San Francisco has a waiting list though, so I arrived 10 minute before they opened every other time I went.
A lot of this process was made less daunting by the fact that there is an incredible number of tutorials and information available on the web. I spent a lot of time on the Park Tools website and the Bike Wagon YouTube channel. I imagine this made me sound less clueless at the Bike Kitchen and just simply misinformed.
The fact that friction shifters actually have "friction" in their name is a subtle signal that you shouldn't load them down with grease. The ghost shifting problems are much better now.

Faster Shadows

Martin Robinson (Martin Robinson) — Sun, 13 Mar 2011 00:00:00 EST

One area we paid particular attention last year at Igalia was the performance of shadow rendering in web content. While web shadows are not yet ubiquitous, a slow shadow implementation can cause poor scrolling and redraw performance, ruining your day like an afternoon full of stepping in wet cement. Until recently, the GTK+ (and all Cairo) ports had one of these slow shadow implementations. Scrolling through identi.ca posts could lock Epiphany's user interface for several painful seconds. Alex and I set out to improve this situation.

I separate shadowed web content into three categories: CSS box and text shadows, canvas shadows and SVG shadows. Even though a different standard addresses each of these categories of shadows, the process of drawing the shadow remains very similar. The web rendering engine simply draws a copy of the shadowed object underneath itself with a solid fill (the shadow fill). This "shadow image" may also be blurred with a certain radius. For each of the standards that I mentioned, it is sufficient to blur the shadow image by estimating a Gaussian blur on its pixels.

A Gaussian blur is just the application of the normal distribution to the pixel values of an image. The value of a pixel in the blurred image is calculated by combining the values of the pixels around it in the original image. The contribution of a particular surrounding pixel decreases with the distance from the target pixel. Until recently the Cairo port performed an actual Gaussian blur with an expensive two-dimensional kernel. The first speed improvement involved making Ariya Hidayat's fast blurring algorithm (for the Qt port) cross-platform and using it for our own shadows. This algorithm estimates a Gaussian blur by performing multiple one-dimension motion blurs. This decreases the number of pixels that need to be read to calculate a target pixel and also increases the number of cache hits during the blurring calculation.

In the course of our work we realized that we were not clipping the shadowed area when blurring. For instance, identi.ca has one long column of content that extends down the page with a box blur. Instead of blurring just the part of the column visible in the viewport, we were blurring the entire box. This simple fix alone was enough to make most sites usable.

Alex performed the final and most novel optimization for box shadows. Here's an image extremely typical web content with a shadowed div.

If we remove everything but the shadow we are left with what you see below. A lot of this information is repeated. In fact, we can divide the image into nine regions of unique information. Obviously we are calculating the same few values over and over again. Keep in mind that the slowest part of creating a blurry shadow is estimating the Gaussian.

Each of non-corner regions of the box are made up of the same row of pixels repeated down their length. In fact, if we had only one a single row of those blurred pixels we could just copy it into the region where the shadow belongs. This is precisely what Alex's implementation does. Instead of rendering the large shadowed area above, we can simply calculate a smaller box and copy the data into the target image.

I'm quite pleased with the performance of shadows now. There are a few more optimizations we could do, such as extending the tiling optimization to inset shadows. In fact, the Mac port has made our implementation cross-platform and has already added support for inset shadows. Hopefully more WebKit ports will soon be able to benefit from these optimizations.

Node.js Knockout

Martin Robinson (Martin Robinson) — Mon, 30 Aug 2010 00:00:00 EDT

I participated in the Node.js Knockout this year, kindly hosted by Fortnight Labs. Our team became a bit smaller as we neared the competition, so we decided to make something dead simple. When you're making software, it's good to have a vision statement. Even if you don't finish the software, you'll at least have a well-constructed sentence, which is progress. Frankly, I was too embarrassed to mention my vision statement to my teammate, Josh.

Each tweet is a tiny burst of light illuminating a cold, dark world.

My opinion is that if you're not inspired after reading that, you've probably lost the ability to feel actual human emotions such as hope, rage, and yellow. We decided to make a Twitter visualization for people like you.

I'd never written anything on node.js before this competition and it did nothing but impress me. Node.js is built from the ground up with asynchronicity in mind, so ideally you aren't bound by the overhead of creating one thread per connection. Think Twisted. We also used Socket.IO, which made WebSockets sinfully pleasant.

My partner, Josh, handled the server-side part of the application and I generally worked on the client-side¹. My favorite challenge was calculating the Robinson Projection for mapping latitude and longitude onto the map surface. It turns out that Robinson, unlike most projection progenitors, relied on lookup tables and interpolation rather than providing a set of formulas only. I was easily able to port some Robinson Projection code from Java (which had itself been ported from C). It was some time later that I realized I had to scale the resulting numbers by the limits of projection output. It all seemed so obvious later on.

In any case, I hope this amuses you for a couple seconds. The code will be on github after it's been cleaned up and licensed.

I clumsily contributed a last minute cache optimization which had a negligible effect on the output.

Moving

Martin Robinson (Martin Robinson) — Mon, 30 Aug 2010 00:00:00 EDT

My blog has a new URL. Please update your RSS subscriptions accordingly!

Looking back at the WebKit GTK hackfest

Martin Robinson (Martin Robinson) — Sat, 23 Jan 2010 00:00:00 EST

In December, I attended the WebKit GTK hackfest which has been summed up nicely in many other places. Some of the things I worked on (apart from getting my luggage):

With the closing of 20736 WebKit GTK should now properly support windows with RGBA colormaps. This means that WebKit GTK can now be used to create nice desktop widgets or non-rectangular applications without worrying about BadMatch errors. Surprisingly non-rectangular windows is one of the most requested features for Titanium and we now support it on OS X, Linux and Windows via the transparent-background property in tiapp.xml.

The first big part of my clipboard / drag-and-drop reorganization also landed. Once I get the remaining patches together, DOM DataTransfer access to WebKit GTK clipboard and drag-and-drop data will work properly. With luck one day you'll be able to drop your photos and videos onto your web browser and have them upload seamlessly.

The hackfest itself was in A Coruña, a picturesque Galician city right on the coast of the Atlantic (which I managed to call the Pacific once, against all odds). I also had a rather abrupt introduction to the bounty of European languages — Spain has four recognized regional languages apart from Spanish (Castilian).

Xan and me, probably trying to convince him that drag-and-drop is the future.

On a personal note, the hackfest solidified my love for open source software. I want to keep doing this, however possible. Evan Martin made a really good point about open source at some point during the hackfest. He said that people sometimes think that being open source means that you can simply host an archive of your source code somewhere. There is more to it though, including having an open and active community and understanding and responding to your users. Some of this is just good software development practice, but being open source requires doing even more work to open the development process.

One important aspect of this work is having community discussions and allowing the community to help make decisions. One look at the WebKit development mailing list shows this working. There are many interests tied up with WebKit and some of them are direct competitors, yet somehow they manage to coordinate development on a humongous project. LWN.net recently posted a great article related to this topic, which I think exemplifies what not to do.

Managing the Python GIL via RAII

Martin Robinson (Martin Robinson) — Tue, 06 Oct 2009 00:00:00 EDT

One of the killer features of C++ is RAII. RAII means that the amount of special-case cleanup code in the case of exceptions or early exits is minimized. For more on exactly how this happens, I recommend the Wikipedia article linked above.

This feature became useful to me when making our Python Kroll module work properly with the Python GIL. There were two usage patterns I was interested in for this work.

Letting Python code in other threads run

PyThreadState* threadState = PyEval_SaveThread();
...do some expensive non-Python work here...
PyEval_RestoreThread(threadState);

Taking back the GIL to use the Python API

PyGILState gstate = PyGILState_Ensure();
...use the Python API here...
PyGILState_Release(gstate);

The problem with this approach is that if an uncaught exception escapes from anywhere between the ellipses, the cleanup code (PyEval_RestoreThread or PyGILState_Release) will not run, likely leaving the program in a bad state. It turns out that RAII has a very elegant solution to this problem.


    class PyLockGIL
    {
        PyLockGIL() : gstate(PyGILState_Ensure())
        { }

        ~PyLockGIL()
        {
            PyGILState_Release(gstate);
        }

        PyGILState_STATE gstate;
    };

    class PyAllowThreads
    {
        PyAllowThreads() : threadState(PyEval_SaveThread())
        { }

        ~PyAllowThreads()
        {
            PyEval_RestoreThread(threadState);
        }

        PyThreadState* threadState;
    };

The acquisition and release of the GIL is just wrapped in the constructors and destructors of objects that are allocated on the stack. Here is the previous example using these new objects:


{
    PyLockGIL lock
    ...do some expensive non-Python work here...
}

{
    PyAllowThreads allow;
    ...use the Python API here...
}

Notice that braces can be used to fine tune the amount of code with the desired GIL state. The destructor for these objects will be called as soon as they go out of scope.

End-run around registration-free COM

Martin Robinson (Martin Robinson) — Thu, 23 Apr 2009 00:00:00 EDT

With the introduction of registration-free COM, Windows now allows you to create COM objects without registering their DLLs in the registry. This means that you can include your COM DLLs with your application files and run the application without being an administrator and mucking with regsvr32. Instead you include (or embed in your executable/DLL) a manifest which specifies the location of the COM DLL relative to your application.

This "new" (it was introduced with Windows XP, I believe) approach is not flexible enough for Titanium, which doesn't know until after run-time which WebKit DLL it will be using. Writing this manifest file into the application directory and restarting or copying the DLLs into the application directory is a less than ideal solution. While the copy may work in Vista because of the wonky UAC virtualization of write permissions (that's an entirely different rant), copying an entire DLL and its dependencies at startup is expensive and defeats the purpose of DLLs.

This situation bothered me until I stumbled upon this blog post in the murky underbelly of the interblogs. It basically describes an approach using the somewhat thinly documented (perhaps intentionally) activation context API.

This Windows nightmare has a happy ending in that we were able to simply point an activation context at the WebKit DLL's manifest path. Here's what it looks like (this is originally from Gopalakrishna's blog which I linked to above):


// This is a normal CoCreateInstance -- done as if the COM DLL
// was actually registered with the system -- it will fail
IMyComponent* pObj = NULL;
HRESULT hr = CoCreateInstance(
    CLSID_MyComponent, 
    NULL, 
    CLSCTX_INPROC_SERVER, 
    IID_IMyComponent,
    (void**)&pObj);

ACTCTX actctx;
ZeroMemory(&actctx, sizeof(actctx));
actctx.cbSize = sizeof(actctx);
actctx.lpSource = "C:\Path\To\My\Manifest.manifest";
HANDLE pActCtx = CreateActCtx(&actctx);
ULONG_PTR lpCookie;

if(pActCtx != INVALID_HANDLE_VALUE)
{
    if (ActivateActCtx(pActCtx, &lpCookie))
    {
       // CoCreateInstance should succeed now. 
       HRESULT hr = CoCreateInstance(
           CLSID_MyComponent, 
           NULL, 
           CLSCTX_INPROC_SERVER, 
           IID_IMyComponent,
           (void**)&pObj);
       DeactivateActCtx(0, lpCookie);
    }
    ReleaseActCtx(pActCtx);
}

Avoiding the Logging Performance Hit

Martin Robinson (Martin Robinson) — Sat, 04 Apr 2009 00:00:00 EDT

Sometimes you have a function or a method which more often throws away its arguments than actually uses them. Quite possibly the most common example of this situation is logging. Often you'll see a snippet like this:


log.Debug("Processing " + index + " of " + count);

It's just the way things are that debug log statements are used more liberally than error statements. Even if log.debug(...) doesn't actually log it's argument, your program will still pay the penalty of doing the string concatenation continuously. As a result many programmers are forced to write code that looks like this:


if (log.IsDebug()
{
    log.Debug("Processing " + index + " of " + count);
}

Now the internal state of the logger must be examined externally for every debug statement. D solves this issue through lazily evaluated functional arguments. In D you can change the signature of your function to look like this:


void Debug(lazy char[] entry)
{
   if (debug)
     fwritefln(logfile, entry());
}

This is really just syntactic sugar around D's delegate parameters in which you can pass a function as an argument. The end result, though, is that the statement that composed the argument won't be evaluated until it is actually used.

Titanium and Kroll are written in C++ though, which doesn't have the same kind of syntactic niceties that D does. Recently, when putting the first bits of the logging system into Kroll, I decided that I wasn't satisfied with the Java-style debugging which I wrote above. I wanted to fully encapsulate the behavior of the logger and avoid paying the penalty of uneeded string processing. One common approach for C/C++ is to use macros, but as Walter Bright explains in his essay on lazy argument evaluation, this isn't the best solution.

Luckily, C and C++ have variadic arguments which allow us to create both easy-to-read logging code and to avoid paying a penalty for those statements which should essentially be no-ops. I harnessed the power of a seldom-used friend from the C library for this task: vsnprintf. vsnprintf operates much like snprintf, except that instead of taking a variable list of arguments it takes a va_list (basically a variadic list of arguments that can be passed forward to other functions). Here is the resulting code from Kroll:


  std::string Logger::Format(const char* format, va_list args)
  {
    // Protect the buffer
    Poco::Mutex::ScopedLock lock(this->mutex);
 
    vsnprintf(Logger::buffer, LOGGER_MAX_ENTRY_SIZE - 1, format, args);
    Logger::buffer[LOGGER_MAX_ENTRY_SIZE - 1] = '\0';
    std::string text = buffer;
    return text;
  }

  void Logger::Log(Level level, const char* format, va_list args)
  {
    Poco::Logger& loggerImpl = Poco::Logger::get(name);
 
    // Don't do formatting when this logger filters the message.
    // This prevents unecessary string manipulation.
    if (level >= (Level) loggerImpl.getLevel())
    {
      std::string messageText = Logger::Format(format, args);
      this->Log(level, messageText);
    }
  }

  void Logger::Debug(const char* format, ...)
  {
    va_list args;
    va_start(args, format);
    this->Log(LDEBUG, format, args);
  }

A typical logging statement looks like this:


log.Debug("Processing %i of %i", index, count);

DBus and Threads

Martin Robinson (Martin Robinson) — Fri, 06 Mar 2009 00:00:00 EST

Sometimes you'll be using Dbus with threads and notice intermittent segfaults with stack traces like this;


#0  0xb4d58d29 in _dbus_watch_invalidate (watch=0x0) at dbus-watch.c:147
#1  0xb4d57066 in free_watches (transport=0x979c0d0) at dbus-transport-socket.c:82
#2  0xb4d57e7e in socket_disconnect (transport=0x979c0d0) at dbus-transport-socket.c:908
#3  0xb4d561e8 in _dbus_transport_disconnect (transport=0x979c0d0) at dbus-transport.c:494
#4  0xb4d56c68 in _dbus_transport_queue_messages (transport=0x979c0d0) at dbus-transport.c:1137
#5  0xb4d3dc24 in _dbus_connection_get_dispatch_status_unlocked (connection=0x979c4b0) at dbus-connection.c:3983
#6  0xb4d3b695 in check_for_reply_and_update_dispatch_unlocked (connection=0x979c4b0, pending=0xb3c00550) at dbus-connection.c:2235
#7  0xb4d3b883 in _dbus_connection_block_pending_call (pending=0xb3c00550) at dbus-connection.c:2337
#8  0xb4d501c1 in dbus_pending_call_block (pending=0xb3c00550) at dbus-pending-call.c:707
#9  0xb4d7aedf in dbus_g_proxy_end_call_internal (proxy=0x9776f90, call_id=30, error=0xb4d2d30c, first_arg_type=158972744, args=0xb4d2d2e4 "��Ҵ") at dbus-gproxy.c:2256
#10 0xb4d7b90e in dbus_g_proxy_call (proxy=0x9776f90, method=0xb4dca4bb "Get", error=0xb4d2d30c, first_arg_type=64) at dbus-gproxy.c:2584

This is happening because DBus doesn't do locking by default. I presume this is for performance reasons. Luckily, the solution to this problem is very simple. Simply run dbus_thread_init_default() before you try accessing DBus.

Compiling D Source is Easy

Martin Robinson (Martin Robinson) — Sun, 18 Jan 2009 00:00:00 EST

There are two compilers available for D. The original D compiler (the one written by Walter) is DMD. It has an open-source front-end (the part that generate the IL) and a proprietary, closed-source back-end (the part that converts the IL into machine code). Two other, fully open-source compilers exist, GDC and LDC, which use the GCC and LLVM back-ends respectively.

I like using GDC, because it's open-source and has better support for shared library linking on Unices. It's fairly easy to switch between GDC and DMD, as GDC includes a gdmd command which emulates the command-line behavior of DMD.

Previously I was using DSSS to build my D projects. While DSSS is very nice in many ways, it felt odd using a build tool that was really only popular in the D community. After we started using SCons at work, I was positively pickled to find that it supported building D code. SCons works just as easily for D code as it does for building other things.

Say you have a source file containing your ground-breaking implementation of the Smash Mouth Optimization (see Computational Football vol. 23), SmashMouth.d. To build this with SCons, simply create a SConstruct file that looks like this:

BuildDir('build', 'src')
e = Environment()
e.Program('build/SmashMouth.d')

This assumes that you have your source in a folder called src and you want to build into a folder called build (I know, I know...this is a pretty brazen assumption). Running scons should produce something like this:

scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
gcc -o build/Test build/Test.o -lgphobos -lpthread -lm -lgphobos -lgphobos -lgphobos
scons: done building targets.

You can see here that scons decided to build using GDC and Phobos. The SCons D scanner is pretty smart about what compilers you have on your system. It's also configurable. Okay, here comes the magical wonderland part.

What happens when you decide that you want SmashMouthOptimizer to be a shared library? You have some options here.

You could simply look through all the man pages of the D compilers you are using, hunt for all the relevant linking options, and finally write conditional code in your Makefile to use them.

On the other hand, you could just change your SConstruct to look like this:

BuildDir('build', 'src')
e = Environment()
e.SharedLibrary('build/SmashMouth.d')

Then let scons do this:

scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
gdmd -I. -c -ofbuild/SmashMouth.os build/SmashMouth.d
gcc -o build/libSmashMouth.so -shared build/SmashMouth.os
scons: done building targets.

Different People

Martin Robinson (Martin Robinson) — Sat, 03 Jan 2009 00:00:00 EST

Being There warms my heart for several reasons. One of the most notable is that Hal Ashby managed to turn a Cheech and Chong song into some kind of religious experience with umbrellas. There is also the unexpected vignette during the ending, which many people seem to hate. I have no idea why.

One of the lesser known reasons though, is that it exposes you to some truly awesome music from somewhere you didn't expect. If I had to construct a rigid philosophical framework, I think this would be a major tenet. You might be asking, "What is that unexpected source, Martin?"

Here's the rundown: Buffy Sainte-Marie is talking to Big Bird about some of their interpersonal problems. Big Bird, of course, is suffering unshakable angst, so Buffy is forced to crack it in typical Sesame Street fashion. The music starts at 2:05, by the way.

Embedding Python in D

Martin Robinson (Martin Robinson) — Thu, 01 Jan 2009 00:00:00 EST

Oooooooooh.
It's some snakes.
It's some snakes.

A while back I began spending some time acquainting myself with D. For those who don't know, D is a relatively new system programming language with all kinds of interesting features like very flexible garbage collection, mixins and an easy-to-use yet powerful template system. D was created by Walter Bright, who also created the first native C++ compiler and Empire (of all things).

Anyhow, one of the things that I really like about D is that it is maintains link compatibility with C ¹. This means that D code can make calls into libraries like those from GTK in the same way that C++ code can. This mitigates the need or the desire to rewrite useful tools. Many times simple binding layers can be written to expose C-style APIs in D-style ways.

In my case, I'm writing an application which uses the GTK bindings for D, GtkD, and embeds a Python interpreter. Creating the D header from Python.h turns out to be pretty easy with bcd, a tool which tries to automate most of the process ².

The first step is to run bcd against the header you want to use. For me this looked like:

./bcdgen /usr/include/python2.5/Python.h Python -C -A

In this case "-C" means "C-mode" (as opposed to C++-mode) and "-A" tells bcdgen to continue working on all includes recursively (since Python.h is really a meta-header). Unfortunately this gives me:

In file included from /usr/include/stdio.h:906,
              from /usr/include/python2.5/Python.h:32:
/usr/include/bits/stdio2.h: In function 'int sprintf(char*, const char*, ...)':
/usr/include/bits/stdio2.h:35: error: '__builtin_va_arg_pack' was not declared in this scope
/usr/include/bits/stdio2.h: In function 'int snprintf(char*, size_t, const char*, ...)':
...

This is a bug with Intrepid Ibex's version of gccxml. Luckily there are workarounds. Once bcdgen finishes it's work, it should create a bcd/Python/Python.d, which is where all the Python.h definitions landed. If you open it up and take a look, you can see bcdgen has converted them to D definitions. Way to go automated software development tools!

Generally some work is required before the D compiler accepts this file and the bcd page lists some general guidelines for this task. I'm familiar with Python's C interface, but definitely not intimate. This is a secret code that I'm using which means I spent a bit of time fixing weird D compiler errors. It wasn't bad though and I was was able to resolve most of the compile errors without much trouble. I'm my next post I'll write a little bit about that.

D does not have link compatibility with C++. Many C++ projects have have external APIs which are C-compatible though. One example from recent memory is the KJS API from WebKit.

There is a really great project called CeleriD, which has already done a lot of this work. CeleriD is really geared toward extending Python rather than embedding it though. I decided to strike out on my own as an educational gesture³ and to avoid depending on distutils for my build process.

Gesture because it's hard to believe this is can be considered larnin'⁴, which is coincidentally the way I feel about most professional software training programs.

The best thing about the <sup> tag is that it's a formatting markup and a greeting. 'Sup <sup>?