I am one of our QWS developers. QWS, the Qt Window System, is the heart of Qt for Embedded Linux, formally Qtopia Core, formally Qt/Embedded. What’s great about working on embedded is that you have a view of the system as a whole - the complete stack. So it’s my job to have a pretty good idea how QPainter commands you write in a widget’s paint event end up as voltage levels rapidly alternating between +3.3v and 0v on wires going to your LCD. While QWS handles all the usual window system tasks such as keyboard focus and mouse events, the biggest component is probably graphics. The window system is inherently part of the graphics stack and I actually spend a lot of my time working with the graphics team.

Over the last year our clear focus has been on performance. This performance push has been in all areas on all platforms and architectures. When it comes to graphics, there’s a very broad range of hardware Qt can expect to see - from a simple MMU-less ARM with only a frame-buffer all the way to scary gamer PCs with thousands of graphics cards & neon lights installed. Our lives are made complicated because if we’re not careful, one can end up writing code which runs faster on that little arm than it does on the gamer PC. This post hopes to explain why this is so.

Let’s begin by categorising the range of hardware available:

First, we need to differentiate Unified Memory Architecture (UMA) devices from those with dedicated graphics memory. Generally, high-end hardware will have dedicated graphics memory whereas low-end devices will just use system memory (sometimes reserving a memory region, sometimes not). This is pretty strait forward on PCs - You can almost tell from a PC’s price tag if it has dedicated graphics memory. Sadly, in the world of embedded devices, this is not the case. High-end devices often have UMA and low-end devices (especially set-top-boxes) have dedicated graphics memory.

The next differentiation is the graphics operations supported by the hardware. Generally they are wide ranging but can be loosely categorised as:

1) No acceleration (framebuffer only)
2) Blitter & alpha-blending hardware
3) Path based 2D vector graphics
4) Fixed-function 3D
5) Programmable 3D

Hardware with no acceleration whatsoever or a simple video overlay is the most common we see in embedded devices. This will always be the case until someone figures out how to design and manufacture silicon for free. Blitter and alpha-blending hardware is almost non-existent on desktops these days, but it does seem to still be around in the current generation of embedded hardware. Path based 2D vector graphics is pretty new and looks ready to replace blitter-only style hardware. NOTE: This does not refer to hardware which can draw a 1-pixel wide, non-anti-aliased, non-dashed, solid-colour line without clipping. Fixed-function 3D tends to be the older generation of desktop graphics processors. Generally, fixed function has pretty much been replaced with programmable 3D. This is even the case on mobile hardware.

So, there’s five categories of operations and two types of memory architecture leading to ten different overall types of graphics hardware. I’ve collected an example of each, just so you know we don’t make this stuff up.

Type UMA Non-UMA None Marvel PXA270 Various* Blitter NXP PNX8935** Fujitsu Lime MB86276*** 2D vector Freescale i.MX35 Fixed-3D Freescale i.MX31 nVidia GeForce 2 Programmable-3D TI OMAP3530 AMD Radeon HD 4600

* Various: Some devices use dedicated framebuffer memory to reduce load on the system memory bus
** NXP PNX8935: http://www.nxp.com/applications/set_top_box/ip_stb/stb225/
*** Fujitsu Lime MB86276: http://www.fujitsu.com/downloads/MICRO/fma/pdf/MB86276.pdf

The next question then becomes: How can Qt off-load graphics operations to these different types of hardware? Well this is done through Qt’s QPaintEngine API. The idea is that Qt applications (& Qt itself of course) always uses QPainter, which in turn uses one of the paint engines. To take advantage of graphics acceleration, we write a new paint engine (like the OpenGL ES 2.0 engine we’ve added in 4.5.0). The advantage is that existing applications can benefit from new rendering back ends and new applications can still work on older or less advanced hardware (albeit with lower performance). There seems to be a misconception in the community that Qt is out-of-date because it has no OpenGL scene graph API. While that statement is technically correct, Qt does have QGraphicsView scene graph API which uses QPainter. Because it uses QPainter, if OpenGL (for example) is available, it can be used as the rendering back end.

So, now that’s cleared up, what QPaintEngines are there and do we have all the hardware acceleration types covered by them?

Well, for devices with no acceleration, Qt will use it’s raster paint engine. The raster engine has seen some very impressive optimizations in Qt 4.5, as Gunnar has previously blogged about. For higher end graphics hardware, there’s usually a nice high-level API which is powerful enough to express all of QPainter. I.e. OpenGL & OpenVG. The trouble we’ve recently hit is the hardware in-between, I.e. those with blitters but not much else. Such hardware is not powerful enough to express the whole of QPainter, so we must fall back to the raster paint engine for unsupported operations. The raster paint engine needs a pointer to the memory it renders to (and reads from). On UMA systems, this is not a problem as the buffer is obviously in system memory (that’s all there is!). It’s on systems with dedicated graphics memory where the fun begins…

First, on many systems, you simply can not map graphics memory into your process’ address space - The architecture simply has no way to do it. On such systems, the buffer must be copied to system memory, rendered to with the raster engine, then copied back. If this happens _every_ time you switch between a fall-back and the hardware, it’s going to be _slow_!!

On some systems (particularly PowerPC for some reason?), the graphics controller sits on the SoC’s external bus and can be addressed directly by an application. All that needs to happen is for the kernel to configure the process’s page table to point to the graphics controller’s memory range. It’s then up to the graphics controller to access data in it’s dedicated memory on behalf of the host CPU. Although this kind of set-up does allow the raster paint engine to get a pointer to graphics memory, all accesses go over this external bus - which is usually slow. On PC/x86 architecture, things get more even more complicated, the kernel has to fiddle with lots more hardware, cache controllers, PCI bus controllers, IOMMUs, etc. However, in all cases, if you’re lucky enough to get a pointer to graphics memory, all access must go over a slower external bus.

So now we know what’s going on, what conclusion can we draw? Well, reading & writing to external graphics memory is slow. If your on non-UMA, don’t have OpenGL or OpenVG available, but do want to use your blitter then you’d better make sure your mostly using QPainter::drawPixmap(). NOTE: Graphics view’s cache modes can help you out a lot there - see Andreas’ previous posts! Otherwise falling back to the raster engine is going to be slow. Fortunatly, this type of hardware is (finally) on it’s way out.

NOTE: I should also mention that there’s a similar issue with X11. There’s no API to get a pointer to an X pixmap and X11 does not provide enough API to implement the whole of QPainter. While the X11 paint engine does not inherit from the raster paint engine, it does make use of software fall-backs which involve copying the pixmap, executing the fall-back and then uploading the result. It’s for that reason that we’ve added the raster graphics system which uses system memory (via the MITSHM extension) in Qt 4.5. On desktop, this is a fairly temporary measure until our OpenGL 2.0 engine & graphics system is in a fit state to take over all of Qt’s rendering. No promises, but we hope that can happen for Qt 4.6. For X11 on low-end embedded devices (like the n810), MITSHM provides a pretty decent long term solution.

So, when we look to the future of Qt’s graphics architecture and the required paint engines, I think we’re well on the way to having all the bases covered:

Type UMA Non-UMA None Raster Raster* Blitter DirectFB DirectFB** 2D vector OpenVG*** OpenVG*** Fixed-3D OpenGL (ES) 1.x OpenGL (ES) 1.x Programmable-3D OpenGL (ES) 2.x OpenGL (ES) 2.x****

* When using raster on NUMA, rendering is actually done in system memory first, then flushed to VRAM
** This is the one which is going to be slow when doing anything other than QPainter::drawPixmap()
*** It shouldn’t be a big surprise we’re researching an OpenVG paint engine!
**** Qt 4.5 contains a new paint engine for OpenGL ES 2.x which we’re now making work on desktop OpenGL 2.0

I just want to finish by asking you to take another look at the above table. Do you notice anything interesting? All of the graphics systems (apart from DirectFB) are cross platform which means, when we make something faster in one engine, all platforms will benefit.

In the comments to my previous post, a few pointed out that the states & transitions API we’re researching seems generally useful (i.e. outside of Kinetic context). Hence, we’ve released a stand-alone state machine add-on that you can play with. It’s been tested with Qt 4.4 and 4.5. Enjoy, and don’t be shy about giving feedback.

Now into something a bit different. Today’s Graphics Dojo example is related to WebKit (again), namely using it as a basis for WYSIWYG (what-you-see-is-what-you-get) editor. Thanks to Apple engineers and other WebKit hackers, WebKit’s editing features have improved a lot so that it is perfectly possible to use it to let the user edit the web page.

These days, rich-text editing is supported quite well in many web browsers. Take a look at Mozilla support for editing, which gives an idea on how to start doing it. Head also to the short overview on HTML and JS-based editors to get more information.

The editor demo is available at the usual git repository. Too lazy to use git? Just grab the archive (65 KB) and unpack it. Check the editor under the htmleditor directory. You need Qt 4.5, which has just been released.

To get an idea what this little editor can do, have a look at this short screencast.

A couple of things I showed in the demo are character formatting (bold, italic, underline, font name, size, colors), paragraph settings (style, alignment, indentation, lists), image and hyperlink support. You can also insert any HTML code. For editing convenience, undo and redo are even possible. Switching to "HTML Source" tab will reveal the raw HTML code for the page (note the syntax highlight).

Since this is just a quick-and-dirty example, there are few things which are not done the right way. For example, due to the lack of our QtWebKit public API (which we plan to address for Qt 4.6), I need to resort to JavaScript code for some functionalities. Also, things like paragraph alignment, heading style, font size, etc are stateless, mainly because there is no easy path to get the state value. Of course, this can be solved using a few DOM and JS magic tricks commonly employed in popular online editors, however I am still looking for a much cleaner way. Feel free to point me to some leads and/or give some feedback!

Commit by fullmetalcoder :: r970 /trunk/3rdparty/qcodeedit2/lib/document/qdocument.cpp: (link) Small fix in document font change.

In the spirit of doing releases and more specifically, Jasons doing releases (or at least blogging about them), I am happy to present to you the latest release of the Qt for S60 port! Go ahead, download it in the background while you keep reading:

Download Qt for S60 “Garden”

Following fresh on the heels of the Qt 4.5.0 release, our latest release, commonly referred to as “Garden”, is a real showcase for all the hard work that we have done over the past 4 months. Espen set the bar pretty high when he mentioned our ambitions for Garden back in December, but I am quite proud to say that we pulled it all off in the end. One of the most obvious differences you’ll notice from the previous (Temple) release is that this one is based on Qt 4.5. That means we got a bunch of performance improvements and other features practically for free!

One of the main goals of this release was to cleverly disguise Qt applications as native S60 applications and see if we could fool our grandmothers. This was no easy feat considering that S60 has interchangeable themes, input methods, and special menus and adding to that the fact that our previous release looked like a certain OS from 1995, we knew we had some work to do. Well, grandma put on her reading specs and fired up her XpressMusic 5800 to have a look and she was pleasantly surprised!

She immediately saw the application themed correctly and could input text like she did in other S60 applications (they call her “CrazyThumbs” at bingo) and after trying some other Qt applications she could use menus and even drag-n-drop worked! Well, needless to say she had no idea that she was using the same application framework that powered the desktop environment on her Linux box!

Ok, far fetched fantasies aside, there are tonnes of other improvements in the Garden release. We have rewritten large parts of our event loop and also QWidget so that QApplication and QWidgets can be easily embedded in native S60 applications. In fact, we even introduced a new static library called ‘qtmain’ to hijack the entry point and construct a full S60 application before calling an application’s main() function. With the full S60 environment at our disposal we could start integrating with things like the status pane and menu pane thus making functions like QWidget::setWindowTitle() work properly. Finally, we ported a few new classes such as QLocale, QClipboard and QDesktopServices. There are lots of other things I could mention, but I don’t want to steal all the joy from reading the changelog

To get you up and running with the build, here is Alessandro to explain the details:

Finally, here is a video from Espen where he takes you on a walk through the Garden (release):

The next steps for us will be to evaluate where we go next with the port. I won’t cover that here today since it warrants a blog post of its own so I’ll keep you in suspense. Enjoy!

Getting help and providing feedback

Pre-releases are not supported, but we still want your feedback of course. We have set up a special mailing list for feedback on the S60 port:

qts60-feedback@trolltech.com

This list is read by the developers working on the port, so please join if you want to provide technical feedback, bug reports or suggestions to us directly. In order to join the mailing list send a mail to:

qts60-feedback-request@trolltech.com

There is also a Nokia Forum available here.

Solid Dragon

It’s time to present the little hack that I’ve been working on for a while now: A Mac OS X backend (source code) for Solid, KDE’s hardware introspection library. To make a good screen shot, and to show how easy it is to use Solid, here’s also a small hardware browser:

The backend is still in progress, and not all Solid hardware interfaces are implemented yet. Still, the generic interface can be used to introspect any device and read out its properties. You can even introspect the famous Dont_Steal_Mac_OS_X device

One of the nice things about Solid is that it can be used from non-KDE applications as well - it just requires Qt (*)

64-bit Leopard

Next to playing with Solid, the release of Qt 4.5 was a good excuse to play with 64-bit KDE on 10.5 (Leopard).

First, we need the magic CMake switch to build 64-bit Mac OS X applications: -DCMAKE_OSX_ARCHITECTURES=”x86_64″ (**)

Unfortunately, some of kdelib’s files rely on deprecated API which doesn’t exist in the 64-bit versions of OS X’s libraries. So after some fixing and some more commenting out of non-compiling stuff, I finally got my first KDE 64-bit application:

minibrowser.app/Contents/MacOS/minibrowser: Mach-O 64-bit executable x86_64

Screen shot: see above

 

(*) Well, there’s also a D-Bus dependency

(**) You can produce universal binaries by separating the desired architectures with a semicolon, e.g. -DCMAKE_OSX_ARCHITECTURES=”x86_64;i386;ppc;ppc64″ to get all four currently supported architectures. Unfortunately, I have no clue how to create universal binaries for the third party libs that KDE depends on (e.g. D-Bus) - any automake expert, please leave a comment (or convince the projects to switch to a more humane build system).

Of course, I cheated. The trick is actually by using Google’s own iPhone web application, which is pure HTML and JavaScript. Despite the name Google Talk, unfortunately it has no support for voice chat, rather only text chat.

The secret is to find the actual URL and load it in QWebView. Since initially it presents a login page, this particular client also automatically fills the username and password for you, getting them from its own login page. Overall, the code is very simple and does not even reach 200 lines, including some rudimentary error handling (wrong password, failed login). The user interface was created completely using Qt Designer. Careful readers might notice that I deliberately added the flick support (kinetic scrolling) using Flick Charm. This quick-and-dirty chat client was completed in less than quarter an hour.

document.getElementById("webgtalk").style.maxWidth="800px";

Check the code at the usual Graphics Dojo git repository, find it under the webgtalk folder. This example requires Qt 4.5, which has just been released, in particular due to the use document.getElementByid. Exercise for the reader: tweak the little JavaScript snippets used in this program so that it works also with Qt 4.4. Another challenge: there is also
chat client for iGoogle which performs better but then it requires Flash plugin.

Commit by fullmetalcoder :: r969 /trunk/ (72 files in 11 dirs): (link) Updated svn:eol-style property.
Copied qxs/ data from QCE to Edyuk to stay up-to-date.
Added a missing icon to Edyuk.

Commit by fullmetalcoder :: r968 /trunk/src/lib/ (15 files in 2 dirs): (link) Added configuration for QCE formatting to Edyuk config dialog.

Commit by fullmetalcoder :: r967 /trunk/3rdparty/qcodeedit2/ (3 files in 2 dirs): (link) Small changes in QFormatConfig for more flexibility.

Commit by fullmetalcoder :: r966 /trunk/3rdparty/qcodeedit2/ (7 files in 2 dirs): (link) Some small improvements to config widgets.

Commit by fullmetalcoder :: r965 /trunk/3rdparty/qcodeedit2/designer-plugin/ (4 files): (link) Added simple color picker widget to the designer plugin.

Commit by fullmetalcoder :: r964 /trunk/3rdparty/qcodeedit2/ (22 files in 5 dirs): (link) Added per-language QXF files (content not tuned though) for all supported languages.
Made sure svn:eol-style property was set to native on all source files.

Commit by fullmetalcoder :: r963 /trunk/3rdparty/qcodeedit2/qce.dox: (link) Adjusted inline HTML in qce.dox : <li> tags are now closed to make folding behave properly. (figure out a way for the syntax to gracefully allow some tags like <li> and <br> not to be closed?)

Commit by fullmetalcoder :: r962 /trunk/3rdparty/qcodeedit2/qxs/ (dox.qnfa formats.qxf xml.qnfa): (link) Fixed tag name highlighting in XML syntax (tagnames with numbers are now handled properly).
Added XML embed in Doxygen syntax, as should have been done from the beginning.

Commit by fullmetalcoder :: r961 /trunk/3rdparty/qcodeedit2/lib/ (8 files in 3 dirs): (link) Some API extensions.
Changed the QCE_EXPORT macro definition to avoid easier embed into an app.

Commit by fullmetalcoder :: r960 /trunk/3rdparty/qcodeedit2/ (2 files in 2 dirs): (link) Added auto-loading of per-language format files (.qxf) if present in the same dir as the associated .qnfa
Updated QCE::Demo format config to take thes format files into account when found.

Commit by fullmetalcoder :: r959 /trunk/3rdparty/qcodeedit2/lib/widgets/ (formatconfig.ui qformatconfig.cpp): (link) Some small adjustements in format config widget.

Commit by fullmetalcoder :: r958 /trunk/3rdparty/qcodeedit2/lib/widgets/ (2 files): (link) Improved the color picker look when it holds an invalid color and made it possible to clear the color (i.e make it invalid so that app as to use proper default value) using a right click (context menu event is caught actually which may not be the smartest thing...)

Commit by fullmetalcoder :: r957 /trunk/3rdparty/qcodeedit2/designer-plugin/designer-plugin.pro: (link) Added missing project file for QCE Designer plugin.