Tuesday, April 21, 2009

Multi-Process Mobile Desktop Concept

With the introduction of the Atom 330 processor, dual-core computing was realized in the mobile world. While still keeping the overall power consumption down, these chips add real power and additional capabilities to the system.

In order to take advantage of this new power, software must be designed with multi-threading in mind. Graphical applications that use a GUI toolkit, often perform background work on one thread, and run the Graphics on another thread. Most of the time, multi-threading can be a headache as developers run into race-conditions, deadlocks, and other problems related to multi-threading. Many times multi-threaded apps are discouraged and avoided by some developers.

In addition, when a thread in an application crashes, it usually brings the entire app down with it. For plugin-centric applications like firefox, this can be a huge annoyance to the end-user when a little ad-blocker plugin brings down your entire browsing session.

So how does a developer take advantage of the multiple core processing power and avoid the issues related to multi-threaded development? The answer may be a multi-process design.

Google Chrome wasn't the first to implement a multi-process design, nor will it be the last. Each tab in the Chrome browser is a separate process. Google did this to increase stability and security. And it works. When a tab crashes in Chrome, the tab dies... Chrome keeps on chugging.

Should the same concept apply to mobile computing? I don't see why it shouldn't. The hardware is becoming cheaper, and multi-core chips are here to stay.

A Multi-Threaded Mobile Desktop

To start out with a mobile desktop, one must first understand the mobile environment. Mobile devices usually hand held, or dash-mounted (in the car), with anywhere from a 4"-8" screen. Because the screen is smaller, fonts and other graphical elements need to scale so they are readable by the user. Applications on these usually run full screen.

The desktop in a mobile environment also differs slightly from a desktop on a personal computer. A PC desktop is usually a place where you can put files, and launch applications. It is usually run by the file-manager like nautilus in GNOME and explorer in windows. A mobile desktop is also a place where you can launch applications. But it probably isn't a good place to dump files on because of the lack of screen space. In reality, the mobile desktop is an application that launches other applications. For our multi-process desktop, we will take this idea one step further.

Building a mobile desktop

Writing apps takes time. Writing plugins using a very specific set of tools and interfaces is often much faster. The desktop in our example will act as a server: it will facilitate communication between plugins. Launch them at the users request, and close them when they are no longer needed. Below is a c++ example of a simple plugin interface:

class PluginInterface

virtual ~PluginInterface();

virtual void OnCommandRecieved(CommandClass command) = 0;
virtual void init(DesktopServer ServerInstance) = 0;
virtual void run() = 0;

Why is communication needed? In a mobile environment, resources are scarce. The audio device is an excellent example of a scarce resource. If you are using your mobile GPS to guide you through traffic while listening to music, you don't want your directions to mix with the music. Not only will you not understand where to go, your music enjoyment will be interrupted by a mix of disharmony and chaos. The GPS application/plugin, needs to ask for control of the audio device. It can send a message to the server to "fade" the music volume down to 0, pause, and then fade back in the music when the GPS is done giving it's direction. That sounds much better than Ozzy Ozborne being interrupted by "Turn Left in 30 meters".

By using DBus, a common *nix IPC messaging system, plugins can communicate with each other, and with the server.

Plugins, however, are usually in-process code. How do they fit in a multi-process environment? Simple, they don't. They need a "harness" in order to run as a separate process. In the case of our mobile desktop, a lightweight "core" will be used. The main function of this "core" is to setup communication with the server, load the plugin, and make sure the plugin gets whatever the server sends it. Below is an simple example of our core application:

int main(int argc, char** argv)
if(argc < 2)
printf("usage: core [pluginname]");

string pluginname = string(argv[1]);

PluginInterface *plugin = LoadPlugin(pluginname);

DesktopServer *Server = new Server();




In our desktop, we will have a dock plugin, an app/plugin launcher plugin, an audio device manager plugin and a notification plugin. When any additional plugins or apps are loaded, an icon will appear on the dock so users can switch between that plugin and others. When a plugin wants to use the audio device, it will ask the audio device plugin for control. Likewise when a plugin wants to notify the user of something, like an incomming call, it can send a message to the notification plugin.

Here is an example of a command being sent to the audio device plugin:

void Speak()
string volumelevel = Server.Evaluate("AudioDevice:GetVolumeLevel");

Server.Send("AudioDevice:Fade '0'");

SpeakSomething("Turn Left in 100 Meters");

Server.Send("AudioDevice:Resume '0'");
Server.Send("AudioDevice:Fade '"+volumelevel+"'");


The server then parses the message, grabs the plugin name "AudioDevice" and forwards the message to the Audio Device plugin:

void OnServerMessageReceived(string message)
string plugin_name = GetPluginName(message);
Plugin plugin = GetPlugin(plugin_name);



Below is a visual example of the server core+plugin concept:

When it's all run together, the result is a tightly integrated stable and secure desktop system like the visual below:

Concept Drawbacks

It would be dishonest to present a concept without pointing out the drawbacks, obvious or not. The first drawback is that applications that are used in the system that aren't plugins or do not have plugin-wrappers have a huge disadvantage. These applications won't be able to integrate at the level the plugins will. They likely won't know about the notification plugin, or the audio device plugin. Second, this system forces the developer to learn yet another plugin system, etc.


A multi-process desktop has many advantages. Stability: plugins can't crash the desktop or any other plugin; Security: precautions made by the OS help partition what the plugin can do; Integration: plugins can talk to each other to provide the ultimate user experience; Rapid Development: plugins are faster to develop than entire apps.

Finally, OpenICE, the Open In-Car-Entertainment group plans on implementing the multi-process desktop for use in in-vehicle infotainment systems in the 2.0 release in 2010. Stay tuned...

Tuesday, April 7, 2009

OpenICE - Computer Enhanced Cars

No matter how you look at it, technology in cars has come a long way since the cassette tape. When you buy a Car today, you most likely were offered by the dealer an upgraded "technology package". This usually includes upgraded speakers, GPS navigation, Bluetooth and the ability to play music from your iPod.

With all the technology available today, do these enhancements really help enhance our experience. What is the future of technology in cars? I hope to point to a opensource software project that looks to answer the question.

There are many closed-source software solutions looking to become the center of In Vehicle Infotainment (IVI). Streetdeck an advanced car interface for the Windows XP operating system just released their 2.0 version last month. Centrafuse, another software application for Windows was seen on a number of devices at CES this year. Many of us have seen the Microsoft/Ford Sync commercials on TV. So where is the Open Source alternative? Enter OpenICE.

OpenICE (Open In Car Entertainment/Infotainment) is a open source project that aims to create a standard for IVI systems. OpenICE also develops software to meet these standards.

OpenICE categorizes IVI into 4 sections:

* Entertainment
* Information
* Connectivity
* Mobility

OpenICE maintains a "Plays Everything" policy by utilizing open source technology. Unlink many closed-source systems, the OpenICE media playback software, nGhost Media Desktop, can play back a number of non-patent encumbered media codecs like Ogg-Vorbis/Theora and FLAC. NGhost Media desktop also supports plugins for mplayer and phonon, which allows backends such as gstreamer among others to be used.

The Future:
The future of entertainment is in smart devices. Smart phones, mobile internet devices, and Ultra Mobile PCs are examples of such devices. In the future, OpenICE hopes to integrate with these devices so that the moment you walk into the car, the car recognizes these devices, and the car's interface displays and controls information from the devices. Drivers will be able to play mp3's from their phone on their car stereo system via bluetooth or wifi. They'll be able to answer phone calls by voice activation allowing the driver to focus on driving.

OpenICE has developed plugins to the nGhost media desktop to give real-time car statistics to the driver. In the current version of the plugin, the driver can view Engine RPM, Speed, Engine Coolant Temperature, and Engine Load.

OpenICE has also developed plugins to display weather and traffic information to drivers.

The Future:
OpenICE hopes to extend information into read even more data from the vehicle, including error codes. OpenICE hopes to integrate traffic information into the navigation and automatically reroute you to avoid traffic.

OpenICE also hopes to integrate automation into the vehicle. By selecting a user preference, the car may automatically roll down windows and shut off the air conditioning to save energy when traveling at certain speeds. The volume on your music may automatically adjust to compensate for wind noise when traveling at high speeds. The vehicle may warn you that you are driving to aggressively and recommend different ways to save gas by adjusting your driving behavior.

OpenICE hopes to achieve connectivity with mobile devices. Handsfree communication is essential in IVI environments. OpenICE uses the nohands project to provide handsfree communication with cellular phones with bluetooth.

Streaming media over bluetooth, upnp via wifi, are also ways in which OpenICE hopes to connect the driver to his mobile devices.

The Vehicle is a very different environment then most. The driver must keep his attention on the road where it belongs. Any technology added to the car must aid the driver and not distract him.

OpenICE implements guidelines for graphical interface so the driver will be able to use the system, and still keep his attention on the road.

Currently mobile graphical interfaces are designed for use in handheld devices. Because of the requirements of the Vehicle's environment, where the drivers attention is only on the interface for brief moments, no other mobile graphical interface existed that really tailored to life on the road. OpenICE developed it's own customized desktop (nghost) and dock (icepanel) to satisfy these specific requirements.

Video of nGhost,icepanel, and Compiz-Fusion's cube rotate:

OpenICE is a young project with big goals. It recently (yesterday) released it's development roadmap where hopefully, many of these future ideas and concepts will become a reality. As with any young open source project, the more developers that join and contribute, the better the experience for everyone will be who uses the software.

Hopefully, we will soon see some aftermarket products based on the OpenICE platform. Eventually, even auto makers will include Open Standards in their "technology" offerings.