Optimization of real-time video processing on Nokia N95 smartphone

To work with real-time video one needs at least two things - direct access to bitmaps and fast way to get frames from a video camera. To achieve the first goal one needs to keep in mind that Bitmaps are located on a shared heap that needs to be locked before any direct access can safely take place. This is because some other applications and/or memory management can shift things around so if you get an address of a Bitmap it might become invalid while you are still accessing Bitmap elements.

Direct access

In this example, we access CFbsBitmap iSrcBitmap created elsewhere:

iSrcBitmap->LockHeap(ETrue); // lock global heap (once regardless of the number of bitmaps)
TUint8* ptr = (TUint8*)iSrcBitmap->DataAddress();

// direct access with the code that never leaves
ptr[33] = 255;

iSrcBitmap->UnlockHeap(ETrue); // unlock global heap

TIP: My experience shows that in bitmap-intensive applications in order to avoid unexplained crashes it is a good idea to change the priority of an application e.g. add the following string in AppUi::ConstructL()
RProcess().SetPriority (EPriorityHigh);

YUV to RGB conversion

2. In order to outperform BitBlt() function that is particularly slow when copies frames from a video camera (YUV format)
to RGB bitmaps one has to use his/her own conversion function from say YUV244 to RGB 16M. If done properly, this will speed up your code approximately 3 times. For example, YUV244 uses a unique gray (Y) and shared color (U, V) for two neighboring pixels in such a way that 4 bytes in YUV244 format result in 6 BGR bytes. In short U,Y1,V,Y2 -> G1,B1,R1,G2,B2,R2 according to a following formula:

     U -=128;
     V -=128;
     R1 = Y1+1.370705*V;
     G1 = Y1-0.698001*V-0.337633*U;
     B1 = Y1+1.732446*U;
     R2 = Y2+1.370705*V;
     G2 = Y2-0.698001*V-0.337633*U;
     B2 = Y2+1.732446*U;

By the way, somebody just gave me an excellent idea to use Y1, Y2 as gray channel in case you don't need color. In this case there is no need for conversion and multiplications! Note that in general Y is not an average RGB and is calculated as Y = R.299+G*.587+B*.114 which actually looks better then (R+G+B)/3

TIP: A good idea is to clip the resulting RGB values to make sure they stay within 0..255
In order to avoid above multiplications that are too slow one has to create look up tables for each value of intensity from 0 to 255. A good starting point to learn more about this conversion is http://local.wasp.uwa.edu.au/~pbourke/dataformats/yuv/

Landscape Mode

3. Also a good idea for using all capabilities of your video camera is to switch to a Landscape mode right away regardless of how you are going to hold a phone. It is especially important for image recognition and localization applications since it guarantees a wide vertical angle (so a user don't need to scan vertically) and a narrow horizontal angle (so the azimuth error of localization is small). In order to switch to a Landscape mode do two things. First, update your AppUi::ConstructL():

BaseConstructL(CAknAppUi::EAknEnableSkin|EAppOrientationLandscape); // code in your AppUi::ConstructL()

And second, Choose an additional option : "Read User Environment" in your mmp file ->Options->Capabilities.

TIP: Note that in a Landscape mode the longest screen dimension (320 pixel) corresponds to the width of a bitmap.