Multimedia
From UIQ Books
Contents |
Multimedia
Mobile applications frequently include multimedia content. We use images, animations, audio and video to enrich the user experience.
This chapter gives you an introduction to using multimedia and covers some of the common application level multimedia requirements:
- load images
- store images
- perform some basic image manipulation
- use the camera to take a photograph
- load and play back audio clips
- record and save audio clips
- load and playback a video clip
- use the FM radio via the Tuner API.
SignedAppPhase3
In the previous chapter we developed a simple file management application, SignedAppPhase2. We use this as the starting point for our multimedia examples and add multimedia functions to create SignedAppPhase3.
Figure 1 SignedAppPhase3 list view and menu
The New command enables us to record a sound clip or take a photograph. The Open command allows us to open the sample media.
Figure 2 New command and image display with rotate function
Sample media is provided in the folder SignedAppPhase3\ExampleFiles. Copy them to the application folder in the emulator \epoc32\winscw\c\Private\20000462 where is the path to where the emulator is installed on your PC.
After adding the multimedia features to SignedAppPhase3, we submitted it to the Test House and obtained Symbian Signed. The signed application is included in the download.
Symbian Signed Requirements
Capabilities
Prior to proceeding with any development work, we should consider any implications that Symbian OS platform security may impose. Capabilities are described in the Symbian Signed chapter. Multimedia APIs may require your application to have one or more of these three capabilities:
-
UserEnvironment(User) -
SurroundingsDD(System) -
MultimediaDD(System).
Of these, UserEnvironment can be granted by a user; SurroundingsDD and MultimediaDD can be granted by passing Symbian Signed. MultimediaDD used to be a Device Manufacturer capability. You must use Open Signed with a Publisher ID to test an application that uses MultimediaDD.
To work out which capabilities you will need, consult the SDK documentation for the classes that your application will use. Occasionally, you may need to use trial and error where the SDK is unclear.
The trailing DD in both the SurroundingsDD and the MultimediaDD capabilities, along with their description, is an indication that these capabilities are only required if we need to gain direct access to the internals of device drivers. The vast majority of applications simply want to use the functionality provided by the device drivers and do not need such low level access.
The MultimediaDD capability is only required if applications wish to manipulate the priority or preferences of the audio record or playback channel it has opened. Few third party applications should manipulate these features, as they may affect the incoming phone call ringing or other alarm type sounds. Symbian Signed test Criterion GEN-01 requires that applications do not adversely affect the use of other system features. Blocking incoming phone call ringing is likely to cause your application to fail this test.
No capabilities are required for the image manipulation, audio and video playback example applications in this chapter. UserEnvironment is required for sound and video recording and using the camera.
Symbian Signed Test Criteria
When we assessed SignedAppPhase2 against the Symbian Signed Test Criteria in the Symbian Signed Test Criteria section, we identified that further work was required to pass all the test cases. We have added the privacy statement for CON-02 and billable event dialog for CON-03. While these tests from Test Criteria v2.11.0 are likely to be removed in v3.0.0, it will always be important that you check your application against the current Test Criteria and make any necessary changes.
Figure 3 Privacy statement and billable event warning dialog
Once we have added the multimedia functionality described in this chapter, our application will be ready for final testing and submission via the Symbian Signed website.
Images
Symbian OS provides an Image Conversion Library (ICL) which offers support for image file encoding and decoding, plus scaling and rotation of bitmaps. The ICL provides encode and decode support for the following image types:
| Format | Encode support | Decode support |
| BMP (Bitmap) | Yes | Yes |
| EXIF (Exchangable Image File format) | Yes | Yes |
| GIF (Graphics Interchange Format) | Single frame, no transparency | Single & multi frame, bitmap mask support |
| JPEG (Joint Photographic Experts Group) | Yes | Yes |
| MBM (Multi Bitmap) | Single frame | Single & multi frame |
| MNG (Multiple image Network Graphic) | Yes | Yes |
| PNG (Portable Network Graphics) | No transparency | Bitmap mask support |
| SMS OTA (Over The Air) | No | Yes |
| TIFF (Tagged Image File Format) | No | LittleEndian and BigEndian sub-type support |
| WBMP (Wireless Bitmap) | No | Yes |
| ICO (Icon) | No | Single & multi frame |
| WMF (Windows Meta File) | No | Std, apm and clp sub-type support |
Since the ICL uses a plug-in architecture to perform the decoding tasks, the set of decoders on your mobile phone may vary from the set within the SDK. The Sony Ericsson P1i supports BMP, GIF, JPEG, MBM, PNG and WBMP formats. We recommend that you check support and test your application on the mobile phones that you wish to support.
Figure 4 Image decoding and encoding
Image Decoding
Image decoding is the process of loading an image and converting it into the internal CFbsBitmap format used by Symbian OS applications to display it on screen. Input formats include BMP, JPG and GIF.
The image decoding classes use asynchronous methods to perform the actual processing. Instead of using threads, they use active objects. This has several consequences:
- Our application must have an active scheduler in the thread performing the image decoding. A standard UIQ application primary thread always has a scheduler as it is provided by the application framework.
- Our application needs to wrap the image processing functions in an active object.
- Our application can process other events between the time an image processing event is started and the time it completes.
The CImageDecoder class provides the basic image decoding functionality. Since the ICL comprises a set of plug-ins, typically one plug-in per image format, a mechanism of associating the correct plug-in with the content is required. Variants of the CImageDecoder class allow applications to specify different attributes. For our application we simply want to be able to load an image file, irrespective of its underlying format. This can be achieved by requesting the CImageDecoder to use the underlying plug-ins to determine the file type on our behalf.
Image Size and Color Depth
Once the CImageDecoder object has been created, we should request information about the underlying image. We are particularly interested in the image size and the color depth attributes.
Prior to loading the image, we need to create space, in the form of a CFbsBitmap object, to store the image data. The CFbsBitmap needs to be the same width and height as the original image.
Color depth refers to the number of different colors that each pixel can represent. GIF images use 8-bits per pixel, so each pixel can represent one color from a palette of 256, but BMP and JPG formats are capable of representing millions of colors. Similarly, the screen has a color depth and this will vary between different models of mobile phone. The first UIQ phone, the Sony Ericsson P800, had a 4096 color screen. The MOTO Z8 has a 16-million color screen.
When loading, the values representing colors in the image need to be translated to the equivalent value for those colors used by the screen, particularly if an image format uses a palette to store color information. Optimal display speed is achieved when the color depth of an image matches the display mode of the window within which the image is displayed. This is because no translation of the individual pixels is required between the CFbsBitmap data and the data displayed within a window. Since windows are normally created using a default display mode it is preferable to create the CFbsBitmap with the same display mode and only translate the image once, when it is loaded.
Translating the image when it is loaded is only possible if the underlying plug-in supports color translations. This is indicated by the TFrameInfo::ECanDither flag returned within the TFrameInfo object. We use this in our example application. If it is important for your application to always load images into the current display mode but the plug-in does not support TFrameInfo::ECanDither, simply create a CFbsBitmap of the required color depth then copy the image into a this second CFbsBitmap after it has been loaded. The copy operation in the form of the DrawBitmap() method automatically performs the color depth translation.
Loading the Image
The code for our image loading is:
CImageLoader::~CImageLoader()
{
Cancel();
}
CImageLoader::CImageLoader(CImageDisplayControl& aDisplay):
CActive(CActive::EPriorityStandard),iDisplay(aDisplay)
{
CActiveScheduler::Add(this);
}
void CImageLoader::StartLoadImageL(
const TFileName& aFileName)
{
iLoader=CImageDecoder::FileNewL(CEikonEnv::Static()->FsSession(),aFileName);
// Create a bitmap within which to store the loaded image.
TFrameInfo frameInfo(iLoader->FrameInfo(0));
TInt ret=KErrNoMemory;
iBitmap=new CFbsBitmap();
if (iBitmap)
{
if (frameInfo.iFlags&TFrameInfo::ECanDither)
ret=iBitmap->Create(frameInfo.iOverallSizeInPixels,
CEikonEnv::Static()->ScreenDevice()->DisplayMode());
else
ret=iBitmap->Create(frameInfo.iOverallSizeInPixels,frameInfo.iFrameDisplayMode);
}
if (ret!=KErrNone)
{
delete(iBitmap);
iBitmap=NULL;
delete(iLoader);
iLoader=NULL;
User::Leave(ret);
}
// Start loading the image into our bitmap, RunL() called when complete.
iLoader->Convert(&iStatus,*iBitmap,0);
SetActive();
}
void CImageLoader::RunL()
{
iDisplay.LoadComplete(iStatus.Int(),iBitmap);
iBitmap=NULL; // We no longer own it.
delete(iLoader);
iLoader=NULL;
}
void CImageLoader::DoCancel()
{
iLoader->Cancel();
delete(iBitmap);
iBitmap=NULL;
delete(iLoader);
iLoader=NULL;
}
As you can see, this is a relatively simple active object. The CImageDisplayControl object referenced by the constructor would create and own this object. When an image is required to be loaded, the StartLoadImageL() method is called passing the fully defined file name. At some point in the future, the RunL() method is called by the active scheduler. The image has been fully loaded by the time this method gets called. In our application we call back to our owning object, informing them of the success or otherwise of the load request and pass any created bitmap object. The CImageDisplayControl::LoadComplete(iStatus.Int(),iBitmap) method is called back.
The StartLoadImageL() demonstrates that simply leaving if we fail to create objects is usually not sufficient. In this example, if we called iBitmap = new(ELeave) CFbsBitmap() and the leave occurred, we would have a CImageDecoder object created and stored in the iLoader property. It is likely that if the StartLoadImageL() did leave, we would not tidy up or delete the CImageLoader object; instead, we would probably inform the user of a problem and allow them to try again.
Trying again means recalling StartLoadImageL(). This would create a second CImageDecoder object, store the handle in iLoader overwriting any previous object handle which leads to a memory leak of the original iLoader. An alternative implementation of StartLoadImageL() using the cleanup stack is:
void CImageLoader::StartLoadImageL(
const TFileName& aFileName)
{
CImageDecoder*decoder=CImageDecoder::FileNewL(CEikonEnv::Static()->FsSession(),aFileName);
CleanupStack::PushL(decoder);
// Create a bitmap within which
// to store the loaded image.
TFrameInfo frameInfo(decoder->FrameInfo(0));
CFbsBitmap* bmp=new(ELeave)CFbsBitmap();
CleanupStack::PushL(bmp);
if (frameInfo.iFlags&TFrameInfo::ECanDither)
User::LeaveIfError(
bmp->Create(frameInfo.iOverallSizeInPixels,
CEikonEnv::Static()->ScreenDevice()->DisplayMode()));
else
User::LeaveIfError(
bmp->Create(frameInfo.iOverallSizeInPixels,frameInfo.iFrameDisplayMode));
// Only now we cant leave can we take ownership of the objects in object property.
iBitmap=bmp;
iLoader=decoder;
CleanupStack::Pop(2);
// Start loading the image into our bitmap, RunL() called when complete.
iLoader->Convert(&iStatus,*iBitmap,0);
SetActive();
}
Both implementations perform the same task. In general, the second version is shorter, faster and arguably easier to follow since there are fewer branches to understand.
| Pro tip:
A common error when creating bitmaps is failing to take any notice of the return value of the |
Image Encoding
Image encoding is the process of taking a bitmap used internally by an application and converting to a specified output format. Internal bitmaps are represented by the CFbsBitmap object. Output formats include BMP, JPG and GIF. Since the ICL uses a plug-in architecture to perform the encoding tasks, the set of encoders on each model of mobile phone may vary from the set within the SDK.
Image formats typically comprise a primary type along with a set of sub-types. These types and sub-types are properties of the set of plug-ins that support image saving. In our application, we enumerate this information, presenting it to the user to make the choice of saved format. The information is presented within a dialog comprising a text editor to accept the target file name and two choice lists, one for the primary type and one for the sub-types associated with that primary type.
Figure 5 SignedAppPhase3 save image dialog
The set of primary file types presented to the user is generated with:
void CImageSaverDialog::PreLayoutDynInitL()
{
CImageEncoder::GetImageTypesL(iImageTypes);
CEikChoiceList* cl=static_cast<CEikChoiceList*>(Control(2));
CDesCArrayFlat* items=new(ELeave)CDesC16ArrayFlat(4);
CleanupStack::PushL(items);
TInt count=iImageTypes.Count();
for (TInt i=0;i<count;i++)
items->AppendL(iImageTypes[i]->Description());
cl->SetArrayL(items);
CleanupStack::Pop(items);
cl->SetCurrentItem(0);
}
And the set of image sub-types is generated with the following code:
void CImageSaverDialog::SetImageSubTypesL()
{
CEikChoiceList* cl=static_cast<CEikChoiceList*>(Control(2));
TUid type=iImageTypes[cl->CurrentItem()]->ImageType();
// Create the sub-types list based on primary type.
cl=static_cast<EikChoiceList*>(Control(3));
TRAPD(err, CImageEncoder::GetImageSubTypesL(type,iImageSubTypes));
if (err==KErrNotFound)
{ // No sub-types are reported via an error!
cl->SetArrayL(R_IMAGE_SAVE_NO_SUB_TYPES_ARRAY);
cl->SetCurrentItem(0);
}
else
{
User::LeaveIfError(err);
// Create list of sub-types to display to the user.
CDesCArrayFlat* items=new(ELeave)CDesCArrayFlat(4);
CleanupStack::PushL(items);
TInt count=iImageSubTypes.Count();
for (TInt i=0;i<count;i++)
items->AppendL(iImageSubTypes[i]->Description());
cl->SetArrayL(items); // Takes ownership.
CleanupStack::Pop(items);
cl->SetCurrentItem(0);
}
}
In the case of sub-types it is possible that no sub-types for a particular primary type are supported. In this case, the GetImageSubTypesL() method leaves with the error code KErrNotFound. In our application, we choose to display a choice list containing a single entry entitled None. If a particular plug-in supports a range of sub-types, we can enumerate them as shown above.
Once the name, type and sub-type information has been collected we can begin the process of encoding the image, storing the resultant information in the specified file. As with decoding, the encoding operation is asynchronous and our application wraps this within an active object:
CImageSaver::~CImageSaver()
{
Cancel();
}
CImageSaver::CImageSaver(CImageDisplayControl& aDisplay) :
CActive(CActive::EPriorityStandard),iDisplay(aDisplay)
{
CActiveScheduler::Add(this);
}
void CImageSaver::StartSaveImageL(
const TFileName& aFileName,
const TUid& aImageType,
const TUid& aImageSubType,
CFbsBitmap* aBitmap)
{
iSaver=CImageEncoder::FileNewL(
CEikonEnv::Static()->FsSession(),
aFileName,CImageEncoder::EOptionNone,
aImageType, aImageSubType);
iSaver->Convert(&iStatus,*aBitmap);
SetActive();
}
void CImageSaver::RunL()
{
iDisplay.SaveComplete(iStatus.Int());
delete(iSaver);
iSaver=NULL;
}
void CImageSaver::DoCancel()
{
iSaver->Cancel();
delete(iSaver);
iSaver=NULL;
}
As with image decoding, the active object is not too difficult to understand. The CImageDisplayControl object referenced by the constructor would create and own this object. When an image is required to be saved, the StartSaveImageL() method is called passing the fully defined file name, type, subtype and image data. In due course, the RunL() method is called by the active scheduler. The image has been saved by the time this method gets called. In our application, we call back to our owning object, informing them of the success or failure of the save request. The CImageDisplayControl::SaveComplete(iStatus.Int()) method is called back.
Image Processing
The multimedia framework provides a number of additional image processing functions such as rotation, scaling and mirroring. In our example, we demonstrate the usage of the CBitmapRotator class to perform bitmap rotation. Mirroring uses the same class but different parameters. Scaling is performed by the CBitmapScaler class which follows an identical pattern to the usage of the CBitmapRotator class.
Our application supports rotating an image. As with encoding and decoding, the image rotation is performed asynchronously. We wrap this within an active object as shown:
CImageRotator::~CImageRotator()
{
Cancel();
}
CImageRotator::CImageRotator(CImageDisplayControl& aDisplay) :
CActive(CActive::EPriorityStandard),iDisplay(aDisplay)
{
CActiveScheduler::Add(this);
}
void CImageRotator::StartRotateImageL(
CFbsBitmap* aBitmap)
{
iRotator=CBitmapRotator::NewL();
// Take ownership now we cant leave.
iBitmap=aBitmap;
// Start rotating the image into our bitmap.
// RunL() called when complete.
iRotator->Rotate(&iStatus,*iBitmap,
CBitmapRotator::ERotation90DegreesClockwise);
SetActive();
}
void CImageRotator::RunL()
{
iDisplay.RotateComplete(iStatus.Int(),iBitmap);
iBitmap=NULL; // We no longer own it.
delete(iRotator);
iRotator=NULL;
}
void CImageRotator::DoCancel()
{
iRotator->Cancel();
delete(iBitmap);
iBitmap=NULL;
delete(iRotator);
iRotator=NULL;
}
In our example, we have chosen to store the rotated image in the same CFbsBitmap as it was originally stored. This reduces the amount of memory required to store the image since a second target CFbsBitmap is not required but because the operation is asynchronous it is entirely possible that a redraw event occurs while in the middle of the rotation. Attempting to draw from a partially rotated bitmap will at best produce an apparently corrupt image. In our application, we avoid this by transferring ownership to the CImageRotator object and passing it back when the image rotation completes. In this way we only ever draw from the CFbsBitmap when no rotation is in progress.
As previously mentioned, both image rotation and image saving are asynchronous operations. The main reason for this is to ensure your application remains responsive to other events. This includes the possibility for a user to choose either the Rotate or Save menu options before the first request has completed. Attempting to rotate an image while in the middle of saving may result in a panic and is not recommended. A CBase-42 panic will occur if you attempt to start a rotate (or save) operation while the current rotate (or save) operation is in progress since an active object cannot represent two simultaneous outstanding events.
Robust applications need to handle this situation. One Symbian Signed test, GEN-02, is for the Test House to try and break your application. A panic usually results in failure.
In the case of rotating or saving images, the time between starting the operating and it completing is relatively small. The likelihood of a user managing to request a second operation while the first is running is very low. In such cases, rather than presenting a Cancel type dialog it is usually acceptable to simply ignore the second request. Almost no-one in the real world will ever manage to hit the condition and notice that their request was ignored, yet we remove the possibility of our application crashing via a panic.
To support this requirement, rather than run any state machine to track our internal status we can simply use the active object state. Therefore our RotateImageL() code that wraps around the CImageRotator class is:
void CImageDisplayControl::RotateImageL()
{
if (iBitmap && !iImageRotator->IsActive() && !iImageSaver->IsActive())
{
iImageRotator->StartRotateImageL(iBitmap);
iBitmap=NULL;
DrawNow(); // Bitmap removed from display, while being manipulated.
}
}
Image Processing Summary
In this section, we have discovered how to add some basic image processing support to our application outside of the ability to load Symbian native MBM files.
As you have seen, the image processing functionality is largely asynchronous. Robust applications need to track the state of the image processing sub-systems to ensure the application remains in a stable state irrespective of user actions. This type of application programming is typical of real-time systems.
Alternative Image Support: CQikContent
As an alternative to using the low level image conversion library to load and display images UIQ provides a powerful UI level object, CQikContent, designed to simplify programming by hiding a number of the complexities that we encountered in the previous section. We have already seen CQikContent in action within the ListView2 example application. We present further examples of its use in the QContent example which we discuss in this section.
Overview
At a high level, details such as the graphics file format are often a distraction from the task in hand. For example, if we wish to display a graphic icon to represent a particular piece of information, we are only interested in what information we are trying to present and not which file format the graphic icon happens to be stored in.
Similarly, details about how to actually display the image, for example, whether there is a bitmap and icon pair or simply a bitmap and whether to scale or crop the image are implementation details. Objects that want to display images strongly prefer such details to be dealt with by the image class itself leaving it to focus on which images to display.
A CQikContent object encapsulates these concepts to aid the higher level programming tasks.
The QContent application demonstrates numerous ways in which we can construct a CQikContent object.
Construction from an MBM File
Probably the most common way to construct a CQikContent is by referencing a bitmap and mask pair stored within the application MBM file.
icon=CQikContent::NewL(NULL, KMbmFile, EMbmQcontentIcon2, EMbmQcontentIcon2mask);
As a shortcut to referencing the application specific MBM file, the KMbmFile descriptor can be set to the value “*”. The underlying system code explicitly checks for this value and works out your full MBM file name from the application name and the file system path the application was loaded from. The remaining enums, EMbmQcontentIcon2 and EMbmQcontentIcon2mask are automatically generated in an application specific MBG file when you compile your image file list defined in the MMP file
Construction from a Resource File
Rather than explicitly define which images are required within your C++ code it is possible to define the images in your application resource file and use that information to construct a CQikContent. The QContent example demonstrates the usage of both the QIK_CONTENT and QIK_CONTENT_MBM resource structures:
RESOURCE QIK_CONTENT r_qik_content_icon
{
uri = "c:\\Private\\EDEAD023\\image.gif";
}
RESOURCE QIK_CONTENT_MBM r_command_icon
{
bmpid=EMbmQcontentIcon0;
bmpmask=EMbmQcontentIcon0mask;
}
These resources are read by the following code fragments:
// From a QIK_CONTENT.
{
TResourceReader rr;
HBufC8* q=iEikonEnv>AllocReadResourceAsDes8LC(R_QIK_CONTENT_ICON);
rr.SetBuffer(q);
CQikContent* icon=CQikContent::NewL(NULL,rr);
CleanupStack::PopAndDestroy(q);
}
// … other code removed for clarity.
// From a QIK_CONTENT_MBM.
{
TResourceReader rr;
HBufC8* q=iEikonEnv->AllocReadResourceAsDes8LC(R_COMMAND_ICON);
rr.SetBuffer(q);
CQikContent* icon=CQikContent::NewL(NULL,rr);
CleanupStack::PopAndDestroy(q);
}
The first will construct a CQikContent containing an image loaded from a GIF format file named image.gif which is stored at the root of the C drive. The second loads an image and bitmap mask from the application-specific MBM file.
Construction from a CGulIcon
Your application may already be using icons in the form of CGulIcon objects. Rather than fully construct a CQikContent using the original source images of a CGulIcon you can clone the CGulIcon into a CQikContent. Cloning is considerably faster than constructing the objects from scratch as little underlying bitmap manipulation is required.
The following code fragment constructs a CGulIcon and clones it into a CQikContent.
// Create GulIcon for example purposes only. CGulIcon* gi=iEikonEnv->CreateIconL(KMbmFile, EMbmQcontentIcon3, EMbmQcontentIcon3mask); CleanupStack::PushL(gi); // Create a CQikConent from a CGulIcon. CQikContent* icon=new(ELeave)CQikContent; CleanupStack::PushL(icon); icon->CloneL(gi); CleanupStack::Pop(icon); // For this example application purposes only. CleanupStack::PopAndDestroy(gi);
Construction from a CQikContent
Your application may need to display the same graphic image in several different places. Rather than loading the graphic image each time a new CQikContent object is required, you can load the images once then clone the originals to generate the other CQikContent objects.
The following code demonstrates how you might create a masterIcon then clone it.
CQikContent* masterIcon=CQikContent::NewL(NULL, KMbmFile, EMbmQcontentIcon2, EMbmQcontentIcon2mask); CQikContent* icon= masterIcon->CloneL();
Construction from an External File
A core feature of the CQikContent class is the ability to load images from non Symbian native image files. We have already seen how this can be achieved using a QIK_CONTENT resource definition. The following code fragment shows how you might load images directly in C++:
icon1=CQikContent::NewL(NULL,_L("c:\\Image.bmp"));
icon2=CQikContent::NewL(NULL,_L("c:\\Image.gif"));
icon3=CQikContent::NewL(NULL,_L("c:\\Image.jpg"));
icon4=CQikContent::NewL(NULL,_L("c:\\Image.png"));
icon5=CQikContent::NewL(NULL,_L("c:\\Image.wbmp"));
In this code fragment we load five images, each from a different file format.
As we stated at the beginning of the chapter, the list of image formats supported by a particular phone is dependent on the set of image decoders available on that phone.
If you observe the QContent application closely, you should notice that those icons we load from MBM files are displayed immediately whereas those loaded from non-Symbian native file formats are displayed after a slight delay. This tiny delay is due to the CQikContent code having to perform some reasonably compute intensive operations to load and decode the external image file formats. This same delay is observed in our SignedAppPhase3 application when we use the CImageDecoder classes.
Camera
Symbian OS provides an abstract onboard camera API called ECam, which is implemented by handset manufacturers according to the capabilities of each particular mobile phone. ECam provides a hardware agnostic interface for applications to communicate with and control any onboard camera hardware. This API allows applications to:
- control camera settings, for example setting contrast, brightness and zoom settings
- capture images
- capture video clips.
The precise settings available are dependent on the hardware within a phone. The API accommodates variation in hardware through various informational services.
Since cameras capture information about the user and the current user environment, applications that wish to use the camera API are required to have the UserEnvironment capability.
Symbian Signed Requirements
Symbian Signed test GEN-01 requires applications do not affect the use of system features or other applications. In the case of the camera resource, this means that our application should relinquish control of the camera if the user chooses to task away from our application. When returning to our application it would be quite reasonable to take back control of the camera as if we were using it. This ensures that we do not affect any other application that requires usage of the camera, such as the Camera application itself.
Not all phones contain cameras. Attempting to use the camera APIs on such phones will at best not work and at worst panic our application. Panics cause applications to fail Symbian Signed testing. Our application does not depend on the camera and contains functionality that still makes sense on mobile phones that have no camera. We should therefore support phones that do not contain a camera. This may not be possible if you application must use the camera, in which case you should disable application installation onto phones except those that contain cameras.
In our application, we report that the functionality is not available should the user attempt to use the camera on a mobile phone that does not have one:
TInt count=CCamera::CamerasAvailable();
if (count<1)
User::Leave(KErrNotSupported);
Using the Camera
To use the camera to take a photograph our application needs to:
- create a
CCameraobject - reserve usage of the camera, ensuring our application has exclusive use of the camera
- switch the camera on
- optionally configure the camera settings
- optionally display images within a viewfinder
- capture an image
- switch power off
- release the camera, so other applications can use it.
As with the image processing classes, the camera class uses asynchronous methods to perform the actual processing. Using active objects has the following consequences:
- Our application must have an active scheduler in the thread performing the camera operations. A standard UIQ application primary thread always has a scheduler as it is provided by the application framework.
- Event completion is indicated through call backs. We have to implement the callback interface.
- Our application can process other events between the time the camera object is created and the time a preview image is available to capture. We must ensure our application handles such events appropriately.
Camera Callback Interface
The camera API supports two callback interfaces, MCameraObserver and MCameraObserver2. Neither of these interfaces has been deprecated. The main difference is in the information delivered as part of the captured image. Our application demonstrates the usage of both these interfaces but the interface chosen depends on a compile time #define within the CameraView.cpp file. The final application uses the MCameraObserver2 interface, which we recommend, although you may wish to check with the mobile phone manufacturer.
| Pro tip:
We recommend using the |
Capturing an Image
When the camera view is ViewActivatedL() it starts the camera. When ViewDeactivated() is called we shut down usage of the camera. This ensures we comply with the Symbian Signed GEN-01 test.
Firstly, we ensure there is a least one camera on the phone, and leave if none exists. This ensures that our application does not panic if users attempt to take photos on phones that do not contain cameras.
void CCameraDisplayControl::StartViewFinderL()
{
// Check to see if we have a camera on the current phone.
TInt count=CCamera::CamerasAvailable();
if (count<1)
User::Leave(KErrNotSupported);
#ifdef USE_CAMERA_OBSERVER_INTERFACE
iCamera=CCamera::NewL(*this,0);
#else
iCamera=CCamera::NewL(*this,0,EPriorityNormal);
#endif
iCamera->Reserve();
}
Secondly, we create an instance of the CCamera object, the CCameraDisplayControl class implements the MCameraObserver or MCameraObserver2 interface depending on the compile time #ifdef.
Using the MCameraObserver Interface
The following section explains the processes and callbacks that occur if an application chooses to use the MCameraObserver interface.
Our first requirement for using the camera is to Reserve() usage. This is an asynchronous function which calls back the ReserveComplete() method.
void CCameraDisplayControl::ReserveComplete(TInt aError)
{
if (aError==KErrNone)
{
iCamera->PowerOn(); // Calls back PowerOnComplete().
}
else
{
TBuf<128>bb;
iEikonEnv->Format128(bb,R_STR_CAMERA_RESERVE_ERROR,aError);
iEikonEnv->InfoMsg(bb);
CloseCamera();
}
}
If we are able to reserve usage, we then need to power on the camera by calling the PowerOn() method. This method is asynchronous which eventually calls back the PowerOnComplete() method.
void CCameraDisplayControl::PowerOnComplete(TInt aError)
{
TCameraInfo info;
info.iImageFormatsSupported=0;
if (aError==KErrNone)
{
iState=ECameraStatePoweredOn;
iCamera->CameraInfo(info);
TRAP(aError,
// Tell view finder draw directly to our scn space
TRect rect(Rect());
rect.Move(PositionRelativeToScreen());
iCamera->StartViewFinderDirectL(iEikonEnv->WsSession(),
*iEikonEnv->ScreenDevice(), Window(),rect);
iCamera->PrepareImageCaptureL(CCamera::EFormatJpeg, info.iNumImageSizesSupported-1);
);
}
if (aError!=KErrNone)
{
TBuf<128>bb;
iEikonEnv->Format128(bb,R_STR_CAMERA_POWER_ERROR, aError,info.iImageFormatsSupported);
iEikonEnv->InfoMsg(bb);
CloseCamera();
}
}
Assuming the power on request completed without error, we now request the camera to present content directly to the screen to act as a viewfinder. This is achieved using the StartViewFinderDirectL() method. Since this method uses direct screen access, we need to adjust the rectangle our window occupies relative to the screen, such that, the output from the camera is positioned correctly for our application.
Finally, we tell the camera to PrepareImageCaptureL(). By calling this method now, we minimize any latency between attempting to take a photo and the image actually being captured.
We must provide some sort of shutter button for the user to take a photo. In our application, this is achieved by tapping on the screen, depressing the Action key, entering the numeric value 5 or pressing enter. Do not forget that some UIQ 3 mobile phones do not have touch sensitive screens and may present a standard phone keypad. Therefore, you should consider presenting a number of ways (commands) to achieve the same task.
Whatever input mechanism is used, the TakePictureL() method will be called:
void CCameraDisplayControl::TakePictureL()
{
if (iState==ECameraStateTakingPicture)
User::Leave(KErrInUse);
if (iState!=ECameraStatePoweredOn)
User::Leave(KErrNotSupported);
iCamera->CaptureImage();// Calls back mageBufferReady()
iState=ECameraStateTakingPicture;
}
Since we are a real-time system we need to ensure the application is in the correct state to capture images. We achieve this by recording the states our application progresses though and validating that we are able to take a photo before calling the CaptureImage() method. This eventually calls back the ImageReady() method.
void CCameraDisplayControl::ImageReady(
CFbsBitmap* aBitmap,
HBufC8* aData,
TInt aError)
{
if (aError==KErrNone)
{
RFile file;
_LIT(KPhotoJpg,"Photo.jpg");
aError=file.Replace(iEikonEnv->FsSession(),KPhotoJpg,EFileShareExclusive|EFileWrite);
if (aError==KErrNone)
{
aError=file.Write(*aData);
file.Close();
if (aError==KErrNone)
{
TRAP(aError, iEngine->AddEntryL(KPhotoJpg,EAppCategoryImages);
);
iEikonEnv->InfoMsg(R_STR_PHOTO_SAVED);
}
}
if (aError!=KErrNone)
iEikonEnv->FsSession().Delete(KPhotoJpg);
}
if (aError!=KErrNone)
{
TBuf<128>bb;
iEikonEnv->Format128(bb,R_STR_CAMERA_IMAGE_ERROR,aError);
iEikonEnv->InfoMsg(bb);
}
CloseCamera();
Static_cast<CQikAppUi*>(iEikonEnv->AppUi())->ActivateViewL(KViewIdListView);
}
In this code extract we save the data to file, tell the engine there is a new entry, close down the camera and switch back to our list view.
| Pro tip
The 5 key on a numeric keypad is often used as a fire or Enter button. If you have used Java applications on a mobile phone, pressing the 5 will usually result in an action. |
Using the MCameraObserver2 Interface
The following section explains the processes and callbacks that occur using the MCameraObserver2 interface. The actions to perform are the same as when using the MCameraObserver interface but it is how they are performed that is slightly different.
Our first task is to reserve usage of the camera. Calling the Reserve() function results in a callback to the HandleEvent() method as opposed to the ReserveComplete() method:
void CCameraDisplayControl::HandleEvent(
const TECAMEvent& aEvent)
{
TBuf<128>bb;
if (aEvent.iEventType==KUidECamEventReserveComplete)
{ // Called back after a Reserve() completes.
if (aEvent.iErrorCode==KErrNone)
{
iCamera->PowerOn();
// Calls back HandleEvent() with KUidECamEventPowerOnComplete.
}
else
{
iEikonEnv->Format128(bb, R_STR_CAMERA_RESERVE_ERROR, aEvent.iErrorCode);
iEikonEnv->InfoMsg(bb);
CloseCamera();
}
}
else
// Omitted for clarity, see below…
}
As before, we simply progress to calling the PowerOn() method to switch the camera on. Completion of the PowerOn() request calls back the HandleEvent() method a second time. To work out which event is generating the callback, we check the aEvent.iEvent field of the passed TECAMEvent object.
void CCameraDisplayControl::HandleEvent(
const TECAMEvent& aEvent)
{
TBuf<128>bb;
if (aEvent.iEventType==KUidECamEventReserveComplete)
{ // Omitted for clarity, see above…
}
else if (aEvent.iEventType == KUidECamEventPowerOnComplete)
{ // Called back after a PowerOn() completes.
TInt err=aEvent.iErrorCode;
TCameraInfo info;
info.iImageFormatsSupported=0;
if (err==KErrNone)
{
iState=ECameraStatePoweredOn;
iCamera->CameraInfo(info);
TRAP(err,
TRect rect(Rect());
rect.Move(PositionRelativeToScreen());
iCamera->StartViewFinderDirectL(iEikonEnv->WsSession(),
*iEikonEnv->ScreenDevice(), Window(),rect);
iCamera->PrepareImageCaptureL(Camera::EFormatJpeg, info.iNumImageSizesSupported-1);
);
}
if (err!=KErrNone)
{
iEikonEnv>Format128(bb,R_STR_CAMERA_POWER_ERROR,err,info.iImageFormatsSupported);
iEikonEnv->InfoMsg(bb);
CloseCamera();
}
}
}
As before, we start the viewfinder and tell the camera to prepare for image capture. When the user selects one of the input mechanisms supported to take a photo, the
TakePictureL() method is called. Since we are implementing the MCameraObserver2 interface, the ImageBufferReady() method will be called back when the photo has been taken.
void CCameraDisplayControl::ImageBufferReady(
MCameraBuffer& aCameraBuffer,
TInt aError)
{
if (aError==KErrNone)
{
RFile file;
_LIT(KPhotoJpg2,"Photo.jpg");
aError=file.Replace(iEikonEnv->FsSession(), KPhotoJpg2, EFileShareExclusive|EFileWrite);
if (aError==KErrNone)
{
aError=file.Write(*aCameraBuffer.DataL(aCameraBuffer.iIndexOfFirstFrameInBuffer));
file.Close();
if (aError==KErrNone)
{
TRAP(aError,
iEngine->AddEntryL(KPhotoJpg2,EAppCategoryImages);
);
iEikonEnv->InfoMsg(R_STR_PHOTO_SAVED);
}
}
if (aError!=KErrNone)
iEikonEnv->FsSession().Delete(KPhotoJpg2);
}
if (aError!=KErrNone)
{
TBuf<128>bb;
iEikonEnv->Format128(bb,R_STR_CAMERA_IMAGE_ERROR,aError);
iEikonEnv->InfoMsg(bb);
}
aCameraBuffer.Release();
CloseCamera();
Static_cast<CQikAppUi*>(iEikonEnv->AppUi())->ActivateViewL(KViewIdListView);
}
In this code fragment, we save the data to file, tell the engine there is a new entry, close down the camera and switch back to our list view.
You should note that in both the ImageRead() and ImageBufferReady() method code fragments included above, we have removed the code that generates the fully formed drive, path and file name for the sake of clarity. Our application always creates a file called Photo.jpg in our private folder.
Image Formats and Sizes
In our application, we have chosen not to enumerate the supported image types and sizes or allow a user to choose between them. The Sony Ericsson P990i supports capturing JPG images, so we have chosen to hard code this option. We use image size 640x480 pixels.
To enumerate the supported image formats and sizes associated with them you would perform an action similar to:
TInt i;
TInt mask=0x01;
for (i=0;i<17;i++,mask<<=1)
{
if (info.iImageFormatsSupported&mask)
{ // This format is supported.
CCamera::TFormat format=(CCamera::TFormat)mask;
TSize size;
for (TInt j=0;j<info.iNumImageSizesSupported;i++)
{
iCamera->EnumerateCaptureSizes(size,j,format);
if (size.iWidth>0 && size.iHeight>0)
{ // This format can generate this size image...
}
}
}
}
It should be noted that the iNumImageSizesSupported field actually means the maximum number of supported image sizes irrespective of format, therefore we need to check that the returned size is non zero should we want to use the size information.
CQikCameraCaptureDlg Class
As an alternative to the callback interface, you may find the CQikCameraCaptureDlg is adequate for your application.
To use it, simply call:
CFbsBitmap* image=NULL; CQikCameraCaptureDlg::RunDlgLD(image);
This presents a dialog enabling a user to take a photo. For the application to process as required, a copy of the image is returned as a CFbsBitmap.
Figure 6 Camera dialog
Since cameras capture information about the user and the current user environment, applications that wish to use the CQikCameraCaptureDlg are required to have the UserEnvironment capability. If you choose to use this class you should also ensure you have an appropriate privacy statement dialog displayed when your application is run for the first time.
Camera Summary
We have seen how to add some support for taking photographs within our application. Good application behavior requires that we relinquish control of the camera should the user task to another application. As we have seen, this task is not especially difficult, particularly if the application is structured appropriately.
Multimedia Framework (MMF)
A large number of audio and video file formats have been developed, many of which are widely used on mobile phones today. The multimedia framework is designed to accommodate these different formats by use of a plug-in architecture. Each plug-in is responsible for supporting a specific data type and presenting a common interface to the framework. Writing plug-ins is a specialist task and is beyond the scope of this book.
The audio file formats supported as standard by the multimedia framework are AU, WAV and raw audio. Each of these file formats has variants such as mono/stereo, sample rate and bits-per-sample. Further details are available in the SDK documentation. MMF supports video: however, the phone manufacturer must add the desired plug-ins.
For example, the following audio types are supported on both the Sony Ericsson P1i and MOTO Z8: AAC (various types), AMR-NB, iMelody, MP3 and MIDI. We recommend that you check and test that the mobile phones you are targeting work correctly with the multimedia formats that your application uses.
The MMF makes it possible for your application to work with different format files. Apart from setting any necessary parameters, you need to know nothing about the encoding and decoding of the formats.
Figure 7 Multimedia framework
The tone player and audio streaming APIs are part of the MMF but plug-ins are not used.
If your application contains audio clips, you should check that the audio format you are using is available on the mobile phones you wish to support.
The audio and video playback classes use asynchronous methods to perform the actual processing; in particular, they use active objects. This has several consequences:
- Our application must have an active scheduler in the thread performing the playback. A standard UIQ application primary thread always has a scheduler as it is provided by the application framework.
- Event completion is indicated through callbacks so we have to implement a callback interface. In general, we also have to run a state machine to manage the playback states.
- Our application can process other events between the time a load, record or playback request is started and the time it completes.
Audio
In this section we extend our example application code to play and record audio clips.
Playing Back Audio Clips
Symbian OS supports a number of mechanisms to play back audio. The easiest of these to use is encapsulated by the CMdaAudioPlayerUtility object.
Playing back sound involves two tasks:
- loading the sound data to play
- supplying that data to the underlying hardware to generate the sound.
Loading Operation
Loading audio content from file is usually more difficult that simply opening a file and reading its content. In general, audio files contain information indicating how the audio data has been stored and compressed. To play back a sound successfully, this information needs to be interpreted and acted upon. The multimedia framework automatically interprets audio file information by determining which of the plug-ins is prepared to perform this task on its behalf. Once the correct plug-in has been identified, the audio content can be prepared for playback.
While this approach removes the need for individual applications to understand audio file formats, the disadvantage is that loading audio files can take a long period of time. The multimedia framework takes this into account within the API presented by the CMdaAudioPlayerUtility object. However, this makes the object harder to use since some application level co-ordination is required, particularly between loading and playing a particular audio clip.
Playback Operation
Once an audio clip has been loaded, it can be played back. Short audio clips might be used to confirm user actions or alert the user to events. Audio playback may last a long time, however. In the case of a music player, where a user may use the mobile phone to perform other tasks at the same time, like sending messages, then the user will task away to another application. Applications must therefore be able to play audio clips in the background. In common with loading audio clips, the multimedia framework takes this into account within the API presented by the CMdaAudioPlayerUtility object.
Usage of the CMdaAudioPlayerUtility object requires that we implement the MMdaAudioPlayerCallback interface. The interface comprises two methods:
void MapcInitComplete(TInt aError, const TTimeIntervalMicroSeconds& aDuration); void MapcPlayComplete(TInt aError);
To simply load and play back an audio clip the following code is sufficient:
void CListView::TidyUpSoundPlayer()
// Tidy up sound player resources.
{
if (iSoundPlayer)
{
iSoundPlayer->Close();
delete(iSoundPlayer);
iSoundPlayer=NULL;
}
}
void CListView::MapcInitComplete(
TInt aError,
const TTimeIntervalMicroSeconds& aDuration)
{
if (aError)
TidyUpSoundPlayer();
else
iSoundPlayer->Play();
}
void CListView::MapcPlayComplete(TInt aError)
// The sound we were playing has finished, successfully or otherwise.
{
TidyUpSoundPlayer();
}
void CListView::PlayAudioEntryL()
// The entry has been classified as an audio entry.
{
TFileName name;
iEngine->EntryFullName(name);
iSoundPlayer=CMdaAudioPlayerUtility::NewFilePlayerL(name,*this);
}
This code will successfully load and play back audio clips but it does not handle all of the events that could take place. In common with other real-time components, one of the reasons for audio playback to occur in background, or asynchronously, is to allow other events to occur. The user may choose to select a second audio clip to play before the first has completed. In this case, the PlayAudioEntryL() method will be called before the first audio clip has finished playing. Apart from overwriting the iSoundPlayer property with a new object handle, numerous undesirable downstream consequences are possible when the audio clip playback completes. At best the application will survive until application exit when a memory leak will be detected. At worse your application will panic..
A simple solution to this problem, and one that is adequate for our application, is to change the PlayAudioEntryL() method to cancel any currently playing audio clip before attempting to load and play an alternate audio clip. We also need to adjust the TidyUpSoundPlayer() method to take into account an additional real-time state: the audio clip is being played back at the time the method is called.
void CListView::TidyUpSoundPlayer()
// Tidy up sound player resources.
{
if (iSoundPlayer)
{
// Cancel any playback, harmless if not playing.
// Will panic if we have not completed the initialization yet.
if (iSoundInitialized)
iSoundPlayer->Stop();
iSoundPlayer->Close();
delete(iSoundPlayer);
iSoundPlayer=NULL;
}
}
void CListView::MapcInitComplete(
TInt aError,
const TTimeIntervalMicroSeconds& aDuration)
// The sound has been loaded.
{
if (aError)
{
TidyUpSoundPlayer();
return;
}
iSoundInitialized=ETrue;
iSoundPlayer->Play();// Calls back MapcPlayComplete().
}
void CListView::MapcPlayComplete(TInt aError)
// The sound we were playing has finished,
// successfully or otherwise.
{
TidyUpSoundPlayer();
}
void CListView::PlayAudioEntryL()
// The entry has been classified as an audio entry. Play it back.
{
// If any sound is currently playing we need to ensure we stop it in this application since
// we re-use the iSoundPlayer property for a different sound.
TidyUpSoundPlayer();
// Create a sound player to play back the currently selected file.
TFileName name;
iEngine->EntryFullName(name);
iLevel=ESoundLevelMedium;
iSoundInitialized=EFalse;
iSoundPlayer=CMdaAudioPlayerUtility::NewFilePlayerL(name,*this);
}
Another event that can occur is that the application is asked to exit while in the middle of playing back an audio clip. To accommodate such an event, we add a CListView destructor:
CListView::~CListView()
{
TidyUpSoundPlayer();
}
Synchronizing Audio Clips with Events
A common requirement is for audio clips to be synchronized with an event occurring within the system. If a significant amount of time is required to load a particular audio clip before it can be played back, the event may have passed. In some cases, playing a sound a half or one second after an event has occurred may be acceptable, in others it may not. Additionally, if a particular audio clip is played relatively frequently the amount of processing required to continually load the audio clip results in slower overall system performance and reduced battery life.
To reduce the delay between requesting a sound to be played and the sound being heard, audio clips can be pre-loaded, using one CMdaAudioPlayerUtility object per audio clip. The primary disadvantage to this is approach is a significant amount of RAM is required to hold the audio clip data. UIQ 3 provides a specific class, CMMFDevSound, for low latency audio playback, although it also uses RAM to hold the clip.
Playing Back Tones
Audio tones, such as sine waves or DTMF telephony signals can be played back using the CMdaAudioToneUtility class. This class follows a very similar pattern to the CMdaAudioPlayerUtility class. The object is instantiated and data is prepared asynchronously. Once the data has been prepared, the MatoPrepareComplete() method is called back. This is equivalent to the MapcInitComplete() callback. When asynchronous playback completes, the MatoPlayComplete() method is called. This is equivalent to the MapcPlayComplete() callback.
Further details of this class are available in the UIQ 3 SDK documentation.
Streaming Audio
Audio streaming enables applications to play audio without the entire audio clip necessarily being present. The audio data is held in a buffers and playback takes place from them. During playback, incoming data is added to the buffers. To provide uninterrupted sound output, it is important the buffers are, on average, filled faster than the data rate of the sound playback. Large buffers enable playback to continue for longer if the data is interrupted. The CMdaAudioOutputStream class is used to stream audio content. As with the CMdaAudioToneUtility, this class follows a very similar pattern to the CMdaAudioPlayerUtility class.
Further details of this class are available in the UIQ 3 SDK documentation.
Audio Recording
Recording of audio clips is achieved using the CMdaAudioRecorderUtility object. In theory, you can use this class to record audio in any of the audio formats for which a controller plug-in exists, however, we were restricted to recording to WAV on the emulator and AMR on a real phone. You should carefully check the audio record support that is available on the phones that you are targeting.
Our application presents a list of the plug-ins available along with their available configuration parameters:
Figure 8 SignedAppPhase3 record sound clip function
The list of plug-ins is presented as file name extensions and is generated with the following code:
CDesCArrayFlat* array=new(ELeave)CDesCArrayFlat(4);
CleanupStack::PushL(array);
RMMFControllerImplInfoArray controllers;
CleanupResetAndDestroyPushL(controllers);
RArray<TUid> mediaIds;
mediaIds.Append(KUidMediaTypeAudio);
CleanupClosePushL(mediaIds);
CMMFControllerPluginSelectionParameters* select = CMMFControllerPluginSelectionParameters::NewLC();
select->SetMediaIdsL(mediaIds, CMMFPluginSelectionParameters::EAllowOnlySuppliedMediaIds);
CMMFFormatSelectionParameters* selectRead = CMMFFormatSelectionParameters::NewLC();
select->SetRequiredPlayFormatSupportL(*selectRead);
CMMFFormatSelectionParameters* selectWrite = CMMFFormatSelectionParameters::NewLC();
select->SetRequiredRecordFormatSupportL(*selectWrite);
select->ListImplementationsL(controllers);
TBuf<KMaxFileName> bb;
TInt count=controllers.Count();
for (TInt i=0;i<count;i++)
{
const RMMFFormatImplInfoArray& recordFormats = controllers[i]->RecordFormats();
TInt rCount=recordFormats.Count();
for (TInt j=0;j<rCount;j++)
{
const CDesC8Array& extensions = recordFormats[j]->SupportedFileExtensions();
TInt eCount=extensions.Count();
for (TInt k=0;k<eCount;k++)
{
bb.Copy(extensions[k]);
array->AppendL(bb);
}
}
}
CleanupStack::PopAndDestroy(5);
The code fragment asks the system to generate a list of all the controllers. For all controllers, we request a list of recording formats. From each of the formats we request the file name extension most commonly associated with the format. In our code, we need to convert the extension, from ASCII to Unicode so we can present it to the user. The code that performs the conversation is as follows:
bb.Copy(extensions[k]);
It assumes all characters in the filename extension have a one to one mapping between ASCII and Unicode. This is always true for the filename extensions we are manipulating.
Filling in the Choice Lists
Prior to being able to add any data to the four remaining choice lists, we have to create a CMdaAudioRecorderUtility object and open a file with the required file name extension.
iRecorder=CMdaAudioRecorderUtility::NewL(*this); iRecorder->OpenFileL(iFileName);
The open file request is asynchronous. When complete, it will callback the MoscoStateChangeEvent() method which is defined in the MMdaObjectStateChangeObserver interface definition.
Once we receive the callback informing us the file has been successfully created, we can request the data type, bit rate, sample rate and channel information required to seed the remainder of our choice lists. It should be noted that if we request some information but none is available, the data supply methods will leave with a KErrNotSupported error.
void CAudioRecordView::CreateFormatChoicesL(void)
// Create the data sets from which the choice
// list content is generated.
{
TInt err;
TRAP(err,iRecorder->GetSupportedDestinationDataTypesL(iDataTypes));
if (err!=KErrNotSupported)
User::LeaveIfError(err);
TRAP(err,iRecorder->GetSupportedBitRatesL(iBitRates));
if (err!=KErrNotSupported)
User::LeaveIfError(err);
TRAP(err,iRecorder->GetSupportedSampleRatesL(iSampleRates));
if (err!=KErrNotSupported)
User::LeaveIfError(err);
TRAP(err,iRecorder->GetSupportedNumberOfChannelsL(iChannels));
if (err!=KErrNotSupported)
User::LeaveIfError(err);
}
Since we are presenting the information within a UI we are able to handle blank choice lists. Therefore we need to TRAP() any leave and continue collecting information as shown in the code fragment above.
Once we have collected the information we can inform the choice lists to display the first entry:
CEikChoiceList* cl;
cl=LocateControlByUniqueHandle<CEikChoiceList>(EAppChoiceList2);
if (iDataTypes.Count()>0)
cl->SetCurrentItem(0);
cl->DrawNow();
cl=LocateControlByUniqueHandle<CEikChoiceList>(EAppChoiceList3);
if (iBitRates.Count()>0)
cl->SetCurrentItem(0);
cl->DrawNow();
cl=LocateControlByUniqueHandle<CEikChoiceList>(EAppChoiceList4);
if (iSampleRates.Count()>0)
cl->SetCurrentItem(0);
cl->DrawNow();
cl=LocateControlByUniqueHandle<CEikChoiceList>(EAppChoiceList5);
if (iChannels.Count()>0)
cl->SetCurrentItem(0);
cl->DrawNow();
Our application allows the user to choose between any of the parameters the underlying plug-ins report is available.
If the user chooses a different file extension we have to close down the current file and CMdaAudioRecorderUtility object, create a new one and replace the recording parameters with those associated with this new format. The following code detects the change in the Types list:
void CAudioRecordView::HandleControlEventL(
CCoeControl* aControl,
TCoeEvent aEventType)
{
if (aEventType==EEventStateChanged)
{
CEikChoiceList* cl=LocateControlByUniqueHandle<CEikChoiceList>(EAppChoiceList1);
if (cl==aControl)
{ // Audio types changed content...
OpenRecordFileL();
}
}
}
When the user is ready to record, we need to set up the recording parameters and start recording:
void CAudioRecordView::StartRecordingL()
// Set up the recorder with the user chosen parameters and begin recording.
{
#ifdef __WINS__
iRecorder->SetDestinationDataTypeL(KMMFFourCCCodePCMU8);
#else
iRecorder->SetDestinationDataTypeL(KMMFFourCCCodeAMR);
iRecorder->SetDestinationBitRateL(12200);
iRecorder->SetDestinationSampleRateL(8000);
iRecorder->SetDestinationNumberOfChannelsL(TMdaAudioDataSettings::EChannelsMono);
iRecorder->SetGain(iRecorder->MaxGain());
#endif
iRecorder->SetPosition(TTimeIntervalMicroSeconds(0));
TRAPD(err,iRecorder->RecordL());
}
Recording is stopped by calling:
iRecorder->Stop();
Audio Summary
We have seen how to add audio playback and recording to applications. Your application must track the current state of the audio subsystems so you do not attempt to perform an action while a previously requested action is still running
11.8 Video
Symbian OS supports some basic video services, including the ability to record and play back video clips. As with the audio and image components of the multimedia framework, support for video is based on plug-ins. Unlike the audio and image components Symbian OS does not provide any standard plug-ins; this task is delegated to the mobile phone manufacturer. In practice, most mobile phones support the 3GPP file format standard since this has been specially developed for UMTS mobile phones. The Sony Ericsson P1i and MOTO Z8 phones both support 3GPP and MPEG-4.
Loading Video Clips
Prior to playing back a video clip we have to load and prepare it for playback. Since loading a video clip can take a long time, this task is performed asynchronously by the multimedia framework code.
Video clips can be considered as a large number of bitmaps that are drawn on the screen in rapid succession. To present smooth playback, these images need to be displayed at a constant rate and remain synchronized with any audio component. Rather than require each application perform such tasks, applications should use the CVideoPlayerUtility class.
Since the CVideoPlayerUtility class will perform playback for us, it needs to know where on screen the playback will occur. In our application, we want playback to occur within the rectangle belonging to the window where we are displaying any other output. The code to create a CVideoPlayerUtility and start loading a video clip is:
void CVideoDisplayControl::NewVideoSelectedL()
// Open the entry currently selected in the list view.
{
TRect rect(Rect());
rect.Move(PositionRelativeToScreen());
iVideoPlayer=CVideoPlayerUtility::NewL(*this,
EMdaPriorityNormal,
EMdaPriorityPreferenceNone,
iEikonEnv->WsSession(),
*iEikonEnv->ScreenDevice(),
Window(),rect,rect);
TFileName name;
iEngine->EntryFullName(name);
TRAPD(err,
iVideoPlayer->OpenFileL(name);
iEikonEnv->BusyMsgL(R_STR_LOADING);
);
if (err!=KErrNone)
{
CloseVideo();
User::Leave(err);
}
}
Once the iVideoPlayer->OpenFileL(name) request completes, the MvpuoOpenComplete() method is called back.
void CVideoDisplayControl::MvpuoOpenComplete(
TInt aError)
{
if (aError==KErrNone)
iVideoPlayer->Prepare();
else
{
iEikonEnv->BusyMsgCancel();
TBuf<128>bb;
iEikonEnv->Format128(bb,R_STR_OPEN_ERROR,aError);
iEikonEnv->InfoMsg(bb);
}
}
Prior to playing back the video clip or query any of its properties, we have to call the CVideoPlayerUtility:: Prepare() method. This in turn will call back the MvpuoPrepareComplete() method.
void CVideoDisplayControl::MvpuoPrepareComplete(
TInt aError)
{
iEikonEnv->BusyMsgCancel();
if (aError==KErrNone)
{
TRAP(aError,
iVideoPlayer->SetPositionL(0);
iVideoPlayer->GetFrameL(EColor64K);
iState=EVidDispStateGetFrame;
);
}
if (aError!=KErrNone)
{
TBuf<128>bb;
iEikonEnv->Format128(bb,R_STR_PREPARE_ERROR,aError);
iEikonEnv->InfoMsg(bb);
}
}
In our application, we have chosen to load the first frame of the video clip to display on screen while the video is not being played back. We achieve this with the SetPositionL() and GetFrameL() methods. The GetFrameL() method is asynchronous, eventually calling back the MvpuoFrameReady() method.
void CVideoDisplayControl::MvpuoFrameReady(
CFbsBitmap& aFrame,
TInt aError)
{
iState=EVidDispStateIdle;
if (aError==KErrNone)
{
iBitmap=(&aFrame);
DrawNow();
}
if (aError!=KErrNone)
{
TBuf<128>bb;
bb.Format(_L("FrameErr %d"),aError);
iEikonEnv->InfoMsg(bb);
}
}
As you can see, this method presents a CFbsBitmap handle that we can use to display the first frame as our primary content.
You may have noticed that we track the various CVideoPlayerUtility states through the iState property. In our application we have defined the following states:
-
EVidDispStateClosed, theCVideoPlayerUtilityclass is closed and we are not performing any video-related tasks. -
EVidDispStateGetFrame, theCVideoPlayerUtilityhas successfully been opened and a call to obtain the first frame to display as our primary content has been made. -
EVidDispStateIdle, our application is currently idle, displaying the first frame. We must be in this state to begin playback of a video clip. -
EVidDispStatePlaying, our application is currently playing back the video clip.
We use the iState information in two places: firstly to know when to draw the bitmap frame and, secondly, to know when we are able to actually playback the video clip.
Playing Back Video Clips
Once successfully loaded, we can play back the video clip with the following code:
void CVideoDisplayControl::PlayL()
// Start the video clip playback.
{
if (iState!=EVidDispStateIdle)
User::Leave(KErrNotSupported);
iVideoPlayer->SetPositionL(0); // From the start.
iVideoPlayer->Play();
iState=EVidDispStatePlaying;
DrawNow();
}
When playback completes, the MvpuoPlayComplete() method will be called:
void CVideoDisplayControl::MvpuoPlayComplete(
TInt aError)
// Called back by the framework when iVideoPlayer->Play() completes.
{
// We want to be able to replay so dont close the image down here.
iState=EVidDispStateIdle;
DrawNow();
}
In our application, we remove the single static frame displayed while the video clip is not playing, and replace it with a red square so that you can more easily see when playback commences. This helps demonstrate how you might construct a state machine to handle various states that occur in systems such as video playback.
As with the asynchronous functions in audio and image processing, we need to add some functionality to handle events such as a second playback request while the first is running. When we handled this condition for image rotation or saving, we choose to simply ignore the request. Since video clips are often minutes or even hours long, it is very easy for a user to choose the Replay menu option while playback is already in progress. As a minimum, we should present some kind of feedback to the user. In more consumer-friendly applications, we might choose to interpret the Replay event to mean ‘cancel playing the current clip and restart from the beginning’. We have chosen to simply present some feedback to the user. An alternative option could be to disable the Replay menu option while the video clip was playing and restoring it at a later date. Whichever option you choose needs to fit with your application requirements. Not handling the condition is likely to cause problems.
Recording video clips
The CVideoRecorderUtility class facilitates video recording. As with video playback the record class requires you:
- create a
CVideoRecorderUtilityobject providing it with a class implementing theMVideoRecorderUtilityObserverinterface - open a file to record the video content to
- prepare the recorder for recording
- record the content.
We have not implemented the above in our application but the required process follows an almost identical pattern to video playback. Selection of the format to save the video is very similar to the selection of format to save images. In particular, the CVideoRecorderUtility::GetSupportedVideoTypesL() method will scan the set of available plug-ins to generate the set of supported video types.
| Pro tip:
Since recording video clips capture information about the user and the current user environment, applications that wish to use record video content are required to have the |
Tuner API
See the printed book for this section.
Code
Code for this chapter
Multimedia_code.zip
