Using Gestures to Control Tablet Media Player

 Download CD Content

Overview

If you recall from Chapter 16, Creating an MP3 Player, we created an MP3 player that allows the user to write out a command that is recognized and a certain behavior is executed, such as playing or stopping a file. Although it obviously works, it is definitely not an easy solution from a user standpoint, as the need to write the word 'play' to play a file is very tedious. In this chapter, we look at a much better way to control these types of applications by using gestures to control a media player (see Figure 18.1).

Figure 18.1: Built-in gestures from Table 18.1, viewed from left to right.

Tablet Gestures

If you have ever used a PalmPilot, you have seen the way a device can take advantage of a type of shorthand called graffiti. The Palm works by allowing the user to write a simple version of complex characters-making entering text an easier objective. The Tablet PC provides a much deeper implementation of a shorthand system known as gestures.

The gestures are available as part of the standard Tablet PC Platform SDK. When you develop an application, the gestures are not enabled by default. Rather, if you want to enable one of the built-in gestures, you must instruct the application that you are doing so. Most gestures have a 'hot point,' which is a distinguishing point in the geometry of the gesture. The hot point can be used to determine where the gesture was performed. The gestures APIs provide the HotPoint property of the Gesture event, making it possible to determine the hot point for a given gesture. Not all gestures have a specific hot point; for these gestures, the starting point is reported as the hot point. Using both Figure 18.1 and Table 18.1, you can see an example of the gestures and the location of the hot point.

Building the Media Player

Microsoft provides the Media Player ActiveX control as a simple way to offer media player capabilities. We are going to take advantage of this control as we build an application that can open various movie files and that is controlled by gestures.

To begin this application, create a new Windows Forms application and add an OpenFileDialog to it. You can leave its name as the default given. In the middle of the form, add an InkPicture control, and in the upper right of the form, add a separate one (should look like Figure 18.2) leaving their names as InkPicture1 and InkPicture2.

Figure 18.2: Position the controls on the form.

Now, let's add the media player control to the form. Choose Project | Add Reference to display the Add Reference dialog box. Next, choose the COM tab and then locate the Windows Media Player OCX control. You can position the media player control on the form as seen in Figure 18.3. This control should approximately cover the InkPlayer control we placed on the form earlier. We're going to use the InkPlayer control to allow us to annotate on the movie.

Figure 18.3: The Media Player OCX positioned on the form.

The next step for this project is to add a Label control and position it above the InkPicture2 control. Change its Text property to read 'Gesture'. The final thing we need is a StatusBar control added to the form. When it's added, it automatically positions itself along the bottom of the form, which is where the status bar is normally placed.

Writing Some Code

Before we get to most of the code, we need to add a reference to the Microsoft Ink API. Therefore, we need to add the following line to the top of the code:

Imports Microsoft.Ink

We'll use the Form_Load event to set several properties very important to our application. First, set the InkPicture1 control to have a transparent background color, remove its border, and then bring it to the front so that it is available for annotating over the video. Here are the three lines of code:

InkPicture1.BackColor = Color.Transparent InkPicture1.BorderStyle = BorderStyle.None InkPicture1.BringToFront()

Now, let's turn our attention to the other InkPicture control. This one will be used to handle the gestures for our application, which will control the features such as opening, playing, and pausing a video. First, we'll set its borderstyle and backcolor so that it separates itself from the rest of the interface and is very visible to the user. As we are interested in using the control to capture gestures only, we are going to set its Collection mode to GestureOnly, and then we'll set the gestures that it will try to recognize. Our application is going to use the following gestures:

Square: Stop

UpDown: Pause

ChevronRight: Play

ArrowRight: Volume increase

ArrowLeft: Volume decrease

Circle: Open

Here is the code:

InkPicture2.BorderStyle = BorderStyle.Fixed3D InkPicture2.BackColor = Color.White InkPicture2.CollectionMode = CollectionMode.GestureOnly InkPicture2.SetGestureStatus(ApplicationGesture.Square, True) InkPicture2.SetGestureStatus(ApplicationGesture.UpDown, True) InkPicture2.SetGestureStatus(ApplicationGesture.ChevronRight, True) InkPicture2.SetGestureStatus(ApplicationGesture.ArrowRight, True) InkPicture2.SetGestureStatus(ApplicationGesture.ArrowLeft, True) InkPicture2.SetGestureStatus(ApplicationGesture.Circle, True)

The last two things we need to handle are the status bar's Text property, which we set to 'No file loaded.', and the filter for the open dialog box so that we can only open MPEG and AVI videos:

OpenFileDialog1.Filter = "MPEG Video (*.mpg)|*.mpg|Video for Windows (*.avi)|*.avi|All Files (*.*)|*.*" StatusBar1.Text = "No file loaded."

Now that we have our gestures, we can use the InkPicture2_Gesture event to determine which gesture was written and then perform the appropriate actions. We'll use an If...Then statement at the beginning of the procedure to check the current gesture's confidence. We can test to see if the recognition confidence is strong, and if so, we will use a Case statement to determine which type of gesture was written and instruct the AxMediaPlayer control appropriately. If the gesture confidence is less than strong, we will exit without performing any activities with the AxMediaPlayer. Before we exit, we set the status bar's Text property to 'Command Not Recognized' so that the user realizes that the command they attempted did not perform any function.

We have already discussed the gestures and what each of them will do, so here is the code:

If e.Gestures(0).Confidence = RecognitionConfidence.Strong Then Select Case e.Gestures(0).Id Case ApplicationGesture.Square AxMediaPlayer1.Stop() StatusBar1.Text = "Stop" Case ApplicationGesture.ChevronRight AxMediaPlayer1.Play() StatusBar1.Text = "Play" Case ApplicationGesture.UpDown AxMediaPlayer1.Pause() StatusBar1.Text = "Pause" Case ApplicationGesture.ArrowLeft AxMediaPlayer1.Volume = AxMediaPlayer1.Volume - 1 StatusBar1.Text = "<- Vol = " & AxMediaPlayer1.Volume.ToString Case ApplicationGesture.Circle OpenFileDialog1.ShowDialog() AxMediaPlayer1.Open(OpenFileDialog1.FileName) Case ApplicationGesture.ArrowRight AxMediaPlayer1.Volume = AxMediaPlayer1.Volume + 1 StatusBar1.Text = "-> Vol = " & AxMediaPlayer1.Volume.ToString StatusBar1.Text = AxMediaPlayer1.FileName End Select Else StatusBar1.Text = "Command Not Recognized" End If

The application is functional at this time, but we have a few additional things we need to handle. First, we need to change the drawing color for both InkPicture controls, depending on which control the pen is entering. We'll use both MouseEnter events for this aspect of the application:

Private Sub InkPicture2_MouseEnter(ByVal sender As Object, ByVal e As System.EventArgs) Handles InkPicture2.MouseEnter InkPicture2.DefaultDrawingAttributes.Color = Color.Red End Sub Private Sub InkPicture1_MouseEnter(ByVal sender As Object, ByVal e As System.EventArgs) Handles InkPicture1.MouseEnter InkPicture1.DefaultDrawingAttributes.Color = Color.Blue End Sub

The final thing we need to add to the application is a way to pause the AxMediaPlayer currently playing file so that we can annotate a given frame. We can use the MouseDown event procedure for InkPicture1 to test the current filename. If there is a filename, we then test to see if the media player is currently playing. If it is, we pause the player so that we can proceed to annotate it. Lastly, we set the status bar's Text property to 'Paused for annotation.'

Here is the code:

Private Sub InkPicture1_MouseDown(ByVal sender As Object, ByVal e As System.Windows.Forms.MouseEventArgs) Handles InkPicture1.MouseDown If axMediaPlayer1.FileName "" and AxMediaPlayer1.PlayState = MPPlayStateConstants.mpPlaying Then AxMediaPlayer1.Pause() StatusBar1.Text = "Paused for annotation." End If End Sub

You can test the application to see if it works. Figure 18.4 displays an open gesture with an OpenFileDialog and Figure 18.5 shows a video being annotated.

Figure 18.4: A file being opened.

Figure 18.5: A video file being annotated.

Summary

In this chapter, we created a video player that was controlled using gestures. The application also used an additional InkPicture control to allow us to annotate over the content of the video. In Chapter 19, Getting Started with Microsoft Agent, we build our first speech-enabled application.