Fundamentals of Audio and Video Programming for Games (Pro-Developer)

Overview

In the past few years , truly incredible advances have been made in 3-D graphics. The advances in digital video on the computer have been no less impressive. It was not so long ago that a jerky, thumbnail- sized image represented the state of the art in computer video. Today full-motion, high-resolution video is commonplace, and consumers can watch television, DVDs, and DV video on relatively low-end machines.

In many ways, 3-D graphics and video are complementary technologies. With 3-D graphics, you can create imaginary worlds that never existed, and explore and interact with them. The tricky part is making it look realistic, and that s where the ingenuity of programmers and artists comes in. Video, although it is two-dimensional and is not inherently interactive, captures the real world in all of its complicated and beautiful detail.

Yet, inside the computer, these two technologies have more in common than it might appear. Consider how a 3-D object is rendered. The object begins as a set of points (vertices), which are mathematically transformed into 2-D screen coordinates. The object s surface details ” for example, the pattern of bricks on a building ” are stored in a bitmap image (a texture), which in effect is pasted onto the sides of the object.

Now consider how a video is rendered on a computer screen. A certain area of the screen is designated to display the video. This might be a rectangle inside a window, or the entire screen. In any case, the area is defined by a set of screen coordinates that form a destination rectangle . Each video frame is a bitmap that gets stretched to fit the destination rectangle. Starting to sound familiar? These are the very same steps used to draw the 3-D object. Of course, there are important differences. For example, the destination rectangle for video is usually defined in screen coordinates, rather than using transformed 3-D vertices. Also, in video, the bitmap is updated many times a second, whereas most 3-D models use static textures. But the final step when the image gets rendered is the same in both cases.

In fact, for many years DirectX has been the underlying technology used to render video in Windows. Unfortunately, for various historical reasons, it was difficult to combine video seamlessly with 3-D graphics. That changed with DirectX 9.0. Now DirectShow uses the latest Direct3D APIs to render video, and 3-D graphics applications can take full advantage of these capabilities. In this section of this book, you ll learn how to do that. You ll draw 3-D objects on top of video cut scenes, draw video on the sides of 3-D objects, create sliding tile puzzles with full-motion video, and generally push the boundaries of what video can do. Our motto is, Why be flat?

In this chapter, we begin with a quick introduction to the DirectShow architecture, which is necessary for understanding the material that follows . However, we won t delve too deeply into the inner workings of DirectShow ” there is plenty of SDK documentation that you can read for that. Also in this chapter, we explore some of the powerful video-mixing capabilities that are provided in DirectX 9.0.

In Chapter 10, we put DirectShow to work in a Direct3D sample application. Before you read that section, you should have a basic understanding of Direct3D programming. You should know how to create a Direct3D device; enable lighting and texturing; set the world, view, and projection transforms; and draw 3-D primitives and meshes. If you have worked through the tutorials in the Direct3D SDK, you ll be able to follow this material with no problem.

In Chapter 11, we show how to create some interesting video-mixing effects. For example, we ll show how to create circular regions of transparency in a video image, using alpha maps. Chapter 12 is the odd and ends chapter, and includes miscellaneous topics that didn t fit anywhere else.

Our focus in these chapters is video playback. DirectShow is a large SDK, and some of the features that we will not cover include video capture, DVD navigation, MPEG-2, and television tuners, among others. If you d like a more general introduction to everything that DirectShow offers, an excellent place to start is Programming Microsoft DirectShow for Digital Video and Television, by Mark Pesce (Microsoft Press).