Fundamentals of Audio and Video Programming for Games (Pro-Developer)

Chapter 1: Getting Started with DirectSound

Figure 1.1: Standard speaker layout for 5.1 sound.

Figure 1.2: Standard speaker layout for 6.1 sound.

Figure 1.3: Standard speaker layout for 7.1 sound.

Chapter 2: Changing the Volume, Panning, and Frequency of Stereo Sound

Figure 2.1: Main dialog box of the Cacophony tool.

Figure 2.2: Add or Edit Sound dialog box for 2-D sounds.

Chapter 3: Moving Sounds in 3-D Space

Figure 3.1: Rumpus 3-D SFX main dialog box.

Figure 3.2: Doppler effect.

Figure 3.3: Add or Edit Sound dialog box for 3-D sounds.

Figure 3.4: Relationship between minimum distance and volume.

Figure 3.5: Inside and outside sound cones.

Figure 3.6: Listener in a 3-D world.

Chapter 4: Adding Special Effects and Environmental Reverb to 3-D Sounds

Figure 4.1: Special Effects group box.

Figure 4.2: Special effects settings for the Distortion effect.

Figure 4.3: Environmental Reverb dialog box.

Figure 4.4: Change Special Effect Settings dialog box for the Chorus effect.

Chapter 5: Understanding Audio Special Effects

Figure 5.1: Spectrum analysis showing a fundamental tone and its harmonics.

Figure 5.2: Dynamic range of human hearing compared to 16-bit digital audio.

Figure 5.3: Block diagram of wet/dry mix functionality.

Figure 5.4: Block diagram of feedback functionality.

Figure 5.5: Sine wave before and after clipping.

Figure 5.6: Three waveform shapes .

Figure 5.7: Two sine waves in quadrature phase.

Figure 5.8: Peaking equalization showing three different bandwidths at the same center frequency.

Figure 5.9: Signal amplitude before and after compression.

Chapter 6: Streaming Sounds into Circular Buffers

Figure 6.1: Circular Streams dialog box.

Figure 6.2: Three Streams dialog box.

Chapter 7: Driving Hardware with Property Sets

Figure 7.1: Examples of occluded and obstructed sound.

Figure 7.2: Property Values dialog box.

Figure 7.3: Sewer pipe environment.

Chapter 8: Building an Application with the Concertina Framework

Figure 8.1: Concertina framework test dialog box.

Chapter 9: Introducing DirectShow and Video Rendering

Figure 9.1: DirectShow filter graph for AVI file playback.

Figure 9.2: Filter Graph Manager.

Figure 9.3: DirectShow GraphEdit utility.

Figure 9.4: AVI file playback in GraphEdit.

Figure 9.5: AVI file playback using the VMR-9.

Figure 9.6: Video-mixing preferences in the VMR-9.

Chapter 10: Taking Video to the Third Dimension

Figure 10.1: VMR plug-in components .

Figure 10.2: VMR mixing mode.

Figure 10.3: VMR pass-through mode.

Figure 10.4: Two fields of interlaced video.

Figure 10.5: Bob and weave modes.

Chapter 11: Customizing Compositors

Figure 11.1: Source and target rectangles in the video 15 puzzle.

Figure 11.2: Texture and vertex coordinates in the compositor.

Figure 11.3: Calculating the destination rectangle.

Figure 11.4: Texture stages in the AlphaBurst compositor.

Figure 11.5: Dependent read for sepia tone shader.

Chapter 12: MultiMon, ProcAmp, Deinterlacing, and Other Odds and Ends

Figure 12.1: Multi-VMR application.

Figure 12.2: DirectShow filter graph for writing AVI files.

Chapter 13: Producing Content with Technical Quality

Figure 13.1: Steps for using chromakey to layer video.

Figure 13.2: Frames before and after the video was repositioned and cropped.

Figure 13.3: Five steps of the production process.

Figure 13.4: Two examples of camcorders that feature three CCD chips.

Figure 13.5: Example of a portable DAT recorder.

Figure 13.6: Example of a small lavalier microphone.

Figure 13.7: Record dialog box with VU meters that show audio levels.

Figure 13.8: Components of a video-editing program.

Chapter 14: Optimizing Quality Throughout Production

Figure 14.1: Microphone showing a cardioid pickup pattern.

Figure 14.2: Music file opened in a digital audio workstation (DAW).

Figure 14.3: Low-end shelf EQ added with an equalizer.

Figure 14.4: EQ settings in a parametric equalizer.

Figure 14.5: EQ settings in a graphic equalizer.

Figure 14.6: Waveform of a voice before noise reduction is applied.

Figure 14.7: Same waveform of a voice after noise reduction is applied.

Figure 14.8: Sound profile of noise in an audio file.

Figure 14.9: Waveform showing a static pop.

Figure 14.10: Same waveform after pop removal was applied.

Figure 14.11: Waveform zoomed-in to show a transient noise.

Figure 14.12: Same waveform after the pencil tool was used to remove the noise.

Figure 14.13: Waveform showing clipped audio.

Figure 14.14: Same waveform after a restoration process is used.

Figure 14.15: Original waveform of a voice track.

Figure 14.16: Same waveform after a portion was replaced with room tone.

Figure 14.17: Same waveform with a portion muted or silenced.

Figure 14.18: Sample waveform of a track from a video.

Figure 14.19: Original waveform showing a pop.

Figure 14.20: Same waveform after the pop was replaced with room tone.

Figure 14.21: Typical audio compressor plug-in.

Figure 14.22: Waveform of the video track in Figure 14.18 after compression is applied.

Figure 14.23: Video-editor timeline showing the original video track synchronized with the processed track.

Figure 14.24: Background audio track layered with an edited voice track.

Figure 14.25: Using a multitrack program to overlap audio segments.

Figure 14.26: Volume graph added to overlapped audio in a multitrack program.

Figure 14.27: Color-correction plug-in.

Figure 14.28: Software video waveform monitor display.

Figure 14.29: Software video vectorscope display.

Appendix B: Multichannel Audio Tool

Figure B.1: Multi-channel audio tool.

Color Insert

Color Plate 1: Cacaphony tool running the DistantCannons file.

Color Plate 2: Rumpus tool running the CarShop file.

Color Plate 3: To calculate echo delays use the formula: echo delay (ms) = 6 * distance to cliff in meters. file.

Color Plate 4: The natural world provides great inspiration for audio occlusion and obstruction effects.

Color Plate 5: Mixing two video streams in DirectShow with the VMR-9 filter.

Color Plate 6: 3-D mesh rendered on top of a video cut scene.

Color Plate 7: Video texture applied to a 3-D scene.

Color Plate 8: Using custom video mixing to create a 15 puzzle.

Color Plate 9: Two videos blended with an alpha map.

Color Plate 10: Gray scale pixel shader.

Color Plate 11: Sepia tone pixel shader.

Color Plate 12: "Grue" (green-blue) pixel shader.

Color Plate 13: Texture bitmaps used in the Pixellator sample.

Color Plate 14: "Fuzzy dots" pixel shader, using a volumetric texture lookup.

Color Plate 15: ProcAmp controls for brightness, contrast, hue, and saturation.

Color Plate 16: Playing multiple videos in a 3-D scene.