Fundamentals of Audio and Video Programming for Games (Pro-Developer)

In this chapter, we will explore methods and tools that you can use to optimize technical quality as you produce audio and video content. We will describe what we mean by technical quality, and then look at how to optimize it in the context of the production process. Although there are specific methods that you can use at different points in the process to improve technical quality, it s best to factor quality into every decision that you make.

The two main parts of the production process are production and postproduction. Production takes place when you record, shoot, or capture content, while postproduction occurs when you edit and distribute content. In this chapter, we will describe the production process in general. In the next chapter, we will follow the production process in detail from beginning to end, and describe how to optimize technical quality at each step.

Understanding Technical Quality

There are many different ways to measure the quality of audio and video content. For example, you can measure the artistic or creative quality of a game by how well it was designed. Technical quality, however, is concerned with the noncreative aspects of a piece of content, for example, the color quality of the video or the presence of distortion in the audio. Although these qualities describe very different aspects of a video work, they are closely related . For example, the quality of an audio recording can affect the story that you want to tell, and the artistic decision to shoot in black and white will affect technical aspects like the video format and lighting.

Technical quality is a very important aspect of an audio or video work because it carries your artistic vision. There is no rule stating that all colors must be bright and realistic, and all sounds must be clear and audible. Except for limits and standards imposed by the medium itself, such as file size , technical quality is under your control. If the technical quality of a piece is as good as it can be, it is completely transparent: only the creative qualities come through. You are the best judge of the quality of your content and are ultimately responsible for your artistic vision.

The reason for making this point is that often content creators feel bound by some arbitrary quality standard. A book about how to make movies tells them never to use this lens, or always use that kind of microphone. One of the joys of making videos and music comes with discovering that technical quality is not dictated by a camera, or a book, or an industry professional .

You are the one in charge of the technical quality of your production. For example, many consumer camcorders come with features that automatically control the light level entering the lens, so many consumers feel that the light levels that the camcorder sets must be right. The truth is that automatic settings are only right if they are right for you . You should not allow a camcorder to dictate quality just because it was expensive and has many intimidating controls. If you look through the lens and do not see what you want to see, the camcorder is wrong, regardless of how technologically advanced the automatic circuitry may be.

Optimizing is often called cleaning . For example, you might clean narration tracks by removing unrelated sounds like dogs barking and electronic pops, or use a clean section of video that doesn t show the camera person s thumb over the lens. The best way to determine what needs to be cleaned, and how to do it, is to put yourself in the place of the audience. You might be able to hear the narration fine over the sound of a dog barking, but what will the audience think when they listen to the narration in your game about the Civil War? They might think you are trying to make an artistic statement with the dog-barking sound, and the distraction could lead them to make a fatal mistake on the battlefield.

The term cleaning is not used as often when referring to video. The types of technical quality optimizations that you are most likely to do with images relate to lighting, color, and problems with equipment and physical media.

Now, let s look at what professional producers , creative people, and technicians think about when determining acceptable technical quality.

Audio

You approach the use of sound in the same way, whether you are creating a movie, video, or game, because people perceive sounds the same regardless of how the sounds come to them.

Recorded sound as used in video production falls into one of the following categories:

• Specific sound, such as the closing of a door or a passing car. Specific sounds are also called sync sounds when they are synchronized to specific events in the video.

• Background sound, such as residential neighborhood ambience (dogs barking, cars passing, birds chirping, and kids playing). The individual sounds that make up a background are not synchronized to anything in the picture, so a background can be looped or repeated.

• Music, which either originates from within a scene (source music) or from outside the scene (underscore). Examples of source music are sound from radios or marching bands, from sources that are seen in the video. Underscore can be added to enhance the emotional content of a scene.

• Dialog, which can be one or more voices synchronized to people in the picture, or a voice or voices of people who are not seen in the shot, such as narration (voice-over). A clean dialog or narration track is very important, because it is typically the most important sound and carries the meaning of a scene.

A perfectly clean audio track is one that contains only the sound(s) that you want, and the sound itself has the quality that you expect. For example, a narration track contains a good-quality recording of the narrator s voice with no distortion, no reverberation , and no barking dogs. If you want underscore music along with the voice, you can add it during the postproduction process with a multitrack editor. A multitrack editor enables you to control the placement and mix of the sounds by enabling you to work with modules or clips of sound layered on tracks. For example, you can mix sync dialog with a train station background in a multitrack editor, and it will sound like the person is in a train station.

If an audio track is not clean, you will have trouble editing and controlling the quality of the final product. For example, if you must use a narration track that includes music, you will not be able to edit the narration without also editing the music. You must be concerned with how an edit will affect the sound of both the dialog and the music. An example of this is attempting to edit video of a speech that includes a radio playing in the background. Whenever you edit the voice, the music on the radio will jump.

Unwanted, nonmusical sound in a recording is called noise. The same sound can be noise in one case and necessary in another. Audio noise can be introduced into a recording either externally or internally:

• External noise is unwanted sound that is picked up by the microphone, from sources such as air conditioners, elevator motors, refrigerators, traffic, surf, talking, wind, and music.

• Internal noise is unwanted sound that is introduced after a sound has been received by a microphone and converted to an electrical signal. This very- low-voltage audio signal is susceptible to interference from a number of sources, such as the microphone itself, cables carrying the signal, electronic components used to amplify or process the signal, the device used to convert the signal to a digital stream, or the software audio processors. Internal noise can also be generated by an external source, such as the electromagnetic radiation from power lines and transformers . Noise includes hum, hiss, pops, clicks, and distortion from electronic components and recording tape. Distortion of the audio signal is most often caused by overmodulation, when electronic components, digital processors, or recording tape receive a signal that is too high for them to effectively process.

Once noise is recorded and mixed with the desired audio, it can rarely be eliminated. It might be possible to reduce noise with software or mask it with a background sound, but the process also affects the desired sound. The best way to ensure noise-free audio is to make sure that noise is not recorded.

Video

Recorded video images are not categorized the same as audio because they are technically very different from audio, and play a different role in our perception. In audio production, you typically record elements discretely and then mix them together to create a finished piece. In video production, on the other hand, you most often do not think in terms of mixing multiple discrete elements because the process is more time-consuming and difficult. In general, photographers shoot complete images. It is typically easier to record or shoot an actor in a real location than it is to shoot the actor and background separately, and composite them later. Compositing is becoming more popular with inexpensive digital systems, but it will be a while before it is used as often as multitrack mixing is used with audio.

As with audio, once an image is recorded on tape or film, elements of the image are locked in place. You can use editing to cut before or after a shot that you want, but it is not easy to add to or remove parts of the actual shot. For example, if a shot of a speaker contains an unwanted airplane flying behind the person s head in the background, you cannot simply remove an airplane layer. True, there are digital video-production systems that enable you to track shapes and perform advanced compositing techniques, with which you could clean up the image. However, these systems are outside the scope of this book, and the processes are often time-consuming and expensive.

For the most part, optimizing video technical quality is limited to working with the color properties. You can do this in several ways. You can change the overall color properties of an image or create composite images using colors. If the image is digital, you can also use effects processes that change the 3-D positioning and shape of images. For example, you can enlarge and reposition an image to remove unwanted video along the edges of the frame.

The best time to optimize quality is during production when you are shooting the video. For example, it is far easier to do something to ensure that the camera does not record an airplane flying overhead than to attempt to remove it in postproduction. After the image has been recorded, you typically optimize quality during postproduction with a digital video editor, such as Avid Media Composer and Adobe Premiere. An editor enables you to apply processors to a video track and to composite images.

As with audio, problems with the technical quality of video are introduced either externally or internally. External technical problems are introduced when images are recorded, and because cameras record light, the problems are typically a result of poor lighting. Poor lighting can sometimes be corrected electronically or digitally in postproduction. There are roughly three types of poor lighting:

• Overall level of lighting is too high or too low. If it is too low, the level of lighting can be raised, however, that also increases the internal noise. If the level of lighting is too high, it can be lowered . As with audio, however, if the level is so high that it has clipped the video, those areas that appear washed out cannot be restored.

• Contrast ratio is too high. If the ratio between the dark and light areas of the image is too great, the detail in the dark areas is lost. If you increase the overall lighting level to bring out the detail, the light areas become clipped. To help correct the problem, the contrast can be reduced.

• Overall color is off. Typically, color problems are introduced by light that has a different color temperature from the type of film that is used or the white balance setting on the video camera. Color temperature describes the color of a light source. Incandescent light is more reddish than sunlight, so a subject that was lit with incandescent light and shot with a camera that was balanced for sunlight will appear very red. Consumer video cameras automatically compensate, but not always in the best way. For example, a camcorder might incorrectly compensate when a scene is lit by both sunlight and incandescent, or worse , adjust the color as you move the camera. In any event, overall color problems can be corrected in postproduction.

Other types of external problems, such as unwanted objects in a shot or bad camera angles, are often thought of as artistic problems. However, if the problem distracts from the central story, it can become a technical problem, and repairing the problem using electronics and software might not produce the best results.

Internal technical problems are introduced after images enter the camera lens. Unlike analog video, video that is stored digitally does not deteriorate when it is transferred or copied , so many of the internal technical problems of the old analog days are now history. For example, we no longer have to be concerned with inconsistent color caused by a consumer tape format, such as VHS. Any internal technical problems typically arise inside the camcorder before the video is converted to a digital stream. For example, problems can come from the lens, the CCD chips, the analog processors and amplifiers , and the analog to digital converter. The following list describes categories of technical problems:

• Noise. Problems appear as small flashing sparkles throughout the image. Video noise is created by electronic components that amplify or process the analog video. To minimize noise, use high-quality video hardware and minimize the number of analog components in the video-processing chain.

• Low resolution. Problems arise when detail is missing and an image appears out of focus, or soft . To maintain resolution, you should eliminate unnecessary video-processing steps. For example, save resolution by transferring video digitally from a tape to a computer, instead of capturing through analog composite video outputs and inputs. By transferring digitally, you bypass two converters, a number of amplifiers and processors, and a cable and connectors, all of which can lower resolution and add noise.

• Glitches and drop-outs. These are quick flashes that are usually restricted to one or two lines of a video frame. They are caused by imperfections in the videotape or some other momentary loss of signal.

• Digital artifacts. These consist of various types of aberrations caused by problems with a computer or digital device, or a codec, the most common of which include blockiness and dropped frames . Blockiness occurs when a codec is unable to reproduce the needed detail, and blocks appear in the image.

• Clipping. This problem occurs when the brightest areas of an image exceed the upper threshold of the video components, resulting in a loss of detail and a washed-out appearance. However, unlike audio, a clipped video image might not be overmodulated. Often video processors include circuitry that electronically clips or prevents the signal from going over the upper limits defined by the standard. Therefore, an image can contain clipped areas without adversely affecting the recording or transmission. In shooting high-contrast scenes, you may need to clip areas if there is no way to avoid it. For example, when shooting a shaded subject against a bright background, the best solution is to increase the light level on the subject. However, if this is not possible, you can adjust the exposure for the subject and allow the background to be slightly clipped.

Internal problems are often difficult to repair because they cannot be fixed by adjusting overall color properties. With advances in camcorder technology, tape and noise problems are becoming rarer.

The following list describes some of the basic ways to work with technical quality in postproduction:

• Editing. The easiest way to maintain technical quality is to simply edit around the problem areas. For example, you can use a different take of a reading. If the audio portion is useable but the video is not, you can insert video of another shot.

• Color correction. Next to editing, the easiest way to adjust technical quality is with color correction. The methods available for correcting color depend on whether the video is digital data or an analog signal and, if analog, at what point the video signal is in the processing chain. Most often, you will work with the video as digital data, which gives you a great deal of control over many details of the image. With an analog signal, you have more control early in the process when the video is in its red, green, and blue (RGB) components, but after the video has been processed into a composite signal, you have less control. Processors and filters enable you to correct the overall color of images by modifying the following video properties:

  • Luminance or brightness. This property describes video intensity. In composite video, luminance is the black-and-white portion of the analog signal. When you adjust luminance, the image gets brighter or darker . If you turn luminance all the way off, the image is black.

  • Chrominance or saturation. This property describes color intensity. In composite video, chrominance is the color portion of the analog signal. When you adjust chrominance, colors become more or less intense . If you turn chrominance all the way off, the image is black and white.

  • Hue or phase. This property describes the overall hue of the colors. In composite video, the phase or timing relationship between the image portion of the chrominance signal and a reference that is called a color burst determines the hue. As you adjust hue, the colors change to red, green, blue, and all the shades in between.

  • Setup. This property describes the black level. In composite video, setup is the lowest or darkest part of the black-and-white portion of the analog signal. When you increase setup, the black portions become lighter. Setup does not apply to digital video.

  • Gamma. This property describes the mid-tone intensity. Gamma adjustments are usually made to digital and component video. Typically, gamma is used to adjust the contrast between the light and dark areas of an image. If you change gamma, the mid-tone shades become lighter or darker. You can also adjust the gamma of red, green, and blue individually.

  • RGB. This property describes the intensity of the red, green, and blue components, which can only be adjusted if the video is analog component or digital. If you adjust the three components equally, the overall video intensity changes.

• Compositing or layering. There are basically two ways to layer video: superimposing and keying. Superimposing is simply placing one image on top of another so they both appear together. Keying is a technique that creates transparent areas of an image based on a color or luminance level. The four frames in Figure 13.1 demonstrate a simple use of chromakey .

Figure 13.1: Steps for using chromakey to layer video.

The background of the first image is green. The chromakey process is applied to the image, and the keyer is carefully adjusted to create a transparent mask, where every instance of that shade of green is found, as in the second frame. Then the masked image is composited over the third frame, resulting in the fourth frame. Using these techniques, you could cut out a bad portion of one image by creating a mask and replacing it with something else.

• Positioning and cropping. Positioning tools enable you to move the video frame to any location and position in three dimensions. For example, you can make a frame smaller and position it in a corner, and then add perspective so it appears that the frame is facing to the side. Cropping tools, which also only work with digital video, enable you to remove portions of the top, bottom, or sides of a frame. You can use cropping to eliminate portions or change the aspect ratio of a frame. The two images in Figure 13.2 show a frame before and after it was enlarged slightly and cropped.

Figure 13.2: Frames before and after the video was repositioned and cropped.