Fundamentals of Audio and Video Programming for Games (Pro-Developer)

In previous chapters, we ve used the VMR filter s mixing mode to play several videos at once. An interesting alternative is to use multiple instances of the VMR, each residing in a separate filter graph (see Figure 12.1), but all connected to the same allocator-presenter. This approach gives you full control over the filter graphs, so that you can pause, start, and seek each video independently of the others.

We demonstrate this idea in the Carousel sample located in the AVBook\Video projects directory. This sample renders different videos on the sides of four objects that look vaguely like onions. (This is what happens when you let a programmer use a 3-D modeling application.) Each onion-shaped object can be brought to the foreground by pressing the 1, 2, 3, or 4 key. Pressing the ˜P key pauses or resumes the video in the foreground.

In order to control the video streams independently without disrupting the frame rate of our 3-D scene, the Carousel sample does not draw any frames during the PresentImage method. Instead, it uses the PresentImage method to copy each video frame to a private texture surface. The 3-D rendering happens on the application thread, rather than the filter graph s streaming thread, and therefore the 3-D frame rate is independent of the video frame rate. The application always renders from the private textures, not from the VMR surfaces. (This can only work if all of the VMR filters are in pass-through mode. In mixing mode, the VMR makes BeginScene and EndScene calls to mix the video, and your application will end up fighting with the VMR for the device, as described in Chapter 10.)

Figure 12.1: Multi-VMR application.

In a multi-VMR application, you assign a unique ID number to each VMR instance. Specify this ID when you call IVMRSurfaceAllocatorNotify9::AdviseSurfaceAllocator . Then, when the VMR calls methods on your allocator-presenter, it provides the ID number as the dwUserID parameter. (Up to now, we have ignored this parameter.) The ID gives you a way to match the method invocation with the correct VMR instance.

All of this requires some modifications to our earlier code. To begin with, the allocator-presenter must keep a separate pool of surfaces for each VMR. We therefore define a new class, named VideoSource , which holds an array of IDirect3DSurface9 pointers, along with the private texture that we will use to copy the frames. Each time the application adds a new VMR instance, the allocator-presenter adds a new VideoSource object to an internal list:

HRESULT CAllocator::AttachVMR(DWORD_PTR dwUserID, IUnknown* pVMR) { CLock lock(&m_CritSec); // First, check to see if this ID was already used. VideoSource *pSource = GetVideoSource(dwUserID); if (pSource != NULL) { return E_INVALIDARG; // Duplicate ID. } // Add a new VideoSource object to our list. pSource = new VideoSource(m_pDevice, pVMR, dwUserID); if (!pSource) { return E_OUTOFMEMORY; } m_surfaces.push_back(pSource); return S_OK; }

The application calls AttachVMR when it initializes a new VMR instance, as shown in the following code example. Note that this method replaces the IVMRSurfaceAllocator9::AdviseNotify method.

HRESULT CVmrGame::InitializeVMR() { // Assign the next ID number. DWORD id = m_dwNextId++; // Create the object that manages the filter graph. CVMRGraph *pGraph = new CVMRGraph(); if (!pGraph) { return E_OUTOFMEMORY; } // Create a new VMR and initialize it for renderless mode. CComPtr<IVMRSurfaceAllocatorNotify9> pNotify; HRESULT hr = pGraph->InitVMR_RenderlessMode(&pNotify); // Hook up the VMR and the allocator-presenter. HMONITOR hMonitor = g_d3d->GetAdapterMonitor(D3DADAPTER_DEFAULT); hr = pNotify->SetD3DDevice(m_pDevice, hMonitor); hr = pNotify->AdviseSurfaceAllocator(id, m_pAlloc); hr = m_pAlloc->AttachVMR(id, pNotify); // New method! // Add the CVMRGraph object to an STL map, indexed by ID. m_graphs.insert(GraphMap::value_type(id, pGraph)); return hr; }

In the new version, many of the allocator-presenter methods simply call corresponding methods on the VideoSource object. For example, here is the new version of InitalizeDevice .

STDMETHODIMP CAllocator::InitializeDevice(DWORD_PTR dwUserID, VMR9AllocationInfo* pAllocInfo, DWORD* pNumBuffers) { CLock lock(&m_CritSec); // Look up the VideoSource object by the ID. VideoSource *pSource = GetVideoSource(dwUserID); if (!pSource) { return E_INVALIDARG; } // Create the surfaces. return pSource->CreateSurfaces(pAllocInfo, *pNumBuffers); }

The GetSurface and TerminateDevice methods follow the same pattern. In each case, the corresponding method on the VideoSource class contains most of the code that previously was in the allocator-presenter class. For example, the CreateBuffer method creates the surfaces for the VMR, along with the private texture.

HRESULT VideoSource::CreateBuffers(VMR9AllocationInfo* pAlloc, DWORD cBuffers) { // Allocate an array for the surface pointers. ReleaseSurfaces(); m_pSurf = new IDirect3DSurface9* [cBuffers]; if (m_pSurf == NULL) { return E_OUTOFMEMORY; } // Create offscreen surfaces. pAlloc->dwFlags = VMR9AllocFlag_OffscreenSurface; HRESULT hr = m_pNotify->AllocateSurfaceHelper(pAlloc, &cBuffers, m_pSurf); if (FAILED(hr)) { return hr; } m_dwNumSurfaces = cBuffers; // Create the texture. It must be a render target. AdjustTextureSize(pAlloc); pAlloc->dwFlags = VMR9AllocFlag_TextureSurface VMR9AllocFlag_3DRenderTarget; cBuffers = 1; CComPtr<IDirect3DSurface9> pTexSurf; hr = m_pNotify->AllocateSurfaceHelper(pAlloc, &cBuffers, &pTexSurf); if (FAILED(hr)) { return hr; } pTexSurf->GetContainer(IID_IDirect3DTexture9, (void**)&m_pTexture); ScaleVideoTexture(); FixDepthBuffer(pAllocInfo); return hr; }

The texture is created with the D3DUSAGE_DYNAMIC flag, so that it can be modified. The PresentImage method copies the video frame to the texture.

STDMETHODIMP CAllocator::PresentImage( DWORD_PTR dwUserID, VMR9PresentationInfo* pPresInfo) { CLock lock(&m_CritSec); VideoSource* pSurf = GetVideoSource(dwUserID); if (!pSurf) { return E_INVALIDARG; } // Copy the video surface to the texture. CComPtr<IDirect3DSurface9> pTexSurf; pSurf->GetTexture()->GetSurfaceLevel(0, &pTexSurf); m_pDevice->StretchRect(pPresInfo->lpSurf, NULL, pTexSurf, NULL, D3DTEXF_NONE); return S_OK; }

Because the PresentImage method no longer draws the scene, we must provide a way for the application to get the textures from the allocator-presenter. We have added a method named GetTexture for this purpose. However, it s crucial that the allocator-presenter not modify any textures while the application is using them. Therefore, we also add a pair of methods named LockTextures and UnlockTextures . The first method holds the allocator-presenter s critical section, and the second releases it.

IDirect3DTexture9* CAllocator::GetTexture(DWORD_PTR dwUserID) { VideoSource *pSource = GetVideoSource(dwUserID); if (!pSource) { return NULL; } return pSource->GetTexture(); } void CAllocator::LockTextures() { m_CritSec.Enter(); } void CAllocator::UnlockTextures() { m_CritSec.Leave(); }

Rendering the scene is now simply a matter of calling LockTextures to hold the critical section, calling GetTexture for each VMR instance, and assigning the textures to the four meshes. When we re done, we call UnlockTextures to release the critical section.

HRESULT CVmrGame::Render() { m_Device.Clear(D3DCOLOR_XRGB(0x00, 0x80, 0x80)); HRESULT hr = m_pDevice->BeginScene(); if (FAILED(hr)) { return hr; } // Lock the allocator-presenter textures. m_pAlloc->LockTextures(); // Iterate through the filter graphs and get the texture for // each VMR. Assign the texture to a mesh. GraphMap::iterator iter = m_graphs.begin(); inti=0; for (; iter != m_graphs.end(); iter++, i++) { DWORD id = iter->first; IDirect3DTexture9 *pTex = m_pAlloc->GetTexture(iter->first); CMesh* p1 = m_pScene->GetMesh(i); p1->SetTexture(1, pTex); } // Now draw the scene and unlock the textures. m_pScene->Render(m_pDevice); m_pDevice->EndScene(); m_pAlloc->UnlockTextures(); hr = m_pDevice->Present(NULL, NULL, NULL, NULL); return hr; }

You can see that we ve been very careful throughout about holding critical sections on any public allocator-presenter methods. This is crucial to avoid race conditions.