Video Decoding using pyNvVideoCodec

The video batch decoder is responsible for reading an MP4 video as tensors. The actual decoding is done per frame using NVIDIA’s PyNvVideoCodec API. The video decoder is generic enough to be used across the sample applications. The code associated with this class can be found in the samples/common/python/nvcodec_utils.py file.

There are two classes responsible for the decoding work:

1. VideoBatchDecoder and

2. nvVideoDecoder

The first class acts as a wrapper on the second class which allows us to:

1. Stay consistent with the API of other decoders used throughout CVCUDA

2. Support batch decoding.

3. Use accelerated ops in CVCUDA to perform the necessary color conversion from NV12 to RGB after decoding the video.

VideoBatchDecoder

Let’s get started by understanding how this class is initialized in its __init__ method. We use PyNvDemuxer to read a few properties of the video. The decoder instance and CVCUDA color conversion tensors both are allocated when needed upon the first use.

Note: Due to the nature of NV12, representing it directly as a CVCUDA tensor is a bit challenging. Be sure to read through the explanation in the comments of the code shown below to understand more.

self.logger = logging.getLogger(__name__)
self.input_path = input_path
self.batch_size = batch_size
self.device_id = device_id
self.cuda_ctx = cuda_ctx
self.cuda_stream = cuda_stream
self.cvcuda_perf = cvcuda_perf
self.total_decoded = 0
self.batch_idx = 0
self.decoder = None
self.cvcuda_RGBtensor_batch = None
nvDemux = nvvc.PyNvDemuxer(self.input_path)
self.fps = nvDemux.FrameRate()
self.logger.info("Using PyNvVideoCodec decoder version: %s" % nvvc.__version__)

Once things are defined and initialized, we would start the decoding when a call to the __call__ function is made.

def __call__(self):
    self.cvcuda_perf.push_range("decoder.pyVideoCodec")

    # docs_tag: begin_alloc_videobatchdecoder_pyvideocodec
    # Check if we need to allocate the decoder for its first use.
    if self.decoder is None:
        self.decoder = nvVideoDecoder(
            self.input_path, self.device_id, self.cuda_ctx, self.cuda_stream
        )

Next, we call the nvdecoder instance to actually do the decoding and stack the image tensors up to form a 4D tensor.

# Get the NHWC YUV tensor from the decoder
cvcuda_YUVtensor = self.decoder.get_next_frames(self.batch_size)

# Check if we are done decoding
if cvcuda_YUVtensor is None:
    self.cvcuda_perf.pop_range()
    return None

# Check the code for the color conversion based in the pixel format
cvcuda_code = pixel_format_to_cvcuda_code.get(self.decoder.pixelFormat)
if cvcuda_code is None:
    raise ValueError(f"Unsupported pixel format: {self.decoder.pixelFormat}")

# Check layout to make sure it is what we expected
if cvcuda_YUVtensor.layout != "NHWC":
    raise ValueError("Unexpected tensor layout, NHWC expected.")

# this may be different than batch size since last frames may not be a multiple of batch size
actual_batch_size = cvcuda_YUVtensor.shape[0]

Once the video batch is ready, we use CVCUDA’s cvtcolor_into function to convert its data from NV12 format to RGB format. We will use pre-allocated tensors to do the color conversion to avoid allocating same tensors on every batch.

# Create a CVCUDA tensor for color conversion YUV->RGB
# Allocate only for the first time or for the last batch.
if not self.cvcuda_RGBtensor_batch or actual_batch_size != self.batch_size:
    self.cvcuda_RGBtensor_batch = cvcuda.Tensor(
        (actual_batch_size, self.decoder.h, self.decoder.w, 3),
        nvcv.Type.U8,
        nvcv.TensorLayout.NHWC,
    )

# Convert from YUV to RGB. Conversion code is based on the pixel format.
cvcuda.cvtcolor_into(self.cvcuda_RGBtensor_batch, cvcuda_YUVtensor, cvcuda_code)

self.total_decoded += actual_batch_size

The final step is to pack all of this data into a special CVCUDA samples object called as Batch. The Batch object helps us keep track of the data associated with the batch, the index of the batch and optionally any filename information one wants to attach (i.e. which files did the data come from).

nvVideoDecoder

This is a class offering hardware accelerated video decoding functionality using pyNvVideoCodec. It reads an MP4 video file, decodes it and returns a CUDA accessible Tensor per frame. Please consult the documentation of the pyNvVideoCodec to learn more about its capabilities and APIs.

For use in CVCUDA, this class defines the following functions which decode data to a tensor in a given CUDA stream.

class nvVideoDecoder:
    def __init__(self, enc_file, device_id, cuda_ctx, stream):
        """
        Create instance of HW-accelerated video decoder.
        :param enc_file: Full path to the MP4 file that needs to be decoded.
        :param device_id: id of video card which will be used for decoding & processing.
        :param cuda_ctx: A cuda context object.
        """
        self.device_id = device_id
        self.cuda_ctx = cuda_ctx
        self.input_path = enc_file
        self.stream = stream
        # Demuxer is instantiated only to collect required information about
        # certain video file properties.
        self.nvDemux = nvvc.PyNvDemuxer(self.input_path)
        self.nvDec = nvvc.CreateDecoder(
            gpuid=0,
            codec=self.nvDemux.GetNvCodecId(),
            cudacontext=self.cuda_ctx.handle,
            cudastream=self.stream.handle,
            enableasyncallocations=False,
        )

        self.w, self.h = self.nvDemux.Width(), self.nvDemux.Height()
        self.pixelFormat = self.nvDec.GetPixelFormat()
        # In case sample aspect ratio isn't 1:1 we will re-scale the decoded
        # frame to maintain uniform 1:1 ratio across the pipeline.
        sar = 8.0 / 9.0
        self.fixed_h = self.h
        self.fixed_w = int(self.w * sar)

    # frame iterator
    def generate_decoded_frames(self):
        for packet in self.nvDemux:
            for decodedFrame in self.nvDec.Decode(packet):
                nvcvTensor = nvcv.as_tensor(
                    nvcv.as_image(decodedFrame.nvcv_image(), nvcv.Format.U8)
                )
                if nvcvTensor.layout == "NCHW":
                    # This will re-format the NCHW tensor to a NHWC tensor which will create
                    # a copy in the CUDA device decoded frame will go out of scope and the
                    # backing memory will be available by the decoder.
                    yield cvcuda.reformat(nvcvTensor, "NHWC")
                else:
                    raise ValueError("Unexpected tensor layout, NCHW expected.")

    def get_next_frames(self, N):
        decoded_frames = list(itertools.islice(self.generate_decoded_frames(), N))
        if len(decoded_frames) == 0:
            return None
        elif len(decoded_frames) == 1:  # this case we dont need stack the tensor
            return decoded_frames[0]
        else:
            # convert from list of tensors to a single tensor (NHWC)
            tensorNHWC = cvcuda.stack(decoded_frames)
            return tensorNHWC