media/v4l/dev-decoder.rst

1 .. SPDX-License-Identifier: GPL-2.0
6 Memory-to-Memory Stateful Video Decoder Interface
9 A stateful video decoder takes complete chunks of the bytestream (e.g. Annex-B
34 5. Single-planar API (see :ref:`planar-apis`) and applicable structures may be
35    used interchangeably with multi-planar API, unless specified otherwise,
36    depending on decoder capabilities and following the general V4L2 guidelines.
41 7. Given an ``OUTPUT`` buffer A, then A' represents a buffer on the ``CAPTURE``
44 .. _decoder-glossary:
79    Good at sub-partitioning the picture into variable sized structures.
103    Instantaneous Decoder Refresh; a type of a keyframe in an H.264/HEVC-encoded
108    can be decoded fully on its own.
111    a processing unit in image and video compression formats based on linear
112    block transforms (e.g. H.264, VP8, VP9); codec-specific, but for most of
129 resume point
132    SPS/PPS/IDR sequence (H.264/HEVC); a resume point is required to start decode
133    of a new stream, or to resume decoding after a seek.
153    additional (non-visual) information contained inside encoded bytestream;
170 .. kernel-render:: DOT
186        qi -> Initialization [ label = "open()" ];
188        Initialization -> CaptureSetup [ label = "CAPTURE\nformat\nestablished" ];
190        CaptureSetup -> Stopped [ label = "CAPTURE\nbuffers\nready" ];
192        Decoding -> ResChange [ label = "Stream\nresolution\nchange" ];
193        Decoding -> Drain [ label = "V4L2_DEC_CMD_STOP" ];
194        Decoding -> EoS [ label = "EoS mark\nin the stream" ];
195        Decoding -> Seek [ label = "VIDIOC_STREAMOFF(OUTPUT)" ];
196        Decoding -> Stopped [ label = "VIDIOC_STREAMOFF(CAPTURE)" ];
197        Decoding -> Decoding;
199        ResChange -> CaptureSetup [ label = "CAPTURE\nformat\nestablished" ];
200        ResChange -> Seek [ label = "VIDIOC_STREAMOFF(OUTPUT)" ];
202        EoS -> Drain [ label = "Implicit\ndrain" ];
204        Drain -> Stopped [ label = "All CAPTURE\nbuffers dequeued\nor\nVIDIOC_STREAMOFF(CAPTURE)" ];
205        Drain -> Seek [ label = "VIDIOC_STREAMOFF(OUTPUT)" ];
207        Seek -> Decoding [ label = "VIDIOC_STREAMON(OUTPUT)" ];
208        Seek -> Initialization [ label = "VIDIOC_REQBUFS(OUTPUT, 0)" ];
210        Stopped -> Decoding [ label = "V4L2_DEC_CMD_START\nor\nVIDIOC_STREAMON(CAPTURE)" ];
211        Stopped -> Seek [ label = "VIDIOC_STREAMOFF(OUTPUT)" ];
218    client may call :c:func:`VIDIOC_ENUM_FMT` on ``OUTPUT``.
221      format set on ``CAPTURE``.
224      In particular whether or not the decoder has a full-fledged bytestream
228    :c:func:`VIDIOC_ENUM_FMT` on ``CAPTURE``.
230    * Only the formats supported for the format currently active on ``OUTPUT``
234      the client must first set that coded format on ``OUTPUT`` and then
235      enumerate formats on ``CAPTURE``.
247      decoder for given raw pixel format and the coded format currently set on
250 4. Supported profiles and levels for the coded format currently set on
257 1. Set the coded format on ``OUTPUT`` via :c:func:`VIDIOC_S_FMT`.
286      resolution instantly based on the width and height returned by
298       and the client must ensure it matches its needs afterwards.
300 2.  Allocate source (bytestream) buffers via :c:func:`VIDIOC_REQBUFS` on
325     Alternatively, :c:func:`VIDIOC_CREATE_BUFS` on the ``OUTPUT`` queue can be
353 3.  Start streaming on the ``OUTPUT`` queue via :c:func:`VIDIOC_STREAMON`.
371     * If the client has not set the coded resolution of the stream on its own,
373       :c:func:`VIDIOC_TRY_FMT` or :c:func:`VIDIOC_REQBUFS` on the ``CAPTURE``
386        A client capable of acquiring stream parameters from the bytestream on
388        to non-zero values matching the coded size of the stream, skip this step
390        rely on any driver queries regarding stream parameters, such as
405 1.  Call :c:func:`VIDIOC_G_FMT` on the ``CAPTURE`` queue to get format for the
450           returned by :c:func:`VIDIOC_G_FMT` on ``CAPTURE``.
452     * The following selection targets are supported on ``CAPTURE``:
464           ``CAPTURE``; defaults to ``V4L2_SEL_TGT_CROP_DEFAULT``; read-only on
478           read-only on hardware without additional compose/scaling capabilities.
488        successfully parses the stream metadata. The client must not rely on the
491 3.  **Optional.** Enumerate ``CAPTURE`` formats via :c:func:`VIDIOC_ENUM_FMT` on
517 4.  **Optional.** Set the ``CAPTURE`` format via :c:func:`VIDIOC_S_FMT` on the
534    * Setting the ``CAPTURE`` format will reset the compose selection rectangles
535      to their default values, based on the new resolution, as described in the
538 5. **Optional.** Set the compose rectangle via :c:func:`VIDIOC_S_SELECTION` on
553          read-only on hardware without additional compose/scaling capabilities.
559          returned by :c:func:`VIDIOC_G_FMT` on ``CAPTURE``.
564       supported one to meet codec and hardware requirements. The client needs
567 6.  If all the following conditions are met, the client may resume the decoding
578     In that case, the remaining steps do not apply and the client may resume
585       on the ``CAPTURE`` queue.
592     buffers on the ``CAPTURE`` queue until a buffer marked with the
596     on the ``CAPTURE`` queue to stop streaming.
601        :c:func:`VIDIOC_STREAMOFF` on it would abort the sequence and trigger a
618 10. Allocate ``CAPTURE`` buffers via :c:func:`VIDIOC_REQBUFS` on the
651     Alternatively, :c:func:`VIDIOC_CREATE_BUFS` on the ``CAPTURE`` queue can be
697 11. Call :c:func:`VIDIOC_STREAMON` on the ``CAPTURE`` queue to start decoding
708 The content of the source ``OUTPUT`` buffers depends on the active coded pixel
709 format and may be affected by codec-specific extended controls, as stated in
713 buffer queues and memory-to-memory devices. In addition, the order of decoded
723   on ``CAPTURE`` (e.g. if it does not contain encoded data, or if only
727   on ``CAPTURE`` (if the encoded data contained more than one frame, or if
731 * a buffer queued to ``OUTPUT`` may result in a buffer being produced on
736 * buffers may become available on the ``CAPTURE`` queue without additional
759      buffers are out-of-order compared to the ``OUTPUT`` buffers): ``CAPTURE``
788   Change` sequence needs to be followed,
790 * if a ``V4L2_EVENT_EOS`` event is pending, the `End of Stream` sequence needs
797 of details depending on the decoder capabilities. Specifically:
807 further operations on corresponding decoder file handle will return the -EIO
809 alternatively reinitialize the instance by stopping streaming on both queues,
816 The seek does not require any specific operation on the ``CAPTURE`` queue, but
841    queue until a suitable resume point is found.
846       from a resume point (e.g. SPS or a keyframe). Any queued ``OUTPUT``
848       resume point is found.  While looking for a resume point, the decoder
851       Some hardware is known to mishandle seeks to a non-resume point. Such an
853       being made available on the ``CAPTURE`` queue. Drivers must ensure that
871 4. After a resume point is found, the decoder will start returning ``CAPTURE``
894    produced.  For example, given the sequence of operations on the
899    any of the following results on the ``CAPTURE`` queue is allowed:
910    To achieve instantaneous seek, the client may restart streaming on the
935 * bit-depth of the bitstream has been changed.
964        with ``V4L2_BUF_FLAG_LAST`` will result in a -EPIPE error from
977        :c:func:`VIDIOC_STREAMOFF` and :c:func:`VIDIOC_STREAMON` on the
986    streaming. Calling :c:func:`VIDIOC_STREAMOFF` on the ``OUTPUT`` queue would
1047         marked with ``V4L2_BUF_FLAG_LAST`` will result in a -EPIPE error from
1059       dequeued. It is a deprecated behavior and the client must not rely on it.
1067    * ``V4L2_DEC_CMD_START`` - the decoder will not be reset and will resume
1070    * a pair of :c:func:`VIDIOC_STREAMOFF` and :c:func:`VIDIOC_STREAMON` on the
1071      ``CAPTURE`` queue - the decoder will resume the operation normally,
1075    * a pair of :c:func:`VIDIOC_STREAMOFF` and :c:func:`VIDIOC_STREAMON` on the
1076      ``OUTPUT`` queue - any pending source buffers will be returned to the
1081    Once the drain sequence is initiated, the client needs to drive it to
1083    issuing :c:func:`VIDIOC_STREAMOFF` on any of the ``OUTPUT`` or ``CAPTURE``
1086    will fail with -EBUSY error code if attempted.
1104 1. Setting the format on the ``OUTPUT`` queue may change the set of formats
1105    supported/advertised on the ``CAPTURE`` queue. In particular, it also means
1106    that the ``CAPTURE`` format may be reset and the client must not rely on the
1109 2. Enumerating formats on the ``CAPTURE`` queue always returns only formats
1112 3. Setting the format on the ``CAPTURE`` queue does not change the list of
1113    formats available on the ``OUTPUT`` queue. An attempt to set a ``CAPTURE``
1118 4. Enumerating formats on the ``OUTPUT`` queue always returns the full set of
1121 5. While buffers are allocated on any of the ``OUTPUT`` or ``CAPTURE`` queues,
1122    the client must not change the format on the ``OUTPUT`` queue. Drivers will
1123    return the -EBUSY error code for any such format change attempt.