[PATCH v9 RESEND 0/1] Virtio video device specification

[Alexander Gordeev]

Hi,

Update 05.06.2024:
I'm resending the patch because now the virtio mailing lists seem to be
functional again. This time without the RFC mark. I think it is already
good enough for merging. I'm working on updating the implementation. If
there are any minor changes necessary, we still have a lot of time
before the next release to fix them.

This is the 9th version of virtio-video device patch. There are some
major changes and now I think it is good enough conceptually, but may
need a few minor changes, when the driver gets updated. Sorry, it took
way longer, than I expected. There are a lot of details to care about
and also a lot of distractions. Please review.

Changelog v8 -> v9:
1. Chenged representation of device capabilities/parameters to
Type-Length-Value (TLV).
TLV representation is already used in the kernel, see [1][2]. It is
mentioned in the virtio sound device specification. It is also
compatible with struct v4l2_ext_control, that is used to set V4L2
controls in batches. Here I'm trying to extend the approach onto many
other things: device capabilities, formats, buffers/resources, crop
settings. This way it is possible to get all the device capabilities in
one call, then set all the parameters/controls in one call. For us this
is important due to significant hypercall delays. I hope this might be
applicable in the V4L2 core as well in the future, so I'm very
interested in comments from linux-media community.

2. Changed back the representation of capabilities to pre v7 model.
Video decoders and encoders are basically converters between
multidimensional shapes of all supported encoded formats and all
supported raw formats. This was roughly the way the capabilities were
represented in v1-v6 (excluding the controls), then in v7 and v8 I
tried to do it closer to the V4L2 discovery process. Now I think this
was a mistake, the previously used method may look more complex, but it
fits better. This approach is similar to the way the capabilities are
described in gstreamer (they don't seem to have explicit links between
the sets though).

3. Replaced the internal device's per stream resource queues with
virtqueues.
The idea briefly: we had to avoid blocking the descriptors for a long
time, so I implemented delayed responses in v8. Then there is no
natural limit for the number of elements in the resource queues
anymore. What to do if the queue becomes full? So there are a bunch of
things to define in the spec. I decided to replace these queues with
virtqueues, because the virtqueues are already defined well enough.
There are of course some upsides and downsides. I think upsides are
overwhelming and I'd be happy to discuss this if anyone is interested.

4. Renamed commandq to mainq since commands can be queued to per stream
queues as well.

5. Renamed STREAM_CREATE -> STREAM_OPEN, STREAM_DESTROY -> STREAM_CLOSE.
Now the per stream virtqueues are opened rather than created.

6. Enabled RESET on the output queue too.
I didn't know how to handle a VIDIOC_STREAMOFF on the capture queue
coming from guest user-space otherwise.

7. Merged ATTACH_BACKING command into SET_PARAMS.
I felt, that it really belongs to the parameters alongside the
num_resources, it allows the driver to see if the resources are detached
after a SET_PARAMS or not, also might be helpful during debugging. Still
there is an issue: SG lists can be quite large. If it is large and the
results are returned on the eventq, then this significantly increases
the eventq's descriptor sizes. Avoided this by adding a requirement for
the device to cap the TLV length to 8.

8. Added DRM format modifiers.
9. Added FWHT codec.
10. Addressed review comments.
11. The spec size went down from 24 pages to 19 pages.

Full PDF: https://drive.google.com/file/d/1gj-kOBp1uvCB5IzrtOTjf-zunP3f-EFq/view?usp=sharing
PDF with the video section only: https://drive.google.com/file/d/1fH8XaSYkwkb2beB9thS7FOvFysUM22Q5/view?usp=sharing
Please note, that external links are broken in the short version.

[1] https://docs.kernel.org/userspace-api/netlink/intro.html?highlight=tlv
[2] https://docs.kernel.org/sound/kernel-api/writing-an-alsa-driver.html?highlight=tlv
[3] https://lore.kernel.org/linux-media/20230922100303.GF19112@pendragon.ideasonboard.com/

Alexander Gordeev (1):
  virtio-video: Add virtio video device specification

 conformance.tex                           |    4 +
 content.tex                               |    1 +
 device-types/video/description.tex        | 1607 +++++++++++++++++++++
 device-types/video/device-conformance.tex |   22 +
 device-types/video/driver-conformance.tex |   20 +
 introduction.tex                          |   21 +
 6 files changed, 1675 insertions(+)
 create mode 100644 device-types/video/description.tex
 create mode 100644 device-types/video/device-conformance.tex
 create mode 100644 device-types/video/driver-conformance.tex

-- 
2.40.1