[libcamera-devel] [virtio-dev] Re: [RFC PATCH v6] virtio-video: Add virtio video device specification

Fri May 5 14:47:32 CEST 2023

Hi All,

Coming in late, thanks to lei/lore spotting the libcamera keyword.

+ Cc: libcamera-devel to raise awareness of the discussion there.

Quoting Alexander Gordeev (2023-05-05 10:57:29)
> On 03.05.23 17:53, Cornelia Huck wrote:
> > On Wed, May 03 2023, Alex Bennée <alex.bennee at linaro.org> wrote:
> >
> >> Cornelia Huck <cohuck at redhat.com> writes:
> >>
> >>> On Fri, Apr 28 2023, Alexander Gordeev <alexander.gordeev at opensynergy.com> wrote:
> >>>
> >>>> On 27.04.23 15:16, Alexandre Courbot wrote:
> >>>>> But in any case, that's irrelevant to the guest-host interface, and I
> >>>>> think a big part of the disagreement stems from the misconception that
> >>>>> V4L2 absolutely needs to be used on either the guest or the host,
> >>>>> which is absolutely not the case.
> >>>>
> >>>> I understand this, of course. I'm arguing, that it is harder to
> >>>> implement it, get it straight and then maintain it over years. Also it
> >>>> brings limitations, that sometimes can be workarounded in the virtio
> >>>> spec, but this always comes at a cost of decreased readability and
> >>>> increased complexity. Overall it looks clearly as a downgrade compared
> >>>> to virtio-video for our use-case. And I believe it would be the same for
> >>>> every developer, that has to actually implement the spec, not just do
> >>>> the pass through. So if we think of V4L2 UAPI pass through as a
> >>>> compatibility device (which I believe it is), then it is fine to have
> >>>> both and keep improving the virtio-video, including taking the best
> >>>> ideas from the V4L2 and overall using it as a reference to make writing
> >>>> the driver simpler.
> >>>
> >>> Let me jump in here and ask another question:
> >>>
> >>> Imagine that, some years in the future, somebody wants to add a virtio
> >>> device for handling video encoding/decoding to their hypervisor.
> >>>
> >>> Option 1: There are different devices to chose from. How is the person
> >>> implementing this supposed to pick a device? They might have a narrow
> >>> use case, where it is clear which of the devices is the one that needs to
> >>> be supported; but they also might have multiple, diverse use cases, and
> >>> end up needing to implement all of the devices.
> >>>
> >>> Option 2: There is one device with various optional features. The person
> >>> implementing this can start off with a certain subset of features
> >>> depending on their expected use cases, and add to it later, if needed;
> >>> but the upfront complexity might be too high for specialized use cases.
> >>>
> >>> Leaving concrete references to V4L2 out of the picture, we're currently
> >>> trying to decide whether our future will be more like Option 1 or Option
> >>> 2, with their respective trade-offs.
> >>>
> >>> I'm slightly biased towards Option 2; does it look feasible at all, or
> >>> am I missing something essential here? (I had the impression that some
> >>> previous confusion had been cleared up; apologies in advance if I'm
> >>> misrepresenting things.)
> >>>
> >>> I'd really love to see some kind of consensus for 1.3, if at all
> >>> possible :)
> >>
> >> I think feature discovery and extensibility is a key part of the VirtIO
> >> paradigm which is why I find the virtio-v4l approach limiting. By
> >> pegging the device to a Linux API we effectively limit the growth of the
> >> device specification to as fast as the Linux API changes. I'm not fully
> >> immersed in v4l but I don't think it is seeing any additional features
> >> developed for it and its limitations for camera are one of the reasons
> >> stuff is being pushed to userspace in solutions like libcamera:
> >>
> >>    How is libcamera different from V4L2?
> >>
> >>    We see libcamera as a continuation of V4L2. One that can more easily
> >>    handle the recent advances in hardware design. As embedded cameras have
> >>    developed, all of the complexity has been pushed on to the developers.
> >>    With libcamera, all of that complexity is simplified and a single model
> >>    is presented to application developers.
> >
> > Ok, that is interesting; thanks for the information.
> >
> >>
> >> That said its not totally our experience to have virtio devices act as
> >> simple pipes for some higher level protocol. The virtio-gpu spec says
> >> very little about the details of how 3D devices work and simply offers
> >> an opaque pipe to push a (potentially propriety) command stream to the
> >> back end. As far as I'm aware the proposals for Vulkan and Wayland
> >> device support doesn't even offer a feature bit but simply changes the
> >> graphics stream type in the command packets.
> >>
> >> We could just offer a VIRTIO_VIDEO_F_V4L feature bit, document it as
> >> incompatible with other feature bits and make that the baseline
> >> implementation but it's not really in the spirit of what VirtIO is
> >> trying to achieve.
> >
> > I'd not be in favour of an incompatible feature flag,
> > either... extensions are good, but conflicting features is something
> > that I'd like to avoid.
> >
> > So, given that I'd still prefer to have a single device: How well does
> > the proposed virtio-video device map to a Linux driver implementation
> > that hooks into V4L2?
> 
> IMO it hooks into V4L2 pretty well. And I'm going to spend next few
> months making the existing driver fully V4L2 compliant. If this goal
> requires changing the spec, than we still have time to do that. I don't
> expect a lot of problems on this side. There might be problems with
> Android using V4L2 in weird ways. Well, let's see. Anyway, I think all
> of this can be accomplished over time.
> 
> > If the general process flow is compatible and it
> > is mostly a question of wiring the parts together, I think pushing that
> > part of the complexity into the Linux driver is a reasonable
> > trade-off. Being able to use an existing protocol is nice, but if that
> > protocol is not perceived as flexible enough, it is probably not worth
> > encoding it into a spec. (Similar considerations apply to hooking up the
> > device in the hypervisor.)
> 
> I very much agree with these statements. I think this is how it should
> be: we start with a compact but usable device, then add features and
> enable them using feature flags. Eventually we can cover all the
> use-cases of V4L2 unless we decide to have separate devices for them
> (virtio-camera, etc). This would be better in the long term I think.

Camera's definitely have their quirks - mostly because many usecases are
hard to convey over a single Video device node (with the hardware) but I
think we might expect that complexity to be managed by the host, and
probably offer a ready made stream to the guest. Of course how to handle
multiple streams and configuration of the whole pipeline may get more
difficult and warrant a specific 'virtio-camera' ... but I would think
the basics could be covered generically to start with.

It's not clear who's driving this implementation and spec, so I guess
there's more reading to do.

Anyway, I've added Cc libcamera-devel to raise awareness of this topic
to camera list.

I bet Laurent has some stronger opinions on how he'd see camera's exist
in a virtio space.

--
Regards

Kieran

> If everybody wants a single device that much, I think it should be based
> on the current virtio-video. virtio-v4l2 spec can be developed out of
> tree for those, who need 100% compatibility right now. Maybe we can link
> it in the virtio-video, when it is ready?
> 
> > Sorry about asking all those basic questions, but I really rely on the
> > judgment of people familiar with the infrastructure and use cases so
> > that we end up with a specification that is actually usable in the long
> > term. Too many details are likely to confuse me :)
> 
> No worries. :) Thank you!
> 
> --
> Alexander Gordeev
> Senior Software Engineer
> 
> OpenSynergy GmbH
> Rotherstr. 20, 10245 Berlin
> 
> Phone: +49 30 60 98 54 0 - 88
> Fax: +49 (30) 60 98 54 0 - 99
> EMail: alexander.gordeev at opensynergy.com
> 
> www.opensynergy.com
> 
> Handelsregister/Commercial Registry: Amtsgericht Charlottenburg, HRB 108616B
> Geschäftsführer/Managing Director: Régis Adjamah
> 
> Please mind our privacy notice<https://www.opensynergy.com/datenschutzerklaerung/privacy-notice-for-business-partners-pursuant-to-article-13-of-the-general-data-protection-regulation-gdpr/> pursuant to Art. 13 GDPR. // Unsere Hinweise zum Datenschutz gem. Art. 13 DSGVO finden Sie hier.<https://www.opensynergy.com/de/datenschutzerklaerung/datenschutzhinweise-fuer-geschaeftspartner-gem-art-13-dsgvo/>
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: virtio-dev-unsubscribe at lists.oasis-open.org
> For additional commands, e-mail: virtio-dev-help at lists.oasis-open.org
>