[libcamera-devel] RFC: libcamera pipeline hints

[Naushir Patuck]

Hi David,

On Wed, 31 Mar 2021 at 12:17, David Plowman <david.plowman at raspberrypi.com>
wrote:

> Hi Naush
>
> Thanks for bringing us back to this topic, I think it's very important!
>
> On Tue, 30 Mar 2021 at 09:50, Naushir Patuck <naush at raspberrypi.com>
> wrote:
> >
> > Hi,
> >
> > I've talked about this in passing, but thought it might be worth having
> a dedicated topic to discuss the idea of applications passing "hints" to
> pipeline handlers.
> >
> > The idea of hints would be to allow the application some sort of high
> level control of bits of the pipeline handler logic.  Hints are entirely
> optional, and may not be handled by the pipeline handler, so an application
> cannot expect certain behavior based on hints it sets.  Here are some
> examples of hints that the Raspberry Pi pipeline handler would find useful
> in its operation:
> >
> > Fast FPS mode
> > This would allow us to explicitly choose a high fps sensor mode, and
> possibly disable running certain processing algorithms that would limit
> framerate.  In the past we would also switch to running 3A on 1 out of N
> frames to reduce the inter-frame processing latency.
>
> So from our early experiences dealing with folks using the Raspberry
> Pi libcamera-apps, the most likely kinds of requests are:
>
> - I want a fast framerate mode, or I don't care about framerate
> - I want full FOV modes, or I don't care about the FOV
> - I want a high bit-depth mode, or I don't care about bit-depth.
>
> I still wonder whether the only solution that will really keep
> everyone happy is the one where we give out a list of all the
> available modes, and the application can choose one directly. Of
> course we could supply a method that chooses one of these modes from
> the list according to said "hints", and even have a mode of operation
> where the list of modes is never made explicit and the application
> merely supplies hints. I guess the latter solution could be a stepping
> stone to the others.
>

Yes, I agree, this is a painful one.  I'll expand my thoughts on this a bit.
The hints should not necessarily be used as a mechanism for manual
mode selection.  Although they could - e.g. for a fast fps hint, we would
likely want to choose the fast fps sensor mode, likewise with a full FoV
hint.  This fast fps hint would also instruct the pipeline handler to
possibly
run the IPA control algorithms at a lower rate than the sensor rate to
reduce latencies and chance for frame drops.

A separate mechanism for manual mode selection is definitely needed
for our users, but that should come under a separate topic.


>
> >
> > Strinct buffer matching
> > This would tell the pipeline handler to prioritise matching embedded and
> image buffers and allow dropping frames in cases where having an embedded
> data buffer is a requirement (e.g. in specialized sensors delivering CV
> data).  I had a patch where this was a const bool in our pipeline handler
> that has not been merged.
> >
> > AE in RAW mode
> > This is one that I am trying to work through now.  Some of our users
> would like to run AE with only a RAW output to ensure the bayer frame is
> exposed correctly.  Other users would like to run only  RAW output without
> ever needing AE at runtime and switching to manual mode.  In the latter
> case, we can simply avoid running the ISP and IPA at all and ensure we get
> the highest throughput.  Arguably you could say that the application can
> set manual controls and we can deduce from that, but controls can be set at
> runtime, and this would add lots of complexity to this approach.
>
> The idea of swapping between running the ISP for the stats and not
> running it sounds quite hair-raising, and I vote to ban it!
>

Yes, fully agree! Having a hint available before we start streaming makes
this
switch much easier to handle in the pipeline handler.


>
> David
>
> >
> > I'm sure there are going to be other use cases that could make use of
> hints like these and other vendors that may want them to optimise their
> pipelines in a similar way.  I would like to hear from folks on what they
> think about this?  If using hints is an acceptable approach, I could
> prototype some code to discuss an implementation.
> >
> > Thanks,
> > Naush
> >
> > _______________________________________________
> > libcamera-devel mailing list
> > libcamera-devel at lists.libcamera.org
> > https://lists.libcamera.org/listinfo/libcamera-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.libcamera.org/pipermail/libcamera-devel/attachments/20210331/3d2d0740/attachment.htm>