[libcamera-devel] RFC: libcamera pipeline hints

[David Plowman]

Hi Naush

Thanks for bringing us back to this topic, I think it's very important!

On Tue, 30 Mar 2021 at 09:50, Naushir Patuck <naush at raspberrypi.com> wrote:
>
> Hi,
>
> I've talked about this in passing, but thought it might be worth having a dedicated topic to discuss the idea of applications passing "hints" to pipeline handlers.
>
> The idea of hints would be to allow the application some sort of high level control of bits of the pipeline handler logic.  Hints are entirely optional, and may not be handled by the pipeline handler, so an application cannot expect certain behavior based on hints it sets.  Here are some examples of hints that the Raspberry Pi pipeline handler would find useful in its operation:
>
> Fast FPS mode
> This would allow us to explicitly choose a high fps sensor mode, and possibly disable running certain processing algorithms that would limit framerate.  In the past we would also switch to running 3A on 1 out of N frames to reduce the inter-frame processing latency.

So from our early experiences dealing with folks using the Raspberry
Pi libcamera-apps, the most likely kinds of requests are:

- I want a fast framerate mode, or I don't care about framerate
- I want full FOV modes, or I don't care about the FOV
- I want a high bit-depth mode, or I don't care about bit-depth.

I still wonder whether the only solution that will really keep
everyone happy is the one where we give out a list of all the
available modes, and the application can choose one directly. Of
course we could supply a method that chooses one of these modes from
the list according to said "hints", and even have a mode of operation
where the list of modes is never made explicit and the application
merely supplies hints. I guess the latter solution could be a stepping
stone to the others.

>
> Strinct buffer matching
> This would tell the pipeline handler to prioritise matching embedded and image buffers and allow dropping frames in cases where having an embedded data buffer is a requirement (e.g. in specialized sensors delivering CV data).  I had a patch where this was a const bool in our pipeline handler that has not been merged.
>
> AE in RAW mode
> This is one that I am trying to work through now.  Some of our users would like to run AE with only a RAW output to ensure the bayer frame is exposed correctly.  Other users would like to run only  RAW output without ever needing AE at runtime and switching to manual mode.  In the latter case, we can simply avoid running the ISP and IPA at all and ensure we get the highest throughput.  Arguably you could say that the application can set manual controls and we can deduce from that, but controls can be set at runtime, and this would add lots of complexity to this approach.

The idea of swapping between running the ISP for the stats and not
running it sounds quite hair-raising, and I vote to ban it!

David

>
> I'm sure there are going to be other use cases that could make use of hints like these and other vendors that may want them to optimise their pipelines in a similar way.  I would like to hear from folks on what they think about this?  If using hints is an acceptable approach, I could prototype some code to discuss an implementation.
>
> Thanks,
> Naush
>
> _______________________________________________
> libcamera-devel mailing list
> libcamera-devel at lists.libcamera.org
> https://lists.libcamera.org/listinfo/libcamera-devel