[libcamera-devel] RFC: libcamera pipeline hints

[Naushir Patuck]

Hi,

I've talked about this in passing, but thought it might be worth having a
dedicated topic to discuss the idea of applications passing "hints" to
pipeline handlers.

The idea of hints would be to allow the application some sort of high level
control of bits of the pipeline handler logic.  Hints are entirely
optional, and may not be handled by the pipeline handler, so an application
cannot expect certain behavior based on hints it sets.  Here are some
examples of hints that the Raspberry Pi pipeline handler would find useful
in its operation:

Fast FPS mode
This would allow us to explicitly choose a high fps sensor mode, and
possibly disable running certain processing algorithms that would limit
framerate.  In the past we would also switch to running 3A on 1 out of N
frames to reduce the inter-frame processing latency.

Strinct buffer matching
This would tell the pipeline handler to prioritise matching embedded and
image buffers and allow dropping frames in cases where having an embedded
data buffer is a requirement (e.g. in specialized sensors delivering CV
data).  I had a patch where this was a const bool in our pipeline handler
that has not been merged.

AE in RAW mode
This is one that I am trying to work through now.  Some of our users would
like to run AE with only a RAW output to ensure the bayer frame is
exposed correctly.  Other users would like to run only  RAW output without
ever needing AE at runtime and switching to manual mode.  In the latter
case, we can simply avoid running the ISP and IPA at all and ensure we get
the highest throughput.  Arguably you could say that the application can
set manual controls and we can deduce from that, but controls can be set at
runtime, and this would add lots of complexity to this approach.

I'm sure there are going to be other use cases that could make use of hints
like these and other vendors that may want them to optimise their pipelines
in a similar way.  I would like to hear from folks on what they think about
this?  If using hints is an acceptable approach, I could prototype some
code to discuss an implementation.

Thanks,
Naush
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.libcamera.org/pipermail/libcamera-devel/attachments/20210330/9e1feb15/attachment.htm>