[libcamera-devel] Application requesting statistics buffers

[Naushir Patuck]

Hi Laurent,

On Mon, 29 Jun 2020 at 00:40, Laurent Pinchart
<laurent.pinchart at ideasonboard.com> wrote:
>
> Hi Naush,
>
> On Fri, Jun 26, 2020 at 12:56:21PM +0100, Naushir Patuck wrote:
> > Hi,
> >
> > As part of the work (just submitted for review) on providing a focus
> > FoM to the application through libcamera controls, one of the options
> > I explored was to allow the application to explicitly request the ISP
> > statistics buffer along with the image frame through the usual Request
> > mechanism.  The Raspberry Pi pipeline handler is written with enough
> > flexibility to allow this to happen.  However, one thing missing is a
> > way for the application to request (and provide) the statistics buffer
> > as an explicit format.
>
> You know you ask very good and very annoying questions, right ? :-)

Thanks for the feedback :)

>
> > In the v4l2 world, we have a specific format
> > (V4L2_META_FMT_BCM2835_ISP_STATS) for the ISP statistics buffer
> > defined.  In order to create a libcamera::format (from formats.yaml),
> > it seems I must have an equivalent DRM format defined.  However, from
> > what I can tell, DRM formats are not defined for metadata types, only
> > image data, is that correct?  Would it be reasonable for me to define
> > an equivalent DRM_FORMAT_BCM2835_ISP_STATS for this purpose?
>
> I wonder if we shouldn't use an entirely different mechanism here. The
> PixelFormat class, as its name implies, describes pixel formats, and is
> ill-fitted for this purpose. Same for StreamConfiguration. I would
> rather create a property that reports the statistics format (as it is
> immutable), and extending the Request class to add a statistics buffer
> explicitly. We'll have to figure out how to handle that in
> FrameBufferAllocator and most probably in other parts of the stack.
>
> Maybe this would create too much complexity and interfacing with the
> existing streams mechanism would be best, but I think we need to
> evaluate all options before deciding which one to pick.

Yes, we could use a different mechanism here, and that would
complicate the APIs.  But it does provide a nice separation between
pixel and metadata formats.  I am happy to prototype some things out
if you need help.

>
> > This may seem a somewhat isolated use case, but I can see that some
> > specific applications could want to do their own analysis of the
> > statistics data for their purposes.  In this example, a user wanted to
> > create a microscope application...
>
> Would the user also have a need for embedded data ?

I see no reason why not.  Right now, it does not serve a purpose (for
Raspberry Pi at least) since we extract the important bits from the
embedded data and convert them to libcamera::controls to return as
Metadata.  However, there could be some sensors that provide different
things that the application wants, and may not be suitable for
libcamera controls.

Regards,
Naush

>
> > Right now I will not worry about the other problem with this where the
> > application will have to provide a stats buffer on every Request,
> > similar to the problem with RAW captures - but a zero-copy mechanism
> > is in the works from what I can tell :)
> >
> > Let me know your thoughts on how we can achieve this.
>
> --
> Regards,
>
> Laurent Pinchart