[libcamera-devel] Software gathering of image statistics for libcamera on Librem 5

[Jacopo Mondi]

Hi Pavel

On Fri, Dec 09, 2022 at 02:29:44PM +0100, Pavel Machek wrote:
> Hi!
>
> > > Due to small ammount of computation, this is not really job for GPU.
> > >
> > > Thanks for rkisp pointer. I'm quite confused by that code, not knowing
> > > much about rkisp. Does it have some special support in hardware, like
> > > providing one stream of frames for application and second for
> > > statistics gathering?
> >
> > Not really through frame streams but through parameters buffers. ISPs
> > do -not-produce frames (well, they do, but not from a user point of
> > view) which will still be captured through a video node, but are
> > rather subdevices with at least (in the simplest use case)
> >
> > 1) one input video input stream (from camera sensor/CSI receiver)
> > 2) one output video stream (to DMA engines)
> > 3) one parameters input
> > 4) one statistics output
> >
> > Through 1) you queue RAW buffers which will be crunched to produce
> > statistics available from 4) and apply on the frames transformation
> > according to the parameters set through 3). The transformed frames
> > will be sent to the memory and captured from 2).
> >
> > ISP do indeed perform transformations and corrections on the images
> > (debayer, awb etc) but more importantly produce statistics.
>
> Ok, thanks for description.
>
> > The IPA module you're looking at handles the closed phase loop  that
> > upon receiving statistic from the ISP, compute the parameters for next
> > frame using statistics and control values from the sensor.
>
> IPA == "image processing algorithms", right?
>

Yes! It's a libcamera term, sorry for giving it for granted.

> > > For Librem 5 (and possibly PinePhone and similar), I'd like to snoop
> > > on frames provided to the application, and set up exposure
> > > accordingly. But I don't see any code in libcamera accessing image
> > > data.
> >
> > We don't. All statistics are computed by the ISP, as we never really
> > considered software based implementation to be actually usable in
> > practice and you keep getting back mentions of GPU because it's not
> > only about statistics but also about performing computationally
> > intensive transformations on the image buffers, such as debayering
> > raw frames into formats consumable by users.
> >
> > Are you working with YUV or RAW sensors ?
>
> RAW sensors are the important ones.
>

Ack

> > > Closest I could get is Image class (src/apps/common/image.h) used by
> > > file sinks in cam application. Would it be acceptable to move that
> > > into libcamera and use it to peek at the pixels?
> >
> > I here assume you're using the simple pipeline handler. If you want a
> > component that implements AEGC you need to receive as input the frame
> > data and generate as output the sensor's control values for
> > CID_EXPOSURE and CID_ANALOGUE_GAIN. I would model that as a soft-IPA
> > module that receives the dmabuf of a just-completed frame, maps it,
> > computes statistics and use them to generate control values for the
> > sensor.
>
> AEGC == "auto exposure gain contrast"?
>

Yes

> Anyway, I'm a bit of worried about mapping the frames once in
> libcamera and second time in application. Syscalls are probably fast
> enough but it is kind of ugly.
>

You certainly pay a price for the double mapping. I'm not sure how it
compares to the time it take for the actual computations though.


> > However, if you use RAW frames, either you do SW debayering which is
> > rather intensive, or you can only produce RAW frames. If you use YUV
>
> SW debayering for preview is not a problem. You need to scale image
> down to fit the screen at the same time, so... you can do simple
> thing (R, G+G/2, B).
>
> > Also the requirement that for each frame you have to compute stats and
> > generate values will stall the pipeline and reduce frame rate. There
> > are several hacks you could attempt, like computing stats on every x
> > frames, but it really sounds like a proof of concept at best.
>
> Umm, better hack would be to compute stats with only part of the
> buffer (and only take every n-th pixel into account). That needs to be
> done, anyway, when photographer asks for "spot metering". I'm
> confident good-enough statistics can be done in 10% CPU... and we have
> 4 CPUs available.
>
> > All of this for AEGC, which produces values to be applied on the
> > -sensor-. What about AWB ? Are you doing software AWB too ? How
> > are you realizing it, by changing what parameters ?
>
> Yes, millipixels is doing software AWB. I'd not call it important (and
> don't know details), but it is "fast enough". Code is here:
>
> https://source.puri.sm/Librem5/millipixels/-/blob/master/quickpreview.c

I do there see the colorspace conversion and a software debayering
implementation. I guess one can plumb AWB there by adjusting the color
space transfer function matrices coefficients  or by piping some color
gain adjustment routine after that.

>
> > I actually suggest to consider GPU, but seeing a pure software
> > implementation running would be nice for a comparison. However, I
> > would be hesitant in having anything like that upstream for the above
> > mentioned reasons.
>
> Well, there's no other good place to put camera support :-(. We should
> not really AE/AF support in each application, and it would not be
> acceptable in kernel.

That's a good point.

I'm not sure yet how we could model that if not with a module that
acts a soft-IPA, but that means double mapping, in the library and in
the application at display time (unless your application is smart and
does 0 copy rendering)

>
> Best regards,
> 								Pavel
>
> --
> People of Russia, stop Putin before his war on Ukraine escalates.


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <https://lists.libcamera.org/pipermail/libcamera-devel/attachments/20221209/41a785c0/attachment.sig>