[libcamera-devel] Software gathering of image statistics for libcamera on Librem 5

[Kieran Bingham]

Hi Pavel,

Quoting Pavel Machek (2022-12-09 10:55:38)
> Hi!
> 
> > > I'm playing with camera on Librem 5. So far I added af/ae/awb support
> > > to millipixels done completely in software, but I guess libcamera
> > > would be more suitable place for it. If you have some hints/example
> > > where to look at, it would be nice.
> > 
> > The area you are looking at is essentially doing the role of a software
> > based ISP processor. It would be a component that a pipeline handler
> > could choose to instantiate, to support any algorithms that use the data
> > you produce. It can be difficult to produce 'generic' data that any
> > > IPA
> 
> I don't need full ISP. I just need to gather enough statistics.
> 
> > But how that would fit in the libcamera design would need further
> > consideration ... falling back to software ISP ... should really be
> > avoided ... but perhaps there's benefits for early bring up - or simply
> > being able to manage a simple way of getting AWB/AE for raw streams,
> > without an ISP (as we have on the RKISP1), so I think it has some
> > value.
> 
> ..to do AWB/AE/AF. If I wanted to, I'd only need to access 200 pixels
> or so.
> 
> > Ideally - we shouldn't rely on the CPU where possible though, and that's
> > why several people have been looking at a GPU/OpenGL based option. I
> > could imagine a software fallback would be a part of that pairing of
> > SoftwareISP component.
> 
> Due to small ammount of computation, this is not really job for GPU.
> 
> Thanks for rkisp pointer. I'm quite confused by that code, not knowing
> much about rkisp. Does it have some special support in hardware, like
> providing one stream of frames for application and second for
> statistics gathering?

Yes. But when capturing RAW frames, it doesn't have the ability to
provide statistics, so that means for RAW capture - we can't have a 3a
loop running, and can only run with manual parameters.


> For Librem 5 (and possibly PinePhone and similar), I'd like to snoop
> on frames provided to the application, and set up exposure
> accordingly. But I don't see any code in libcamera accessing image
> data.
> 
> Closest I could get is Image class (src/apps/common/image.h) used by
> file sinks in cam application. Would it be acceptable to move that
> into libcamera and use it to peek at the pixels?

You can peek at pixels with the MappedFrameBuffer class.

 include/libcamera/internal/mapped_framebuffer.h
 src/libcamera/mapped_framebuffer.cpp

If we only need 200 pixels to process, I'd be very happy to see a CPU
based implementation that could be used on 'any' platform without an
ISP.


 
> Best regards,
>                                                                 Pavel
> -- 
> People of Russia, stop Putin before his war on Ukraine escalates.