[libcamera-devel] "Software ISP" -- where to put it, neccessary hooks

[Laurent Pinchart]

On Fri, Jul 07, 2023 at 11:40:53AM +0100, Kieran Bingham wrote:
> Quoting Laurent Pinchart (2023-07-07 11:29:38)
> > On Wed, Jul 05, 2023 at 06:05:55PM +0100, Kieran Bingham wrote:
> > > Quoting Pavel Machek via libcamera-devel (2023-06-30 14:25:18)
> > > > Hi!
> > > > 
> > > > I'm looking for guidance. I have software-only auto-exposure code, but
> > > > it is in wrong places. Code is at
> > > > 
> > > > https://gitlab.com/tui/libcamera/-/tree/hacks-ae?ref_type=heads
> > > > 
> > > > I tried to move the code to src/ipa/libipa/simple_ae.cpp , but I could
> > > > not get the result to link if I did that.
> > > > 
> > > > Currently code is called from SimpleCameraData::bufferReady , is that
> > > > okay thing to do?
> > > 
> > > So the tricky thing here is that you are trying to call libipa from a
> > > pipeline handler - that's not quite right.
> > > 
> > > So we likely need to determine if the simple-pipeline handler should
> > > (optionally?) instantiate an IPA module which can then incorporate the
> > > software ISP implementation. I expect this would only instantiate and
> > > use the IPA when a RAW sensor is connected.
> > 
> > That's what I would recommend too. We should have a soft ISP component
> > for the pipeline handler to instantiate, with a corresponding IPA
> > module. The interface of the soft ISP should be generic, defined as an
> > abstract class with pure virtual functions, with a derived class that
> > implements a CPU-based soft ISP. Later, we'll develop a GPU-based soft
> > ISP as a second implementation of the interface, and the simple pipeline
> > handler will be able to use either.
> > 
> > > I suspect we would have a new IPA under src/ipa/softisp/ which would do
> > > the implementation of 3a using a CPU/GPU depending on resource
> > > availability.
> > 
> > The code in src/ipa/softisp/ should only control the ISP algorithms, not
> > implement the CPU-based image processing. Ideally that IPA module would
> > be used with either the CPU- and GPU-based implementations on the
> > pipeline handler side.
> 
> ah yes - good point - there should be the soft-isp component that the
> pipeline handler instantiates, which would be src/libcamera/softisp/ ?

That's a good location for now, we can always reshuffle things later.

> with src/ipa/softisp/ as the algorithm code to correspond to that and
> can make decisions based on the processing.
> 
> So there should be a new component under src/libcamera/softisp/ whic
> maps the buffers with MappedFrameBuffer and allows the direct image
> processing to occur. That's the starting point I believe where your
> existing code would live, and that would be possible to link directly
> with the simple pipeline hander as it could include
> "libcamera/internal/softisp.h"
> 
> > > From there any helpers that might be shared across IPA modules could
> > > live in libipa, but anything that is specific to the new 'softisp' would
> > > be under src/ipa/softisp/ ... so you would have
> > > src/ipa/softisp/algorithms/cpu_ae.cpp perhaps.
> > > 
> > > > My divideUpExposure code has origins in rapsberry pi code; I could not
> > > > find a way to share it nicely so I cut&pasted. If there's a better way
> > > > (or better code to reuse), I'd like to know.
> > > 
> > > I believe code reuse from the RPi code is fine, but you would need to
> > > ensure that the new code file matches the RPi IPA licence which is BSD I
> > > believe.

-- 
Regards,

Laurent Pinchart