[libcamera-devel] "Software ISP" -- where to put it, neccessary hooks

[Kieran Bingham]

Quoting Laurent Pinchart (2023-07-07 11:29:38)
> Hello,
> 
> On Wed, Jul 05, 2023 at 06:05:55PM +0100, Kieran Bingham wrote:
> > Quoting Pavel Machek via libcamera-devel (2023-06-30 14:25:18)
> > > Hi!
> > > 
> > > I'm looking for guidance. I have software-only auto-exposure code, but
> > > it is in wrong places. Code is at
> > > 
> > > https://gitlab.com/tui/libcamera/-/tree/hacks-ae?ref_type=heads
> > > 
> > > I tried to move the code to src/ipa/libipa/simple_ae.cpp , but I could
> > > not get the result to link if I did that.
> > > 
> > > Currently code is called from SimpleCameraData::bufferReady , is that
> > > okay thing to do?
> > 
> > So the tricky thing here is that you are trying to call libipa from a
> > pipeline handler - that's not quite right.
> > 
> > So we likely need to determine if the simple-pipeline handler should
> > (optionally?) instantiate an IPA module which can then incorporate the
> > software ISP implementation. I expect this would only instantiate and
> > use the IPA when a RAW sensor is connected.
> 
> That's what I would recommend too. We should have a soft ISP component
> for the pipeline handler to instantiate, with a corresponding IPA
> module. The interface of the soft ISP should be generic, defined as an
> abstract class with pure virtual functions, with a derived class that
> implements a CPU-based soft ISP. Later, we'll develop a GPU-based soft
> ISP as a second implementation of the interface, and the simple pipeline
> handler will be able to use either.
> 
> > I suspect we would have a new IPA under src/ipa/softisp/ which would do
> > the implementation of 3a using a CPU/GPU depending on resource
> > availability.
> 
> The code in src/ipa/softisp/ should only control the ISP algorithms, not
> implement the CPU-based image processing. Ideally that IPA module would
> be used with either the CPU- and GPU-based implementations on the
> pipeline handler side.

ah yes - good point - there should be the soft-isp component that the
pipeline handler instantiates, which would be src/libcamera/softisp/ ?

with src/ipa/softisp/ as the algorithm code to correspond to that and
can make decisions based on the processing.

So there should be a new component under src/libcamera/softisp/ whic
maps the buffers with MappedFrameBuffer and allows the direct image
processing to occur. That's the starting point I believe where your
existing code would live, and that would be possible to link directly
with the simple pipeline hander as it could include
"libcamera/internal/softisp.h"

--
Kieran



> > From there any helpers that might be shared across IPA modules could
> > live in libipa, but anything that is specific to the new 'softisp' would
> > be under src/ipa/softisp/ ... so you would have
> > src/ipa/softisp/algorithms/cpu_ae.cpp perhaps.
> > 
> > > My divideUpExposure code has origins in rapsberry pi code; I could not
> > > find a way to share it nicely so I cut&pasted. If there's a better way
> > > (or better code to reuse), I'd like to know.
> > 
> > I believe code reuse from the RPi code is fine, but you would need to
> > ensure that the new code file matches the RPi IPA licence which is BSD I
> > believe.
> 
> -- 
> Regards,
> 
> Laurent Pinchart