[libcamera-devel] softISP for libcamera

[Laurent Pinchart]

On Tue, Dec 05, 2023 at 11:16:34PM +0300, Andrey Konovalov via libcamera-devel wrote:
> On 05.12.2023 20:32, Pavel Machek wrote:
> >>>> With "-Dpipelines=simple/linaro -Dipas=simple/linaro" the libcamera/ipa/soft_ipa_interface.h
> >>>> will be generated OK.
> >>>
> >>> Thank you, that got me something working.
> >>>
> >>> I believe modifying the code to support bayer8 was fairly easy (but I
> >>> did not look at the results yet).
> >>>
> >>> Unfortunately processing is quite slow:
> >>>
> > ...
> >>> 58759.057586 (0.45 fps) cam0-stream0 seq: 000111 bytesused: 2752812
> >>
> >> On a sm8250 board (pre-release version of RB5) with imx219 camera I am getting:
> >>
> >> 1) <snip>cam -c1 -C --stream pixelformat=RGB888,width=3278,height=2462
> >>
> >> 548.839035 (5.00 fps) cam0-stream0 seq: 000003 bytesused: 24211308
> > 
> > Seems better, yes. PinePhone is slow and there's probably still
> > problem with the caches.
> > 
> >> That said, debayering and statistics calculation on CPU aren't very fast.
> >> But your numbers still look too low for me...
> >>
> >> The statistics calculations for AWB aren't optimized in the current
> >> patch set (use all the pixels instead of a subset of them), but even
> >> dropping the stats calculations completely could improve the fps by
> >> a factor of two or so.
> > 
> > Yes, but we'd need factor on 20 or so...
> > 
> > I'd really like to take still photos. It is one of main use cases.
> > 
> > That means
> > 
> > 1) showing some kind of preview, best at 30fps but 10fps would
> > do. Fortunately screen does have way lower resolution than sensor, and
> > low-quality debayer+downscale is rather easy to do. Just treat GRRB
> > block as one pixel...
> > 
> > 2) saving full resolution photo. Best saving raw bayer data.
> > 
> > For that, it would be very nice to have statistics computation
> > decoupled from actual debayer.
> 
> We are very likely to go this route (decoupling statistics computation
> from debayer).
> 
> > Most of the time, it is enough to
> > sample 100 (or so) points in the picture to get good enough statistics
> > for AE, and some more for auto focus. No need to process full frame.
> > 
> > Application can just do debayer+downscale, and display preview, maybe
> > using GPU. Anyway CPU is fast enough to do it at 10fps and screen
> > resolution. But when user presses the trigger, we still have full
> > frame to save, and can save it to disk or (do one-off high-quality
> > jpeg processing).
> > 
> >>> Am I right that you only use top 8 bits from the bayer10 data?
> >>
> >> Yes, this is correct.
> > 
> > And that's wrong, right? :-).
> 
> Yes, this way we do loose part of image data. Using the full 10 bits
> for further processing would improve the image quality.
> 
> > Let me explain. RGB888 has 8 bits per pixel, non-linear. So debayer
> > should really take 10 bits, and apply the correction. It may be called
> > gamma or something, I'm sure list knows.
> 
> Right. Going from RAW10's 10 bits (linear) to RGB888's 8 bits (non-linear)
> should be better done during the gamma correction stage. Rather than cutting
> the 2 LS bits off before any processing is done.

For a GPU-based ISP, possibly, but for a CPU-based implementation, the
extra processing will be very costly. I really wouldn't bother with the
2 LSBs.

I also don't expect any tone mapping in a CPU-based implementation to be
honest, unless the implementation is meant for testing purpose only, for
instance to prototype code before moving the implementation to the GPU.
In production use case, anything beyond very basic processing will
likely consume too much CPU.

> The excuse is that in the current patch set there is no support for gamma
> at all (which also makes the picture look dark and low contrast). And I was
> also saving some CPU cycles in the easiest way.

-- 
Regards,

Laurent Pinchart