Soft ISP TODO list (was: Re: [PATCH v6 17/18] libcamera: software_isp: Apply black level compensation)

[Laurent Pinchart]

On Mon, Apr 22, 2024 at 09:28:08PM +0200, Pavel Machek wrote:
> Hi!
> 
> > > > - Or if we add support for non-linear gains, as mentioned by Pavel.
> > 
> > Sensors are mostly linear these days. That is, until they reach
> > saturation of course. I don't think the soft ISP needs a linearization
> > LUT.
> 
> That's not what Martijn measured:
> https://blog.brixit.nl/fixing-the-megapixels-sensor-linearization/
> Librem 5 is reasonably modern.
> 
> Basically it is linear in 0..70% range, and then it is linear again in
> 90%..100% range, but at different slope. Take a look, it is fun article.

I did, and I don't reach the same conclusion. Obviously sensors
saturate, so close to saturation there will be non linearities. This is
not something that really requires compensation as far as I understand.
What linearization LUTs are meant to compensate for, unless I'm
mistaken, is non-linearities in the linear part of the dynamic range.

On a side note, I wonder if lens shading could play a role in shape of
the curve in the above article. Central pixels will saturate quicker
than pixels closer to the corners. It would be interesting to make the
same measurements taking only the central part of the image into
account. Another interesting experiment would be to measure the sensor
linearity without the lens (but that will be difficult in many cases
without destroying the camera).

> > > > - If we split the processing to pre-bayer and post-bayer parts, we should
> > > >   probably work with uint16_t or float's, which may have impact on performance.
> > > > 
> > > > - Pavel couldn't get a better performance by using SIMD CPU instructions for
> > > >   debayering.  Applying a CCM matrix may be a different matter.  Anyway, SIMD on
> > > >   CPU is hard to use and may differ on architectures, so the question is whether
> > > >   it's worth to invest into it.
> > 
> > Good question :-)
> 
> Oh, so good news is you write SIMD code once with gcc intristics, and
> gcc does its magic. You don't have to know assembly for that, but it
> certainly helps to look at the assembly if it looks reasonable.

There are also potentially interesting helper libraries such as
https://github.com/vectorclass/version2 (I haven't checked the license
compatibility).

> (Debayering needs too much of byte shuffling to work well with SIMD
> extensions. Matrix multiply could be better, but I'm not sure if 3x3
> matrix is big enough to get advantage).

-- 
Regards,

Laurent Pinchart