[PATCH v6 17/18] libcamera: software_isp: Apply black level compensation

[Milan Zamazal]

Hi,

thank you all for your comments and clarifications.  Let me summarize the
discussion, with my notes merged in, before I get lost in it:

- Since our current black level and color gain operations are linear, we average
  only the same color pixels and the lookup table is applied after averaging the
  pixels, the current debayering implementation is correct.  It doesn't matter
  whether we do "average pixels -> subtract black -> multiply by color gain ->
  apply gamma" or "subtract black -> multiply by color gain -> average pixels ->
  apply gamma".

- This may no longer hold if we change the operations, for example:

- The optimization suggested by Hans (preprocessing while copying lines from the
  input buffer).  It may or may not have a significant impact on the performance
  and may or may not have a significant impact on the image quality.  I think we
  need some experiments here.  Performance on the CPU is generally challenging,
  so if we can demonstrate a performance benefit, we should consider including
  it (most likely configurable), if it doesn't turn the implementation into a
  complete mess.

- Or if we add support for non-linear gains, as mentioned by Pavel.

- Laurent suggested adding a CCM matrix between debayering and gamma to achieve
  a reasonable image quality.  According to Hans, this is a heavy operation on
  CPUs.  Laurent still sees CPU processing as useful for experiments.  I think
  we can make this simply configurable (if it gets implemented).

- If we split the processing to pre-bayer and post-bayer parts, we should
  probably work with uint16_t or float's, which may have impact on performance.

- Pavel couldn't get a better performance by using SIMD CPU instructions for
  debayering.  Applying a CCM matrix may be a different matter.  Anyway, SIMD on
  CPU is hard to use and may differ on architectures, so the question is whether
  it's worth to invest into it.

- GPU processing should make many of these things easier and more things
  possible.

- Do we already know how to map the buffers to GPU efficiently?

My conclusions are:

- The current CPU debayering computation itself is fine at the moment and we
  shouldn't change it without a good reason.  It can be replaced in future if
  needed, once we have a better understanding of what and how we can
  realistically achieve.  General cleanups, like moving table computations
  elsewhere, would be still useful already now.

- We can experiment with whatever mentioned above to get the better
  understanding, but:

- GPU processing is clearly a priority.

- We have also other items in the TODO file!  (I already work on some of them.)

- We should probably change the e-mail thread subject.

Thanks,
Milan