[libcamera-devel] [RFC PATCH 0/2] libcamera: converter: introduce software converter for debayering and AWB

[Mattijs Korpershoek]

Hi Andrey,

Thank you for this work.

I'm very interested by this series, mostly because I would like to add
YUYV -> NV12 pixel format conversion in this same class (to simplify
some Android use cases I'm working on).

I'm not familiar enough with the libcamera codebase to judge how well
this is implemented but I'm looking forward to see comments on this patchset.

On dim., août 06, 2023 at 21:01, Andrey Konovalov via libcamera-devel <libcamera-devel at lists.libcamera.org> wrote:

> Here is a draft implementation of Bayer demosaicing which follows the
> Converter interface and runs on CPU.
>
> It also includes a naive version of Grey World AWB. Just for demo purposes
> (to make the final image looking a bit nicer). Otherwise, Converter isn't
> the right place for AWB - only the statistics should be gathered here,
> and the rest belongs to an IPA module.
>
> Currently this software converter supports single output only, but adding
> the second stream for statistics is under consideration.
>
> As libcamera::Converter currently assumes a media device underneath the
> convertor, I wasn't able to avoid hacking the simple pipeline handler to make
> it work with the software converter. For the same reason ConverterFactory
> is not used for now.
>
> Only RAW10P format from the sensor is currently supported, but adding more
> Bayer formats wouldn't be a problem. Out of 10 bits, only 8 most significant
> ones are used to lessen the load on CPU. Simple bilinear interpolation is
> used for the same reason.
>
> AWB simplifications:
> - a naive implementation of "Grey World" algorithm: all pixels are used (no
>   brightest and darkest pixels excluded from the calculations)
> - to lessen the load on CPU, works on raw Bayer data and takes red values from
>   red pixels, blue values from blue, and green values from green pixels only. 
> - to lessen the load on CPU, the red/green/blue gains calculated from the
>   current frame data are applied to the next frame. These gains are purely
>   in software (no V4L2 controls are set).
>
> No performance analysis or tuning have been done yet. On RB5 board this
> software convertor gives:
>   ~ 5 fps at 3278x2462 (camera sensor runs at 15 fps)
>   ~ 18..19 fps at 1918x1078 (out of 30 fps)
>   ~ 18..19 fps at 1638x1230 (out of 30 fps)
>   ~ 30 fps at 638x478 (out of 30 fps)
> (The resolutions above are the output ones; demosaic filter drops 1 pixel
> from each side of the frame, so that 3280x2464 from the camera sensor becomes
> 3278x2462 etc)

Could you detail a bit how this was tested? Did you validate this using
"cam" ?

Also, is there a reason for testing this with real hardware?

Can't this be validated/developped upon the Virtual Media Controller
Driver (vimc) ?

https://docs.kernel.org/admin-guide/media/vimc.html

>
> Andrey Konovalov (2):
>   libcamera: converter: add software converter
>   [DNI] libcamera: pipeline: simple: a hack to use sotware converter
>     with qcom-camss
>
>  .../internal/converter/converter_softw.h      |  90 ++++
>  src/libcamera/converter/converter_softw.cpp   | 430 ++++++++++++++++++
>  src/libcamera/converter/meson.build           |   3 +-
>  src/libcamera/pipeline/simple/simple.cpp      |  22 +-
>  4 files changed, 542 insertions(+), 3 deletions(-)
>  create mode 100644 include/libcamera/internal/converter/converter_softw.h
>  create mode 100644 src/libcamera/converter/converter_softw.cpp
>
> -- 
> 2.34.1