[libcamera-devel] [RFC PATCH 0/2] libcamera: converter: introduce software converter for debayering and AWB

[Mattijs Korpershoek]

Hi Jacopo,

On mer., août 30, 2023 at 16:38, Jacopo Mondi <jacopo.mondi at ideasonboard.com> wrote:

> Hi Mattijs,
>
> On Wed, Aug 30, 2023 at 01:30:23PM +0200, Mattijs Korpershoek via libcamera-devel wrote:
>> Hi Andrey,
>>
>> Thank you for this work.
>>
>> I'm very interested by this series, mostly because I would like to add
>> YUYV -> NV12 pixel format conversion in this same class (to simplify
>> some Android use cases I'm working on).
>
> Have you considered using libYUV for such task ? It also feels this
> can be contained in the Android HAL layer where we already use libYUV ?

Yes, using libYUV is what I plan to do for this.

I will look into doing this only in the Android HAL layer but it feels
easier to plumb this conversion as a Converter class.

>
> Thanks
>    j
>
>>
>> I'm not familiar enough with the libcamera codebase to judge how well
>> this is implemented but I'm looking forward to see comments on this patchset.
>>
>> On dim., août 06, 2023 at 21:01, Andrey Konovalov via libcamera-devel <libcamera-devel at lists.libcamera.org> wrote:
>>
>> > Here is a draft implementation of Bayer demosaicing which follows the
>> > Converter interface and runs on CPU.
>> >
>> > It also includes a naive version of Grey World AWB. Just for demo purposes
>> > (to make the final image looking a bit nicer). Otherwise, Converter isn't
>> > the right place for AWB - only the statistics should be gathered here,
>> > and the rest belongs to an IPA module.
>> >
>> > Currently this software converter supports single output only, but adding
>> > the second stream for statistics is under consideration.
>> >
>> > As libcamera::Converter currently assumes a media device underneath the
>> > convertor, I wasn't able to avoid hacking the simple pipeline handler to make
>> > it work with the software converter. For the same reason ConverterFactory
>> > is not used for now.
>> >
>> > Only RAW10P format from the sensor is currently supported, but adding more
>> > Bayer formats wouldn't be a problem. Out of 10 bits, only 8 most significant
>> > ones are used to lessen the load on CPU. Simple bilinear interpolation is
>> > used for the same reason.
>> >
>> > AWB simplifications:
>> > - a naive implementation of "Grey World" algorithm: all pixels are used (no
>> >   brightest and darkest pixels excluded from the calculations)
>> > - to lessen the load on CPU, works on raw Bayer data and takes red values from
>> >   red pixels, blue values from blue, and green values from green pixels only.
>> > - to lessen the load on CPU, the red/green/blue gains calculated from the
>> >   current frame data are applied to the next frame. These gains are purely
>> >   in software (no V4L2 controls are set).
>> >
>> > No performance analysis or tuning have been done yet. On RB5 board this
>> > software convertor gives:
>> >   ~ 5 fps at 3278x2462 (camera sensor runs at 15 fps)
>> >   ~ 18..19 fps at 1918x1078 (out of 30 fps)
>> >   ~ 18..19 fps at 1638x1230 (out of 30 fps)
>> >   ~ 30 fps at 638x478 (out of 30 fps)
>> > (The resolutions above are the output ones; demosaic filter drops 1 pixel
>> > from each side of the frame, so that 3280x2464 from the camera sensor becomes
>> > 3278x2462 etc)
>>
>> Could you detail a bit how this was tested? Did you validate this using
>> "cam" ?
>>
>> Also, is there a reason for testing this with real hardware?
>>
>> Can't this be validated/developped upon the Virtual Media Controller
>> Driver (vimc) ?
>>
>> https://docs.kernel.org/admin-guide/media/vimc.html
>>
>> >
>> > Andrey Konovalov (2):
>> >   libcamera: converter: add software converter
>> >   [DNI] libcamera: pipeline: simple: a hack to use sotware converter
>> >     with qcom-camss
>> >
>> >  .../internal/converter/converter_softw.h      |  90 ++++
>> >  src/libcamera/converter/converter_softw.cpp   | 430 ++++++++++++++++++
>> >  src/libcamera/converter/meson.build           |   3 +-
>> >  src/libcamera/pipeline/simple/simple.cpp      |  22 +-
>> >  4 files changed, 542 insertions(+), 3 deletions(-)
>> >  create mode 100644 include/libcamera/internal/converter/converter_softw.h
>> >  create mode 100644 src/libcamera/converter/converter_softw.cpp
>> >
>> > --
>> > 2.34.1