[PATCH v3 00/16] libcamera: introduce Software ISP and Software IPA
Hans de Goede
hdegoede at redhat.com
Wed Feb 14 18:03:45 CET 2024
Hi All,
On 2/14/24 18:01, Hans de Goede wrote:
> Hi All,
>
> Here is v3 of the patch-set to add Software ISP support
> to libcamera / to the simple pipeline-handler.
Note as Kieran pointed out in an offlist email, the new CI has
already caught a couple of issues with this v3:
Unit test configuration failed to build:
- https://gitlab.freedesktop.org/camera/libcamera-softisp/-/jobs/55028866#L357
Debian 10 has an issue with some of the documentation additions:
- https://gitlab.freedesktop.org/camera/libcamera-softisp/-/jobs/55028856#L439
Clang picks up warnings in the debayer_cpu implementation:
- https://gitlab.freedesktop.org/camera/libcamera-softisp/-/jobs/55028858#L508
Clang picks up warnings in the swstats implementation:
- https://gitlab.freedesktop.org/camera/libcamera-softisp/-/jobs/55028858#L549
I'll start looking into these next week at the earliest, feel
free to beat me to it :)
Regards,
Hans
> Changes in v3 vs v2:
> - The Software ISP now is always build when building the simple pipeline
> handler, no more -Dpipelines=simple/simple. Instead whether the SoftISP
> is used or not is based on the media-controller driver. For now it is
> only enabled for the "qcom-camss" driver (Andrey)
> - SoftISP factory has been removed, there is a just a single SoftwareISP class now
> - Fix the multi-threading issues (Andrey)
> - Fully document the SharedMemObject and DmaHeap classes (Andrey)
> - Drop SwStats base (integrate into SwStatsCpu class)
> - Move headers for classes only used by the SoftISP into src/libcamera/software_isp
> - Move SwStats / SwDebayer docs into .cpp and extend it
> - Rename many foo_bar symbols to fooBar
> - Add constexpr kFooBar values for varies hardcoded sizes like
> the yHistoGram having 16 bins and the gammalookuptable having 1024 entries
> - Make startFrame() and finishFrame() normal methods instead of
> using function pointers for these
> - Document how/why an array of src pointers is passed to
> the debayer functions
> - 12bpp unpacked raw bayer input support (Kieran)
> - Add "Software ISP benchmarking" doc
>
> Changes in v2 vs v1:
> - Integrated Dennis, Martti and Toon's auto-exposure algorithm
> based on the paper which they found which gives us a nice
> relatively simple AEC + AGC algorithm
> TODO: Add link to paper to source + commit-message
> - Integrated Dennis' doxygen comments for all new classes, no more warnings!
> - Move AWB gain calculations to the IPA (Andrey)
> - Added 8 bpp and 10 bpp unpacked raw bayer input support (Hans)
> - Use dma-buf for the output FrameBuffer-s (Andrey)
> - Memcpy data from input FrameBuffers to a heap buffer on a line by line
> basis to speedup debayering from uncached mem (Hans)
> - This addresses all "open issues" from the v1 posting
>
> Changes in v1 vs RFC-v2:
> - Add and use SwStats[Cpu] and Debayer[Cpu] classes
> - Rename linaro soft-IPA and soft-ISP implementations to simple following
> the simple pipeline hander
> - Integrate Pavel's swstats and debayer improvements
>
> This has been tested on:
> - Qualcomm RB5 board with a mezzanine board equipped with RPi camera v2 (Andrey)
> - Lenovo x13s, sc8280xp (Bryan)
> - Pinephone (Pavel)
> - Debix Model A (Milan)
> - Lenovo Thinkpad Yoga X1 yoga gen7/8, Alder Lake/Raptor Lake IPU6EP +
> ov2740 10bpp packed + unpacked (Dennis / Hans)
> - Dell Latitude 9420, Tiger Lake IPU6 + ov01a1s (RGBI) sensor
> (Martti, requires IGIG_GBGR_IGIG_GRGB patches on top)
>
> git branch for this v3 patch set:
> https://gitlab.freedesktop.org/camera/libcamera-softisp/-/commits/SoftwareISP-v06
>
> git branch for v2 of the patch set:
> https://gitlab.freedesktop.org/camera/libcamera-softisp/-/commits/SoftwareISP-v05
>
> git branch for v1 of the patch set:
> https://gitlab.freedesktop.org/camera/libcamera-softisp/-/commits/SoftwareISP-v04
>
> Old RFC-v2 cover-letter with small updates:
>
> Here is an implementation of Software ISP and Software IPA
> which provide debayering and implementation of image processing
> algorithms for systems without a hardware ISP, or in cases when
> there are no public drivers for the hardware ISP present in the
> system.
>
> The Software IPA has auto exposure/AGC and AWB. For the AGC
> the analogue gain control of the camera sensor is used (if
> available).
>
> The 6th patch renames some variables in the simple pipeline
> handler for the Software ISP to use the same buffer handling
> code as the Converter currently does. This lets one to
> avoid adding extra code to the pipeline handler, but also
> makes the Software ISP and the Converter mutually exclusive.
>
> The Software ISP / IPA are intended to be used with the simple
> pipeline handler. The pipeline handler doesn't interact with
> the Software IPA directly - the Software IPA is hidden behind
> the Software ISP interface. To use the Software ISP the build
> must be configured with -Dpipelines=simple -Dipas=simple .
>
> This patch set uses SharedMemObject class used by the RPi pipeline
> handler - the second patch in the series moves the header file
> to a common directory.
>
> This patch set has been tested on Qualcomm RB5 board with
> a mezzanine board equipped with RPi camera v2 (not the
> standard RB5 camera mezzanine).
>
> git branch for the RFC-v2 patch set:
> https://gitlab.freedesktop.org/camera/libcamera-softisp/-/commits/SoftwareISP-v03
>
> RFC-v1 of the patch set:
> https://patchwork.libcamera.org/cover/19262/
>
> Changes in RFC v2 vs RFC v1:
> - patches are restructured and reordered
> - the Software IPA is hidden behind the Software ISP interface. The
> pipeline handler doesn't interact with the Software IPA directly
> - instantiation of the Software ISP / IPA is configurable (at build
> time)
> - comment explaining the implementation limitations is added to
> SwIspLinaro::IspWorker::debayerRaw10P()
>
> Regards,
>
> Hans
>
>
>
> Andrei Konovalov (1):
> libcamera: shared_mem_object: reorganize the code and document the
> SharedMemObject class
>
> Andrey Konovalov (8):
> libcamera: pipeline: simple: fix size adjustment in validate()
> libcamera: internal: Move dma_heaps.[h,cpp] to common directories
> libcamera: dma_heaps: extend DmaHeap class to support system heap
> libcamera: internal: Move SharedMemObject class to a common directory
> libcamera: ipa: add Soft IPA
> libcamera: introduce SoftwareIsp
> libcamera: pipeline: simple: rename converterBuffers_ and related vars
> libcamera: pipeline: simple: enable use of Soft ISP and Soft IPA
>
> Hans de Goede (7):
> libcamera: software_isp: Add SwStatsCpu class
> libcamera: software_isp: Add Debayer base class
> libcamera: software_isp: Add DebayerCpu class
> libcamera: swstats_cpu: Add support for 8, 10 and 12 bpp unpacked
> bayer input
> libcamera: debayer_cpu: Add support for 8, 10 and 12 bpp unpacked
> bayer input
> libcamera: debayer_cpu: Add BGR888 output support
> libcamera: Add "Software ISP benchmarking" documentation
>
> Documentation/Doxyfile.in | 1 +
> Documentation/index.rst | 1 +
> Documentation/meson.build | 1 +
> Documentation/software-isp-benchmarking.rst | 82 ++
> .../libcamera/internal}/dma_heaps.h | 14 +-
> include/libcamera/internal/meson.build | 3 +
> .../libcamera/internal}/shared_mem_object.h | 94 ++-
> .../internal/software_isp/debayer_params.h | 48 ++
> .../internal/software_isp/meson.build | 7 +
> .../internal/software_isp/software_isp.h | 98 +++
> .../internal/software_isp/swisp_stats.h | 38 +
> include/libcamera/ipa/meson.build | 1 +
> include/libcamera/ipa/soft.mojom | 28 +
> meson_options.txt | 2 +-
> src/ipa/simple/data/meson.build | 9 +
> src/ipa/simple/data/soft.conf | 3 +
> src/ipa/simple/meson.build | 25 +
> src/ipa/simple/soft_simple.cpp | 308 +++++++
> src/libcamera/dma_heaps.cpp | 164 ++++
> src/libcamera/meson.build | 3 +
> src/libcamera/pipeline/rpi/vc4/dma_heaps.cpp | 90 --
> src/libcamera/pipeline/rpi/vc4/meson.build | 1 -
> src/libcamera/pipeline/rpi/vc4/vc4.cpp | 5 +-
> src/libcamera/pipeline/simple/simple.cpp | 232 ++++--
> src/libcamera/shared_mem_object.cpp | 191 +++++
> src/libcamera/software_isp/debayer.cpp | 29 +
> src/libcamera/software_isp/debayer.h | 104 +++
> src/libcamera/software_isp/debayer_cpu.cpp | 779 ++++++++++++++++++
> src/libcamera/software_isp/debayer_cpu.h | 157 ++++
> src/libcamera/software_isp/meson.build | 8 +
> src/libcamera/software_isp/software_isp.cpp | 349 ++++++++
> src/libcamera/software_isp/swstats_cpu.cpp | 336 ++++++++
> src/libcamera/software_isp/swstats_cpu.h | 167 ++++
> 33 files changed, 3186 insertions(+), 192 deletions(-)
> create mode 100644 Documentation/software-isp-benchmarking.rst
> rename {src/libcamera/pipeline/rpi/vc4 => include/libcamera/internal}/dma_heaps.h (65%)
> rename {src/libcamera/pipeline/rpi/common => include/libcamera/internal}/shared_mem_object.h (66%)
> create mode 100644 include/libcamera/internal/software_isp/debayer_params.h
> create mode 100644 include/libcamera/internal/software_isp/meson.build
> create mode 100644 include/libcamera/internal/software_isp/software_isp.h
> create mode 100644 include/libcamera/internal/software_isp/swisp_stats.h
> create mode 100644 include/libcamera/ipa/soft.mojom
> create mode 100644 src/ipa/simple/data/meson.build
> create mode 100644 src/ipa/simple/data/soft.conf
> create mode 100644 src/ipa/simple/meson.build
> create mode 100644 src/ipa/simple/soft_simple.cpp
> create mode 100644 src/libcamera/dma_heaps.cpp
> delete mode 100644 src/libcamera/pipeline/rpi/vc4/dma_heaps.cpp
> create mode 100644 src/libcamera/shared_mem_object.cpp
> create mode 100644 src/libcamera/software_isp/debayer.cpp
> create mode 100644 src/libcamera/software_isp/debayer.h
> create mode 100644 src/libcamera/software_isp/debayer_cpu.cpp
> create mode 100644 src/libcamera/software_isp/debayer_cpu.h
> create mode 100644 src/libcamera/software_isp/meson.build
> create mode 100644 src/libcamera/software_isp/software_isp.cpp
> create mode 100644 src/libcamera/software_isp/swstats_cpu.cpp
> create mode 100644 src/libcamera/software_isp/swstats_cpu.h
>
More information about the libcamera-devel
mailing list