[libcamera-devel] [PATCH v2 00/18] libcamera: introduce Software ISP and Software IPA
Hans de Goede
hdegoede at redhat.com
Sat Jan 13 15:35:51 CET 2024
Hi All,
Ugh I forgot to fill in the subject part of the cover-letter, sorry.
I've fixed the subject in this reply, for any replies
to the cover-letter please use this subject.
Regards,
Hans
On 1/13/24 15:22, Hans de Goede wrote:
> Hi All,
>
> Here is v2 of the patch-set to add Software ISP support
> to libcamera / to the simple pipeline-handler.
>
> Changes in v2 vs v1:
> - Integrated Dennis, Martti and Toon's auto-exposure algorithm
> based on the paper which they found which gives us a nice
> relatively simple AEC + AGC algorithm
> TODO: Add link to paper to source + commit-message
> - Integrated Dennis' doxygen comments for all new classes, no more warnings!
> - Move AWB gain calculations to the IPA (Andrey)
> - Added 8 bpp and 10 bpp unpacked raw bayer input support (Hans)
> - Use dma-buf for the output FrameBuffer-s (Andrey)
> - Memcpy data from input FrameBuffers to a heap buffer on a line by line
> basis to speedup debayering from uncached mem (Hans)
> - This addresses all "open issues" from the v1 posting
>
> Changes in v1 vs RFC-v2:
> - Add and use SwStats[Cpu] and Debayer[Cpu] classes
> - Rename linaro soft-IPA and soft-ISP implementations to simple following
> the simple pipeline hander
> - Integrate Pavel's swstats and debayer improvements
>
> This has been tested on:
> - Qualcomm RB5 board with a mezzanine board equipped with RPi camera v2 (Andrey)
> - Lenovo x13s, sc8280xp (Bryan)
> - Pinephone (Pavel)
> - Debix Model A (Milan)
> - Lenovo Thinkpad Yoga X1 yoga gen7/8, Alder Lake/Raptor Lake IPU6EP +
> ov2740 10bpp packed + unpacked (Dennis / Hans)
> - Dell Latitude 9420, Tiger Lake IPU6 + ov01a1s (RGBI) sensor
> (Martti, requires IGIG_GBGR_IGIG_GRGB patches on top)
>
> git branch for this v2 patch set:
> https://gitlab.freedesktop.org/camera/libcamera-softisp/-/commits/SoftwareISP-v05
>
> git branch for v1 of the patch set:
> https://gitlab.freedesktop.org/camera/libcamera-softisp/-/commits/SoftwareISP-v04
>
> Old RFC-v2 cover-letter with small updates:
>
> Here is an implementation of Software ISP and Software IPA
> which provide debayering and implementation of image processing
> algorithms for systems without a hardware ISP, or in cases when
> there are no public drivers for the hardware ISP present in the
> system.
>
> The implementation of the Software ISP is a reference one.
> A naive AWB alorithm is implemented as part of a function which
> does debayering and statistics calculations - the algorithm part
> is to be moved to the IPA in the next version of the patch set.
> And for debayering itself there is already a more efficient
> implementation by Hans de Goede. This patch set is currently using
> the earlier debayering implementation as it is less lines of code
> and is OK for the initial discussion.
> Only RAW10P format from the sensor is currently supported, but
> other debayering functions can be easily added (which of them to
> call is decided based on the input format).
>
> The Software IPA has only auto exposure and AGC. For the AGC
> the analogue gain control of the camera sensor is used (if
> available). The algorithm is very much simplified, and is
> mostly included as a reference code.
>
> The 6th patch renames some variables in the simple pipeline
> handler for the Software ISP to use the same buffer handling
> code as the Converter currently does. This lets one to
> avoid adding extra code to the pipeline handler, but also
> makes the Software ISP and the Converter mutually exclusive.
>
> The Software ISP / IPA are intended to be used with the simple
> pipeline handler. The pipeline handler doesn't interact with
> the Software IPA directly - the Software IPA is hidden behind
> the Software ISP interface. To use the Software ISP the build
> must be configured with
> -Dpipelines=simple/simple -Dipas=simple/simple
> and the Converter must not be used (the latter is hardcoded
> in the simple pipeline handler).
> If the build is configured with just
> -Dpipelines=simple
> the Software ISP / IPA are not used, and the simple pipeline
> handler works in the same way as without this patchset.
>
> "simple" in the
> -Dpipelines=simple/simple -Dipas=simple/simple
> is the name of the Software ISP / IPA implementation.
> -Dpipelines=simple/<something else> -Dipas=simple/<something else>
> should work too if another implementation were added. What
> implementation to use is the build time choice, only one
> implementation is included into the build.
>
> This patch set uses SharedMemObject class used by the RPi pipeline
> handler - the second patch in the series moves the header file
> to a common directory.
>
> This patch set has been tested on Qualcomm RB5 board with
> a mezzanine board equipped with RPi camera v2 (not the
> standard RB5 camera mezzanine).
>
> git branch for the RFC-v2 patch set:
> https://gitlab.freedesktop.org/camera/libcamera-softisp/-/commits/SoftwareISP-v03
>
> RFC-v1 of the patch set:
> https://patchwork.libcamera.org/cover/19262/
>
> Changes in RFC v2 vs RFC v1:
> - patches are restructured and reordered
> - the Software IPA is hidden behind the Software ISP interface. The
> pipeline handler doesn't interact with the Software IPA directly
> - instantiation of the Software ISP / IPA is configurable (at build
> time)
> - comment explaining the implementation limitations is added to
> SwIspLinaro::IspWorker::debayerRaw10P()
>
> Regards,
>
> Hans
>
>
>
> Andrey Konovalov (10):
> libcamera: pipeline: simple: fix size adjustment in validate()
> libcamera: internal: Move dma_heaps.[h,cpp] to common directories
> libcamera: dma_heaps: extend DmaHeap class to support system heap
> libcamera: internal: Move SharedMemObject class to a common directory
> libcamera: introduce SoftwareIsp class
> libcamera: ipa: add Soft IPA common files
> libcamera: ipa: Soft IPA: add a Simple Soft IPA implementation
> libcamera: software_isp: add Simple SoftwareIsp implementation
> libcamera: pipeline: simple: rename converterBuffers_ and related vars
> libcamera: pipeline: simple: enable use of Soft ISP and Soft IPA
>
> Dennis Bonke (1):
> libcamera: internal: Document the SharedMemObject class
>
> Hans de Goede (7):
> libcamera: software_isp: Add SwStats base class
> libcamera: software_isp: Add SwStatsCpu class
> libcamera: software_isp: Add Debayer base class
> libcamera: software_isp: Add DebayerCpu class
> libcamera: swstats_cpu: Add support for 8 and 10 bpp unpacked bayer
> input
> libcamera: debayer_cpu: Add support for 8 and 10 bpp unpacked bayer
> input
> libcamera: debayer_cpu: Add BGR888 output support
>
> Documentation/Doxyfile.in | 1 +
> .../libcamera/internal}/dma_heaps.h | 14 +-
> include/libcamera/internal/meson.build | 4 +
> .../libcamera/internal}/shared_mem_object.h | 57 +-
> include/libcamera/internal/software_isp.h | 231 ++++++
> .../libcamera/internal/software_isp/debayer.h | 132 ++++
> .../internal/software_isp/debayer_cpu.h | 141 ++++
> .../internal/software_isp/debayer_params.h | 43 ++
> .../internal/software_isp/meson.build | 11 +
> .../internal/software_isp/swisp_simple.h | 163 +++++
> .../internal/software_isp/swisp_stats.h | 34 +
> .../libcamera/internal/software_isp/swstats.h | 215 ++++++
> .../internal/software_isp/swstats_cpu.h | 51 ++
> include/libcamera/ipa/meson.build | 1 +
> include/libcamera/ipa/soft.mojom | 29 +
> meson_options.txt | 3 +-
> src/ipa/simple/common/meson.build | 17 +
> src/ipa/simple/common/soft_base.cpp | 73 ++
> src/ipa/simple/common/soft_base.h | 50 ++
> src/ipa/simple/meson.build | 12 +
> src/ipa/simple/simple/data/meson.build | 9 +
> src/ipa/simple/simple/data/soft.conf | 3 +
> src/ipa/simple/simple/meson.build | 26 +
> src/ipa/simple/simple/soft_simple.cpp | 273 ++++++++
> .../{pipeline/rpi/vc4 => }/dma_heaps.cpp | 55 +-
> src/libcamera/meson.build | 3 +
> src/libcamera/pipeline/rpi/vc4/meson.build | 1 -
> src/libcamera/pipeline/rpi/vc4/vc4.cpp | 5 +-
> src/libcamera/pipeline/simple/simple.cpp | 177 +++--
> src/libcamera/software_isp.cpp | 62 ++
> src/libcamera/software_isp/debayer.cpp | 22 +
> src/libcamera/software_isp/debayer_cpu.cpp | 661 ++++++++++++++++++
> src/libcamera/software_isp/meson.build | 27 +
> src/libcamera/software_isp/swisp_simple.cpp | 238 +++++++
> src/libcamera/software_isp/swstats.cpp | 22 +
> src/libcamera/software_isp/swstats_cpu.cpp | 261 +++++++
> 36 files changed, 3035 insertions(+), 92 deletions(-)
> rename {src/libcamera/pipeline/rpi/vc4 => include/libcamera/internal}/dma_heaps.h (65%)
> rename {src/libcamera/pipeline/rpi/common => include/libcamera/internal}/shared_mem_object.h (62%)
> create mode 100644 include/libcamera/internal/software_isp.h
> create mode 100644 include/libcamera/internal/software_isp/debayer.h
> create mode 100644 include/libcamera/internal/software_isp/debayer_cpu.h
> create mode 100644 include/libcamera/internal/software_isp/debayer_params.h
> create mode 100644 include/libcamera/internal/software_isp/meson.build
> create mode 100644 include/libcamera/internal/software_isp/swisp_simple.h
> create mode 100644 include/libcamera/internal/software_isp/swisp_stats.h
> create mode 100644 include/libcamera/internal/software_isp/swstats.h
> create mode 100644 include/libcamera/internal/software_isp/swstats_cpu.h
> create mode 100644 include/libcamera/ipa/soft.mojom
> create mode 100644 src/ipa/simple/common/meson.build
> create mode 100644 src/ipa/simple/common/soft_base.cpp
> create mode 100644 src/ipa/simple/common/soft_base.h
> create mode 100644 src/ipa/simple/meson.build
> create mode 100644 src/ipa/simple/simple/data/meson.build
> create mode 100644 src/ipa/simple/simple/data/soft.conf
> create mode 100644 src/ipa/simple/simple/meson.build
> create mode 100644 src/ipa/simple/simple/soft_simple.cpp
> rename src/libcamera/{pipeline/rpi/vc4 => }/dma_heaps.cpp (54%)
> create mode 100644 src/libcamera/software_isp.cpp
> create mode 100644 src/libcamera/software_isp/debayer.cpp
> create mode 100644 src/libcamera/software_isp/debayer_cpu.cpp
> create mode 100644 src/libcamera/software_isp/meson.build
> create mode 100644 src/libcamera/software_isp/swisp_simple.cpp
> create mode 100644 src/libcamera/software_isp/swstats.cpp
> create mode 100644 src/libcamera/software_isp/swstats_cpu.cpp
>
More information about the libcamera-devel
mailing list