[libcamera-devel] [PATCH v1 00/10] Raspberry Pi: Platform configuration and buffer allocation improvements

[Naushir Patuck]

Hi,

One big issue that is plaguing libcamera users on Raspberry Pi is memory usage.
All our frame buffers have to be allocated in CMA space because of hardware
restrictions.  Unfortunately CMA space is very limited (particularly on the
Raspberry Pi Zero), and is prone to frequent fragmentation as display/rendering
buffers also use this pool. Not to mention that the kernel is also allowed to
allocate in the CMA pool for its own needs. As a result, some of our libcamera
applications intermittently fail to run because of buffer allocation failure,
and some use-cases (e.g. > 20MPix captures) will simply not be able to run on
some platforms at all.

This patch series attempts to try and fix this by easing the memory usage of
the Raspberry Pi pipeline handler. It does this by using a platform configuration
file which (amongst other things) instructs the pipeline handler on how many
internal buffers it should allocate. There may be a performance penality (e.g
dropped frames, or longer time taken for initial AE/AWB convergence) that have
been documented.  But if the application can guarantee certain behavior, we can
ensure the pipeline can still process requests with the same performance as we
have today. The Raspberry Pi libcamera-apps and picamera2 framework already use
this behavior, so the change should be transparent to applications built on
these, apart from using significantly less memory.

Patches 1 to 4 and the framework for using a configuration file to setup the
buffer allocation parameters

Patch 5 to 7 improve the buffer handling dealing with dropped frames, allowing
us to use fewer internal buffers.

Patch 8 adds a parameter to modify drop frame behavior - not related to this
problem, but a nice to have.

Patches 9 and 10 add configuration files for minimal memory configuration, and
another for optimal memory usage for libcamera-apps and picamera2.

I've considered how these various performance tuning parameters could be made
part of the core StreamConfiguration structure or even as a Control.  However,
they are entirely platform specific, and I am struggling to come up with a clean
way to put this into libcamera core with polluting it with various platform specific
parameters.  If others have any ideas or suggestions on using alternate means
of passing platform specific parameters into the pipeline handler, please do
comment on this review.

Thanks,
Naush

Naushir Patuck (10):
  libcamera: pipeline: Add a platform configuration file helper
  pipeline: raspberrypi: Add a pipeline config structure
  pipeline: raspberrypi: Split out ISP Output0 buffer allocation
  pipeline: raspberrypi: Read config parameters from a file
  pipeline: raspberrypi: Disable StreamOn for ISP Output0/1 when
    dropping frames
  pipeline: raspberrypi: Reorder startup drop frame initialisation
  pipeline: raspberrypi: Add a parameter to disable startup drop frames
  pipeline: raspberrypi: Allow pipeline handler to always use the newest
    frame
  pipeline: raspberrypi: Add minimal memory usage config file
  pipeline: raspberrypi: Add libcamera-apps and picamera2 config file

 include/libcamera/internal/pipeline_handler.h |   2 +
 .../pipeline/raspberrypi/data/default.json    |  28 +++
 .../pipeline/raspberrypi/data/meson.build     |  10 +
 .../raspberrypi/data/minimal_mem.json         |  28 +++
 .../pipeline/raspberrypi/data/rpi_apps.json   |  28 +++
 .../pipeline/raspberrypi/meson.build          |   2 +
 .../pipeline/raspberrypi/raspberrypi.cpp      | 215 +++++++++++++++---
 src/libcamera/pipeline_handler.cpp            |  57 +++++
 8 files changed, 339 insertions(+), 31 deletions(-)
 create mode 100644 src/libcamera/pipeline/raspberrypi/data/default.json
 create mode 100644 src/libcamera/pipeline/raspberrypi/data/meson.build
 create mode 100644 src/libcamera/pipeline/raspberrypi/data/minimal_mem.json
 create mode 100644 src/libcamera/pipeline/raspberrypi/data/rpi_apps.json

-- 
2.25.1