[libcamera-devel] [PATCH v1 00/10] Raspberry Pi: Platform configuration and buffer allocation improvements

[Naushir Patuck]

On Fri, 14 Oct 2022 at 14:18, Naushir Patuck <naush at raspberrypi.com> wrote:

> Hi,
>
> One big issue that is plaguing libcamera users on Raspberry Pi is memory
> usage.
> All our frame buffers have to be allocated in CMA space because of hardware
> restrictions.  Unfortunately CMA space is very limited (particularly on the
> Raspberry Pi Zero), and is prone to frequent fragmentation as
> display/rendering
> buffers also use this pool. Not to mention that the kernel is also allowed
> to
> allocate in the CMA pool for its own needs. As a result, some of our
> libcamera
> applications intermittently fail to run because of buffer allocation
> failure,
> and some use-cases (e.g. > 20MPix captures) will simply not be able to run
> on
> some platforms at all.
>
> This patch series attempts to try and fix this by easing the memory usage
> of
> the Raspberry Pi pipeline handler. It does this by using a platform
> configuration
> file which (amongst other things) instructs the pipeline handler on how
> many
> internal buffers it should allocate. There may be a performance penality
> (e.g
> dropped frames, or longer time taken for initial AE/AWB convergence) that
> have
> been documented.  But if the application can guarantee certain behavior,
> we can
> ensure the pipeline can still process requests with the same performance
> as we
> have today. The Raspberry Pi libcamera-apps and picamera2 framework
> already use
> this behavior, so the change should be transparent to applications built on
> these, apart from using significantly less memory.
>
> Patches 1 to 4 and the framework for using a configuration file to setup
> the
> buffer allocation parameters
>
> Patch 5 to 7 improve the buffer handling dealing with dropped frames,
> allowing
> us to use fewer internal buffers.
>

I should add that this particular change requires a firmware update to work
correctly.
If you want to test this out, it's best to run "rpi-update" to get the
latest firmware.

Naush


>
> Patch 8 adds a parameter to modify drop frame behavior - not related to
> this
> problem, but a nice to have.
>
> Patches 9 and 10 add configuration files for minimal memory configuration,
> and
> another for optimal memory usage for libcamera-apps and picamera2.
>
> I've considered how these various performance tuning parameters could be
> made
> part of the core StreamConfiguration structure or even as a Control.
> However,
> they are entirely platform specific, and I am struggling to come up with a
> clean
> way to put this into libcamera core with polluting it with various
> platform specific
> parameters.  If others have any ideas or suggestions on using alternate
> means
> of passing platform specific parameters into the pipeline handler, please
> do
> comment on this review.
>
> Thanks,
> Naush
>
> Naushir Patuck (10):
>   libcamera: pipeline: Add a platform configuration file helper
>   pipeline: raspberrypi: Add a pipeline config structure
>   pipeline: raspberrypi: Split out ISP Output0 buffer allocation
>   pipeline: raspberrypi: Read config parameters from a file
>   pipeline: raspberrypi: Disable StreamOn for ISP Output0/1 when
>     dropping frames
>   pipeline: raspberrypi: Reorder startup drop frame initialisation
>   pipeline: raspberrypi: Add a parameter to disable startup drop frames
>   pipeline: raspberrypi: Allow pipeline handler to always use the newest
>     frame
>   pipeline: raspberrypi: Add minimal memory usage config file
>   pipeline: raspberrypi: Add libcamera-apps and picamera2 config file
>
>  include/libcamera/internal/pipeline_handler.h |   2 +
>  .../pipeline/raspberrypi/data/default.json    |  28 +++
>  .../pipeline/raspberrypi/data/meson.build     |  10 +
>  .../raspberrypi/data/minimal_mem.json         |  28 +++
>  .../pipeline/raspberrypi/data/rpi_apps.json   |  28 +++
>  .../pipeline/raspberrypi/meson.build          |   2 +
>  .../pipeline/raspberrypi/raspberrypi.cpp      | 215 +++++++++++++++---
>  src/libcamera/pipeline_handler.cpp            |  57 +++++
>  8 files changed, 339 insertions(+), 31 deletions(-)
>  create mode 100644 src/libcamera/pipeline/raspberrypi/data/default.json
>  create mode 100644 src/libcamera/pipeline/raspberrypi/data/meson.build
>  create mode 100644
> src/libcamera/pipeline/raspberrypi/data/minimal_mem.json
>  create mode 100644 src/libcamera/pipeline/raspberrypi/data/rpi_apps.json
>
> --
> 2.25.1
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.libcamera.org/pipermail/libcamera-devel/attachments/20221014/5f11694a/attachment.htm>