[libcamera-devel] [PATCH v1 00/10] Raspberry Pi: Platform configuration and buffer allocation improvements

[Kieran Bingham]

Quoting Naushir Patuck via libcamera-devel (2022-10-14 15:55:46)
> On Fri, 14 Oct 2022 at 14:18, Naushir Patuck <naush at raspberrypi.com> wrote:
> 
> > Hi,
> >
> > One big issue that is plaguing libcamera users on Raspberry Pi is memory
> > usage.
> > All our frame buffers have to be allocated in CMA space because of hardware
> > restrictions.  Unfortunately CMA space is very limited (particularly on the
> > Raspberry Pi Zero), and is prone to frequent fragmentation as
> > display/rendering
> > buffers also use this pool. Not to mention that the kernel is also allowed
> > to
> > allocate in the CMA pool for its own needs. As a result, some of our
> > libcamera
> > applications intermittently fail to run because of buffer allocation
> > failure,
> > and some use-cases (e.g. > 20MPix captures) will simply not be able to run
> > on
> > some platforms at all.
> >
> > This patch series attempts to try and fix this by easing the memory usage
> > of
> > the Raspberry Pi pipeline handler. It does this by using a platform
> > configuration
> > file which (amongst other things) instructs the pipeline handler on how
> > many
> > internal buffers it should allocate. There may be a performance penality
> > (e.g
> > dropped frames, or longer time taken for initial AE/AWB convergence) that
> > have
> > been documented.  But if the application can guarantee certain behavior,
> > we can
> > ensure the pipeline can still process requests with the same performance
> > as we
> > have today. The Raspberry Pi libcamera-apps and picamera2 framework
> > already use
> > this behavior, so the change should be transparent to applications built on
> > these, apart from using significantly less memory.
> >
> > Patches 1 to 4 and the framework for using a configuration file to setup
> > the
> > buffer allocation parameters
> >
> > Patch 5 to 7 improve the buffer handling dealing with dropped frames,
> > allowing
> > us to use fewer internal buffers.
> >
> 
> I should add that this particular change requires a firmware update to work
> correctly.
> If you want to test this out, it's best to run "rpi-update" to get the
> latest firmware.

Mostly from curiosity - is there a 'standard' way for the Raspberry Pi
pipeline handler to interogate or query the rpi firmware to do a check
in case it is not the correct version?

--
Kieran
 
> Naush
> 
> 
> >
> > Patch 8 adds a parameter to modify drop frame behavior - not related to
> > this
> > problem, but a nice to have.
> >
> > Patches 9 and 10 add configuration files for minimal memory configuration,
> > and
> > another for optimal memory usage for libcamera-apps and picamera2.
> >
> > I've considered how these various performance tuning parameters could be
> > made
> > part of the core StreamConfiguration structure or even as a Control.
> > However,
> > they are entirely platform specific, and I am struggling to come up with a
> > clean
> > way to put this into libcamera core with polluting it with various
> > platform specific
> > parameters.  If others have any ideas or suggestions on using alternate
> > means
> > of passing platform specific parameters into the pipeline handler, please
> > do
> > comment on this review.
> >
> > Thanks,
> > Naush
> >
> > Naushir Patuck (10):
> >   libcamera: pipeline: Add a platform configuration file helper
> >   pipeline: raspberrypi: Add a pipeline config structure
> >   pipeline: raspberrypi: Split out ISP Output0 buffer allocation
> >   pipeline: raspberrypi: Read config parameters from a file
> >   pipeline: raspberrypi: Disable StreamOn for ISP Output0/1 when
> >     dropping frames
> >   pipeline: raspberrypi: Reorder startup drop frame initialisation
> >   pipeline: raspberrypi: Add a parameter to disable startup drop frames
> >   pipeline: raspberrypi: Allow pipeline handler to always use the newest
> >     frame
> >   pipeline: raspberrypi: Add minimal memory usage config file
> >   pipeline: raspberrypi: Add libcamera-apps and picamera2 config file
> >
> >  include/libcamera/internal/pipeline_handler.h |   2 +
> >  .../pipeline/raspberrypi/data/default.json    |  28 +++
> >  .../pipeline/raspberrypi/data/meson.build     |  10 +
> >  .../raspberrypi/data/minimal_mem.json         |  28 +++
> >  .../pipeline/raspberrypi/data/rpi_apps.json   |  28 +++
> >  .../pipeline/raspberrypi/meson.build          |   2 +
> >  .../pipeline/raspberrypi/raspberrypi.cpp      | 215 +++++++++++++++---
> >  src/libcamera/pipeline_handler.cpp            |  57 +++++
> >  8 files changed, 339 insertions(+), 31 deletions(-)
> >  create mode 100644 src/libcamera/pipeline/raspberrypi/data/default.json
> >  create mode 100644 src/libcamera/pipeline/raspberrypi/data/meson.build
> >  create mode 100644
> > src/libcamera/pipeline/raspberrypi/data/minimal_mem.json
> >  create mode 100644 src/libcamera/pipeline/raspberrypi/data/rpi_apps.json
> >
> > --
> > 2.25.1
> >
> >