[libcamera-devel] [PATCH v1 00/10] Raspberry Pi: Platform configuration and buffer allocation improvements

[Naushir Patuck]

On Mon, 17 Oct 2022 at 10:00, Kieran Bingham <
kieran.bingham at ideasonboard.com> wrote:

> Quoting Naushir Patuck via libcamera-devel (2022-10-14 15:55:46)
> > On Fri, 14 Oct 2022 at 14:18, Naushir Patuck <naush at raspberrypi.com>
> wrote:
> >
> > > Hi,
> > >
> > > One big issue that is plaguing libcamera users on Raspberry Pi is
> memory
> > > usage.
> > > All our frame buffers have to be allocated in CMA space because of
> hardware
> > > restrictions.  Unfortunately CMA space is very limited (particularly
> on the
> > > Raspberry Pi Zero), and is prone to frequent fragmentation as
> > > display/rendering
> > > buffers also use this pool. Not to mention that the kernel is also
> allowed
> > > to
> > > allocate in the CMA pool for its own needs. As a result, some of our
> > > libcamera
> > > applications intermittently fail to run because of buffer allocation
> > > failure,
> > > and some use-cases (e.g. > 20MPix captures) will simply not be able to
> run
> > > on
> > > some platforms at all.
> > >
> > > This patch series attempts to try and fix this by easing the memory
> usage
> > > of
> > > the Raspberry Pi pipeline handler. It does this by using a platform
> > > configuration
> > > file which (amongst other things) instructs the pipeline handler on how
> > > many
> > > internal buffers it should allocate. There may be a performance
> penality
> > > (e.g
> > > dropped frames, or longer time taken for initial AE/AWB convergence)
> that
> > > have
> > > been documented.  But if the application can guarantee certain
> behavior,
> > > we can
> > > ensure the pipeline can still process requests with the same
> performance
> > > as we
> > > have today. The Raspberry Pi libcamera-apps and picamera2 framework
> > > already use
> > > this behavior, so the change should be transparent to applications
> built on
> > > these, apart from using significantly less memory.
> > >
> > > Patches 1 to 4 and the framework for using a configuration file to
> setup
> > > the
> > > buffer allocation parameters
> > >
> > > Patch 5 to 7 improve the buffer handling dealing with dropped frames,
> > > allowing
> > > us to use fewer internal buffers.
> > >
> >
> > I should add that this particular change requires a firmware update to
> work
> > correctly.
> > If you want to test this out, it's best to run "rpi-update" to get the
> > latest firmware.
>
> Mostly from curiosity - is there a 'standard' way for the Raspberry Pi
> pipeline handler to interogate or query the rpi firmware to do a check
> in case it is not the correct version?
>

Unfortunately not in an easy way.
We could run a shell command "vcgencmd version" to get a sha
hash, but I would find that really ugly to do in the pipeline hanalder.




>
> --
> Kieran
>
> > Naush
> >
> >
> > >
> > > Patch 8 adds a parameter to modify drop frame behavior - not related to
> > > this
> > > problem, but a nice to have.
> > >
> > > Patches 9 and 10 add configuration files for minimal memory
> configuration,
> > > and
> > > another for optimal memory usage for libcamera-apps and picamera2.
> > >
> > > I've considered how these various performance tuning parameters could
> be
> > > made
> > > part of the core StreamConfiguration structure or even as a Control.
> > > However,
> > > they are entirely platform specific, and I am struggling to come up
> with a
> > > clean
> > > way to put this into libcamera core with polluting it with various
> > > platform specific
> > > parameters.  If others have any ideas or suggestions on using alternate
> > > means
> > > of passing platform specific parameters into the pipeline handler,
> please
> > > do
> > > comment on this review.
> > >
> > > Thanks,
> > > Naush
> > >
> > > Naushir Patuck (10):
> > >   libcamera: pipeline: Add a platform configuration file helper
> > >   pipeline: raspberrypi: Add a pipeline config structure
> > >   pipeline: raspberrypi: Split out ISP Output0 buffer allocation
> > >   pipeline: raspberrypi: Read config parameters from a file
> > >   pipeline: raspberrypi: Disable StreamOn for ISP Output0/1 when
> > >     dropping frames
> > >   pipeline: raspberrypi: Reorder startup drop frame initialisation
> > >   pipeline: raspberrypi: Add a parameter to disable startup drop frames
> > >   pipeline: raspberrypi: Allow pipeline handler to always use the
> newest
> > >     frame
> > >   pipeline: raspberrypi: Add minimal memory usage config file
> > >   pipeline: raspberrypi: Add libcamera-apps and picamera2 config file
> > >
> > >  include/libcamera/internal/pipeline_handler.h |   2 +
> > >  .../pipeline/raspberrypi/data/default.json    |  28 +++
> > >  .../pipeline/raspberrypi/data/meson.build     |  10 +
> > >  .../raspberrypi/data/minimal_mem.json         |  28 +++
> > >  .../pipeline/raspberrypi/data/rpi_apps.json   |  28 +++
> > >  .../pipeline/raspberrypi/meson.build          |   2 +
> > >  .../pipeline/raspberrypi/raspberrypi.cpp      | 215 +++++++++++++++---
> > >  src/libcamera/pipeline_handler.cpp            |  57 +++++
> > >  8 files changed, 339 insertions(+), 31 deletions(-)
> > >  create mode 100644
> src/libcamera/pipeline/raspberrypi/data/default.json
> > >  create mode 100644 src/libcamera/pipeline/raspberrypi/data/meson.build
> > >  create mode 100644
> > > src/libcamera/pipeline/raspberrypi/data/minimal_mem.json
> > >  create mode 100644
> src/libcamera/pipeline/raspberrypi/data/rpi_apps.json
> > >
> > > --
> > > 2.25.1
> > >
> > >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.libcamera.org/pipermail/libcamera-devel/attachments/20221017/f69459e1/attachment.htm>