[libcamera-devel] [PATCH v1 00/10] Raspberry Pi: Platform configuration and buffer allocation improvements

[Laurent Pinchart]

Hi Naush,

On Mon, Oct 17, 2022 at 10:10:43AM +0100, Naushir Patuck via libcamera-devel wrote:
> On Mon, 17 Oct 2022 at 10:00, Kieran Bingham wrote:
> > Quoting Naushir Patuck via libcamera-devel (2022-10-14 15:55:46)
> > > On Fri, 14 Oct 2022 at 14:18, Naushir Patuck wrote:
> > > > Hi,
> > > >
> > > > One big issue that is plaguing libcamera users on Raspberry Pi is memory usage.
> > > > All our frame buffers have to be allocated in CMA space because of hardware
> > > > restrictions.  Unfortunately CMA space is very limited (particularly on the
> > > > Raspberry Pi Zero), and is prone to frequent fragmentation as display/rendering
> > > > buffers also use this pool. Not to mention that the kernel is also allowed to
> > > > allocate in the CMA pool for its own needs. As a result, some of our libcamera
> > > > applications intermittently fail to run because of buffer allocation failure,
> > > > and some use-cases (e.g. > 20MPix captures) will simply not be able to run on
> > > > some platforms at all.
> > > >
> > > > This patch series attempts to try and fix this by easing the memory usage of
> > > > the Raspberry Pi pipeline handler. It does this by using a platform configuration
> > > > file which (amongst other things) instructs the pipeline handler on how many
> > > > internal buffers it should allocate. There may be a performance penality (e.g
> > > > dropped frames, or longer time taken for initial AE/AWB convergence) that have
> > > > been documented.  But if the application can guarantee certain behavior, we can
> > > > ensure the pipeline can still process requests with the same performance as we
> > > > have today. The Raspberry Pi libcamera-apps and picamera2 framework already use
> > > > this behavior, so the change should be transparent to applications built on
> > > > these, apart from using significantly less memory.
> > > >
> > > > Patches 1 to 4 and the framework for using a configuration file to setup the
> > > > buffer allocation parameters
> > > >
> > > > Patch 5 to 7 improve the buffer handling dealing with dropped frames, allowing
> > > > us to use fewer internal buffers.
> > >
> > > I should add that this particular change requires a firmware update to work correctly.
> > > If you want to test this out, it's best to run "rpi-update" to get the
> > > latest firmware.
> >
> > Mostly from curiosity - is there a 'standard' way for the Raspberry Pi
> > pipeline handler to interogate or query the rpi firmware to do a check
> > in case it is not the correct version?
> 
> Unfortunately not in an easy way.
> We could run a shell command "vcgencmd version" to get a sha
> hash, but I would find that really ugly to do in the pipeline hanalder.

Not great indeed.

Is this something we can improve going forward ? I'm thinking of a
message to query the vc4 firmware version at runtime, that the ISP
driver would use to retrieve the version. We could then expose it to
userspace through the ISP video nodes.

Is there a "traditional" version number for the firmware, or just a git
sha1 ?

> > > > Patch 8 adds a parameter to modify drop frame behavior - not related to this
> > > > problem, but a nice to have.
> > > >
> > > > Patches 9 and 10 add configuration files for minimal memory configuration, and
> > > > another for optimal memory usage for libcamera-apps and picamera2.
> > > >
> > > > I've considered how these various performance tuning parameters could be made
> > > > part of the core StreamConfiguration structure or even as a Control. However,
> > > > they are entirely platform specific, and I am struggling to come up with a clean
> > > > way to put this into libcamera core with polluting it with various platform specific
> > > > parameters.  If others have any ideas or suggestions on using alternate means
> > > > of passing platform specific parameters into the pipeline handler, please do
> > > > comment on this review.
> > > >
> > > > Thanks,
> > > > Naush
> > > >
> > > > Naushir Patuck (10):
> > > >   libcamera: pipeline: Add a platform configuration file helper
> > > >   pipeline: raspberrypi: Add a pipeline config structure
> > > >   pipeline: raspberrypi: Split out ISP Output0 buffer allocation
> > > >   pipeline: raspberrypi: Read config parameters from a file
> > > >   pipeline: raspberrypi: Disable StreamOn for ISP Output0/1 when
> > > >     dropping frames
> > > >   pipeline: raspberrypi: Reorder startup drop frame initialisation
> > > >   pipeline: raspberrypi: Add a parameter to disable startup drop frames
> > > >   pipeline: raspberrypi: Allow pipeline handler to always use the newest
> > > >     frame
> > > >   pipeline: raspberrypi: Add minimal memory usage config file
> > > >   pipeline: raspberrypi: Add libcamera-apps and picamera2 config file
> > > >
> > > >  include/libcamera/internal/pipeline_handler.h |   2 +
> > > >  .../pipeline/raspberrypi/data/default.json    |  28 +++
> > > >  .../pipeline/raspberrypi/data/meson.build     |  10 +
> > > >  .../raspberrypi/data/minimal_mem.json         |  28 +++
> > > >  .../pipeline/raspberrypi/data/rpi_apps.json   |  28 +++
> > > >  .../pipeline/raspberrypi/meson.build          |   2 +
> > > >  .../pipeline/raspberrypi/raspberrypi.cpp      | 215 +++++++++++++++---
> > > >  src/libcamera/pipeline_handler.cpp            |  57 +++++
> > > >  8 files changed, 339 insertions(+), 31 deletions(-)
> > > >  create mode 100644 src/libcamera/pipeline/raspberrypi/data/default.json
> > > >  create mode 100644 src/libcamera/pipeline/raspberrypi/data/meson.build
> > > >  create mode 100644 src/libcamera/pipeline/raspberrypi/data/minimal_mem.json
> > > >  create mode 100644 src/libcamera/pipeline/raspberrypi/data/rpi_apps.json

-- 
Regards,

Laurent Pinchart