<div dir="ltr"><div><br></div><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, 14 Oct 2022 at 14:18, Naushir Patuck <<a href="mailto:naush@raspberrypi.com">naush@raspberrypi.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi,<br>
<br>
One big issue that is plaguing libcamera users on Raspberry Pi is memory usage.<br>
All our frame buffers have to be allocated in CMA space because of hardware<br>
restrictions. Unfortunately CMA space is very limited (particularly on the<br>
Raspberry Pi Zero), and is prone to frequent fragmentation as display/rendering<br>
buffers also use this pool. Not to mention that the kernel is also allowed to<br>
allocate in the CMA pool for its own needs. As a result, some of our libcamera<br>
applications intermittently fail to run because of buffer allocation failure,<br>
and some use-cases (e.g. > 20MPix captures) will simply not be able to run on<br>
some platforms at all.<br>
<br>
This patch series attempts to try and fix this by easing the memory usage of<br>
the Raspberry Pi pipeline handler. It does this by using a platform configuration<br>
file which (amongst other things) instructs the pipeline handler on how many<br>
internal buffers it should allocate. There may be a performance penality (e.g<br>
dropped frames, or longer time taken for initial AE/AWB convergence) that have<br>
been documented. But if the application can guarantee certain behavior, we can<br>
ensure the pipeline can still process requests with the same performance as we<br>
have today. The Raspberry Pi libcamera-apps and picamera2 framework already use<br>
this behavior, so the change should be transparent to applications built on<br>
these, apart from using significantly less memory.<br>
<br>
Patches 1 to 4 and the framework for using a configuration file to setup the<br>
buffer allocation parameters<br>
<br>
Patch 5 to 7 improve the buffer handling dealing with dropped frames, allowing<br>
us to use fewer internal buffers.<br></blockquote><div><br></div><div>I should add that this particular change requires a firmware update to work correctly.</div><div>If you want to test this out, it's best to run "rpi-update" to get the latest firmware.</div><div><br></div><div>Naush</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
Patch 8 adds a parameter to modify drop frame behavior - not related to this<br>
problem, but a nice to have.<br>
<br>
Patches 9 and 10 add configuration files for minimal memory configuration, and<br>
another for optimal memory usage for libcamera-apps and picamera2.<br>
<br>
I've considered how these various performance tuning parameters could be made<br>
part of the core StreamConfiguration structure or even as a Control. However,<br>
they are entirely platform specific, and I am struggling to come up with a clean<br>
way to put this into libcamera core with polluting it with various platform specific<br>
parameters. If others have any ideas or suggestions on using alternate means<br>
of passing platform specific parameters into the pipeline handler, please do<br>
comment on this review.<br>
<br>
Thanks,<br>
Naush<br>
<br>
Naushir Patuck (10):<br>
libcamera: pipeline: Add a platform configuration file helper<br>
pipeline: raspberrypi: Add a pipeline config structure<br>
pipeline: raspberrypi: Split out ISP Output0 buffer allocation<br>
pipeline: raspberrypi: Read config parameters from a file<br>
pipeline: raspberrypi: Disable StreamOn for ISP Output0/1 when<br>
dropping frames<br>
pipeline: raspberrypi: Reorder startup drop frame initialisation<br>
pipeline: raspberrypi: Add a parameter to disable startup drop frames<br>
pipeline: raspberrypi: Allow pipeline handler to always use the newest<br>
frame<br>
pipeline: raspberrypi: Add minimal memory usage config file<br>
pipeline: raspberrypi: Add libcamera-apps and picamera2 config file<br>
<br>
include/libcamera/internal/pipeline_handler.h | 2 +<br>
.../pipeline/raspberrypi/data/default.json | 28 +++<br>
.../pipeline/raspberrypi/data/meson.build | 10 +<br>
.../raspberrypi/data/minimal_mem.json | 28 +++<br>
.../pipeline/raspberrypi/data/rpi_apps.json | 28 +++<br>
.../pipeline/raspberrypi/meson.build | 2 +<br>
.../pipeline/raspberrypi/raspberrypi.cpp | 215 +++++++++++++++---<br>
src/libcamera/pipeline_handler.cpp | 57 +++++<br>
8 files changed, 339 insertions(+), 31 deletions(-)<br>
create mode 100644 src/libcamera/pipeline/raspberrypi/data/default.json<br>
create mode 100644 src/libcamera/pipeline/raspberrypi/data/meson.build<br>
create mode 100644 src/libcamera/pipeline/raspberrypi/data/minimal_mem.json<br>
create mode 100644 src/libcamera/pipeline/raspberrypi/data/rpi_apps.json<br>
<br>
-- <br>
2.25.1<br>
<br>
</blockquote></div></div>