[PATCH v3 5/8] libcamera: simple: Consider raw output configurations
Kieran Bingham
kieran.bingham at ideasonboard.com
Thu Mar 20 23:43:59 CET 2025
Quoting Milan Zamazal (2025-03-05 19:26:12)
> When generating simple pipeline configuration, raw output configurations
> are generated but later filtered out. Let's include processed and/or
> raw output configurations depending on the requested stream roles.
>
> Raw and processed formats are handled separately. The intention is that
> in case both raw and processed formats are requested then the raw
> formats should be left intact to the extent possible and not be
> influenced by the processed formats (implying that raw parameters
> compatible with the processed output requirements must be requested by
> the application).
>
> Raw output configurations are marked by setting their colorSpace to
> ColorSpace::Raw, which is the only place there suitable for the purpose.
>
> This is another preparatory patch without making raw outputs working.
I've identified two interesting effects which I've bisected to this
patch:
First:
Without this patch on x13s:
[libcamera] kbingham at charm:~/iob/libcamera/build/gcc$ cam -c1 --metadata -C10
[1:44:39.464209549] [37696] INFO IPAManager ipa_manager.cpp:137 libcamera is not installed. Adding '/home/kbingham/iob/libcamera/build/gcc/src/ipa' to the IPA search path
[1:44:39.471858239] [37696] INFO Camera camera_manager.cpp:327 libcamera v0.4.0+149-628e8c80
[1:44:39.497573824] [37697] WARN CameraSensor camera_sensor_legacy.cpp:616 'ov5675 24-0010': Rotation control not available, default to 0 degrees
[1:44:39.502155101] [37697] INFO IPAProxy ipa_proxy.cpp:151 libcamera is not installed. Loading IPA configuration from '/home/kbingham/iob/libcamera/src/ipa/simple/data'
[1:44:39.502180672] [37697] WARN IPAProxy ipa_proxy.cpp:177 Configuration file 'ov5675.yaml' not found for IPA module 'simple', falling back to 'uncalibrated.yaml'
[1:44:39.502468206] [37697] INFO IPAProxy ipa_proxy.cpp:151 libcamera is not installed. Loading IPA configuration from '/home/kbingham/iob/libcamera/src/ipa/simple/data'
[1:44:39.512547578] [37697] INFO Pipeline pipeline_handler.cpp:618 libcamera is not installed. Loading platform configuration file from '/home/kbingham/iob/libcamera/src/libcamera/pipeline/virtual/data/virtual.yaml'
Using camera /base/soc at 0/cci at ac4c000/i2c-bus at 1/camera at 10 as cam0
[1:44:39.514127115] [37696] INFO Camera camera.cpp:1203 configuring streams: (0) 2584x1944-ABGR8888
[1:44:39.520536247] [37697] WARN CameraSensor camera_sensor_legacy.cpp:501 'ov5675 24-0010': No sensor delays found in static properties. Assuming unverified defaults.
[1:44:39.520899922] [37697] INFO IPASoft soft_simple.cpp:251 IPASoft: Exposure 4-2016, gain 1-15.9922 (0.149922)
cam0: Capture 10 frames
6279.664529 (0.00 fps) cam0-stream0 seq: 000000 bytesused: 20093184
SensorTimestamp = 6279664529000
6279.697607 (30.23 fps) cam0-stream0 seq: 000001 bytesused: 20093184
SensorTimestamp = 6279697607000
6279.731336 (29.65 fps) cam0-stream0 seq: 000002 bytesused: 20093184
SensorTimestamp = 6279731336000
6279.831491 (9.98 fps) cam0-stream0 seq: 000003 bytesused: 20093184
SensorTimestamp = 6279831491000
6279.898368 (14.95 fps) cam0-stream0 seq: 000004 bytesused: 20093184
SensorTimestamp = 6279898368000
6279.965133 (14.98 fps) cam0-stream0 seq: 000005 bytesused: 20093184
SensorTimestamp = 6279965133000
6280.031911 (14.97 fps) cam0-stream0 seq: 000006 bytesused: 20093184
SensorTimestamp = 6280031911000
6280.098577 (15.00 fps) cam0-stream0 seq: 000007 bytesused: 20093184
SensorTimestamp = 6280098577000
6280.165320 (14.98 fps) cam0-stream0 seq: 000008 bytesused: 20093184
SensorTimestamp = 6280165320000
6280.232225 (14.95 fps) cam0-stream0 seq: 000009 bytesused: 20093184
SensorTimestamp = 6280232225000
(Note the 15 FPS), and pipewire/gstreamer/cheese are all 'streaming'...
But with this patch the rate jumps to a more reasonable 30 FPS seen by
qcam and cam:
[libcamera] kbingham at charm:~/iob/libcamera/build/gcc$ cam -c1 --metadata -C10
[1:46:59.040578435] [38346] INFO IPAManager ipa_manager.cpp:137 libcamera is not installed. Adding '/home/kbingham/iob/libcamera/build/gcc/src/ipa' to the IPA search path
[1:46:59.048420757] [38346] INFO Camera camera_manager.cpp:327 libcamera v0.4.0+150-d0e1caa0
[1:46:59.074259713] [38347] WARN CameraSensor camera_sensor_legacy.cpp:616 'ov5675 24-0010': Rotation control not available, default to 0 degrees
[1:46:59.078796773] [38347] INFO IPAProxy ipa_proxy.cpp:151 libcamera is not installed. Loading IPA configuration from '/home/kbingham/iob/libcamera/src/ipa/simple/data'
[1:46:59.078822605] [38347] WARN IPAProxy ipa_proxy.cpp:177 Configuration file 'ov5675.yaml' not found for IPA module 'simple', falling back to 'uncalibrated.yaml'
[1:46:59.079110243] [38347] INFO IPAProxy ipa_proxy.cpp:151 libcamera is not installed. Loading IPA configuration from '/home/kbingham/iob/libcamera/src/ipa/simple/data'
[1:46:59.087987917] [38347] INFO Pipeline pipeline_handler.cpp:618 libcamera is not installed. Loading platform configuration file from '/home/kbingham/iob/libcamera/src/libcamera/pipeline/virtual/data/virtual.yaml'
Using camera /base/soc at 0/cci at ac4c000/i2c-bus at 1/camera at 10 as cam0
[1:46:59.089522196] [38346] INFO Camera camera.cpp:1203 configuring streams: (0) 1288x972-ABGR8888
[1:46:59.096422962] [38347] WARN CameraSensor camera_sensor_legacy.cpp:501 'ov5675 24-0010': No sensor delays found in static properties. Assuming unverified defaults.
[1:46:59.096786533] [38347] INFO IPASoft soft_simple.cpp:251 IPASoft: Exposure 4-2016, gain 1-15.9922 (0.149922)
cam0: Capture 10 frames
6419.234779 (0.00 fps) cam0-stream0 seq: 000000 bytesused: 5007744
SensorTimestamp = 6419234779000
6419.268213 (29.91 fps) cam0-stream0 seq: 000001 bytesused: 5007744
SensorTimestamp = 6419268213000
6419.301359 (30.17 fps) cam0-stream0 seq: 000002 bytesused: 5007744
SensorTimestamp = 6419301359000
6419.334744 (29.95 fps) cam0-stream0 seq: 000003 bytesused: 5007744
SensorTimestamp = 6419334744000
6419.368129 (29.95 fps) cam0-stream0 seq: 000004 bytesused: 5007744
SensorTimestamp = 6419368129000
6419.401516 (29.95 fps) cam0-stream0 seq: 000005 bytesused: 5007744
SensorTimestamp = 6419401516000
6419.434902 (29.95 fps) cam0-stream0 seq: 000006 bytesused: 5007744
SensorTimestamp = 6419434902000
6419.468289 (29.95 fps) cam0-stream0 seq: 000007 bytesused: 5007744
SensorTimestamp = 6419468289000
6419.501673 (29.95 fps) cam0-stream0 seq: 000008 bytesused: 5007744
SensorTimestamp = 6419501673000
6419.535058 (29.95 fps) cam0-stream0 seq: 000009 bytesused: 5007744
SensorTimestamp = 6419535058000
But alas I note that now gstreamer (and cheese, and pipewire) now fail:
[libcamera] kbingham at charm:~/iob/libcamera/build/gcc$ gst-launch-1.0 libcamerasrc ! autovideosink
Setting pipeline to PAUSED ...
[1:46:09.897725501] [38325] INFO IPAManager ipa_manager.cpp:137 libcamera is not installed. Adding '/home/kbingham/iob/libcamera/build/gcc/src/ipa' to the IPA search path
[1:46:09.903176776] [38325] INFO Camera camera_manager.cpp:327 libcamera v0.4.0+150-d0e1caa0
[1:46:09.935374613] [38330] WARN CameraSensor camera_sensor_legacy.cpp:616 'ov5675 24-0010': Rotation control not available, default to 0 degrees
[1:46:09.941677813] [38330] INFO IPAProxy ipa_proxy.cpp:151 libcamera is not installed. Loading IPA configuration from '/home/kbingham/iob/libcamera/src/ipa/simple/data'
[1:46:09.941734112] [38330] WARN IPAProxy ipa_proxy.cpp:177 Configuration file 'ov5675.yaml' not found for IPA module 'simple', falling back to 'uncalibrated.yaml'
[1:46:09.942172574] [38330] INFO IPAProxy ipa_proxy.cpp:151 libcamera is not installed. Loading IPA configuration from '/home/kbingham/iob/libcamera/src/ipa/simple/data'
[1:46:09.949343846] [38330] INFO Pipeline pipeline_handler.cpp:618 libcamera is not installed. Loading platform configuration file from '/home/kbingham/iob/libcamera/src/libcamera/pipeline/virtual/data/virtual.yaml'
Pipeline is live and does not need PREROLL ...
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
[1:46:09.950895937] [38331] ERROR Stream stream.cpp:215 Range format is ambiguous
[1:46:09.950940309] [38331] ERROR Stream stream.cpp:215 Range format is ambiguous
[1:46:09.950962495] [38331] ERROR Stream stream.cpp:215 Range format is ambiguous
[1:46:09.950982129] [38331] ERROR Stream stream.cpp:215 Range format is ambiguous
[1:46:09.951001711] [38331] ERROR Stream stream.cpp:215 Range format is ambiguous
[1:46:09.951020564] [38331] ERROR Stream stream.cpp:215 Range format is ambiguous
ERROR: from element /GstPipeline:pipeline0/GstLibcameraSrc:libcamerasrc0: Internal data stream error.
Additional debug info:
../../src/gstreamer/gstlibcamerasrc.cpp(662): gst_libcamera_src_task_enter (): /GstPipeline:pipeline0/GstLibcameraSrc:libcamerasrc0:
streaming stopped, reason not-negotiated (-4)
Execution ended after 0:00:00.000438306
Setting pipeline to NULL ...
Freeing pipeline ...
Debugging a little, at stream.ccp:215
diff --git a/src/libcamera/stream.cpp b/src/libcamera/stream.cpp
index 978d72752b00..22f348c2df87 100644
--- a/src/libcamera/stream.cpp
+++ b/src/libcamera/stream.cpp
@@ -213,6 +213,8 @@ std::vector<Size> StreamFormats::sizes(const PixelFormat &pixelformat) const
if (!discrete) {
if (ranges.size() != 1) {
LOG(Stream, Error) << "Range format is ambiguous";
+ for (auto range: ranges)
+ LOG(Stream, Error) << range;
return {};
}
Reports:
[1:51:01.855863590] [40484] ERROR Stream stream.cpp:215 Range format is ambiguous
[1:51:01.855901452] [40484] ERROR Stream stream.cpp:217 (4x2)-(1288x972)/(+4,+2)
[1:51:01.855920305] [40484] ERROR Stream stream.cpp:217 (4x2)-(2584x1944)/(+4,+2)
So we're somehow setting up the stream capabilities incorrectly here
which breaks things... (note that ranges.size is 2, not 1?)
Perhaps we're incorrectly adding the raw stream sizes to the processed stream
sizes ?
--
Kieran
>
> Signed-off-by: Milan Zamazal <mzamazal at redhat.com>
> ---
> src/libcamera/pipeline/simple/simple.cpp | 72 ++++++++++++++++--------
> 1 file changed, 49 insertions(+), 23 deletions(-)
>
> diff --git a/src/libcamera/pipeline/simple/simple.cpp b/src/libcamera/pipeline/simple/simple.cpp
> index 0c6e0f44..92b9608b 100644
> --- a/src/libcamera/pipeline/simple/simple.cpp
> +++ b/src/libcamera/pipeline/simple/simple.cpp
> @@ -25,6 +25,7 @@
> #include <libcamera/base/log.h>
>
> #include <libcamera/camera.h>
> +#include <libcamera/color_space.h>
> #include <libcamera/control_ids.h>
> #include <libcamera/pixel_format.h>
> #include <libcamera/request.h>
> @@ -389,6 +390,8 @@ private:
> const MediaPad *acquirePipeline(SimpleCameraData *data);
> void releasePipeline(SimpleCameraData *data);
>
> + void setUpFormatSizes(std::map<PixelFormat, std::vector<SizeRange>> formats);
> +
> std::map<const MediaEntity *, EntityData> entities_;
>
> MediaDevice *converter_;
> @@ -1191,15 +1194,56 @@ SimplePipelineHandler::generateConfiguration(Camera *camera, Span<const StreamRo
> data->processedRequested_ = true;
> }
>
> - /* Create the formats map. */
> - std::map<PixelFormat, std::vector<SizeRange>> formats;
> + /* Create the formats maps. */
> + std::map<PixelFormat, std::vector<SizeRange>> processedFormats;
> + std::map<PixelFormat, std::vector<SizeRange>> rawFormats;
>
> for (const SimpleCameraData::Configuration &cfg : data->configs_)
> - if (!cfg.raw)
> - for (PixelFormat format : cfg.outputFormats)
> - formats[format].push_back(cfg.outputSizes);
> + for (PixelFormat format : cfg.outputFormats)
> + (cfg.raw ? rawFormats : processedFormats)[format].push_back(cfg.outputSizes);
> +
> + if (data->processedRequested_ && processedFormats.empty()) {
> + LOG(SimplePipeline, Error)
> + << "Processed stream requsted but no corresponding output configuration found";
> + return nullptr;
> + }
> + if (data->rawRequested_ && rawFormats.empty()) {
> + LOG(SimplePipeline, Error)
> + << "Raw stream requsted but no corresponding output configuration found";
> + return nullptr;
> + }
> +
> + setUpFormatSizes(processedFormats);
> + setUpFormatSizes(rawFormats);
> +
> + /*
> + * Create the stream configurations. Take the first entry in the formats
> + * map as the default, for lack of a better option.
> + *
> + * \todo Implement a better way to pick the default format
> + */
> + for (StreamRole role : roles) {
> + bool raw = (role == StreamRole::Raw);
> + auto formats = (raw ? rawFormats : processedFormats);
> + StreamConfiguration cfg{ StreamFormats{ formats } };
> + cfg.pixelFormat = formats.begin()->first;
> + cfg.size = formats.begin()->second[0].max;
> + if (raw)
> + cfg.colorSpace = ColorSpace::Raw;
> +
> + config->addConfiguration(cfg);
> + }
> +
> + config->validate();
> +
> + return config;
> +}
>
> +void SimplePipelineHandler::setUpFormatSizes(
> + std::map<PixelFormat, std::vector<SizeRange>> formats)
> +{
> /* Sort the sizes and merge any consecutive overlapping ranges. */
> +
> for (auto &[format, sizes] : formats) {
> std::sort(sizes.begin(), sizes.end(),
> [](SizeRange &a, SizeRange &b) {
> @@ -1219,24 +1263,6 @@ SimplePipelineHandler::generateConfiguration(Camera *camera, Span<const StreamRo
>
> sizes.erase(++cur, sizes.end());
> }
> -
> - /*
> - * Create the stream configurations. Take the first entry in the formats
> - * map as the default, for lack of a better option.
> - *
> - * \todo Implement a better way to pick the default format
> - */
> - for ([[maybe_unused]] StreamRole role : roles) {
> - StreamConfiguration cfg{ StreamFormats{ formats } };
> - cfg.pixelFormat = formats.begin()->first;
> - cfg.size = formats.begin()->second[0].max;
> -
> - config->addConfiguration(cfg);
> - }
> -
> - config->validate();
> -
> - return config;
> }
>
> int SimplePipelineHandler::configure(Camera *camera, CameraConfiguration *c)
> --
> 2.48.1
>
More information about the libcamera-devel
mailing list