[PATCH v3 5/8] libcamera: simple: Consider raw output configurations
Kieran Bingham
kieran.bingham at ideasonboard.com
Wed Apr 2 20:52:53 CEST 2025
Quoting Milan Zamazal (2025-04-02 19:31:02)
> Hi Kieran,
>
> thank you for testing.
>
> Kieran Bingham <kieran.bingham at ideasonboard.com> writes:
>
> > Quoting Milan Zamazal (2025-03-05 19:26:12)
> >> When generating simple pipeline configuration, raw output configurations
> >> are generated but later filtered out. Let's include processed and/or
> >> raw output configurations depending on the requested stream roles.
> >>
> >> Raw and processed formats are handled separately. The intention is that
> >> in case both raw and processed formats are requested then the raw
> >> formats should be left intact to the extent possible and not be
> >> influenced by the processed formats (implying that raw parameters
> >> compatible with the processed output requirements must be requested by
> >> the application).
> >>
> >> Raw output configurations are marked by setting their colorSpace to
> >> ColorSpace::Raw, which is the only place there suitable for the purpose.
> >>
> >> This is another preparatory patch without making raw outputs working.
> >
> > I've identified two interesting effects which I've bisected to this
> > patch:
> >
> > First:
> >
> > Without this patch on x13s:
> >
> > [libcamera] kbingham at charm:~/iob/libcamera/build/gcc$ cam -c1 --metadata -C10
> > [1:44:39.464209549] [37696] INFO IPAManager ipa_manager.cpp:137 libcamera is not installed. Adding '/home/kbingham/iob/libcamera/build/gcc/src/ipa' to the IPA search path
> > [1:44:39.471858239] [37696] INFO Camera camera_manager.cpp:327 libcamera v0.4.0+149-628e8c80
> > [1:44:39.497573824] [37697] WARN CameraSensor camera_sensor_legacy.cpp:616 'ov5675 24-0010': Rotation control not available, default to 0 degrees
> > [1:44:39.502155101] [37697] INFO IPAProxy ipa_proxy.cpp:151 libcamera is not installed. Loading IPA configuration from '/home/kbingham/iob/libcamera/src/ipa/simple/data'
> > [1:44:39.502180672] [37697] WARN IPAProxy ipa_proxy.cpp:177 Configuration file 'ov5675.yaml' not found for IPA module 'simple', falling back to 'uncalibrated.yaml'
> > [1:44:39.502468206] [37697] INFO IPAProxy ipa_proxy.cpp:151 libcamera is not installed. Loading IPA configuration from '/home/kbingham/iob/libcamera/src/ipa/simple/data'
> > [1:44:39.512547578] [37697] INFO Pipeline pipeline_handler.cpp:618 libcamera is not installed. Loading platform configuration file from '/home/kbingham/iob/libcamera/src/libcamera/pipeline/virtual/data/virtual.yaml'
> > Using camera /base/soc at 0/cci at ac4c000/i2c-bus at 1/camera at 10 as cam0
> > [1:44:39.514127115] [37696] INFO Camera camera.cpp:1203 configuring streams: (0) 2584x1944-ABGR8888
> > [1:44:39.520536247] [37697] WARN CameraSensor camera_sensor_legacy.cpp:501 'ov5675 24-0010': No sensor delays found in static properties. Assuming unverified defaults.
> > [1:44:39.520899922] [37697] INFO IPASoft soft_simple.cpp:251 IPASoft: Exposure 4-2016, gain 1-15.9922 (0.149922)
> > cam0: Capture 10 frames
> > 6279.664529 (0.00 fps) cam0-stream0 seq: 000000 bytesused: 20093184
> > SensorTimestamp = 6279664529000
> > 6279.697607 (30.23 fps) cam0-stream0 seq: 000001 bytesused: 20093184
> > SensorTimestamp = 6279697607000
> > 6279.731336 (29.65 fps) cam0-stream0 seq: 000002 bytesused: 20093184
> > SensorTimestamp = 6279731336000
> > 6279.831491 (9.98 fps) cam0-stream0 seq: 000003 bytesused: 20093184
> > SensorTimestamp = 6279831491000
> > 6279.898368 (14.95 fps) cam0-stream0 seq: 000004 bytesused: 20093184
> > SensorTimestamp = 6279898368000
> > 6279.965133 (14.98 fps) cam0-stream0 seq: 000005 bytesused: 20093184
> > SensorTimestamp = 6279965133000
> > 6280.031911 (14.97 fps) cam0-stream0 seq: 000006 bytesused: 20093184
> > SensorTimestamp = 6280031911000
> > 6280.098577 (15.00 fps) cam0-stream0 seq: 000007 bytesused: 20093184
> > SensorTimestamp = 6280098577000
> > 6280.165320 (14.98 fps) cam0-stream0 seq: 000008 bytesused: 20093184
> > SensorTimestamp = 6280165320000
> > 6280.232225 (14.95 fps) cam0-stream0 seq: 000009 bytesused: 20093184
> > SensorTimestamp = 6280232225000
> >
> > (Note the 15 FPS), and pipewire/gstreamer/cheese are all 'streaming'...
> >
> >
> > But with this patch the rate jumps to a more reasonable 30 FPS seen by
> > qcam and cam:
>
> Well, clearly because of the lower resolution. I don't have an
> explanation why the patch should make it different.
Aha - I completley missed that! But yes, that's a good reason at least
for the speed difference ;-)
>
> > [libcamera] kbingham at charm:~/iob/libcamera/build/gcc$ cam -c1 --metadata -C10
> > [1:46:59.040578435] [38346] INFO IPAManager ipa_manager.cpp:137 libcamera is not installed. Adding '/home/kbingham/iob/libcamera/build/gcc/src/ipa' to the IPA search path
> > [1:46:59.048420757] [38346] INFO Camera camera_manager.cpp:327 libcamera v0.4.0+150-d0e1caa0
> > [1:46:59.074259713] [38347] WARN CameraSensor camera_sensor_legacy.cpp:616 'ov5675 24-0010': Rotation control not available, default to 0 degrees
> > [1:46:59.078796773] [38347] INFO IPAProxy ipa_proxy.cpp:151 libcamera is not installed. Loading IPA configuration from '/home/kbingham/iob/libcamera/src/ipa/simple/data'
> > [1:46:59.078822605] [38347] WARN IPAProxy ipa_proxy.cpp:177 Configuration file 'ov5675.yaml' not found for IPA module 'simple', falling back to 'uncalibrated.yaml'
> > [1:46:59.079110243] [38347] INFO IPAProxy ipa_proxy.cpp:151 libcamera is not installed. Loading IPA configuration from '/home/kbingham/iob/libcamera/src/ipa/simple/data'
> > [1:46:59.087987917] [38347] INFO Pipeline pipeline_handler.cpp:618 libcamera is not installed. Loading platform configuration file from '/home/kbingham/iob/libcamera/src/libcamera/pipeline/virtual/data/virtual.yaml'
> > Using camera /base/soc at 0/cci at ac4c000/i2c-bus at 1/camera at 10 as cam0
> > [1:46:59.089522196] [38346] INFO Camera camera.cpp:1203 configuring streams: (0) 1288x972-ABGR8888
> > [1:46:59.096422962] [38347] WARN CameraSensor camera_sensor_legacy.cpp:501 'ov5675 24-0010': No sensor delays found in static properties. Assuming unverified defaults.
> > [1:46:59.096786533] [38347] INFO IPASoft soft_simple.cpp:251 IPASoft: Exposure 4-2016, gain 1-15.9922 (0.149922)
> > cam0: Capture 10 frames
> > 6419.234779 (0.00 fps) cam0-stream0 seq: 000000 bytesused: 5007744
> > SensorTimestamp = 6419234779000
> > 6419.268213 (29.91 fps) cam0-stream0 seq: 000001 bytesused: 5007744
> > SensorTimestamp = 6419268213000
> > 6419.301359 (30.17 fps) cam0-stream0 seq: 000002 bytesused: 5007744
> > SensorTimestamp = 6419301359000
> > 6419.334744 (29.95 fps) cam0-stream0 seq: 000003 bytesused: 5007744
> > SensorTimestamp = 6419334744000
> > 6419.368129 (29.95 fps) cam0-stream0 seq: 000004 bytesused: 5007744
> > SensorTimestamp = 6419368129000
> > 6419.401516 (29.95 fps) cam0-stream0 seq: 000005 bytesused: 5007744
> > SensorTimestamp = 6419401516000
> > 6419.434902 (29.95 fps) cam0-stream0 seq: 000006 bytesused: 5007744
> > SensorTimestamp = 6419434902000
> > 6419.468289 (29.95 fps) cam0-stream0 seq: 000007 bytesused: 5007744
> > SensorTimestamp = 6419468289000
> > 6419.501673 (29.95 fps) cam0-stream0 seq: 000008 bytesused: 5007744
> > SensorTimestamp = 6419501673000
> > 6419.535058 (29.95 fps) cam0-stream0 seq: 000009 bytesused: 5007744
> > SensorTimestamp = 6419535058000
> >
> >
> > But alas I note that now gstreamer (and cheese, and pipewire) now fail:
> >
> > [libcamera] kbingham at charm:~/iob/libcamera/build/gcc$ gst-launch-1.0 libcamerasrc ! autovideosink
> > Setting pipeline to PAUSED ...
> > [1:46:09.897725501] [38325] INFO IPAManager ipa_manager.cpp:137 libcamera is not installed. Adding '/home/kbingham/iob/libcamera/build/gcc/src/ipa' to the IPA search path
> > [1:46:09.903176776] [38325] INFO Camera camera_manager.cpp:327 libcamera v0.4.0+150-d0e1caa0
> > [1:46:09.935374613] [38330] WARN CameraSensor camera_sensor_legacy.cpp:616 'ov5675 24-0010': Rotation control not available, default to 0 degrees
> > [1:46:09.941677813] [38330] INFO IPAProxy ipa_proxy.cpp:151 libcamera is not installed. Loading IPA configuration from '/home/kbingham/iob/libcamera/src/ipa/simple/data'
> > [1:46:09.941734112] [38330] WARN IPAProxy ipa_proxy.cpp:177 Configuration file 'ov5675.yaml' not found for IPA module 'simple', falling back to 'uncalibrated.yaml'
> > [1:46:09.942172574] [38330] INFO IPAProxy ipa_proxy.cpp:151 libcamera is not installed. Loading IPA configuration from '/home/kbingham/iob/libcamera/src/ipa/simple/data'
> > [1:46:09.949343846] [38330] INFO Pipeline pipeline_handler.cpp:618 libcamera is not installed. Loading platform configuration file from '/home/kbingham/iob/libcamera/src/libcamera/pipeline/virtual/data/virtual.yaml'
> > Pipeline is live and does not need PREROLL ...
> > Pipeline is PREROLLED ...
> > Setting pipeline to PLAYING ...
> > New clock: GstSystemClock
> > [1:46:09.950895937] [38331] ERROR Stream stream.cpp:215 Range format is ambiguous
> > [1:46:09.950940309] [38331] ERROR Stream stream.cpp:215 Range format is ambiguous
> > [1:46:09.950962495] [38331] ERROR Stream stream.cpp:215 Range format is ambiguous
> > [1:46:09.950982129] [38331] ERROR Stream stream.cpp:215 Range format is ambiguous
> > [1:46:09.951001711] [38331] ERROR Stream stream.cpp:215 Range format is ambiguous
> > [1:46:09.951020564] [38331] ERROR Stream stream.cpp:215 Range format is ambiguous
> > ERROR: from element /GstPipeline:pipeline0/GstLibcameraSrc:libcamerasrc0: Internal data stream error.
> > Additional debug info:
> > ../../src/gstreamer/gstlibcamerasrc.cpp(662): gst_libcamera_src_task_enter (): /GstPipeline:pipeline0/GstLibcameraSrc:libcamerasrc0:
> > streaming stopped, reason not-negotiated (-4)
> > Execution ended after 0:00:00.000438306
> > Setting pipeline to NULL ...
> > Freeing pipeline ...
> >
> > Debugging a little, at stream.ccp:215
> >
> > diff --git a/src/libcamera/stream.cpp b/src/libcamera/stream.cpp
> > index 978d72752b00..22f348c2df87 100644
> > --- a/src/libcamera/stream.cpp
> > +++ b/src/libcamera/stream.cpp
> > @@ -213,6 +213,8 @@ std::vector<Size> StreamFormats::sizes(const PixelFormat &pixelformat) const
> > if (!discrete) {
> > if (ranges.size() != 1) {
> > LOG(Stream, Error) << "Range format is ambiguous";
> > + for (auto range: ranges)
> > + LOG(Stream, Error) << range;
> > return {};
> > }
> >
> > Reports:
> >
> > [1:51:01.855863590] [40484] ERROR Stream stream.cpp:215 Range format is ambiguous
> > [1:51:01.855901452] [40484] ERROR Stream stream.cpp:217 (4x2)-(1288x972)/(+4,+2)
> > [1:51:01.855920305] [40484] ERROR Stream stream.cpp:217 (4x2)-(2584x1944)/(+4,+2)
> >
> > So we're somehow setting up the stream capabilities incorrectly here
> > which breaks things... (note that ranges.size is 2, not 1?)
> >
> > Perhaps we're incorrectly adding the raw stream sizes to the processed stream
> > sizes ?
>
> No, it's a stupid (and not immediately visible) bug: passing formats to
> setUpFormatSizes by value rather than by reference, see below.
>
> > --
> > Kieran
> >
> >
> >>
> >> Signed-off-by: Milan Zamazal <mzamazal at redhat.com>
> >> ---
> >> src/libcamera/pipeline/simple/simple.cpp | 72 ++++++++++++++++--------
> >> 1 file changed, 49 insertions(+), 23 deletions(-)
> >>
> >> diff --git a/src/libcamera/pipeline/simple/simple.cpp b/src/libcamera/pipeline/simple/simple.cpp
> >> index 0c6e0f44..92b9608b 100644
> >> --- a/src/libcamera/pipeline/simple/simple.cpp
> >> +++ b/src/libcamera/pipeline/simple/simple.cpp
> >> @@ -25,6 +25,7 @@
> >> #include <libcamera/base/log.h>
> >>
> >> #include <libcamera/camera.h>
> >> +#include <libcamera/color_space.h>
> >> #include <libcamera/control_ids.h>
> >> #include <libcamera/pixel_format.h>
> >> #include <libcamera/request.h>
> >> @@ -389,6 +390,8 @@ private:
> >> const MediaPad *acquirePipeline(SimpleCameraData *data);
> >> void releasePipeline(SimpleCameraData *data);
> >>
> >> + void setUpFormatSizes(std::map<PixelFormat, std::vector<SizeRange>> formats);
>
> Should be &formats.
Yikes - yeah - I wouldn't have noticed that either ;-)
I'll retest my side this evening too.
--
Kieran
>
> >> +
> >> std::map<const MediaEntity *, EntityData> entities_;
> >>
> >> MediaDevice *converter_;
> >> @@ -1191,15 +1194,56 @@ SimplePipelineHandler::generateConfiguration(Camera *camera, Span<const StreamRo
> >> data->processedRequested_ = true;
> >> }
> >>
> >> - /* Create the formats map. */
> >> - std::map<PixelFormat, std::vector<SizeRange>> formats;
> >> + /* Create the formats maps. */
> >> + std::map<PixelFormat, std::vector<SizeRange>> processedFormats;
> >> + std::map<PixelFormat, std::vector<SizeRange>> rawFormats;
> >>
> >> for (const SimpleCameraData::Configuration &cfg : data->configs_)
> >> - if (!cfg.raw)
> >> - for (PixelFormat format : cfg.outputFormats)
> >> - formats[format].push_back(cfg.outputSizes);
> >> + for (PixelFormat format : cfg.outputFormats)
> >> + (cfg.raw ? rawFormats : processedFormats)[format].push_back(cfg.outputSizes);
> >> +
> >> + if (data->processedRequested_ && processedFormats.empty()) {
> >> + LOG(SimplePipeline, Error)
> >> + << "Processed stream requsted but no corresponding output configuration found";
> >> + return nullptr;
> >> + }
> >> + if (data->rawRequested_ && rawFormats.empty()) {
> >> + LOG(SimplePipeline, Error)
> >> + << "Raw stream requsted but no corresponding output configuration found";
> >> + return nullptr;
> >> + }
> >> +
> >> + setUpFormatSizes(processedFormats);
> >> + setUpFormatSizes(rawFormats);
> >> +
> >> + /*
> >> + * Create the stream configurations. Take the first entry in the formats
> >> + * map as the default, for lack of a better option.
> >> + *
> >> + * \todo Implement a better way to pick the default format
> >> + */
> >> + for (StreamRole role : roles) {
> >> + bool raw = (role == StreamRole::Raw);
> >> + auto formats = (raw ? rawFormats : processedFormats);
> >> + StreamConfiguration cfg{ StreamFormats{ formats } };
> >> + cfg.pixelFormat = formats.begin()->first;
> >> + cfg.size = formats.begin()->second[0].max;
> >> + if (raw)
> >> + cfg.colorSpace = ColorSpace::Raw;
> >> +
> >> + config->addConfiguration(cfg);
> >> + }
> >> +
> >> + config->validate();
> >> +
> >> + return config;
> >> +}
> >>
> >> +void SimplePipelineHandler::setUpFormatSizes(
> >> + std::map<PixelFormat, std::vector<SizeRange>> formats)
>
> &formats
>
> >> +{
> >> /* Sort the sizes and merge any consecutive overlapping ranges. */
> >> +
> >> for (auto &[format, sizes] : formats) {
> >> std::sort(sizes.begin(), sizes.end(),
> >> [](SizeRange &a, SizeRange &b) {
> >> @@ -1219,24 +1263,6 @@ SimplePipelineHandler::generateConfiguration(Camera *camera, Span<const StreamRo
> >>
> >> sizes.erase(++cur, sizes.end());
> >> }
> >> -
> >> - /*
> >> - * Create the stream configurations. Take the first entry in the formats
> >> - * map as the default, for lack of a better option.
> >> - *
> >> - * \todo Implement a better way to pick the default format
> >> - */
> >> - for ([[maybe_unused]] StreamRole role : roles) {
> >> - StreamConfiguration cfg{ StreamFormats{ formats } };
> >> - cfg.pixelFormat = formats.begin()->first;
> >> - cfg.size = formats.begin()->second[0].max;
> >> -
> >> - config->addConfiguration(cfg);
> >> - }
> >> -
> >> - config->validate();
> >> -
> >> - return config;
> >> }
> >>
> >> int SimplePipelineHandler::configure(Camera *camera, CameraConfiguration *c)
> >> --
> >> 2.48.1
> >>
>
More information about the libcamera-devel
mailing list