<html><head>
</head>
<body><div>Le vendredi 24 mars 2023 à 18:24 +0000, Robert Mader via libcamera-devel a écrit :</div><blockquote type="cite" style="margin:0 0 0 .8ex; border-left:2px #729fcf solid;padding-left:1ex"><p>Hi,<br></p><div class="moz-cite-prefix">On 24.03.23 19:12, Nicolas Dufresne via libcamera-devel wrote:<br></div><div> <br></div><blockquote type="cite" cite="mid:20230324181247.302586-2-nicolas@ndufresne.ca" style="margin:0 0 0 .8ex; border-left:2px #729fcf solid;padding-left:1ex"><pre>From: Nicolas Dufresne <a class="moz-txt-link-rfc2396E" href="mailto:nicolas.dufresne@collabora.com"><nicolas.dufresne@collabora.com></a></pre><pre><br></pre><pre>This add support for selecting the sensor mode. A new read-only</pre><pre>property called sensor-modes is filled when the element reaches</pre><pre>READY state. It contains the list of all available sensor modes,</pre><pre>including the supported framerate range. This is exposed as GstCaps</pre><pre>in the form of sensor/mode,width=X,height=Y,format=Y,framerate=[...].</pre><pre>The format string matches the libcamera format string representation.</pre><pre><br></pre><pre>The application can then select a mode using the read/write sensor-mode</pre><pre>control. The selected mode is also a caps, it will be intersected with</pre><pre>the supported mode and the "best" match will be picked. This allows</pre><pre>application to use simple filter when they want to pick a mode for lets</pre><pre>say a specific framerate (e.g. sensor/mode,framerate=60/1).</pre><pre><br></pre><pre>Signed-off-by: Nicolas Dufresne <a class="moz-txt-link-rfc2396E" href="mailto:nicolas.dufresne@collabora.com"><nicolas.dufresne@collabora.com></a></pre><pre>---</pre><pre> src/gstreamer/gstlibcamera-utils.cpp | 4 +</pre><pre> src/gstreamer/gstlibcamerasrc.cpp | 110 ++++++++++++++++++++++++++-</pre><pre> 2 files changed, 112 insertions(+), 2 deletions(-)</pre><pre><br></pre><pre>diff --git a/src/gstreamer/gstlibcamera-utils.cpp b/src/gstreamer/gstlibcamera-utils.cpp</pre><pre>index 750ec351..c8a8df09 100644</pre><pre>--- a/src/gstreamer/gstlibcamera-utils.cpp</pre><pre>+++ b/src/gstreamer/gstlibcamera-utils.cpp</pre><pre>@@ -416,6 +416,10 @@ gst_libcamera_configure_stream_from_caps(StreamConfiguration &stream_cfg,</pre><pre> stream_cfg.pixelFormat = gst_format_to_pixel_format(gst_format);</pre><pre> } else if (gst_structure_has_name(s, "image/jpeg")) {</pre><pre> stream_cfg.pixelFormat = formats::MJPEG;</pre><pre>+ } else if (gst_structure_has_name(s, "sensor/mode")) {</pre><pre>+ gst_structure_fixate_field(s, "format");</pre><pre>+ const gchar *format = gst_structure_get_string(s, "format");</pre><pre>+ stream_cfg.pixelFormat = PixelFormat::fromString(format);</pre><pre> } else {</pre><pre> g_critical("Unsupported media type: %s", gst_structure_get_name(s));</pre><pre> }</pre><pre>diff --git a/src/gstreamer/gstlibcamerasrc.cpp b/src/gstreamer/gstlibcamerasrc.cpp</pre><pre>index a10cbd4f..2f05a03f 100644</pre><pre>--- a/src/gstreamer/gstlibcamerasrc.cpp</pre><pre>+++ b/src/gstreamer/gstlibcamerasrc.cpp</pre><pre>@@ -147,6 +147,9 @@ struct _GstLibcameraSrc {</pre><pre> </pre><pre> gchar *camera_name;</pre><pre> </pre><pre>+ GstCaps *sensor_modes;</pre><pre>+ GstCaps *sensor_mode;</pre><pre>+</pre><pre> GstLibcameraSrcState *state;</pre><pre> GstLibcameraAllocator *allocator;</pre><pre> GstFlowCombiner *flow_combiner;</pre><pre>@@ -154,7 +157,10 @@ struct _GstLibcameraSrc {</pre><pre> </pre><pre> enum {</pre><pre> PROP_0,</pre><pre>- PROP_CAMERA_NAME</pre><pre>+ PROP_CAMERA_NAME,</pre><pre>+ PROP_SENSOR_MODES,</pre><pre>+ PROP_SENSOR_MODE,</pre><pre>+ PROP_LAST</pre><pre> };</pre><pre> </pre><pre> G_DEFINE_TYPE_WITH_CODE(GstLibcameraSrc, gst_libcamera_src, GST_TYPE_ELEMENT,</pre><pre>@@ -318,6 +324,61 @@ int GstLibcameraSrcState::processRequest()</pre><pre> return err;</pre><pre> }</pre><pre> </pre><pre>+static GstCaps *</pre><pre>+gst_libcamera_src_enumerate_sensor_modes(GstLibcameraSrc *self)</pre><pre>+{</pre><pre>+ GstCaps *modes = gst_caps_new_empty();</pre><pre>+ GstLibcameraSrcState *state = self->state;</pre><pre>+ auto config = state->cam_->generateConfiguration({ libcamera::StreamRole::Raw,</pre><pre>+ libcamera::StreamRole::VideoRecording });</pre><pre>+ if (config == nullptr) {</pre><pre>+ GST_DEBUG_OBJECT(self, "Sensor mode selection is not supported, skipping enumeration.");</pre><pre>+ return modes;</pre><pre>+ }</pre></blockquote><p>I gave this a short try on both my USB camera and on a PPP. In both cases enumeration gets skipped (as intended), but in the later case the following warning gets printed:<br></p><p>> ERROR RkISP1 rkisp1.cpp:684 Can't capture both raw and processed streams<br></p><div> So I wonder if we can come a better check here.<br><br></div></blockquote><div><div>The fact the IPAs / Pipeline manager through stuff on the terminal should probably not justify changing the API usage. The logging needs to be adapted so it does not pretend there has been an error on behalf of the application using the API. For me its seems evident that apps will try different numbers of roles until the camera accepts the configuration.</div></div><div><br></div><blockquote type="cite" style="margin:0 0 0 .8ex; border-left:2px #729fcf solid;padding-left:1ex"><p>When allowing the enumeration by only selecting Raw or VideoRecording I made the following observations:</p><div> </div><ol> <li>enumeration on the USB camera is very slow, takes several seconds</li> <li>on the PPP / RkISP1, even though there are many more modes, the enumeration is super fast - probably a fraction of a second. So a method like this could probably be used for framerate enumeration.<br></li> <li>the mode selection via sensor-mode=sensor/mode,... usually fails though (haven't digged deeper why yet)<br></li></ol></blockquote><div><br></div><div>It fails because of "ERROR RkISP1 rkisp1.cpp:684 Can't capture both raw and processed streams".<br><br>In general, I don't think it is useful to enumerate the modes on PPP, because the method to select sensor mode cannot be supported (you can't pass buffers through the ISP while capturing the raw). In general, this just highlights that sensor mode selection API is currently just a hack, as selecting sensor modes on PPP seems rather important. Currently, only IPU3 and RPi have a sensor/isp model that works with the API.<br><br>Nicolas</div><div><br></div><blockquote type="cite" style="margin:0 0 0 .8ex; border-left:2px #729fcf solid;padding-left:1ex"><p>Regards,</p><p>Robert<br></p><div> <br></div><blockquote type="cite" cite="mid:20230324181247.302586-2-nicolas@ndufresne.ca" style="margin:0 0 0 .8ex; border-left:2px #729fcf solid;padding-left:1ex"><pre>+</pre><pre>+ const libcamera::StreamFormats &formats = config->at(0).formats();</pre><pre>+</pre><pre>+ for (const auto &pixfmt : formats.pixelformats()) {</pre><pre>+ for (const auto &size : formats.sizes(pixfmt)) {</pre><pre>+ config->at(0).size = size;</pre><pre>+ config->at(0).pixelFormat = pixfmt;</pre><pre>+</pre><pre>+ if (config->validate() == CameraConfiguration::Invalid)</pre><pre>+ continue;</pre><pre>+</pre><pre>+ if (state->cam_->configure(config.get()))</pre><pre>+ continue;</pre><pre>+</pre><pre>+ auto fd_ctrl = state->cam_->controls().find(&controls::FrameDurationLimits);</pre><pre>+ if (fd_ctrl == state->cam_->controls().end())</pre><pre>+ continue;</pre><pre>+</pre><pre>+ int minrate_num, minrate_denom;</pre><pre>+ int maxrate_num, maxrate_denom;</pre><pre>+ double min_framerate = gst_util_guint64_to_gdouble(1.0e6) /</pre><pre>+ gst_util_guint64_to_gdouble(fd_ctrl->second.max().get<int64_t>());</pre><pre>+ double max_framerate = gst_util_guint64_to_gdouble(1.0e6) /</pre><pre>+ gst_util_guint64_to_gdouble(fd_ctrl->second.min().get<int64_t>());</pre><pre>+ gst_util_double_to_fraction(min_framerate, &minrate_num, &minrate_denom);</pre><pre>+ gst_util_double_to_fraction(max_framerate, &maxrate_num, &maxrate_denom);</pre><pre>+</pre><pre>+ GstStructure *s = gst_structure_new("sensor/mode",</pre><pre>+ "format", G_TYPE_STRING, pixfmt.toString().c_str(),</pre><pre>+ "width", G_TYPE_INT, size.width,</pre><pre>+ "height", G_TYPE_INT, size.height,</pre><pre>+ "framerate", GST_TYPE_FRACTION_RANGE,</pre><pre>+ minrate_num, minrate_denom,</pre><pre>+ maxrate_num, maxrate_denom,</pre><pre>+ nullptr);</pre><pre>+ gst_caps_append_structure(modes, s);</pre><pre>+ }</pre><pre>+ }</pre><pre>+</pre><pre>+ GST_DEBUG_OBJECT(self, "Camera sensor modes: %" GST_PTR_FORMAT, modes);</pre><pre>+</pre><pre>+ return modes;</pre><pre>+}</pre><pre>+</pre><pre> static bool</pre><pre> gst_libcamera_src_open(GstLibcameraSrc *self)</pre><pre> {</pre><pre>@@ -375,6 +436,7 @@ gst_libcamera_src_open(GstLibcameraSrc *self)</pre><pre> /* No need to lock here, we didn't start our threads yet. */</pre><pre> self->state->cm_ = cm;</pre><pre> self->state->cam_ = cam;</pre><pre>+ self->sensor_modes = gst_libcamera_src_enumerate_sensor_modes(self);</pre><pre> </pre><pre> return true;</pre><pre> }</pre><pre>@@ -462,6 +524,7 @@ gst_libcamera_src_task_enter(GstTask *task, [[maybe_unused]] GThread *thread,</pre><pre> GstLibcameraSrcState *state = self->state;</pre><pre> GstFlowReturn flow_ret = GST_FLOW_OK;</pre><pre> gint ret;</pre><pre>+ g_autoptr(GstCaps) sensor_mode = nullptr;</pre><pre> </pre><pre> g_autoptr(GstStructure) element_caps = gst_structure_new_empty("caps");</pre><pre> </pre><pre>@@ -481,6 +544,16 @@ gst_libcamera_src_task_enter(GstTask *task, [[maybe_unused]] GThread *thread,</pre><pre> roles.push_back(gst_libcamera_pad_get_role(srcpad));</pre><pre> }</pre><pre> </pre><pre>+ if (!gst_caps_is_any(self->sensor_mode)) {</pre><pre>+ sensor_mode = gst_caps_intersect(self->sensor_mode, self->sensor_modes);</pre><pre>+ if (!gst_caps_is_empty(sensor_mode)) {</pre><pre>+ roles.push_back(libcamera::StreamRole::Raw);</pre><pre>+ } else {</pre><pre>+ GST_WARNING_OBJECT(self, "No sensor mode matching the selection, ignoring.");</pre><pre>+ gst_clear_caps(&sensor_mode);</pre><pre>+ }</pre><pre>+ }</pre><pre>+</pre><pre> /* Generate the stream configurations, there should be one per pad. */</pre><pre> state->config_ = state->cam_->generateConfiguration(roles);</pre><pre> if (state->config_ == nullptr) {</pre><pre>@@ -490,7 +563,7 @@ gst_libcamera_src_task_enter(GstTask *task, [[maybe_unused]] GThread *thread,</pre><pre> gst_task_stop(task);</pre><pre> return;</pre><pre> }</pre><pre>- g_assert(state->config_->size() == state->srcpads_.size());</pre><pre>+ g_assert(state->config_->size() == state->srcpads_.size() + (!!sensor_mode));</pre><pre> </pre><pre> for (gsize i = 0; i < state->srcpads_.size(); i++) {</pre><pre> GstPad *srcpad = state->srcpads_[i];</pre><pre>@@ -510,6 +583,12 @@ gst_libcamera_src_task_enter(GstTask *task, [[maybe_unused]] GThread *thread,</pre><pre> gst_libcamera_get_framerate_from_caps(caps, element_caps);</pre><pre> }</pre><pre> </pre><pre>+ if (sensor_mode) {</pre><pre>+ StreamConfiguration &stream_cfg = state->config_->at(state->srcpads_.size());</pre><pre>+ g_assert(gst_caps_is_writable(sensor_mode));</pre><pre>+ gst_libcamera_configure_stream_from_caps(stream_cfg, sensor_mode);</pre><pre>+ }</pre><pre>+</pre><pre> if (flow_ret != GST_FLOW_OK)</pre><pre> goto done;</pre><pre> </pre><pre>@@ -624,6 +703,7 @@ gst_libcamera_src_task_leave([[maybe_unused]] GstTask *task,</pre><pre> g_clear_object(&self->allocator);</pre><pre> g_clear_pointer(&self->flow_combiner,</pre><pre> (GDestroyNotify)gst_flow_combiner_free);</pre><pre>+ gst_clear_caps(&self->sensor_modes);</pre><pre> }</pre><pre> </pre><pre> static void</pre><pre>@@ -659,6 +739,10 @@ gst_libcamera_src_set_property(GObject *object, guint prop_id,</pre><pre> g_free(self->camera_name);</pre><pre> self->camera_name = g_value_dup_string(value);</pre><pre> break;</pre><pre>+ case PROP_SENSOR_MODE:</pre><pre>+ gst_clear_caps(&self->sensor_mode);</pre><pre>+ self->sensor_mode = GST_CAPS(g_value_dup_boxed(value));</pre><pre>+ break;</pre><pre> default:</pre><pre> G_OBJECT_WARN_INVALID_PROPERTY_ID(object, prop_id, pspec);</pre><pre> break;</pre><pre>@@ -676,6 +760,12 @@ gst_libcamera_src_get_property(GObject *object, guint prop_id, GValue *value,</pre><pre> case PROP_CAMERA_NAME:</pre><pre> g_value_set_string(value, self->camera_name);</pre><pre> break;</pre><pre>+ case PROP_SENSOR_MODES:</pre><pre>+ g_value_set_boxed(value, self->sensor_modes);</pre><pre>+ break;</pre><pre>+ case PROP_SENSOR_MODE:</pre><pre>+ g_value_set_boxed(value, self->sensor_mode);</pre><pre>+ break;</pre><pre> default:</pre><pre> G_OBJECT_WARN_INVALID_PROPERTY_ID(object, prop_id, pspec);</pre><pre> break;</pre><pre>@@ -763,6 +853,7 @@ gst_libcamera_src_init(GstLibcameraSrc *self)</pre><pre> /* C-style friend. */</pre><pre> state->src_ = self;</pre><pre> self->state = state;</pre><pre>+ self->sensor_mode = gst_caps_new_any();</pre><pre> }</pre><pre> </pre><pre> static GstPad *</pre><pre>@@ -844,4 +935,19 @@ gst_libcamera_src_class_init(GstLibcameraSrcClass *klass)</pre><pre> | G_PARAM_READWRITE</pre><pre> | G_PARAM_STATIC_STRINGS));</pre><pre> g_object_class_install_property(object_class, PROP_CAMERA_NAME, spec);</pre><pre>+</pre><pre>+ spec = g_param_spec_boxed("sensor-modes", "Sensor Modes",</pre><pre>+ "GstCaps representing available sensor modes.",</pre><pre>+ GST_TYPE_CAPS,</pre><pre>+ (GParamFlags)(G_PARAM_READABLE</pre><pre>+ | G_PARAM_STATIC_STRINGS));</pre><pre>+ g_object_class_install_property(object_class, PROP_SENSOR_MODES, spec);</pre><pre>+</pre><pre>+ spec = g_param_spec_boxed("sensor-mode", "Sensor Mode",</pre><pre>+ "GstCaps representing selected sensor mode.",</pre><pre>+ GST_TYPE_CAPS,</pre><pre>+ (GParamFlags)(GST_PARAM_MUTABLE_READY</pre><pre>+ | G_PARAM_READWRITE</pre><pre>+ | G_PARAM_STATIC_STRINGS));</pre><pre>+ g_object_class_install_property(object_class, PROP_SENSOR_MODE, spec);</pre><pre> }</pre></blockquote><div> </div></blockquote><div><br></div><div><span></span></div></body></html>