libcamerasrc / gstreamer plugin expose sensor-mode configuration

[Jacopo Mondi]

Hi Nicolas

On Mon, Mar 03, 2025 at 11:29:55AM -0500, Nicolas Dufresne wrote:
> While I agree with the above, there is this answer that scratch me a little...
>
> Le vendredi 28 février 2025 à 09:52 +0100, Jacopo Mondi a écrit :
> > > Another exploration could be to place this trial and error in the
> > > GstDeviceProvider. The provider could extend the available caps with
> > > the hidden configuration found by trial and error, and using a private
> > > API, pass the maps of SensorConfiguration and Caps. But this is imply
> > > not possible today.
> > >
> >
> > I'm not sure I 100% understand the above as I'm not gstreamer
> > educated, but if you're suggesting a trial-and-error mechanism to
> > produce an enumeration of modes in gstreamer I would discourage it for
> > all the above reasons. I think operating under the assumption that if
> > you want to configure the sensor you should know what you're doing,
> > is the safest one at the moment.
>
> The GstDeviceMonitor is an API as a service, that runs in the background and
> that let you enumerate all sort of devices like cameras on your computer. It
> also implements hot-plug notification, which is irrelevant to most of the HW
> libcamera supports. Its also used by GStreamer app to enumerate vulkan device,
> audio input/output and is not API limited.
>
> GstDeviceProvider is the class implemented by plugins, Libcamera implements
> one. Since it runs in the background, it is capable of doing more work in
> order to provide more accurate capabilities (which is mostly focused on
> format, resolution and rate, which is what usually matters for the rest of
> the streaming pipeline). I'm fine with discouraging the trial and error
> approach, it is slow, requires the camera to not be in use.

Thanks for the clarifying this.

>
> RPi users do benefit a lot from the fact rpicam-apps does this trial and error,
> and prints it to the console. Its an important tool in understanding and selecting
> the SensorConfiguration, nothing alike is offered by libcamera, and nothing at
> all if you don't have raw capture support. Now that trial and error is currently
> done using StreamerConfiguration against the Raw stream, its only guesses, its not
> possible to automatically translate to SensorConfiguration. But at least, there
> is some tooling to introspect the running system, where you'd otherwise need secret
> access to datasheet, and rely on random comments in forums.
>
> In normal conditions, the IPA does make decisions using heuristics (based on hard
> coded knowledge of the HW). That leads to some functionality being hidden,
> not-accessible. This get noticed by users with one or more focus:
>
>   - Looking for a specific aspect ratio
>   - Looking for a specific field of view
>   - Looking for specific picture rate
>
> They will likely be fine with using fixed SensorConfiguration for their purpose, the
> problem is that there is no reliable source of information on how to pick these
> configurations. rpicam-apps "sensor mdoe", just fill the gap the best they can.

And I guess this is what you've been trying to do with
gst_libcamera_src_enumerate_sensor_modes() in
https://patchwork.libcamera.org/patch/18459/

True that's what rpicam-apps do, and you will probably be able to do
the same by enumerating the RAW modes. However you won't be able to
test the frame durations without configuring the camera, which I'm not
sure it's something that we want to be done in the "enumeration"
phase...


>
> Nicolas