libcamerasrc / gstreamer plugin expose sensor-mode configuration

[Nicolas Dufresne]

While I agree with the above, there is this answer that scratch me a little...

Le vendredi 28 février 2025 à 09:52 +0100, Jacopo Mondi a écrit :
> > Another exploration could be to place this trial and error in the
> > GstDeviceProvider. The provider could extend the available caps with
> > the hidden configuration found by trial and error, and using a private
> > API, pass the maps of SensorConfiguration and Caps. But this is imply
> > not possible today.
> > 
> 
> I'm not sure I 100% understand the above as I'm not gstreamer
> educated, but if you're suggesting a trial-and-error mechanism to
> produce an enumeration of modes in gstreamer I would discourage it for
> all the above reasons. I think operating under the assumption that if
> you want to configure the sensor you should know what you're doing,
> is the safest one at the moment.

The GstDeviceMonitor is an API as a service, that runs in the background and
that let you enumerate all sort of devices like cameras on your computer. It
also implements hot-plug notification, which is irrelevant to most of the HW
libcamera supports. Its also used by GStreamer app to enumerate vulkan device,
audio input/output and is not API limited.

GstDeviceProvider is the class implemented by plugins, Libcamera implements
one. Since it runs in the background, it is capable of doing more work in
order to provide more accurate capabilities (which is mostly focused on
format, resolution and rate, which is what usually matters for the rest of
the streaming pipeline). I'm fine with discouraging the trial and error
approach, it is slow, requires the camera to not be in use.

RPi users do benefit a lot from the fact rpicam-apps does this trial and error,
and prints it to the console. Its an important tool in understanding and selecting
the SensorConfiguration, nothing alike is offered by libcamera, and nothing at
all if you don't have raw capture support. Now that trial and error is currently
done using StreamerConfiguration against the Raw stream, its only guesses, its not
possible to automatically translate to SensorConfiguration. But at least, there
is some tooling to introspect the running system, where you'd otherwise need secret
access to datasheet, and rely on random comments in forums.

In normal conditions, the IPA does make decisions using heuristics (based on hard
coded knowledge of the HW). That leads to some functionality being hidden,
not-accessible. This get noticed by users with one or more focus:

  - Looking for a specific aspect ratio
  - Looking for a specific field of view
  - Looking for specific picture rate

They will likely be fine with using fixed SensorConfiguration for their purpose, the
problem is that there is no reliable source of information on how to pick these
configurations. rpicam-apps "sensor mdoe", just fill the gap the best they can.

Nicolas