libcamerasrc / gstreamer plugin expose sensor-mode configuration

[Nicolas Dufresne]

Le jeudi 27 février 2025 à 16:36 +0100, Jeffrey Katz a écrit :
> Hi Jacopo,
> 
> On 2/27/25 12:18 PM, Jacopo Mondi wrote:
> > I see in the issue a link to this patch series
>  > ... *snip* ...
> > So we allow users of the libcamera API to set a desired sensor
> > configuration but it has to be correct otherwise the camera
> > configuration fails.
> 
> Honestly I think that's reasonable and okay. I don't understand why it 
> would be desirable for a gstreamer pipeline (which are, as far as I 
> understand, typically statically defined) to be able to enumerate sensor 
> modes--maybe for some of the nice auto-negotiation between stages--but 
> in most any setup I can think of, it's about telling libcamera what is 
> desired instead of defaulting to the automagic selection.

The libcamera-apps implementation was based on applying restriction to
a RAW stream. In that context, these where just high level
restrictions, and it made total sense to use caps, so if you only
wanted to restrict the render to 60Hz, possibly giving away some 12bit
mode, you could do just that but forcing lets say 60Hz and 12bit bayer.
Its also very easy for high level user to understand.

The sensor configuration on the other end is exactly what is involved
under the hood to change the "mode" behaviour. Its not a mode (similar
to graphic mode lines) anymore.

The design that got merged into libcamera have several issues though,
that makes it rather inconvenient.

a) You cannot leave anything unset

> Applications shall fully populate all fields of the 
> CameraConfiguration::sensorConfig class members before validating

So you can't do things like force a bit depth, and let the IPA the
freedom to select the other parameters.

b) Configuration extension will break your code

Because of a), if we decide to add a configuration, the old code will
fail to follow the rules and break. For now, the GStreamer element is
in libcamera, so as long as these get updated together, no one will be
hit by this. But it raises quick question on this API extensibility.

c) There is no way to introspect or get a valid default

The first part is like Jacopo said, a kernel API limitation. The second
though could be provided by IPA that knows all their supported sensors
(or have data base for them). It is likely quite some work to revisit
all this, and perhaps long term, we won't need to know about all the
sensors, so not sure how this can work without a little help from the
Linux drivers.

> 
> > I presume the gstreamer patches need to be ported to this new
> > mechanism and resubmitted. Is this something you would be interested in
> > looking at ? cc Nicolas as well.
> 
> I'm always happy when we can upstream something instead of having to 
> maintain an (ugly) patch. I'll have a look if there's a small, clean way 
> to augment libcamerasrc to take advantage of the sensor selection API.

My suggestion would be to either break in down in several properties
(aka sensor-conf-<prop-nameé> or make a single property and use the
GstStructure to get serialization/deserialization for free (so its
usable in gst-launch). Of course it will fail if you don't set all the
field, and it will be unset by default.

Assuming a/b/c get some resolution in the API, a specialized
application could use caps queries to reproduce the old behaviour by
simply changing and testing the configuration. This could easily be
done once and save. I would refrain from putting it back in the
element, as it makes the startup very slow.

Another exploration could be to place this trial and error in the
GstDeviceProvider. The provider could extend the available caps with
the hidden configuration found by trial and error, and using a private
API, pass the maps of SensorConfiguration and Caps. But this is imply
not possible today.

Nicolas