libcamerasrc / gstreamer plugin expose sensor-mode configuration

[Jacopo Mondi]

Hi Nicolas

On Thu, Feb 27, 2025 at 11:37:07PM -0500, Nicolas Dufresne wrote:
> Le jeudi 27 février 2025 à 16:36 +0100, Jeffrey Katz a écrit :
> > Hi Jacopo,
> >
> > On 2/27/25 12:18 PM, Jacopo Mondi wrote:
> > > I see in the issue a link to this patch series
> >  > ... *snip* ...
> > > So we allow users of the libcamera API to set a desired sensor
> > > configuration but it has to be correct otherwise the camera
> > > configuration fails.
> >
> > Honestly I think that's reasonable and okay. I don't understand why it
> > would be desirable for a gstreamer pipeline (which are, as far as I
> > understand, typically statically defined) to be able to enumerate sensor
> > modes--maybe for some of the nice auto-negotiation between stages--but
> > in most any setup I can think of, it's about telling libcamera what is
> > desired instead of defaulting to the automagic selection.
>
> The libcamera-apps implementation was based on applying restriction to
> a RAW stream. In that context, these where just high level
> restrictions, and it made total sense to use caps, so if you only
> wanted to restrict the render to 60Hz, possibly giving away some 12bit
> mode, you could do just that but forcing lets say 60Hz and 12bit bayer.
> Its also very easy for high level user to understand.

It was a clear abuse of the API we only allowed because we had no
better alternative to propose at the time.

Sensor configuration != Stream configuration

see below

>
> The sensor configuration on the other end is exactly what is involved
> under the hood to change the "mode" behaviour. Its not a mode (similar
> to graphic mode lines) anymore.
>
> The design that got merged into libcamera have several issues though,
> that makes it rather inconvenient.
>
> a) You cannot leave anything unset
>
> > Applications shall fully populate all fields of the
> > CameraConfiguration::sensorConfig class members before validating
>
> So you can't do things like force a bit depth, and let the IPA the
> freedom to select the other parameters.
>

Let alone the IPA is not involved in this, the decision of
requiring a full configuration is a direct consequence of the fact we
can't enumerate. If I can't tell what's valid, what should I adjust
the non specified values to ?

If you ask me "something with 12bpp" how should I chose the
resolution, the binning/cropping modes, would you prefer a cropped mode
or a binned one ? A faster one or a slower one ? There is no way we can
"adjust" a sensor configuration like we do for a stream configuration.

So part of this is because we can't enumerate, so we don't know what
valid values to adjust to.

Part of this is also because if you need to control the sensor
configuration, it means you need to control every aspect of it.
Yes, I can give you a 12bpp mode, but if I adjust it to a "wrong"
resolution you will be very unhappy. I could adjust to a 4:3 mode but
you want a 16:9 stream, so the pipeline has to crop somewhere and you
lose part of the field of view. I could adjust to a mode that only
allows a slower framerate but maybe you want 60fps. If you want to
configure the sensor, you need to know what you're doing.

Unfortunately I don't see alternatives to precisely specify the whole
sensor configuration, as the assumption we can adjust it to something
correct for all users is just wrong.

> b) Configuration extension will break your code
>
> Because of a), if we decide to add a configuration, the old code will
> fail to follow the rules and break. For now, the GStreamer element is
> in libcamera, so as long as these get updated together, no one will be
> hit by this. But it raises quick question on this API extensibility.
>

We currently require user to only specify bpp and resolution, because
we don't have kernel APIs to identify cropped/binned modes, and
because the pixel rate configuration goes through a v4l2 control and cannot
be "tested" without actually applying it to the sensor.

Yes, when we'll ask for all fields to be specified for the reasons
explained above, applications which only specify bpp and output res
will break. Nobody likes this but the alternative was to fix the
kernel APIs first and then do the right thing in libcamera. I would
have preferred that, but at the time nobody put
themselves forward for addressing the kernel issue first, and 2 years
have passed already.

> c) There is no way to introspect or get a valid default
>
> The first part is like Jacopo said, a kernel API limitation. The second
> though could be provided by IPA that knows all their supported sensors
> (or have data base for them). It is likely quite some work to revisit

Listing "known to work and vendor approved" modes is indeed a
possibility. Someone wants this to be in the kernel drivers (that will
then have to be "freely configurable" and expose a set of blessed
modes). Someone wants them in userspace (libcamera or a global database
somewhere).

> all this, and perhaps long term, we won't need to know about all the
> sensors, so not sure how this can work without a little help from the
> Linux drivers.

I, personally, wouldn't be opposed to collect listing of supported
modes in a configuration file similar to what we do for tuning data.
Actually I do expect that to the sensor tuning process actually
produces a list of modes that have been "tuned and tested".

However if that's what we want, new kernel interfaces should be designed
accordingly as an API is need to introspect sensors without a listing of
"modes" or freely configurable drivers that do not support modes at
all (if any).

>
> >
> > > I presume the gstreamer patches need to be ported to this new
> > > mechanism and resubmitted. Is this something you would be interested in
> > > looking at ? cc Nicolas as well.
> >
> > I'm always happy when we can upstream something instead of having to
> > maintain an (ugly) patch. I'll have a look if there's a small, clean way
> > to augment libcamerasrc to take advantage of the sensor selection API.
>
> My suggestion would be to either break in down in several properties
> (aka sensor-conf-<prop-nameé> or make a single property and use the
> GstStructure to get serialization/deserialization for free (so its
> usable in gst-launch). Of course it will fail if you don't set all the
> field, and it will be unset by default.
>
> Assuming a/b/c get some resolution in the API, a specialized
> application could use caps queries to reproduce the old behaviour by
> simply changing and testing the configuration. This could easily be
> done once and save. I would refrain from putting it back in the
> element, as it makes the startup very slow.
>
> Another exploration could be to place this trial and error in the
> GstDeviceProvider. The provider could extend the available caps with
> the hidden configuration found by trial and error, and using a private
> API, pass the maps of SensorConfiguration and Caps. But this is imply
> not possible today.
>

I'm not sure I 100% understand the above as I'm not gstreamer
educated, but if you're suggesting a trial-and-error mechanism to
produce an enumeration of modes in gstreamer I would discourage it for
all the above reasons. I think operating under the assumption that if
you want to configure the sensor you should know what you're doing,
is the safest one at the moment.

> Nicolas