[PATCH 2/3] gstreamer: Generate controls from control_ids_*.yaml files

[Jaslo Ziska]

Hi Nicolas,

Nicolas Dufresne <nicolas at ndufresne.ca> writes:
> Le jeudi 08 août 2024 à 10:50 +0100, Kieran Bingham a écrit :
>> Hi Nicolas, Jaslo,
>> 
>> 
> [...]
>
>> > >   
>> > >   af-trigger          : This control starts an autofocus 
>> > >   scan when AfMode is set to AfModeAuto, and can also be 
>> > >   used to terminate a scan early. It is ignored if AfMode 
>> > >   is set to AfModeManual or AfModeContinuous. 
>> > >                         flags: readable, writable, 
>> > >                         controllable
>> > >                         Enum "AfTrigger" Default: 0, 
>> > >                         "start"
>> > >                            (0): start            - Start an 
>> > >                            AF scan. Ignored if a scan is in 
>> > >                            progress. 
>> > >                            (1): cancel           - Cancel 
>> > >                            an AF scan. This does not cause 
>> > >                            the lens to move anywhere else. 
>> > >                            Ignored if no scan is in 
>> > >                            progress. 
>> > 
>> > This one is missing something to make sense, at least from 
>> > GStreamer point of view. When you are not starting or
>> > cancelling, what value this control should change to ? That 
>> > question might be relevant for libcamera globally.
>> > 
>> > Without this third state, I would say this one should not be 
>> > there, and replace with action signals.
>> 
>> 
>> In libcamera - controls only 'exist' in a single request. What 
>> is an
>> 'action signal' in Gstreamer?
>
> Its more or less a method call. The main difference is that you 
> don't need a .h
> and a static link or dlopen to call it, and bindings let you 
> call them without
> the need for per-element code. This allow GStreamer plugins to 
> offer advance
> API. You can check multiudpsink as an example. It has 4 actions, 
> "add",
> "remove", "clear", this is related to the transmission address, 
> and "get-stats",
> which let you get the statistics but for a specific destination 
> rather then the
> combination (which is offered as property). It also adds 2 
> signals, "client-
> added" and "client-remove", which is pretty common to allow a 
> second component
> monitor the sate while application adds/remove clients.
>
> https://gstreamer.freedesktop.org/documentation/udp/multiudpsink.html?gi-language=c#action-signals
>
> This discussion kind of reveals that libcamera controls are many 
> different
> mechanisms under a single umbrella.
>
> You have the controls that has a known state, that you can write 
> in one request,
> and read back on the response of that request. And that you 
> don't expect the
> state to change unless requested. These are 1:1 match with 
> properties
> (properties can be asynchronous, and we have a notifiy signal to 
> let application
> know when applied). The difference is that the yaml that serves 
> as a spec is not
> abstract the ranges and default values. This is manageable, 
> libcamera way of
> specifying control will just pass through the GStreamer element, 
> and the
> libcamera based GStreamer applications will implement per camera 
> glue to make
> things work, just like they would have to do by using the 
> libcamera API
> directly. In reality, folks will hard code their app to the 
> Raspberry Pi way,
> and it will break on other platform, but that is not my battle 
> ;-P
>
> There is metadata (the states in their documentation), which are 
> not expected to
> be set in a request, but only read. This is also good fit for 
> read-only
> properties. Though, I kind of suspect that the specification 
> leave it open in
> regard to these being per-stream (which which case they should 
> be pad
> properties), or for the entire camera (libcamerasrc property).
>
>> 
>> 
>> So - AfTrigger for start and cancel are, in this case 'events'
>> 
>>  - "Start an AFscan at this request"
>>  - "Cancel any active scan"
>
> And finally, there is triggers. This is the same concept as V4L2 
> buttons, except
> that you group multiple actions in the same button by passing an 
> argument. That
> is not a property. If we introduce a classification for 
> settables, metadata and
> triggers, that will be enough for the python generator to avoid 
> this. My
> recommandation Jaslo, would be to try and reduce the set of 
> properties we expose
> in a first pass, or make them opt-in.

Could you maybe elaborate which controls should be which GStreamer 
kind property?

I think I understand it this way: the plain normal libcamera 
controls should be GStreamer properties and the libcamera metadata 
should be read-only GStreamer properties. But what about the other 
controls?

Which kind of libcamera controls do you image to be GStreamer 
actions? And what should be done about the libcamera "triggers"?

And also how should we specify which libcamera control is which 
kind of GStreamer property/action/...? In the main yaml (if this 
would be allowed just for GStreamer) or somewhere else?

Best regards,

Jaslo

>> There are no 'events' in between. Controls are only 'set' in 
>> Requests.
>> Any 'response' is only available in the metadata from a 
>> completed
>> Request.
>> 
>> You can not 'read' the state of a control in the same sense of 
>> a V4L2
>> control.
>> 
>> (Maybe this should change for some use cases, or for an abilty 
>> to set
>> controls directly on a camera, which I believe Stefan has 
>> explored, and
>> not through a request, but it's the current model).
>
> I haven't read the implementation yet, but indeed, this requires 
> a bit of glue.
> There will be a config that is what has been set since the last 
> request (this is
> mostly likely the case already). As soon as we have a response, 
> we can save
> everything, and use that to reply to the get() calls. It always 
> good though to
> filter unsupported controls before making a request, otherwise 
> the warning
> message will repeat every request, and that will be very spammy.
>
>> 
>> > >   analogue-gain : Analogue gain value applied in the sensor 
>> > >   device. The value of the control specifies the gain
>> > > multiplier applied to all colour channels. This value 
>> > > cannot be lower than 1.0. Setting this value means that it
>> > > is now fixed and the AE algorithm may not change it. 
>> > > Setting it back to zero returns it to the control of the AE
>> > > algorithm. See also: exposure-time, ae-enable. Todo: 
>> > > Document the interactions between AeEnable and setting a
>> > > fixed value for this control. Consider interactions with 
>> > > other AE features, such as aperture and aperture/shutter
>> > > priority mode, and decide if control of which features 
>> > > should be automatically adjusted shouldn't better be
>> > > handled through a separate AE mode control.
>> > >                         flags: readable, writable, 
>> > >                         controllable
>> > >                         Float. Range:   -3.402823e+38 - 
>> > >                         3.402823e+38 Default: 
>> > >                         0 
>> > 
>> > I'm not confident there is no range, would be nice if we can 
>> > align to something sensible (might need yaml changes of 
>> > course).
>> 
>> Different cameras will have different ranges ... only 
>> determinable at
>> runtime. We can't specify this in YAML. Though - it should 
>> always be
>> positive at least! (and as you've mentioned elsewhere, if it 
>> was
>
> Well, that would already be an improvement, signed / unsigned. 
> Will it fit 32bit
> ? 64bit, it is float or double ? etc.
>
>> machine readable to say always above 1.0 would be beneficial 
>> too).
>
> Notice that with proper types, there is always -INF and INF 
> defined, and worse
> case, we can define that.
>
>> 
>
> [...]
>
>> 
>> > I think my comments begins to be repetitive, and if we solve 
>> > the earlier issues
>> > we will solve them all. The main thing is that the yaml 
>> > should explicitly define
>> > more stuff rather then relying to text. And using that, we 
>> > can easily adjust
>> 
>> Agreed, I think at least the yaml files could specify if a 
>> control can
>> only be accessed as metadata which would simplify all of the 
>> 'read-only'
>> issues.
>
> Ack, we should propose some improvement to the yaml.
>
>> 
>> > GStreamer properties. A final note, not all of these will be 
>> > supported. I was
>> > wondering if we should add an extra property that tells use 
>> > which one are
>> > actually supported ? We don't need anything fency, a "/" 
>> > seperate list in a
>> > string or similar could do. We can also go for GstStructure.
>> 
>> I don't understand this part fully. Do you mean a new 
>> control/property
>> that gstreamer libcamerasrc will read at runtime from 
>> libcamera? It
>> already has a Camera->controls() that it can read that from.
>> 
>> But you mention GstStructure - do you mean that the 
>> libcamerasrc will
>> generate this ? (Perhaps from the Camera->controls()?)
>
> I think you see core libcamera changes in every comments (must 
> feel discouraging
> :-D), This is in regard to how we glue the two worlds. I'd like 
> to see
> "property" that reports the list of supported controls, or the 
> map of supported
> controls (with the run-time limits).
>
>> 
>> > Nicolas
>> 
>> Kieran