[libcamera-devel] Optic handling questions

[Matthias Fend]

Hi Jacobo,

first of all, thank you very much for your interest and the detailed answer.

Am 16.12.2022 um 13:07 schrieb Jacopo Mondi:
> Hi Matthias
> 
>    thanks for bringing this points to the list. As you said, most of
> the cameras we deal with are either fixed focus or use a simple VCM,
> so learning about more complex use cases is certainly of great value.
> 
> On Fri, Dec 16, 2022 at 11:39:16AM +0100, Matthias Fend via libcamera-devel wrote:
>> Hello Kieran,
>>
>> Am 15.12.2022 um 17:30 schrieb Kieran Bingham:
>>> Hi Matthias,
>>>
>>> Quoting Matthias Fend via libcamera-devel (2022-12-15 13:43:47)
>>>> Hi all,
>>>>
>>>> I have some questions about handling the optic controls. It looks like
>>>> this has been considered, but apart from the use of a simple focus lens
>>>> in IPU3, it doesn't seem to be used extensively yet.
>>>
>>> I think there's already at least a couple of attempts on the list to get
>>> AF in for both RPi and RKISP1.
>>>
>>> ipa: rapsberrypi: Introduce auto-focus (auto-mode) 2022-12-01
>>> - https://patchwork.libcamera.org/project/libcamera/list/?series=3653
>>>
>>> libcamera: rkisp1: Add lens control 2022-08-09
>>> - https://patchwork.libcamera.org/project/libcamera/list/?series=3398
>>>
>>> ipa: rkisp1: Add autofocus algorithm 2022-07-13
>>> https://patchwork.libcamera.org/project/libcamera/list/?series=3276
>>>
>>> ipa: raspberrypi: Introduce an autofocus algorithm 2022-06-13
>>> - https://patchwork.libcamera.org/project/libcamera/list/?series=3174
>>>
>>> Introduce camera lens properties 2022-06-09
>>> - https://patchwork.libcamera.org/project/libcamera/list/?series=3166
>>>
>>> Enabling AF algorithm to get the VCM attributes from the device driver 2022-04-26
>>> - https://patchwork.libcamera.org/project/libcamera/list/?series=3069
>>>
>>> So it seems like there's lots of 'unfinished' developments that need
>>> some attention in some form, or rebasing, or reviewing ...
>>>
>>> (There may be more, this was from me scanning through patchwork).
>>
>> Thank you for gathering all these references!
>> I've looked at a few before and I don't find much of a difference in the
>> concept of optical handling compared to what has already been merged.
>>
>>>
>>>> As I understand it, the current control flow looks something like this:
>>>> - IPA sets v4l2 control (e.g. V4L2_CID_FOCUS_ABSOLUTE)
>>>
>>> IPA's shouldn't really be dealing in V4L2 controls. We're trying to make
>>> IPA's work in libcamera controls. (That may not yet be the case though,
>>> so they may still be using v4l2 controls currently).
>>>
>>>
>>>> - Control is passed to the pipeline handler
>>>> - Pipeline handler reads values from known v4l2 controls (e.g.
>>>> V4L2_CID_FOCUS_ABSOLUTE)
>>>> - Pipeline handler calls dedicated CameraLens methods (e.g.
>>>> setFocusPosition)
>>>> - CameraLens then actually converts to v4l2 control (e.g.
>>>> V4L2_CID_FOCUS_ABSOLUTE) and applies it to sub device
>>>
>>> Yes, ultimately - the CameraLens class should be responsible for taking
>>> any normalised libcamera control and mapping it to any device specific
>>> value.
>>>
>>>
>>>> At first glance, it looks like the controls are being unnecessarily
>>>> converted back and forth once, and I'm wondering if the pipeline handler
>>>> really needs to be included in this way here.
>>>
>>> I think the CameraLens class should be responsible for most of the
>>> abstractions, as it should be responsible for the underlying v4l-subdev
>>> for the VCM.
>>>
>>>
>>>> Currently that would mean that every pipeline handler needs to know how
>>>> to convert all v4l2 lens-specific controls (like V4L2_CID_ZOOM_ABSOLUTE,
>>>> V4L2_CID_IRIS_ABSOLUTE, ...) into CameraLens methods.
>>>> Since the IPAs are already working with v4l2 controls and need to know
>>>> all the lens details, wouldn't it be easier if those controls were
>>>> passed directly to the subdevice?
>>>
>>> IPA's should be abstracted from V4L2. It's the responsibility of the
>>> Pipeline handler, and the helper classes (CameraLens / CameraSensor) to
>>> perform any conversion from libcamera types to V4L2 types.
>>>
>>>
>>>> The pipeline handler's involvement would then be reduced to finding the
>>>> appropriate subdevice and then transparently passing the v4l2 controls
>>>> from the IPA to the device.
>>>>
>>>> But maybe I'm missing something crucial here.
>>>> I would be happy about any hints and I would also be interested to know
>>>> if there are already plans in which direction this topic should develop.
>>>
>>>
>>> There's lots of prior work - but nothings had sufficient momentum to get
>>> completed and merged it seems.
>>>
>>> There was some work that was blocked due to us reworking the internal
>>> handling of the algorithms, which is now done, so some of the above
>>> series need to be rebased or reworked on top of that.
>>>
>>> There's also a recent development to change from Hyperfocal based units
>>> to dioptre based units for focus points - so worth checking on that too.
>>
>> To make sure I got it right, let me summarize what the final goal looks
>> like:
>> - IPAs should use libcamera specific controls to control optics.
>> - The controls should use normalized (e.g. 0.0-1.0) or real units (dB, mm).
>> - The pipeline uses a (generic) visual helper to convert the controls to
>> v4l2 controls
>>
>> Apart from replacing the v4l2 controls in the IPA with libcamera controls
>> and using a common optic helper for all pipelines, it stays as is.
>>
>>>
>>> What are your targets/goals?
>>
>> I think for simple use cases like VCM lenses (which I think is the most
>> common setup) this is a good idea.
>> However, for more advanced optics with multiple lens groups, there are
>> scenarios where:
>> 1. Some of the IPA algorithms require detailed knowledge of optics
>> 2. The IPA must position the lenses absolutely and in their native units
>> (e.g. steps for stepper drivers) accurately.
>> 3. A custom lens driver uses many lens-specific v4l2 controls to somehow
>> offer its numerous individual possibilities
>>
>>
>> To 1
>> Examples of this could be lens light intensity versus lens group position
>> that needs to be considered for AGC, or limited focus range versus other
>> groups that might be of interest for AFC.
>> Things like that are really hard to abstract in a generic lens controller
>> helper.
> 
> Do I get this right in that you're suggesting that algorithms that do
> not control the lens directly (like AGC) need to be instrumented to
> deal with more complex optic setups ?

I don't think this is generally necessary. And that would probably 
confuse a lot of people in a lot of cases.

I wanted to say that there are applications in which algorithms also 
need detailed information about the optics, where this would not be 
expected at first glance. Therefore, not all algorithms have to be 
drilled out accordingly, but it would be nice if something like this 
could still be implemented - even if this is only necessary in a few 
special cases.

> 
>>
>> To 2
>> This may be the case when the IPA needs to move lens groups in relation to
>> other lens groups or for calibration purposes.
>> Perhaps one could also achieve this by first converting the desired absolute
>> position to the normalized range and then hoping that the chain eventually
>> converts it back to the specified value when the v4l2 control is written.
>> But that seems like a somehow unreliable workaround.
>>
> 
> I presume the first conversion described in
> 
> "first converting the desired absolute position to the normalized
> range"
> 
> is due to the fact that calibration data are provided by manufacturer
> in absolute values ? Are there other reasons I am not considering ?

Yes, on the one hand the optics are described with these values and in 
this unit by the manufacturer and additional information can be 
determined or calibration data from another source can be available. An 
absolute mechanical position is relevant for all of them.
No matter what may change in the software over the years, the 
corresponding positions must never change.

> 
>> To 3
>> Since only a limited selection of optic-specific v4l2 controls is available,
>> a driver for more complex optics may be implemented in such a way that it
>> uses existing controls somewhat creatively to map this. Or does it even have
>> to define additional, own v42l controls for itself.
>> This is then probably only the case with a combination of an OOT kernel
>> driver and an OOT IPA module.
>> The libcamera lens controller helper will probably never map all v4l2
>> controls, and if it does, the abstraction most likely won't always match the
>> driver's behavior.
> 
> Let me add one data point here.
> 
> We have a similar problem with sensor's gain values. Sensor gain is
> programmed with device-specific values, while the IPA modules compute
> a normalized value that has to be translated to the device specific
> format before being applied to the subdevice.
> 
> We have CameraSensorHelpers as part of libipa:
> https://git.libcamera.org/libcamera/libcamera.git/tree/src/ipa/libipa/camera_sensor_helper.cpp
> 
> which have functions to translate the generic value to/from the
> device-specific one.
> 
> Right now the IPA modules do the conversion before sending controls to
> the pipeline. In example:
> https://git.libcamera.org/libcamera/libcamera.git/tree/src/ipa/rkisp1/rkisp1.cpp#n258
> 
> In our plans the CameraSensorHelpers will be moved to the pipeline
> handler side and the IPA will send the generic values and the
> translation will happen before setting the controls on the v4l2
> subdevice.
> 
> To continue the analogy, it might be possible that one specific sensor
> will not be able to realize the requested gain with only it's analog
> circuitry, but it will need to also apply some gain in the digital
> domain. A specialized CameraSensorHelper will be capable of handling
> this case opportunely. We're here talking about setups running on a
> specific product, with ad-hoc tuning and an ad-hoc camera sensor
> helper that will probably not fit for the generic use case.

Thanks for this example. At first I also wondered why the gain 
conversion happened in the IPA and not in a general place. I thought 
this was done with foresight to allow the IPA any flexibility it might 
need. But the plans seem to go in a different direction.
The handling of analogue and digital gain is again a great example here. 
In most cases, the AGC algorithm simply wants to set a gain in dB. How 
that ends up in the sensor is completely irrelevant to the algorithm. So 
the plan is absolutely fine here.
But then again there are regulations that, for example, first want to 
increase the analog gain, then open the optic iris to a certain degree 
and only then use the digital gain of the sensor as a last resort.

> 
> Same goes for lenses and optics. Right now we have a single CameraLens
> class (which pairs with CameraSensor as abstraction level) that will
> be complemented by device-specific helpers that will help with
> translating the "generic" values to device specific ones.
> 
> Now, I imagine that as long as you have to translate a single value, this
> whole plan stands. But I understand that if you need 50 different
> values to control your lenses you would need to very specialized IPA,
> which uses very specific controls. In this case you can send all of
> them to the pipeline and decide to apply all of them as they are in
> your very specialized CameraLensHelper. It's not something consumable
> for upstreaming but gives you a way forward I guess.

If the Camera and Lens helpers are provided as a library for the IPA 
modules and these are responsible for converting to V4L2 controls, then 
the IPAs could directly use the convenient functions from them such as 
lens->setFocusPos(). Additionally, a base class could provide a method 
to directly set v4l2 controls. If an IPA thinks it absolutely needs it, 
there would be a way.
But that would mean that the IPA would transfer v4l2 controls to the 
pipeline, which would only be forwarded by the pipeline to the 
appropriate devices.

> 
>>
>> If there were a way to pass on all v4l2 control elements unaltered from the
>> IPA to the subdevice, then the cases mentioned would also be possible.
>> Perhaps this can be implemented as an additional or alternative way to the
>> normalized and abstracted lens controller.
>>
> 
> As said I think it's possible. I wouldn't be happy to have this
> mechanism upstream as it would give a way for sending opaque lists of
> controls to be blindly applied to the subdevices without libcamera
> knowing nothing about them. It's not an issue in itself, but I presume
> that what goes upstream should be as open as possible, or at least
> provide the best support for open solution without any easy shortcuts
> for obfuscation.

Since using the helper's abstracted methods is much easier to use, I 
don't think anyone would be happily fiddling with v4l2 controls, and 
would therefore have no concerns about upstream going in the wrong 
direction.

> 
> Of course, if you're making a product or a very custom solution,
> you're free to take all the shortcuts you want :)

Right ;)
But since something is happening in this area anyway, it might be a good 
time to bring in such side issues as well.
It would be nice if a possibility could be found that does not exclude 
any known scenario.

Thanks a lot
  ~Matthias

> 
> Of course, if there's a way all those device specific controls can be
> standardized as libcamera controls, and an opportune camera lens
> helper could be developed to handle them, it all could be upstreamed with
> no issue even if the setup is certainly more complex than the standard
> VCM lens we usually deal with.
> 
>> Abstracting the optics is certainly a good idea and will work perfectly for
>> most applications. I have tried to show a few things that cannot be realized
>> with it or only with great difficulty.
>> Since I estimate that libcamera is used more than 95% with cameras without
>> moving lenses or with a simple VCM driver, I also understand that other
>> cases will not be considered.
>> Nevertheless, I hope that I was able to bring in a few interesting aspects.
> 
> Thanks, it's very interesting to know about more advanced use cases
> and learn how libcamera could evolve to support them!
> 
> Thanks
>    j
> 
>>
>> ~Matthias
>>
>>>
>>> --
>>> Kieran