[libcamera-devel] Optic handling questions

[Kieran Bingham]

Hi Matthias,

Quoting Matthias Fend via libcamera-devel (2022-12-15 13:43:47)
> Hi all,
> 
> I have some questions about handling the optic controls. It looks like 
> this has been considered, but apart from the use of a simple focus lens 
> in IPU3, it doesn't seem to be used extensively yet.

I think there's already at least a couple of attempts on the list to get
AF in for both RPi and RKISP1.

ipa: rapsberrypi: Introduce auto-focus (auto-mode) 2022-12-01
- https://patchwork.libcamera.org/project/libcamera/list/?series=3653

libcamera: rkisp1: Add lens control 2022-08-09
- https://patchwork.libcamera.org/project/libcamera/list/?series=3398

ipa: rkisp1: Add autofocus algorithm 2022-07-13
https://patchwork.libcamera.org/project/libcamera/list/?series=3276

ipa: raspberrypi: Introduce an autofocus algorithm 2022-06-13
- https://patchwork.libcamera.org/project/libcamera/list/?series=3174

Introduce camera lens properties 2022-06-09
- https://patchwork.libcamera.org/project/libcamera/list/?series=3166

Enabling AF algorithm to get the VCM attributes from the device driver 2022-04-26
- https://patchwork.libcamera.org/project/libcamera/list/?series=3069

So it seems like there's lots of 'unfinished' developments that need
some attention in some form, or rebasing, or reviewing ...

(There may be more, this was from me scanning through patchwork).

> As I understand it, the current control flow looks something like this:
> - IPA sets v4l2 control (e.g. V4L2_CID_FOCUS_ABSOLUTE)

IPA's shouldn't really be dealing in V4L2 controls. We're trying to make
IPA's work in libcamera controls. (That may not yet be the case though,
so they may still be using v4l2 controls currently).


> - Control is passed to the pipeline handler
> - Pipeline handler reads values from known v4l2 controls (e.g. 
> V4L2_CID_FOCUS_ABSOLUTE)
> - Pipeline handler calls dedicated CameraLens methods (e.g. 
> setFocusPosition)
> - CameraLens then actually converts to v4l2 control (e.g. 
> V4L2_CID_FOCUS_ABSOLUTE) and applies it to sub device

Yes, ultimately - the CameraLens class should be responsible for taking
any normalised libcamera control and mapping it to any device specific
value.


> At first glance, it looks like the controls are being unnecessarily 
> converted back and forth once, and I'm wondering if the pipeline handler 
> really needs to be included in this way here.

I think the CameraLens class should be responsible for most of the
abstractions, as it should be responsible for the underlying v4l-subdev
for the VCM.


> Currently that would mean that every pipeline handler needs to know how 
> to convert all v4l2 lens-specific controls (like V4L2_CID_ZOOM_ABSOLUTE, 
> V4L2_CID_IRIS_ABSOLUTE, ...) into CameraLens methods.
> Since the IPAs are already working with v4l2 controls and need to know 
> all the lens details, wouldn't it be easier if those controls were 
> passed directly to the subdevice?

IPA's should be abstracted from V4L2. It's the responsibility of the
Pipeline handler, and the helper classes (CameraLens / CameraSensor) to
perform any conversion from libcamera types to V4L2 types.


> The pipeline handler's involvement would then be reduced to finding the 
> appropriate subdevice and then transparently passing the v4l2 controls 
> from the IPA to the device.
> 
> But maybe I'm missing something crucial here.
> I would be happy about any hints and I would also be interested to know 
> if there are already plans in which direction this topic should develop.


There's lots of prior work - but nothings had sufficient momentum to get
completed and merged it seems.

There was some work that was blocked due to us reworking the internal
handling of the algorithms, which is now done, so some of the above
series need to be rebased or reworked on top of that.

There's also a recent development to change from Hyperfocal based units
to dioptre based units for focus points - so worth checking on that too.

What are your targets/goals?

--
Kieran