[libcamera-devel] Autofocus

[Hanlin Chen]

Hi David, and Kieran,

On Thu, Oct 21, 2021 at 8:29 PM David Plowman
<david.plowman at raspberrypi.com> wrote:
>
> Hi everyone
>
> Another subject I wanted to address was autofocus, as we're not so
> very far away from being able to do some work on this. In the first
> instance, I thought, it would be worth quickly reviewing what Android
> wants/does.
>
> 1. Autofocus Modes
>
> Android seems to define basically 2 controls, an AF "mode" and a
> "trigger". It also reports back its state. Full details are here:
> https://source.android.com/devices/camera/camera3_3Amodes
>
> To summarise, the mode has values like "off" (manual control allowed),
> "auto" (performs a sweep when triggered) and a couple of flavours of
> CAF (continuous autofocus). The "trigger" control simply involves a
> "start" and "cancel". The state that gets reported back seemed
> unsurprising ("I'm doing nothing", "I'm scanning", "I've finished
> scanning and I'm in focus" and so on).
>
> Generally I thought what Android defines here was mostly fine. It
> does seem to conflate certain ideas together all into a single
> mode control. For example:
>
> * It doesn't separate out the "range" that is being searched. There's
>   an "auto" and "auto-macro" mode, but this doesn't allow you to
>   explicitly seratch, for example, the "full" range, or the "full but
>   not extreme macro" range.
>
> * Nor can the speed of the autofocus be specified separately. It has a
>   notion of "fast" CAF and "slow" CAF, but what about a "fast" AF
>   sweep? I think we've done those in the past.
>
> I also thought the "off" mode was unhelpful, as the implication seemed
> to be that you have to set the mode to "off" to move the lens
> yourself. This would be irritating in the following use case:
>
> - Do an AF sweep and capture a picture.
>
> - Then go back to preview, move the lens to hyperfocal, and wait for
>   the next AF trigger.
>
> Do you have to set the mode to "off" (and wait?) before moving the
> lens to hyperfocal, before then going back to "auto"?  It
> would be much easier if you could simply stay in "auto" the whole
> time. Or perhaps it behaves like this and didn't explain it
> clearly. But basically I thought the whole "off" mode was pointless,
> and possibly harmful (rather like earlier Android AEC/AGC
> discussions!).

https://developer.android.com/reference/android/hardware/camera2/CameraMetadata#CONTROL_AF_MODE_OFF
In my understanding, the "off" mode simply means fully manual control
case, i.e., the 3A algorithm won't change lens, but fully adopt by
application.
It's used when users want to control the lens completely, for example,
implementing their own AF algorithm.

You are right that it's easier to stay in "auto" the whole time, and
it's in fact the case with CAF.
In this CAF, the AF algorithm modifies the lens position continually
to attempt to provide a constantly-in-focus image stream.
And the AF status would be changed between "scanning" and
"passive_locked". The typical usage is as follows:
1. Application checks current AF status, if it's not "locked", start AF trigger.
2. The HAL should move the lens to a focused position, and report AF
status to "locked". In case CAF stops moving the lens.
3. Application, take a picture, and send cancel AF trigger
4. The HAL resumes to normal CAF mode and is free to move the lens,
the AF status may change to "passive_locked".
Above is the difference between "passive_scan", "passive_locked" and
"active_scan". Passive means scanning is controlled by CAF algorithms.

On the other hand, Do the "fast" and "slow" modes you mentioned are
from the description in the video and still CAF mode?
If so, it means the requirement to the 3A algorithm, since still
capture prefers "fast" but video recording prefers "smooth".
If you're interested, the related controls can be found in:
https://chromium.googlesource.com/chromium/src/+/refs/heads/main/media/capture/video/chromeos/camera_3a_controller.cc#192
In fact, most devices in CrOS support both CAF modes, and they are
used in all use cases. "Auto" and "Macro" are rarely used.

>
> Android also has an "EDOF" mode which seems to be the same as
> "off" and would therefore seem to be even more pointless!
I don't know any examples supporting "EDOF" either. The "off" should
better be named "manual" though.
>
> 2. Lens Position
>
> Android provides you with a "minimum focus" lens position, and a
> hyperfocal position. It's defined that infinity is always at lens
> position zero. A lens can be "uncalibrated", or it can be "calibrated"
> in which case the units of these numbers are formally dioptres (that's
> 1/metres).
>
> For fixed focus lenses, the minimum focus position should also be
> identically zero; the case that the lens might not be focused at
> infinity does not seem to feature.
>
> See:
> https://developer.android.com/reference/android/hardware/camera2/CameraCharacteristics#LENS_INFO_FOCUS_DISTANCE_CALIBRATION
>
> One thing to be clear on is that for any camera module, not all the
> lens driver chip range will be available owing to the module's
> physical characteristics. And remember that there's no way to find out
> from the hardware exactly what module is attached. I think we'd
> envisage the useful lens driver range and hyperfocal position being
> stored in the device tree.
>
> The V4L2 control can then report its useful range (rather than the
> range supported by the driver chip), and its default position could be
> hyperfocal. For uncalibrated lenses these can be reported as
> properties, with a suitable transform within libcamera so that
> infinity comes out at zero.
>
> For calibrated lenses I guess there needs to be at least a dioptre
> value for the minimum focus and hyperfocal lens positions, so that
> these can be advertised to applications. Perhaps a module database
> would do?
Yes, as far as I know, current CrOS implementations uses a
configuration file or tuning file for those lens characteristics.
>
> But you might in fact need a full mapping between the two. Such a
> thing could easily live in an "rpi.autofocus" algorithm in our JSON
> file, and other platforms would have to make their own
> arrangements. Or could libcamera store something common?
>
> 3. So what should we do?
>
> I'd suggest the following controls:
>
> "AF mode" - just "auto" or "continuous"
>
> "AF range" - "macro", "full" or "normal" (= "full" without the macro
> end)
>
> "AF speed" - "fast" or "slow"
>
> "AF trigger" - "start" or "cancel"
>
> There needs to be an "AF state" reported back in metadata, same idea
> as the Android one. I would say we need "scanning", "focused",
> "failed" and "reset" (when the camera has just started, or a scan was
> cancelled, or the lens moved manually).
We may need to differentiate "passive" and "active" to support the
Android layer.
I would prefer "unfocused", since "failed" is a little sad...
There is a separate status for the lens itself too.
https://developer.android.com/reference/android/hardware/camera2/CaptureResult#LENS_STATE
>
> "lens position" - a value, with 0 being infinity, etc.
> Setting a value in "auto" mode would cancel any sweep that is in
> progress. In CAF mode, I think we'd ignore it.
Hmmm, it seems to depend on how we define the "auto" and "manual" here.
If "auto" means: "Hey AF algorithm, please find a focus for me and
lock it", the lens position is not used here.
>
> And we want a "lens position" property giving the infinity and
> minimum focus positions as the range, and hyperfocal as the default.
>
> At some point someone will want a "is the lens calibrated?" property
> and we'll have to worry about full calibrations. But that could be a
> "phase 2".
>
> As always, I'd love to hear other people's thoughts!
>
> Thanks and best regards
> David