[libcamera-devel] Autofocus

[David Plowman]

Hi everyone

Another subject I wanted to address was autofocus, as we're not so
very far away from being able to do some work on this. In the first
instance, I thought, it would be worth quickly reviewing what Android
wants/does.

1. Autofocus Modes

Android seems to define basically 2 controls, an AF "mode" and a
"trigger". It also reports back its state. Full details are here:
https://source.android.com/devices/camera/camera3_3Amodes

To summarise, the mode has values like "off" (manual control allowed),
"auto" (performs a sweep when triggered) and a couple of flavours of
CAF (continuous autofocus). The "trigger" control simply involves a
"start" and "cancel". The state that gets reported back seemed
unsurprising ("I'm doing nothing", "I'm scanning", "I've finished
scanning and I'm in focus" and so on).

Generally I thought what Android defines here was mostly fine. It
does seem to conflate certain ideas together all into a single
mode control. For example:

* It doesn't separate out the "range" that is being searched. There's
  an "auto" and "auto-macro" mode, but this doesn't allow you to
  explicitly seratch, for example, the "full" range, or the "full but
  not extreme macro" range.

* Nor can the speed of the autofocus be specified separately. It has a
  notion of "fast" CAF and "slow" CAF, but what about a "fast" AF
  sweep? I think we've done those in the past.

I also thought the "off" mode was unhelpful, as the implication seemed
to be that you have to set the mode to "off" to move the lens
yourself. This would be irritating in the following use case:

- Do an AF sweep and capture a picture.

- Then go back to preview, move the lens to hyperfocal, and wait for
  the next AF trigger.

Do you have to set the mode to "off" (and wait?) before moving the
lens to hyperfocal, before then going back to "auto"?  It
would be much easier if you could simply stay in "auto" the whole
time. Or perhaps it behaves like this and didn't explain it
clearly. But basically I thought the whole "off" mode was pointless,
and possibly harmful (rather like earlier Android AEC/AGC
discussions!).

Android also has an "EDOF" mode which seems to be the same as
"off" and would therefore seem to be even more pointless!

2. Lens Position

Android provides you with a "minimum focus" lens position, and a
hyperfocal position. It's defined that infinity is always at lens
position zero. A lens can be "uncalibrated", or it can be "calibrated"
in which case the units of these numbers are formally dioptres (that's
1/metres).

For fixed focus lenses, the minimum focus position should also be
identically zero; the case that the lens might not be focused at
infinity does not seem to feature.

See:
https://developer.android.com/reference/android/hardware/camera2/CameraCharacteristics#LENS_INFO_FOCUS_DISTANCE_CALIBRATION

One thing to be clear on is that for any camera module, not all the
lens driver chip range will be available owing to the module's
physical characteristics. And remember that there's no way to find out
from the hardware exactly what module is attached. I think we'd
envisage the useful lens driver range and hyperfocal position being
stored in the device tree.

The V4L2 control can then report its useful range (rather than the
range supported by the driver chip), and its default position could be
hyperfocal. For uncalibrated lenses these can be reported as
properties, with a suitable transform within libcamera so that
infinity comes out at zero.

For calibrated lenses I guess there needs to be at least a dioptre
value for the minimum focus and hyperfocal lens positions, so that
these can be advertised to applications. Perhaps a module database
would do?

But you might in fact need a full mapping between the two. Such a
thing could easily live in an "rpi.autofocus" algorithm in our JSON
file, and other platforms would have to make their own
arrangements. Or could libcamera store something common?

3. So what should we do?

I'd suggest the following controls:

"AF mode" - just "auto" or "continuous"

"AF range" - "macro", "full" or "normal" (= "full" without the macro
end)

"AF speed" - "fast" or "slow"

"AF trigger" - "start" or "cancel"

There needs to be an "AF state" reported back in metadata, same idea
as the Android one. I would say we need "scanning", "focused",
"failed" and "reset" (when the camera has just started, or a scan was
cancelled, or the lens moved manually).

"lens position" - a value, with 0 being infinity, etc.
Setting a value in "auto" mode would cancel any sweep that is in
progress. In CAF mode, I think we'd ignore it.

And we want a "lens position" property giving the infinity and
minimum focus positions as the range, and hyperfocal as the default.

At some point someone will want a "is the lens calibrated?" property
and we'll have to worry about full calibrations. But that could be a
"phase 2".

As always, I'd love to hear other people's thoughts!

Thanks and best regards
David