[libcamera-devel] [PATCH v3 1/1] libcamera: controls: Controls for driving AF (autofocus) algorithms
Jacopo Mondi
jacopo at jmondi.org
Thu Apr 21 11:24:13 CEST 2022
Hi David,
thanks for this new version
On Wed, Apr 20, 2022 at 05:26:22PM +0100, David Plowman wrote:
> This patch describes a series of controls that allow applications to
> drive AF algorithms:
>
> AfMode - manual, auto or continuous
> AfRange - full, macro or normal
> AfSpeed - fast or slow
> AfWindows - AF window locations
> AfTrigger - start (trigger) an AF scan or cancel
> AfPause - pause continuous AF
> LensPosition - set or retrieve position of lens
> AfState - reports whether scanning/success/failure
> AfPauseState - reports whether continuous AF paused or not
>
> Signed-off-by: David Plowman <david.plowman at raspberrypi.com>
> ---
> src/libcamera/control_ids.yaml | 318 ++++++++++++++++++++++++++-------
> 1 file changed, 258 insertions(+), 60 deletions(-)
>
> diff --git a/src/libcamera/control_ids.yaml b/src/libcamera/control_ids.yaml
> index 9d4638ae..eccf18bc 100644
> --- a/src/libcamera/control_ids.yaml
> +++ b/src/libcamera/control_ids.yaml
> @@ -381,6 +381,264 @@ controls:
> \todo Define how the sensor timestamp has to be used in the reprocessing
> use case.
>
> + - AfMode:
> + type: int32_t
> + description: |
> + Control to set the mode of the AF (autofocus) algorithm. Applications
> + are allowed to set a new mode, and to send additional controls for
> + that new mode, in the same request.
> +
> + An implementation may choose not to implement all the modes.
> +
> + enum:
> + - name: AfModeManual
> + value: 0
> + description: |
> + The AF algorithm is in manual mode. In this mode it will never
> + perform any action nor move the lens of its own accord, but an
> + application can set controls to move the lens.
> +
If, as I understand it, the lens can only be moved with 'LensPosition'
when in manual mode, I would say so explicitly
"... but an application can specify the desired lens
position using the LensPosition control."
> + In this mode the AfState will always report AfStateReset.
I'm tempted to say AfState is useless and can not be reported at all
in this case. For simplicity (for ph/ipa) I'm fine reporting it
regardless, even if seeing 'reset' might be slightly confusing. I
don't have better alternatives to propose though.
Reading the AfStateReset description
- name: AfStateReset
value: 0
description: |
The AF algorithm is in manual mode (AfModeManual) or in auto mode
(AfModeAuto) and a scan has not yet been triggered, or an in-progress
scan was cancelled.
I wonder if this isn't better named AfStateIdle
> + - name: AfModeAuto
> + value: 1
> + description: |
> + The AF algorithm is in auto mode. This means that the algorithm
> + will never move the lens or change state unless the AfTrigger
> + control is used. The AfTrigger control can be used to initiate a
> + focus scan, the results of which will be reported by AfState.
> +
> + If the autofocus algorithm is moved from AfModeAuto to another
> + mode while a scan is in progress, the scan is cancelled
> + immediately, without waiting for the scan to finish.
> +
> + When first entering this mode the AfState will report
> + AfStateReset. When a trigger control is sent, AfState will
> + report AfStateScanning for a period before spontaneously
> + changing to AfStateFocused or AfStateFailed, depending on
> + the outcome of the scan. It will remain in this state until
> + another scan is initiated by the AfTrigger control. If a scan is
> + cancelled (without changing to another mode), AfState will return
> + to AfStateReset.
> + - name: AfModeContinuous
> + value: 2
> + description: |
> + The AF algorithm is in continuous mode. This means that the lens
> + can re-start a scan spontaneously at any moment, without any user
> + intervention. The AfState still reports whether the algorithm is
> + currently scanning or not, though the application has no ability
> + to initiate or cancel scans (though it can "pause" them), nor to
> + move the lens for itself.
> +
> + When set to AfModeContinuous, the system will immediately initiate
> + a scan so AfState will report AfStateScanning, and will settle on
> + one of AfStateFocused or AfStateFailed, depending on the scan result.
> +
> + - AfRange:
> + type: int32_t
> + description: |
> + Control to set the range of focus distances that is scanned. An
> + implementation may choose not to implement all the options here.
> + enum:
> + - name: AfRangeNormal
> + value: 0
> + description: |
> + A wide range of focus distances is scanned, all the way from
> + infinity down to close distances, though depending on the
> + implementation, possibly not including the very closest macro
> + positions.
> + - name: AfRangeMacro
> + value: 1
> + description: Only close distances are scanned.
> + - name: AfRangeFull
> + value: 2
> + description: |
> + The full range of focus distances is scanned just as with
> + AfRangeNormal but this time including the very closest macro
> + positions.
> +
> + - AfSpeed:
> + type: int32_t
> + description: |
> + Control that determines whether the AF algorithm is to move the lens
> + as quickly as possible or more steadily. For example, during video
> + recording it may be desirable not to move the lens too abruptly, but
> + when in a preview mode (waiting for a still capture) it may be
> + helpful to move the lens as quickly as is reasonably possible.
> + enum:
> + - name: AfSpeedNormal
> + value: 0
> + description: Move the lens at its usual speed.
> + - name: AfSpeedFast
> + value: 1
> + description: Move the lens more quickly.
> +
> + - AfWindows:
> + type: Rectangle
> + description: |
> + Sets the focus windows used by the AF algorithm. The units used express
> + a proportion of the ScalerCrop control (or if unavailable, of the entire
> + image), as u0.16 format numbers.
I don't like too much the "or if unavailable", as it means that the
reference rectangle will change depending on the platform
capabilities.
I understand your argument in reply to Jean-Michel comment on the
previous version of the series (quoted here for reference)
"The thing that bothers me a bit about the full pixel array is that, if
you use digital zoom, then you will have to update your AfWindows as
they might otherwise lie outside your final image! If it's defined as
being a region of the scaler crop, then it automatically stays the
same (to the application, at least, the pipeline handler probably has
some work to do).
I note that the Android link you give below says "Focus areas are
relative to the current field of view (getZoom())", which I guess is
partly why they use that slightly arbitrary +/-1000 range. Maybe 1000
is easier to work with than the u0.16 format that I specified?"
Let's reason on the IPA point of view. If the AF algorithm works by
estimating the contrast, will it do so before or after the ScalerCrop
rectangle is applied ? If a sensor provides PDAF statistics, will it
do so on the full pixel array size, the portion of if that gets
processed or on the final image ?
Also, can't the AfWindow rectangles be re-scaled in the ScalerCrop
rectangle, so that they are applied to the actual FOV before being
passed to the AF algorithm ?
One last point about the unit: is the below representation correct,
when it comes to use a Rectangle to represent a proportion ?
Regardless of the reference rectangle, if we have a 4x3 grid, will the
pipeline report 12 rectangles like the ones below, for applications to
chose from ?
(0,0)
+-----+-----+-----+-----+
| | | | |
+-----+-----+-----+-----+
| | | | |
+-----+-----+-----+-----+
| | | | |
+-----+-----+-----+-----+
AfWindows = {
{0, 0, 1/4, 1/3}, {1/4, 0, 1/4, 1/3}, {2/4, 0, 1/4, 1/3}, {3/4, 0, 1/4, 1/3 },
{0, 1/3, 1/4, 1/3}, {1/4, 1/3, 1/4, 1/3}, {2/4, 1/3, 1/4, 1/3}, {3/4, 1/3, 1/4, 1/3 },
{0, 2/3, 1/4, 1/3}, {1/4, 2/3, 1/4, 1/3}, {2/4, 2/3, 1/4, 1/3}, {3/4, 2/3, 1/4, 1/3 }
};
My understanding is that the grid is fixed and depends on the platform
capabilities. I wonder if we shouldn't allow applications to select
grids by index.
Also, do we need a property to report the grid structure and allows
application to select them from, or do you see this conveyed through a
ControlInfo ?
> +
> + In order to be activated, a rectangle must be programmed with non-zero
> + width and height. If no rectangles are programmed in this way, then the
> + system will choose its own single default window in the centre of the
> + image.
> +
> + The details of how the windows are used are platform dependent. We note
> + that when there is more than one AF window, a typical implementation
> + might find the optimal focus position for each one and finally select
> + the window closest to the camera.
What window is more close to the camera ? My understanding is that
windows apply to the sensor produced frame, hence, when it comes to distance,
they all sit on the same focal plan :)
> +
> + size: [n]
> +
> + - AfTrigger:
> + type: int32_t
> + description: |
> + This control starts an autofocus scan when AfMode is set to AfModeAuto,
> + and can also be used to terminate a scan early.
> +
> + It is ignored if AfMode is set to AfModeManual or AfModeContinuous.
> +
> + enum:
> + - name: AfTriggerStart
> + value: 0
> + description: Start an AF scan. Ignored if a scan is in progress.
> + - name: AfTriggerCancel
> + value: 1
> + description: Cancel an AF scan. This does not cause the lens to move
> + anywhere else. Ignored if no scan is in progress.
> +
> + - AfPause:
> + type: int32_t
> + description: |
> + This control has no effect except when in continuous autofocus mode
> + (AfModeContinuous). It can be used to pause any lens movements while
> + (for example) images are captured. The algorithm remains inactive
> + until it is instructed to resume.
> +
> + enum:
> + - name: AfPauseImmediate
> + value: 0
> + description: |
> + Pause the continuous autofocus algorithm immediately, whether or
> + not any kind of scan is underway. AfPauseState will subsequently
> + report AfPauseStatePaused. AfState may report any of AfStateScanning,
> + AfStateFocused or AfStateFailed, depending on the algorithm's state
> + when it received this control.
> + - name: AfPauseDeferred
> + value: 1
> + description: |
> + This is similar to AfPauseImmediate, and if the AfState is currently
> + reporting AfStateFocused or AfStateFailed it will remain in that
> + state and AfPauseState will report AfPauseStatePaused.
> +
> + However, if the algorithm is scanning (AfStateScanning), AfPauseState
> + will report AfPauseStatePausing until the scan is finished, at which
> + point AfState will report one of AfStateFocused or AfStateFailed, and
> + AfPauseState will change to AfPauseStatePaused.
>From here on I see lines getting over 80 cols by a few chars. Can we
align it back ?
> + - name: AfPauseResume
> + value: 2
> + description: |
> + Resume continuous autofocus operation. The algorithm starts again
> + from exactly where it left off, and AfPauseState will report
> + AfPauseStateRunning.
> +
> + - LensPosition:
> + type: float
> + description: |
> + Acts as a control to instruct the lens to move to a particular position
> + and also reports back the position of the lens for each frame.
> +
> + The LensPosition control is ignored unless the AfMode is set to
> + AfModeManual.
> +
> + The units are dioptres divided by the hyperfocal distance. Non-integer
> + values are permitted. For example:
> + 0 moves the lens to infinity.
> + 0.5 moves the lens to twice the hyperfocal distance.
> + 1 moves the lens to the hyperfocal position.
> + And larger values will focus the lens ever closer.
> +
This seems good to me, albeit my understanding of the issue is limited :)
> + - AfState:
> + type: int32_t
> + description: |
> + Reports the current state of the AF algorithm in conjunction with the
> + reported AfMode value and (in continuous AF mode) the AfPauseState value.
> + The possible state changes are described below, though we note the following
> + state transitions that occur when the AfMode is changed.
> +
> + If the AfMode is set to AfModeManual then the AfState will report
> + AfStateReset. This action on its own does not cause the lens to move; the
> + LensPosition control must subsequently be used to achieve this.
> +
> + If the AfMode is set to AfModeAuto then the AfState will report
> + AfStateReset, unless AfTriggerStart is sent at the same time in which
s/unless/until ?
s/at the same time// ?
> + case it will (start a scan and) report AfStateScanning.
> +
> + If the AfMode is set to AfModeContinuous then the AfState will initially
> + report AfStateScanning.
> +
> + enum:
> + - name: AfStateReset
> + value: 0
> + description: |
> + The AF algorithm is in manual mode (AfModeManual) or in auto mode
> + (AfModeAuto) and a scan has not yet been triggered, or an in-progress
> + scan was cancelled.
> + - name: AfStateScanning
> + value: 1
> + description: |
> + The AF algorithm is in auto mode (AfModeAuto), and a scan has been
> + started using the AfTrigger control. The scan can be cancelled by
> + sending AfTriggerCancel at which point the algorithm will either
> + move back to AfStateReset or, if the scan actually completes
> + before the cancel request is processed, to one of
> + AfStateFocused or AfStateFailed.
> +
> + Alternatively the AF algorithm be be in continuous mode (AfModeContinuous)
s/be be/is ?
> + at which point it may enter this state spontaneously whenever it
> + determines that a rescan is needed.
> + - name: AfStateFocused
> + value: 2
> + description: |
> + The AF algorithm is in auto (AfModeAuto) or continuous (AfModeContinuous)
> + mode and a scan has completed with the result that the algorithm believes
> + the image is now in focus.
> + - name: AfStateFailed
> + value: 3
> + description: |
> + The AF algorithm is in auto (AfModeAuto) or continuous (AfModeContinuous)
> + mode and a scan has completed with the result that the algorithm did not
> + find a good focus position.
> +
> + - AfPauseState:
> + type: int32_t
> + description: |
> + Only applicable in continuous (AfModeContinuous) mode, this reports whether
> + the algorithm is currently running, paused or pausing (that is, will pause
> + as soon as any in-progress scan completes).
> +
> + Any change to AfMode will cause AfPauseStateRunning to be reported.
> +
> + enum:
> + - name: AfPauseStateRunning
> + value: 0
> + description: |
> + Continuous AF is running and the algorithm may restart a scan
> + spontaneously.
> + - name: AfPauseStatePausing
> + value: 1
> + description: |
> + Continuous AF has been sent an AfPauseDeferred control, and will pause
> + as soon as any in-progress scan completes (and then report
> + AfPauseStatePaused). No new scans will be start spontaneously until
> + the AfPauseResume control is sent.
> + - name: AfPauseStatePaused
> + value: 2
> + description: |
> + Continuous AF is paused. No further state changes or lens movements
> + will occur until the AfPauseResume control is sent.
> +
Very nice overall! With a few more clarifications I think this is good
to go!
Thanks
j
> # ----------------------------------------------------------------------------
> # Draft controls section
>
> @@ -406,27 +664,6 @@ controls:
> The camera will cancel any active or completed metering sequence.
> The AE algorithm is reset to its initial state.
>
> - - AfTrigger:
> - type: int32_t
> - draft: true
> - description: |
> - Control for AF trigger. Currently identical to
> - ANDROID_CONTROL_AF_TRIGGER.
> -
> - Whether the camera device will trigger autofocus for this request.
> - enum:
> - - name: AfTriggerIdle
> - value: 0
> - description: The trigger is idle.
> - - name: AfTriggerStart
> - value: 1
> - description: The AF routine is started by the camera.
> - - name: AfTriggerCancel
> - value: 2
> - description: |
> - The camera will cancel any active trigger and the AF routine is
> - reset to its initial state.
> -
> - NoiseReductionMode:
> type: int32_t
> draft: true
> @@ -507,45 +744,6 @@ controls:
> The AE algorithm has started a pre-capture metering session.
> \sa AePrecaptureTrigger
>
> - - AfState:
> - type: int32_t
> - draft: true
> - description: |
> - Control to report the current AF algorithm state. Currently identical to
> - ANDROID_CONTROL_AF_STATE.
> -
> - Current state of the AF algorithm.
> - enum:
> - - name: AfStateInactive
> - value: 0
> - description: The AF algorithm is inactive.
> - - name: AfStatePassiveScan
> - value: 1
> - description: |
> - AF is performing a passive scan of the scene in continuous
> - auto-focus mode.
> - - name: AfStatePassiveFocused
> - value: 2
> - description: |
> - AF believes the scene is in focus, but might restart scanning.
> - - name: AfStateActiveScan
> - value: 3
> - description: |
> - AF is performing a scan triggered by an AF trigger request.
> - \sa AfTrigger
> - - name: AfStateFocusedLock
> - value: 4
> - description: |
> - AF believes has focused correctly and has locked focus.
> - - name: AfStateNotFocusedLock
> - value: 5
> - description: |
> - AF has not been able to focus and has locked.
> - - name: AfStatePassiveUnfocused
> - value: 6
> - description: |
> - AF has completed a passive scan without finding focus.
> -
> - AwbState:
> type: int32_t
> draft: true
> --
> 2.30.2
>
More information about the libcamera-devel
mailing list