[libcamera-devel] [PATCH v2 1/1] libcamera: controls: Controls for driving AF (autofocus) algorithms
Jacopo Mondi
jacopo at jmondi.org
Tue Apr 5 22:36:10 CEST 2022
Hi David,
sorry for being late :)
On Thu, Mar 31, 2022 at 04:17:47PM +0100, David Plowman via libcamera-devel wrote:
> This patch describes a series of controls that allow applications to
> drive AF algorithms:
>
> AfMode - manual, auto or continuous
> AfLensPosition - set lens position to macro/hyperfocal/infinity
> AfRange - full, macro or normal
> AfSpeed - fast or slow
> AfWindows - AF window locations
> AfTrigger - start (trigger an AF scan) or cancel
> AfPause - pause continuous AF
> LensPosition - position of lens from lens driver
> AfState - reports the mode, whether scanning/success/failure
> ---
> src/libcamera/control_ids.yaml | 373 +++++++++++++++++++++++++++------
> 1 file changed, 313 insertions(+), 60 deletions(-)
>
> diff --git a/src/libcamera/control_ids.yaml b/src/libcamera/control_ids.yaml
> index 9d4638ae..23607368 100644
> --- a/src/libcamera/control_ids.yaml
> +++ b/src/libcamera/control_ids.yaml
> @@ -406,27 +406,6 @@ controls:
> The camera will cancel any active or completed metering sequence.
> The AE algorithm is reset to its initial state.
>
> - - AfTrigger:
> - type: int32_t
> - draft: true
> - description: |
> - Control for AF trigger. Currently identical to
> - ANDROID_CONTROL_AF_TRIGGER.
> -
> - Whether the camera device will trigger autofocus for this request.
> - enum:
> - - name: AfTriggerIdle
> - value: 0
> - description: The trigger is idle.
> - - name: AfTriggerStart
> - value: 1
> - description: The AF routine is started by the camera.
> - - name: AfTriggerCancel
> - value: 2
> - description: |
> - The camera will cancel any active trigger and the AF routine is
> - reset to its initial state.
> -
> - NoiseReductionMode:
> type: int32_t
> draft: true
> @@ -507,45 +486,6 @@ controls:
> The AE algorithm has started a pre-capture metering session.
> \sa AePrecaptureTrigger
>
> - - AfState:
> - type: int32_t
> - draft: true
> - description: |
> - Control to report the current AF algorithm state. Currently identical to
> - ANDROID_CONTROL_AF_STATE.
> -
> - Current state of the AF algorithm.
> - enum:
> - - name: AfStateInactive
> - value: 0
> - description: The AF algorithm is inactive.
> - - name: AfStatePassiveScan
> - value: 1
> - description: |
> - AF is performing a passive scan of the scene in continuous
> - auto-focus mode.
> - - name: AfStatePassiveFocused
> - value: 2
> - description: |
> - AF believes the scene is in focus, but might restart scanning.
> - - name: AfStateActiveScan
> - value: 3
> - description: |
> - AF is performing a scan triggered by an AF trigger request.
> - \sa AfTrigger
> - - name: AfStateFocusedLock
> - value: 4
> - description: |
> - AF believes has focused correctly and has locked focus.
> - - name: AfStateNotFocusedLock
> - value: 5
> - description: |
> - AF has not been able to focus and has locked.
> - - name: AfStatePassiveUnfocused
> - value: 6
> - description: |
> - AF has completed a passive scan without finding focus.
> -
> - AwbState:
> type: int32_t
> draft: true
> @@ -690,4 +630,317 @@ controls:
> value. All of the custom test patterns will be static (that is the
> raw image must not vary from frame to frame).
>
> + - AfMode:
> + type: int32_t
> + draft: true
Was having all these controls as draft intentional ? The previous
definitions where set to drafts as they mapped to the android
definition, but now that we're defining our own ones...
> + description: |
> + Control to set the mode of the AF (autofocus) algorithm. Applications
> + are allowed to set a new mode, and to send additional controls for
> + that new mode, in the same request.
> +
> + An implementation may choose not to implement all the modes.
> +
> + enum:
> + - name: AfModeManual
> + value: 0
> + description: |
> + The AF algorithm is in manual mode. In this mode it will never
> + perform any action nor move the lens of its own accord, but an
> + application can set controls to move the lens.
> +
> + In this mode the AfState will always report AfStateManual.
> + - name: AfModeAuto
> + value: 1
> + description: |
> + The AF algorithm is in auto mode. This means that the algorithm
> + will never move the lens or change state unless the AfTrigger
> + control is used. The AfTrigger control can be used to initiate a
> + focus scan, the results of which will also be reported by AfState.
> +
> + If the autofocus algorithm is moved from AfModeAuto to another
> + mode while a scan is in progress, the scan is cancelled
> + immediately, without waiting for the scan to finish.
> +
> + When first entering this mode the AfState will report
> + AfStateAuto. When a trigger control is sent, AfState will
> + report AfStateAutoScanning for a period before spontaneously
> + changing to AfStateAutoFocused or AfStateAutoFailed, depending on
> + the outcome of the scan. It will remain in this state until
> + another scan is initiated by the AfTrigger control. If a scan is
> + cancelled (without changing to another mode), AfState will return
> + to AfStateAuto.
> + - name: AfModeContinuous
> + value: 2
> + description: |
> + The AF algorithm is in continuous mode. This means that the lens
> + can re-start a scan spontaneously at any moment, without any user
> + intervention. The AfState still reports whether the algorithm is
> + currently scanning or not, though the application has no ability
> + to initiate or cancel scans (though it can "pause" them), nor to
> + move the lens for itself.
> +
> + When set to AfModeContinuous, the system will immediately initiate
> + a scan so AfState will report AfStateContinuousScanning, and will
> + settle on one of AfStateContinuousFocused or AfStateContinuousFailed,
> + depending on the scan result.
> +
> + - AfLensPosition:
> + type: int32_t
> + draft: true
> + description: |
> + Control to set the position of the lens to one of the following
> + predefined locations. This control only has any effect when the AfMode
> + is set to AfModeManual.
> +
> + This control is distinct from the LensPosition control, which sets the
> + lens position using the lens driver's units.
> + enum:
> + - name: AfLensPositionMacro
> + value: 0
> + description: The closest focal position of the lens.
> + - name: AfLensPositionHyperfocal
> + value: 1
> + description: Hyperfocal position.
> + - name: AfLensPositionInfinity
> + value: 2
> + description: The furthest focal position (usually infinity).
Interesting, bikeshedding on the name apart, is this some 'simplified'
LensState ? Do you think there is a use case for this ? Why would
anyone set the AF to auto mode then use some coarse grained control
like this one ? Wouldn't it make more sense as an 'hint' to the AF
algorithm to restrict the search space ?
(Btw I have a use case for supplying a range of lens positions, which
has been previously estimated according to some pre-defined use cases,
to the AF algorithm in order restrict the AF search range to increase
the speed of a scan. Do you think it's a valid use case and it's worth a
control ?)
> +
> + - AfRange:
> + type: int32_t
> + draft: true
> + description: |
> + Control to set the range of focus distances that is scanned. An
> + implementation may choose not to implement all the options here.
Only valid in Auto and Continuous mode, right ?
(This makes me think that we could easily model Continuous by adding a
TriggerContinuous when in Auto mode. With a TriggerPause we could get rid of
AfPause... well, it's quite a redesign so I'll stop here unless
someone thinks it's a really good idea :)
> + enum:
> + - name: AfRangeNormal
> + value: 0
> + description: |
> + A wide range of focus distances is scanned, all the way from
> + infinity down to close distances, though depending on the
> + implementation, possibly not including the very closest macro
> + positions.
> + - name: AfRangeMacro
> + value: 1
> + description: Only close distances are scanned.
> + - name: AfRangeFull
> + value: 2
> + description: |
> + The full range of focus distances is scanned just as with
> + AfRangeNormal but this time including the very closest macro
> + positions.
> +
> + - AfSpeed:
> + type: int32_t
> + draft: true
> + description: |
> + Control that determines whether the AF algorithm is to move the lens
> + as quickly as possible or more steadily. For example, during video
> + recording it may be desirable not to move the lens too abruptly, but
> + when in a preview mode (waiting for a still capture) it may be
> + helpful to move the lens as quickly as is reasonably possible.
> + enum:
> + - name: AfSpeedNormal
> + value: 0
> + description: Move the lens at its usual speed.
> + - name: AfSpeedFast
> + value: 1
> + description: Move the lens more quickly.
> +
> + - AfWindows:
> + type: Rectangle
> + draft: true
> + description: |
> + Sets the focus windows used by the AF algorithm. The units used express
> + a proportion of the ScalerCrop control (or if unavailable, of the entire
> + image), as u0.16 format numbers.
> +
> + In order to be activated, a rectangle must be programmed with non-zero
> + width and height. If no rectangles are programmed in this way, then the
> + system will choose its own single default window in the centre of the
> + image.
> +
> + The details of how the windows are used are platform dependent. We note
> + that when there is more than one AF window, a typical implementation
> + might find the optimal focus position for each one and finally select
> + the window closest to the camera.
> +
> + size: [platform dependent]
I think this should just be
size: [n]
> +
> + - AfTrigger:
> + type: int32_t
> + draft: true
> + description: |
> + This control starts an autofocus scan when AfMode is set to AfModeAuto,
> + and can also be used to terminate a scan early.
> +
> + It is ignored if AfMode is set to AfModeManual or AfModeContinuous.
> +
> + enum:
> + - name: AfTriggerStart
> + value: 0
> + description: Start an AF scan. Ignored if a scan is in progress.
> + - name: AfTriggerCancel
> + value: 1
> + description: Cancel an AF scan. This does not cause the lens to move
> + anywhere else. Ignored if no scan is in progress.
> +
> + - AfPause:
> + type: int32_t
> + draft: true
> + description: |
> + This control has no effect except when in continuous autofocus mode
> + (AfModeContinuous). It can be used to pause any lens movements while
> + (for example) images are captured. The algorithm remains inactive
> + until it is instructed to resume.
> +
> + enum:
> + - name: AfPauseImmediate
> + value: 0
> + description: |
> + Pause the continuous autofocus algorithm immediately, whether or
> + not any kind of scan is underway. If the AfState was previously
> + reporting AfStateContinuousScanning it will now change to
> + AfStateContinuousScanningPaused, and similarly for
> + AfStateContinuousFocused and AfStateContinuousFailed.
> + - name AfPauseDeferred
> + value: 1
> + description: |
> + Pause the continuous autofocus algorithm as soon as it is no longer
> + scanning. If the AfState is currently reporting
> + AfStateContinuousFocused is will change to
> + AfStateContinuousFocusedPaused, and similarly in the case of
> + AfStateContinuousFailed.
> +
> + If AfState reports AfStateContinuousScanning it will change to
> + AfStateContinuousScanningPausing, and then move to one of
> + AfStateContinuousFocusedPaused or AfStateContinuousFailedPaused
> + when the scan completes.
> + - name: AfPauseResume
> + value: 2
> + description: |
> + Resume continous autofocus operation. The algorithm starts again
> + from exactly where it left off, and the AfState will drop the
> + 'Paused' or 'Pausing' part of the state.
> +
> + - LensPosition:
> + type: int32_t
> + draft: true
> + description: |
> + Acts as a control to instruct the lens to move to a particular position
> + and also reports back the position of the lens for each frame.
> +
> + The units are determined by the lens driver.
Ugh, we cannot do this in the control definition. Driver specificities
cannot surface to the controls definition, which should be platform
independent.
I think we'll have to establish a numerical range (go ahead if you
have any idea) and translate to the actual driver values in the
CameraLens class ?
> +
> + The LensPosition control is ignored unless the AfMode is set to
> + AfModeManual.
> +
> + Note that this control is distinct from AfLensPosition, which allows
> + the lens to be moved to its macro/hyperfocal/infinity position, rather
> + than using lens driver units.
> +
> + - AfState:
> + type: int32_t
> + draft: true
> + description: |
> + Reports the current state of the AF algorithm. The possible state changes
> + are described below, although we add that the following general state
> + changes are also allowed.
wow, there is a lot of states here :)
Let me copy them here
- name: AfStateManual
- name: AfStateAuto
- name: AfStateAutoScanning
- name: AfStateAutoFocused
- name: AfStateAutoFailed
- name: AfStateContinuousScanning
- name: AfStateContinuousScanningPausing
- name: AfStateContinuousScanningPaused
- name: AfStateContinuousFocused
- name: AfStateContinuousFocusedPaused
- name: AfStateContinuousFailed
- name: AfStateContinuousFailedPaused
(please note that AfStateAuto seems mis-aligned, unless I messed up
something while cutting&paste. Please check the compiled doc to see
how it looks like...)
So we have three macro groups
- Manual
- Auto
- Scanning
- Focus
- Failed
- Continuous
- Scanning
- Pausing
- Paused
- Focused
- Paused
- Failed
- Paused
The first group is AfMode (Manual, Auto, Continuous) and can be
deduced by inspecting the AfMode metadata.
The second group is the actual state (Scannng, Focused, Failed)
The third group is specific to continuous mode as it accepts a Pause
trigger.
I understand why you have this, as expressing "The AfState is set to
$something when in AfModeManual" etc etc might get cumbersome, but I
think it might be worth a try
1) Manual mode (the bad)
As we all know there is a delay between when a lens is instructed
to move and when it reaches its final state.
In Manual mode the lens is moved with LensPosition and
AfLensPosition.
To know the current state:
- With LensPosition metadata can be inspected until it doesn't reach the
desired value.
- With AfLensPosition it is not clear to me how to know when the lens
as reched one of the pre-defined positions.
Is it so wrong to report either (Scanning,Focus,Failed) even for
Manual mode ?
2) AutoMode (the good)
This seems the easy one, as (Scanning,Focus,Failed) are already
there
3) Continous (the ugly)
Continuous mode can be paused. That's a shame :)
Would that be that bad keeping a Paused or Pausing state which can
only be returned for Continuous mode.
All in all, can we reduce the number of states to
- Scanning
The AF algorithm is scanning and has not yet reached a
stable result. When running in Auto mode an AF scan is
started by using the AfTrigger control. When in Manual mode
and a new lens position is requested either by sending a
LensPosition or AfLensPosition control, the AF state will
report "Scanning" until the lens doesn't reach the desired
position. Similarly, when running in continuous mode and the
AF algorithm decides to start a scan spontaneously the
AfState will report Scanning until a new stable state is
reached.
- Focus
The AF algorithm has reached a stable state and the image is
now focused. When running in Manual mode the lens has
reached the position requested using either LensPosition and
AfLensPosition.
- Failed
The AF algorithm has completed a scan and without finding a
good focus position. When running in manual mode, the camera
has failed moving the lens to the desired position.
- Paused (copying your below text, please see comments below)
The AF algorithm is in continuous mode (AfModeContinuous) and
was scanning when AfPauseImmediate was sent. The scan will not
complete and the algorithm will remain in this state.
- Pausing
The AF algorithm is in continuous mode (AfModeContinuous) and
was scanning when AfPauseDeferred was sent. The scan will complete
at which point the algorithm moves to the
AfStateContinuousFocusedPaused or AfStateContinuousFailedPaused
state.
> +
> + In any of the AfStateManual or AfStateContinuous states, the AfMode
> + may be set to AfModeAuto and the algorithm will move to the
> + AfStateAuto state. If AfTriggerStart is sent at the same time
> + then the algorithm will move to AfStateAutoScanning state.
> +
> + In any of the AfStateAuto or AfStateManual states, the AfMode may
> + be set to AfModeContinuous and the algorithm will move to
> + AfStateContinuousScanning.
> +
> + In any of the AfStateAuto or AfStateContinuous states, the AfMode may
> + be set to AfModeManual and the algorithm will move to
> + AfStateManual. The lens will not be moved and will be left where
> + it was at that moment.
> +
> + enum:
> + - name: AfStateManual
> + value: 0
> + description: |
> + The AF algorithm is in manual mode (AfModeManual). The LensPosition
> + and AfLensPosition controls can be used directly to move the lens.
> + - name: AfStateAuto
> + value: 1
> + description: |
> + The AF algorithm is in auto mode (AfModeAuto), and has either just
> + been moved into that state, or a scan that was in progress has been
> + cancelled.
> + - name: AfStateAutoScanning
> + value: 2
> + description: |
> + The AF algorithm is in auto mode (AfModeAuto), and a scan has been
> + started using the AfTrigger control. The scan can be cancelled by
> + sending AfTriggerCancel at which point the algorithm will either
> + move back to AfStateAuto or, if the scan actually completes
> + before the cancel request is processed, to one of
> + AfStateAutoFocused or AfStateAutoFailed.
> + - name: AfStateAutoFocused
> + value: 3
> + description: |
> + The AF algorithm is in auto mode (AfModeAuto), and a scan has
> + completed with the result that the algorithm believes the image is
> + now in focus.
> + - name: AfStateAutoFailed
> + value: 4
> + description: |
> + The AF algorithm is in auto mode (AfModeAuto), and a scan has
> + completed with the result that the algorithm did not find a good
> + focus position.
> + - name: AfStateContinuousScanning
> + value: 5
> + description: |
> + The AF algorithm is in continuous mode (AfModeContinuous) and
> + is currently scanning for a good focus position. This occurs when
> + the mode is first set to AfModeContinuous, or may happen
> + spontaneously when the algorithm believes a re-scan is needed.
> + - name: AfStateContinuousScanningPausing
> + value: 6
> + description: |
> + The AF algorithm is in continuous mode (AfModeContinuous) and
> + was scanning when AfPauseDeferred was sent. The scan will complete
> + at which point the algorithm moves to the
> + AfStateContinuousFocusedPaused or AfStateContinuousFailedPaused
> + state.
> + - name: AfStateContinuousScanningPaused
> + value: 7
> + description: |
> + The AF algorithm is in continuous mode (AfModeContinuous) and
> + was scanning when AfPauseImmediate was sent. The scan will not
Why can't the algorithm be paused when it has reached a stable state
but only when scanning ? Won't a pause 'freeze' the algorithm in its
current state no matter what it is ? I understand with
"AfPauseDeferred" that it has to wait for AF to be locked before
stopping it, but with "Immediate" doesn't "pause" apply to all states ?
Thanks
j
> + complete and the algorithm will remain in this state. The scan
> + may be resumed by sending AfPauseResume.
> + - name: AfStateContinuousFocused
> + value: 8
> + description: |
> + The AF algorithm is in continuous mode (AfModeContinuous) and
> + a scan has completed with the algorithm believing it has found a
> + good focus position.
> + - name: AfStateContinuousFocusedPaused
> + value: 9
> + description: |
> + The AF algorithm was in the AfStateContinuousFocused state and
> + has been paused (by either AfPauseImmediate or AfPauseDeferred),
> + or it was in the AfStateContinuousScanningPausing state and the
> + scan has completed successfully. The algorithm will now remain
> + in this state, and may be resumed by sending AfPauseResume.
> + - name: AfStateContinuousFailed
> + value: 10
> + description: |
> + The AF algorithm is in continuous mode (AfModeContinuous) and
> + a scan has completed without finding a good focus position.
> + - name: AfStateContinuousFailedPaused
> + value: 11
> + description: |
> + The AF algorithm was in the AfStateContinuousFailed state and
> + has been paused (by either AfPauseImmediate or AfPauseDeferred),
> + or it was in the AfStateContinuousScanningPausing state and the
> + scan has completed unsuccessfully. The algorithm will now remain
> + in this state, and may be resumed by sending AfPauseResume.
> +
> ...
> --
> 2.30.2
>
More information about the libcamera-devel
mailing list