[libcamera-devel] [PATCH v2 1/1] libcamera: controls: Controls for driving AF (autofocus) algorithms
David Plowman
david.plowman at raspberrypi.com
Mon Apr 4 14:05:50 CEST 2022
Hi
Thanks to those who have had a chance to look at this!
On Fri, 1 Apr 2022 at 10:37, Hanlin Chen <hanlinchen at chromium.org> wrote:
>
> Hi David, thanks for the patch!
> Reviewed-by: Han-Lin Chen <hanlinchen at chromium.org>
>
> On Thu, Mar 31, 2022 at 11:17 PM David Plowman <david.plowman at raspberrypi.com> wrote:
>>
>> This patch describes a series of controls that allow applications to
>> drive AF algorithms:
>>
>> AfMode - manual, auto or continuous
>> AfLensPosition - set lens position to macro/hyperfocal/infinity
>> AfRange - full, macro or normal
>> AfSpeed - fast or slow
>> AfWindows - AF window locations
>> AfTrigger - start (trigger an AF scan) or cancel
>> AfPause - pause continuous AF
>> LensPosition - position of lens from lens driver
>> AfState - reports the mode, whether scanning/success/failure
>> ---
>> src/libcamera/control_ids.yaml | 373 +++++++++++++++++++++++++++------
>> 1 file changed, 313 insertions(+), 60 deletions(-)
>>
>> diff --git a/src/libcamera/control_ids.yaml b/src/libcamera/control_ids.yaml
>> index 9d4638ae..23607368 100644
>> --- a/src/libcamera/control_ids.yaml
>> +++ b/src/libcamera/control_ids.yaml
>> @@ -406,27 +406,6 @@ controls:
>> The camera will cancel any active or completed metering sequence.
>> The AE algorithm is reset to its initial state.
>>
>> - - AfTrigger:
>> - type: int32_t
>> - draft: true
>> - description: |
>> - Control for AF trigger. Currently identical to
>> - ANDROID_CONTROL_AF_TRIGGER.
>> -
>> - Whether the camera device will trigger autofocus for this request.
>> - enum:
>> - - name: AfTriggerIdle
>> - value: 0
>> - description: The trigger is idle.
>> - - name: AfTriggerStart
>> - value: 1
>> - description: The AF routine is started by the camera.
>> - - name: AfTriggerCancel
>> - value: 2
>> - description: |
>> - The camera will cancel any active trigger and the AF routine is
>> - reset to its initial state.
>> -
>> - NoiseReductionMode:
>> type: int32_t
>> draft: true
>> @@ -507,45 +486,6 @@ controls:
>> The AE algorithm has started a pre-capture metering session.
>> \sa AePrecaptureTrigger
>>
>> - - AfState:
>> - type: int32_t
>> - draft: true
>> - description: |
>> - Control to report the current AF algorithm state. Currently identical to
>> - ANDROID_CONTROL_AF_STATE.
>> -
>> - Current state of the AF algorithm.
>> - enum:
>> - - name: AfStateInactive
>> - value: 0
>> - description: The AF algorithm is inactive.
>> - - name: AfStatePassiveScan
>> - value: 1
>> - description: |
>> - AF is performing a passive scan of the scene in continuous
>> - auto-focus mode.
>> - - name: AfStatePassiveFocused
>> - value: 2
>> - description: |
>> - AF believes the scene is in focus, but might restart scanning.
>> - - name: AfStateActiveScan
>> - value: 3
>> - description: |
>> - AF is performing a scan triggered by an AF trigger request.
>> - \sa AfTrigger
>> - - name: AfStateFocusedLock
>> - value: 4
>> - description: |
>> - AF believes has focused correctly and has locked focus.
>> - - name: AfStateNotFocusedLock
>> - value: 5
>> - description: |
>> - AF has not been able to focus and has locked.
>> - - name: AfStatePassiveUnfocused
>> - value: 6
>> - description: |
>> - AF has completed a passive scan without finding focus.
>> -
>> - AwbState:
>> type: int32_t
>> draft: true
>> @@ -690,4 +630,317 @@ controls:
>> value. All of the custom test patterns will be static (that is the
>> raw image must not vary from frame to frame).
>>
>> + - AfMode:
>> + type: int32_t
>> + draft: true
>> + description: |
>> + Control to set the mode of the AF (autofocus) algorithm. Applications
>> + are allowed to set a new mode, and to send additional controls for
>> + that new mode, in the same request.
>> +
>> + An implementation may choose not to implement all the modes.
>> +
>> + enum:
>> + - name: AfModeManual
>> + value: 0
>> + description: |
>> + The AF algorithm is in manual mode. In this mode it will never
>> + perform any action nor move the lens of its own accord, but an
>> + application can set controls to move the lens.
>> +
>> + In this mode the AfState will always report AfStateManual.
>> + - name: AfModeAuto
>> + value: 1
>> + description: |
>> + The AF algorithm is in auto mode. This means that the algorithm
>> + will never move the lens or change state unless the AfTrigger
>> + control is used. The AfTrigger control can be used to initiate a
>> + focus scan, the results of which will also be reported by AfState.
>> +
>> + If the autofocus algorithm is moved from AfModeAuto to another
>> + mode while a scan is in progress, the scan is cancelled
>> + immediately, without waiting for the scan to finish.
>> +
>> + When first entering this mode the AfState will report
>> + AfStateAuto. When a trigger control is sent, AfState will
>> + report AfStateAutoScanning for a period before spontaneously
>> + changing to AfStateAutoFocused or AfStateAutoFailed, depending on
>> + the outcome of the scan. It will remain in this state until
>> + another scan is initiated by the AfTrigger control. If a scan is
>> + cancelled (without changing to another mode), AfState will return
>> + to AfStateAuto.
>> + - name: AfModeContinuous
>> + value: 2
>> + description: |
>> + The AF algorithm is in continuous mode. This means that the lens
>> + can re-start a scan spontaneously at any moment, without any user
>> + intervention. The AfState still reports whether the algorithm is
>> + currently scanning or not, though the application has no ability
>> + to initiate or cancel scans (though it can "pause" them), nor to
>> + move the lens for itself.
>> +
>> + When set to AfModeContinuous, the system will immediately initiate
>> + a scan so AfState will report AfStateContinuousScanning, and will
>> + settle on one of AfStateContinuousFocused or AfStateContinuousFailed,
>> + depending on the scan result.
>> +
>> + - AfLensPosition:
>> + type: int32_t
>> + draft: true
>> + description: |
>> + Control to set the position of the lens to one of the following
>> + predefined locations. This control only has any effect when the AfMode
>> + is set to AfModeManual.
>> +
>> + This control is distinct from the LensPosition control, which sets the
>> + lens position using the lens driver's units.
>> + enum:
>> + - name: AfLensPositionMacro
>> + value: 0
>> + description: The closest focal position of the lens.
>> + - name: AfLensPositionHyperfocal
>> + value: 1
>> + description: Hyperfocal position.
>> + - name: AfLensPositionInfinity
>> + value: 2
>> + description: The furthest focal position (usually infinity).
>> +
>> + - AfRange:
>> + type: int32_t
>> + draft: true
>> + description: |
>> + Control to set the range of focus distances that is scanned. An
>> + implementation may choose not to implement all the options here.
>> + enum:
>> + - name: AfRangeNormal
>> + value: 0
>> + description: |
>> + A wide range of focus distances is scanned, all the way from
>> + infinity down to close distances, though depending on the
>> + implementation, possibly not including the very closest macro
>> + positions.
>> + - name: AfRangeMacro
>> + value: 1
>> + description: Only close distances are scanned.
>> + - name: AfRangeFull
>> + value: 2
>> + description: |
>> + The full range of focus distances is scanned just as with
>> + AfRangeNormal but this time including the very closest macro
>> + positions.
>> +
>> + - AfSpeed:
>> + type: int32_t
>> + draft: true
>> + description: |
>> + Control that determines whether the AF algorithm is to move the lens
>> + as quickly as possible or more steadily. For example, during video
>> + recording it may be desirable not to move the lens too abruptly, but
>> + when in a preview mode (waiting for a still capture) it may be
>> + helpful to move the lens as quickly as is reasonably possible.
>> + enum:
>> + - name: AfSpeedNormal
>> + value: 0
>> + description: Move the lens at its usual speed.
>> + - name: AfSpeedFast
>> + value: 1
>> + description: Move the lens more quickly.
>> +
>> + - AfWindows:
>> + type: Rectangle
>> + draft: true
>> + description: |
>> + Sets the focus windows used by the AF algorithm. The units used express
>> + a proportion of the ScalerCrop control (or if unavailable, of the entire
>> + image), as u0.16 format numbers.
>> +
>> + In order to be activated, a rectangle must be programmed with non-zero
>> + width and height. If no rectangles are programmed in this way, then the
>> + system will choose its own single default window in the centre of the
>> + image.
>> +
>> + The details of how the windows are used are platform dependent. We note
>> + that when there is more than one AF window, a typical implementation
>> + might find the optimal focus position for each one and finally select
>> + the window closest to the camera.
>> +
>> + size: [platform dependent]
>> +
>> + - AfTrigger:
>> + type: int32_t
>> + draft: true
>> + description: |
>> + This control starts an autofocus scan when AfMode is set to AfModeAuto,
>> + and can also be used to terminate a scan early.
>> +
>> + It is ignored if AfMode is set to AfModeManual or AfModeContinuous.
>> +
>> + enum:
>> + - name: AfTriggerStart
>> + value: 0
>> + description: Start an AF scan. Ignored if a scan is in progress.
>> + - name: AfTriggerCancel
>> + value: 1
>> + description: Cancel an AF scan. This does not cause the lens to move
>> + anywhere else. Ignored if no scan is in progress.
>> +
>> + - AfPause:
>> + type: int32_t
>> + draft: true
>> + description: |
>> + This control has no effect except when in continuous autofocus mode
>> + (AfModeContinuous). It can be used to pause any lens movements while
>> + (for example) images are captured. The algorithm remains inactive
>> + until it is instructed to resume.
>> +
>> + enum:
>> + - name: AfPauseImmediate
>> + value: 0
>> + description: |
>> + Pause the continuous autofocus algorithm immediately, whether or
>> + not any kind of scan is underway. If the AfState was previously
>> + reporting AfStateContinuousScanning it will now change to
>> + AfStateContinuousScanningPaused, and similarly for
>> + AfStateContinuousFocused and AfStateContinuousFailed.
>> + - name AfPauseDeferred
Thanks for pointing out the typo here, Jean-Michel. I will fix it in
the next revision (I'll just wait a bit and see if any other reviews
come in..).
>> + value: 1
>> + description: |
>> + Pause the continuous autofocus algorithm as soon as it is no longer
>> + scanning. If the AfState is currently reporting
>> + AfStateContinuousFocused is will change to
>> + AfStateContinuousFocusedPaused, and similarly in the case of
>> + AfStateContinuousFailed.
>> +
>> + If AfState reports AfStateContinuousScanning it will change to
>> + AfStateContinuousScanningPausing, and then move to one of
>> + AfStateContinuousFocusedPaused or AfStateContinuousFailedPaused
>> + when the scan completes.
>> + - name: AfPauseResume
>> + value: 2
>> + description: |
>> + Resume continous autofocus operation. The algorithm starts again
>> + from exactly where it left off, and the AfState will drop the
>> + 'Paused' or 'Pausing' part of the state.
>> +
>> + - LensPosition:
>> + type: int32_t
>> + draft: true
>> + description: |
>> + Acts as a control to instruct the lens to move to a particular position
>> + and also reports back the position of the lens for each frame.
>> +
>> + The units are determined by the lens driver.
>> +
>> + The LensPosition control is ignored unless the AfMode is set to
>> + AfModeManual.
>> +
>> + Note that this control is distinct from AfLensPosition, which allows
>> + the lens to be moved to its macro/hyperfocal/infinity position, rather
>> + than using lens driver units.
>> +
>> + - AfState:
>> + type: int32_t
>> + draft: true
>> + description: |
>> + Reports the current state of the AF algorithm. The possible state changes
>> + are described below, although we add that the following general state
>> + changes are also allowed.
>> +
>> + In any of the AfStateManual or AfStateContinuous states, the AfMode
>> + may be set to AfModeAuto and the algorithm will move to the
>> + AfStateAuto state. If AfTriggerStart is sent at the same time
>> + then the algorithm will move to AfStateAutoScanning state.
>> +
>> + In any of the AfStateAuto or AfStateManual states, the AfMode may
>> + be set to AfModeContinuous and the algorithm will move to
>> + AfStateContinuousScanning.
>> +
>> + In any of the AfStateAuto or AfStateContinuous states, the AfMode may
>> + be set to AfModeManual and the algorithm will move to
>> + AfStateManual. The lens will not be moved and will be left where
>> + it was at that moment.
>> +
>> + enum:
>> + - name: AfStateManual
>> + value: 0
>> + description: |
>> + The AF algorithm is in manual mode (AfModeManual). The LensPosition
>> + and AfLensPosition controls can be used directly to move the lens.
>> + - name: AfStateAuto
>> + value: 1
>> + description: |
>> + The AF algorithm is in auto mode (AfModeAuto), and has either just
>> + been moved into that state, or a scan that was in progress has been
>> + cancelled.
>> + - name: AfStateAutoScanning
>> + value: 2
>> + description: |
>> + The AF algorithm is in auto mode (AfModeAuto), and a scan has been
>> + started using the AfTrigger control. The scan can be cancelled by
>> + sending AfTriggerCancel at which point the algorithm will either
>> + move back to AfStateAuto or, if the scan actually completes
>> + before the cancel request is processed, to one of
>> + AfStateAutoFocused or AfStateAutoFailed.
>> + - name: AfStateAutoFocused
>> + value: 3
>> + description: |
>> + The AF algorithm is in auto mode (AfModeAuto), and a scan has
>> + completed with the result that the algorithm believes the image is
>> + now in focus.
>> + - name: AfStateAutoFailed
>> + value: 4
>> + description: |
>> + The AF algorithm is in auto mode (AfModeAuto), and a scan has
>> + completed with the result that the algorithm did not find a good
>> + focus position.
>> + - name: AfStateContinuousScanning
>> + value: 5
>> + description: |
>> + The AF algorithm is in continuous mode (AfModeContinuous) and
>> + is currently scanning for a good focus position. This occurs when
>> + the mode is first set to AfModeContinuous, or may happen
>> + spontaneously when the algorithm believes a re-scan is needed.
>> + - name: AfStateContinuousScanningPausing
>> + value: 6
>> + description: |
>> + The AF algorithm is in continuous mode (AfModeContinuous) and
>> + was scanning when AfPauseDeferred was sent. The scan will complete
>> + at which point the algorithm moves to the
>> + AfStateContinuousFocusedPaused or AfStateContinuousFailedPaused
>> + state.
>> + - name: AfStateContinuousScanningPaused
>> + value: 7
>> + description: |
>> + The AF algorithm is in continuous mode (AfModeContinuous) and
>> + was scanning when AfPauseImmediate was sent. The scan will not
>> + complete and the algorithm will remain in this state. The scan
>> + may be resumed by sending AfPauseResume.
>> + - name: AfStateContinuousFocused
>> + value: 8
>> + description: |
>> + The AF algorithm is in continuous mode (AfModeContinuous) and
>> + a scan has completed with the algorithm believing it has found a
>> + good focus position.
>> + - name: AfStateContinuousFocusedPaused
>> + value: 9
>> + description: |
>> + The AF algorithm was in the AfStateContinuousFocused state and
>> + has been paused (by either AfPauseImmediate or AfPauseDeferred),
>> + or it was in the AfStateContinuousScanningPausing state and the
>> + scan has completed successfully. The algorithm will now remain
>> + in this state, and may be resumed by sending AfPauseResume.
>> + - name: AfStateContinuousFailed
>> + value: 10
>> + description: |
>> + The AF algorithm is in continuous mode (AfModeContinuous) and
>> + a scan has completed without finding a good focus position.
>> + - name: AfStateContinuousFailedPaused
>> + value: 11
>> + description: |
>> + The AF algorithm was in the AfStateContinuousFailed state and
>> + has been paused (by either AfPauseImmediate or AfPauseDeferred),
>> + or it was in the AfStateContinuousScanningPausing state and the
>> + scan has completed unsuccessfully. The algorithm will now remain
>> + in this state, and may be resumed by sending AfPauseResume.
>> +
>> ...
>> --
>> 2.30.2
>>
I was also wondering if there were any comments on the state machine
diagram or if folks are happy with it. I'm not wild about the
"paused/pausing" thing, but I guess if we want that feature then we
have to distinguish these states. I haven't allowed for transitions
between "AfStateContinuousScanningPausing" and
"AfStateContinuousScanningPaused" which you could imagine might exist,
are people OK with that?
Thanks again
David
More information about the libcamera-devel
mailing list