[libcamera-devel] [PATCH 1/1] libcamera: controls: Controls for driving AF (autofocus) algorithms
Hanlin Chen
hanlinchen at chromium.org
Wed Mar 16 11:19:26 CET 2022
On Wed, Mar 16, 2022 at 12:34 AM David Plowman <
david.plowman at raspberrypi.com> wrote:
> Hi Hanlin, Jean-Michel, everyone
>
> On Mon, 14 Mar 2022 at 11:06, Hanlin Chen <hanlinchen at chromium.org> wrote:
> >
> > Hi David, Thanks for the great patch!
> >
> > On Thu, Mar 10, 2022 at 10:53 PM Jean-Michel Hautbois <
> jeanmichel.hautbois at ideasonboard.com> wrote:
> >>
> >> Hi David,
> >>
> >> Thanks for the patch !
> >>
> >> On 10/03/2022 13:05, David Plowman via libcamera-devel wrote:
> >> > This patch describes a series of controls that allow applications to
> >> > drive AF algorithms:
> >> >
> >> > AfMode - manual, auto or continuous
> >> > AfRange - full, macro or normal
> >> > AfSpeed - fast or slow
> >> > AfWindow - AF window locations
> >> > AfTrigger - start (trigger an AF scan) or cancel
> >> > AfPause - pause continuous AF
> >> > LensPosition - position of lens from lens driver
> >> > AfState - reset, scanning, focused or failed
> >> > ---
> >> > src/libcamera/control_ids.yaml | 271
> +++++++++++++++++++++++++--------
> >> > 1 file changed, 211 insertions(+), 60 deletions(-)
> >> >
> >> > diff --git a/src/libcamera/control_ids.yaml
> b/src/libcamera/control_ids.yaml
> >> > index 9d4638ae..89636d82 100644
> >> > --- a/src/libcamera/control_ids.yaml
> >> > +++ b/src/libcamera/control_ids.yaml
> >> > @@ -406,27 +406,6 @@ controls:
> >> > The camera will cancel any active or completed metering
> sequence.
> >> > The AE algorithm is reset to its initial state.
> >> >
> >> > - - AfTrigger:
> >> > - type: int32_t
> >> > - draft: true
> >> > - description: |
> >> > - Control for AF trigger. Currently identical to
> >> > - ANDROID_CONTROL_AF_TRIGGER.
> >> > -
> >> > - Whether the camera device will trigger autofocus for this
> request.
> >> > - enum:
> >> > - - name: AfTriggerIdle
> >> > - value: 0
> >> > - description: The trigger is idle.
> >> > - - name: AfTriggerStart
> >> > - value: 1
> >> > - description: The AF routine is started by the camera.
> >> > - - name: AfTriggerCancel
> >> > - value: 2
> >> > - description: |
> >> > - The camera will cancel any active trigger and the AF
> routine is
> >> > - reset to its initial state.
> >> > -
> >> > - NoiseReductionMode:
> >> > type: int32_t
> >> > draft: true
> >> > @@ -507,45 +486,6 @@ controls:
> >> > The AE algorithm has started a pre-capture metering
> session.
> >> > \sa AePrecaptureTrigger
> >> >
> >> > - - AfState:
> >> > - type: int32_t
> >> > - draft: true
> >> > - description: |
> >> > - Control to report the current AF algorithm state. Currently
> identical to
> >> > - ANDROID_CONTROL_AF_STATE.
> >> > -
> >> > - Current state of the AF algorithm.
> >> > - enum:
> >> > - - name: AfStateInactive
> >> > - value: 0
> >> > - description: The AF algorithm is inactive.
> >> > - - name: AfStatePassiveScan
> >> > - value: 1
> >> > - description: |
> >> > - AF is performing a passive scan of the scene in
> continuous
> >> > - auto-focus mode.
> >> > - - name: AfStatePassiveFocused
> >> > - value: 2
> >> > - description: |
> >> > - AF believes the scene is in focus, but might restart
> scanning.
> >> > - - name: AfStateActiveScan
> >> > - value: 3
> >> > - description: |
> >> > - AF is performing a scan triggered by an AF trigger
> request.
> >> > - \sa AfTrigger
> >> > - - name: AfStateFocusedLock
> >> > - value: 4
> >> > - description: |
> >> > - AF believes has focused correctly and has locked focus.
> >> > - - name: AfStateNotFocusedLock
> >> > - value: 5
> >> > - description: |
> >> > - AF has not been able to focus and has locked.
> >> > - - name: AfStatePassiveUnfocused
> >> > - value: 6
> >> > - description: |
> >> > - AF has completed a passive scan without finding focus.
> >> > -
> >> > - AwbState:
> >> > type: int32_t
> >> > draft: true
> >> > @@ -690,4 +630,215 @@ controls:
> >> > value. All of the custom test patterns will be static
> (that is the
> >> > raw image must not vary from frame to frame).
> >> >
> >> > + - AfMode:
> >> > + type: int32_t
> >>
> >> Same open question for almost all: why int32_t and not uint8_t ?
> >> We won't have negative values ? Maybe is it yaml ?
>
> The file only seems to contain int32_ts so that's what I copied. Am I
> allowed other things?
>
> >>
> >> > + draft: true
> >> > + description: |
> >> > + Control to set the mode of the AF (autofocus) algorithm.
> Applications
> >> > + are allowed to set a new mode, and to send additional
> controls for
> >> > + that new mode, in the same request.
> >> > + enum:
> >> > + - name: AfModeManual
> >> > + value: 0
> >> > + description: |
> >> > + The AF algorithm is in manual mode. In this mode it will
> never
> >> > + perform any action nor move the lens of its own accord.
> The only
> >> > + autofocus controls that have an immediate effect are AfMode
> (to
> >> > + switch out of manual mode) and LensPosition (so that the
> lens can
> >> > + be moved "manually").
> >>
> >> Alignment changed in the paragraph. I wonder if the part "The only
> >> autofocus [...] "manually")." should be moved in the top description ?
>
> Thanks, I'll fix this up and all the ones below...
>
> >>
> >> > +
> >> > + In this mode the AfState will always report AfStateReset.
> >> > + - name: AfModeAuto
> >> > + value: 1
> >> > + description: |
> >> > + The AF algorithm is in auto mode. This means that the
> algorithm
> >> > + will never move the lens or change state unless the
> AfTrigger
> >> > + control is used. The AfTrigger control can be used to
> initiate a
> >> > + focus scan, the results of which will also be reported
> by AfState.
> >> > +
> >> > + If the autofocus algorithm is moved from AfModeAuto to
> another
> >> > + mode while a scan is in progress, the scan is cancelled
> >> > + immediately, without waiting for the scan to finish.
> >> > +
> >> > + When first entering this mode the AfState will report
> >>
> >> Alignment broken ?
> >>
> >> > + AfStateReset. When a trigger control is sent, AfState will
> >> > + report AfStateScanning for a period before spontaneously
> >> > + changing to AfStateFocused or AfStateFailed, depending on
> the
> >> > + outcome of the scan. It will remain in this state until
> another
> >> > + scan is initiated by the AfTrigger control. If a scan is
> >> > + cancelled (without changing to another mode), AfState will
> return
> >> > + to AfStateReset.
> >> > + - name: AfModeContinuous
> >> > + value: 2
> >> > + description: |
> >> > + The AF algorithm is in continuous mode. This means that
> the lens
> >> > + can re-start a scan spontaneously at any moment, without
> any user
> >> > + intervention. The AfState still reports whether the
> algorithm is
> >> > + currently scanning or not, though the application has no
> ability
> >> > + to initiate or cancel scans, nor move the lens for
> itself.
> >> > +
> >>
> >> Alignement broken ?
> >>
> >> > + When set to AfModeContinuous, the system will immediately
> initiate
> >> > + a scan so AfState will report AfStateScanning, and will
> settle on
> >> > + one of AfStateFocused or AfStateFailed, depending on the
> scan
> >> > + result.
> >> > +
> >> > + The continuous autofocus behaviour can be paused with the
> >> > + AfPause control. Pausing the algorithm does not change the
> value
> >> > + reported by AfState, so that applications can determine the
> >> > + state of the algorithm when the pause control took effect.
> Once
> >> > + un-paused ("resumed"), the algorithm starts again from
> exactly
> >> > + where it left off when it paused.
> >
> > An open question:
> > Should we make Macro and Continuous mandatory? Or they can be optional
> depending on the algorithm implementation?
>
> I'd certainly like CAF (Continuous AF) to be optional. It's a feature
> you could choose not to implement as it's quite a lot of work, and
> hard to make work well (especially without PDAF).
>
> Something like the "Macro" range... don't mind. We could make it
> optional, or define it to be the same as "Normal". What do people
> think?
>
> >>
> >>
> >> I find this difficult to follow without having a state machine in some
> >> drawn way... Is it something we could include in the documentation ?
> >> Seeing which Control changes to which one, which states are then
> >> triggered, etc. ?
> >>
> >> > +
> >> > + - AfRange:
> >> > + type: int32_t
> >> > + draft: true
> >> > + description: |
> >> > + Control to set the range of focus distances that is scanned.
> >> > + enum:
> >> > + - name: AfRangeNormal
> >> > + value: 0
> >> > + description: |
> >> > + A wide range of focus distances is scanned, all the way from
> >> > + infinity down to close distances, though depending on the
> >> > + implementation, possibly not including the very closest
> macro
> >> > + positions.
> >>
> >> Alignment broken ?
> >>
> >> > + - name: AfRangeMacro
> >> > + value: 1
> >> > + description: Only close distances are scanned.
> >> > + - name: AfRangeFull
> >> > + value: 2
> >> > + description: |
> >> > + The full range of focus distances is scanned just as with
> >> > + AfRangeNormal but this time including the very closest macro
> >> > + positions.
> >> > +
> >> > + - AfSpeed:
> >> > + type: int32_t
> >> > + draft: true
> >> > + description: |
> >> > + Control that determines whether the AF algorithm is to move
> the lens
> >> > + as quickly as possible or more steadily. For example, during
> video
> >> > + recording it may be desirable not to move the lens too
> abruptly, but
> >> > + when in a preview mode (waiting for a still capture) it may
> be
> >> > + helpful to move the lens as quickly as is reasonably
> possible.
> >> > + enum:
> >> > + - name: AfSpeedNormal
> >> > + value: 0
> >> > + description: Move the lens at its usual speed.
> >> > + - name: AfSpeedFast
> >> > + value: 1
> >> > + description: Move the lens more quickly.
> >> > +
> >>
> >> Can we somehow configure this speed ?
>
> Good question. You could imagine a continuous value here, though that
> does start to force the developer to do more work, and testing gets
> harder. So I feel slightly inclined to leave this as a binary choice,
> but am certainly open to persuasion.
>
> >>
> >> > + - AfWindow:
> >> > + type: Rectangle
> >> > + draft: true
> >> > + description: |
> >> > + Sets the focus windows used by the AF algorithm. The units
> used express
> >> s/windows/window ?
> >> > + a proportion of the ScalerCrop control (or if unavailable,
> of the entire
> >> > + image), as u0.16 format numbers.
> >> > +
> >> > + In order to be activated, a rectangle must be programmed
> with non-zero
> >> > + width and height. If no rectangles are programmed in this
> way, then the
> >> > + system will choose its own single default window in the
> centre of the
> >> > + image.
> >> > +
> >> > + The details of how the windows are used are platform
> dependent. We note
> >> > + that when there is more than one AF window, a typical
> implementation
> >> > + might find the optimal focus position for each one and
> finally select
> >> > + the window closest to the camera.
> >>
> >> Alignment broken ?
> >>
> >> > +
> >> > + size: [platform dependent]
> >> > +
> >> > + - AfTrigger:
> >> > + type: int32_t
> >> > + draft: true
> >> > + description: |
> >> > + This control starts an autofocus scan when AfMode is set to
> AfModeAuto,
> >> > + and can also be used to terminate a scan early.
> >> > +
> >> > + It is ignored if AfMode is set to AfModeContinuous.
> >> > +
> >> > + enum:
> >> > + - name: AfTriggerStart
> >> > + value: 0
> >> > + description: Start an AF scan. Ignored if a scan is in
> progress.
> >> > + - name: AfTriggerCancel
> >> > + value: 1
> >> > + description: Cancel an AF scan. Ingored if no scan is in
> progress.
> >>
> >> Does cancelling a scan make the lens stay at its current position ? Or
> >> should it be reset ?
>
> Stay at the current position. It's your (or the application's) choice
> whether to reset the lens. I'll make that clearer.
>
> >>
> >> > +
> >> > + - AfPause:
> >> > + type: int32_t
> >> > + draft: true
> >> > + description: |
> >> > + This control has no effect except when in continuous
> autofocus mode
> >> > + (AfModeContinuous). It can be used to pause any lens
> movements while
> >> > + (for example) images are captured. The algorithm remains
> inactive
> >> > + until it is instructed to resume.
> >> > +
> >> > + enum:
> >> > + - name: AfPauseImmediate
> >> > + value: 0
> >> > + description: |
> >> > + Pause the continuous autofocus algorithm immediately,
> whether or
> >> > + not any kind of scan is underway. The AfState will
> continue to
> >> > + report whatever value it had when the control was
> enacted.
> >> > + - name AfPauseDeferred
> >> > + value: 1
> >> > + description: |
> >> > + Pause the continuous autofocus algorithm as soon as it
> is no longer
> >> > + scanning. The AfState will report AfStateFocused or
> AfStateFailed,
> >> > + depending on whether the final scan succeeds or not. If
> no scan is
> >> > + in currently progress, the algorithm will pause immediately.
> >>
> > Just to confirm. Since we removed the AftriggerIdle, I guess the two
> controls AfTriggerStart and AfPauseDeferred should be continuously set
> until the state becomes focused or failed? If so, is it better to mention
> it in the description?
>
> I imagined that these would be "one-shot" things. So you only need to
> send them once and they happen.
>
> I imagine there might also be a slight delay between something being
> in focus and that feeding back to the application, so you wouldn't
> want the algorithm to think "oh, I'm being told to scan again even
> though I've only just finished!"
>
> > For AfPauseImmediate, should it only report the state as failed when
> it's scanning and reset?
>
> So I think if it was "scanning" and gets "AfPauseImmediate", the state
> remains "scanning". If it was "focused" when it gets
> "AfPauseImmediate", then it stays "focused", and so on. The idea is
> that this means you can be sure what state it was in when you sent
> "AfPauseImmediate". If we were to change the state you couldn't be
> sure whether it was focused or not. This could give an application the
> chance to put up a "picture might be blurry" message, for example.
>
> >
> >>
> >> Alignment broken ?
> >>
> >> > + - name: AfPauseResume
> >> > + value: 2
> >> > + description: |
> >>
> >> Alignment broken ?
> >>
> >> > + Resume continous autofocus operation. The algorithm starts
> again
> >> > + from exactly where it left off, with AfState unchanged (one
> of
> >> > + AfStateFocused, AfStateFailed or following AfPauseImmediate
> it
> >> > + might also have been in the AfStateScanning state).
> >> > +
> >> > + - LensPosition:
> >> > + type: int32_t
> >> > + draft: true
> >> > + description: |
> >> > + Acts as a control to instruct the lens to move to a
> particular position
> >> > + and also reports back the position of the lens for each
> frame.
> >> > +
> >> > + The units are determined by the lens driver.
> >> > +
> >> > + The LensPosition control is ignored unless the AfMode is
> set to
> >> > + AfModeManual.
> >> > +
> >> > + - AfState:
> >> > + type: int32_t
> >> > + draft: true
> >> > + description: |
> >> > + Reports the current state of the AF algorithm.
> >> > + enum:
> >> > + - name: AfStateReset
> >> > + value: 0
> >> > + description: |
> >> > + The AF algorithm reports this state when:
> >> > + * It is in manual mode (AfModeManual).
> >> > + * The system has entered auto mode (AfModeAuto)
> but no scan
> >> > + has yet been initiated.
> >> > + * The system is in auto mode (AfModeAuto) and a
> scan has been
> >> > + cancelled.
> >> > + - name: AfStateScanning
> >> > + value: 1
> >> > + description: |
> >> > + AF is performing a scan. This state can be entered
> spontaneously
> >> > + if AfMode is set to AfModeContinuous, otherwise it
> requires the
> >> > + application to use the AfTrigger control to start the
> scan.
> >> > + - name: AfStateFocused
> >> > + value: 2
> >> > + description: |
> >> > + An AF scan has been performed and the algorithm
> believes the
> >> > + scene is in focus.
> >> > + - name: AfStateFailed
> >> > + value: 3
> >> > + description: |
> >> > + An AF scan has been performed but the algorithm has
> not been able
> >> > + to find the best focus position.
> >>
> >> I confirm my wish of having a graphical state machine :-).
> >
> > Vote +1 :-).
>
> I'll look into doing that. Do we have a preferred package? Do I have
> to turn it into Ascii Art? :)
>
How about a state transition table? We could prove its completeness and it
may be easier for rst documentation.
>
> Thanks for all the feedback!
>
> David
>
> >>
> >>
> >> Great work, thanks for it !
> >> JM
> >>
> >> > +
> >> > ...
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.libcamera.org/pipermail/libcamera-devel/attachments/20220316/03aa7217/attachment-0001.htm>
More information about the libcamera-devel
mailing list