[libcamera-devel] [PATCH v2 1/1] libcamera: controls: Controls for driving AF (autofocus) algorithms
David Plowman
david.plowman at raspberrypi.com
Thu Apr 7 11:05:32 CEST 2022
Hi Jacopo
Thanks for the reply!
On Wed, 6 Apr 2022 at 17:34, Jacopo Mondi <jacopo at jmondi.org> wrote:
>
> Hi David,
> + Han-lin for an Android question on metadata
>
> On Wed, Apr 06, 2022 at 02:05:20PM +0100, David Plowman wrote:
> > Hi Jacopo
> >
> > Thanks for taking the time to read all this!
> >
> > On Tue, 5 Apr 2022 at 21:36, Jacopo Mondi <jacopo at jmondi.org> wrote:
> > >
> > > Hi David,
> > > sorry for being late :)
> > >
> > > On Thu, Mar 31, 2022 at 04:17:47PM +0100, David Plowman via libcamera-devel wrote:
> > > > This patch describes a series of controls that allow applications to
> > > > drive AF algorithms:
> > > >
> > > > AfMode - manual, auto or continuous
> > > > AfLensPosition - set lens position to macro/hyperfocal/infinity
> > > > AfRange - full, macro or normal
> > > > AfSpeed - fast or slow
> > > > AfWindows - AF window locations
> > > > AfTrigger - start (trigger an AF scan) or cancel
> > > > AfPause - pause continuous AF
> > > > LensPosition - position of lens from lens driver
> > > > AfState - reports the mode, whether scanning/success/failure
> > > > ---
> > > > src/libcamera/control_ids.yaml | 373 +++++++++++++++++++++++++++------
> > > > 1 file changed, 313 insertions(+), 60 deletions(-)
> > > >
> > > > diff --git a/src/libcamera/control_ids.yaml b/src/libcamera/control_ids.yaml
> > > > index 9d4638ae..23607368 100644
> > > > --- a/src/libcamera/control_ids.yaml
> > > > +++ b/src/libcamera/control_ids.yaml
> > > > @@ -406,27 +406,6 @@ controls:
> > > > The camera will cancel any active or completed metering sequence.
> > > > The AE algorithm is reset to its initial state.
> > > >
> > > > - - AfTrigger:
> > > > - type: int32_t
> > > > - draft: true
> > > > - description: |
> > > > - Control for AF trigger. Currently identical to
> > > > - ANDROID_CONTROL_AF_TRIGGER.
> > > > -
> > > > - Whether the camera device will trigger autofocus for this request.
> > > > - enum:
> > > > - - name: AfTriggerIdle
> > > > - value: 0
> > > > - description: The trigger is idle.
> > > > - - name: AfTriggerStart
> > > > - value: 1
> > > > - description: The AF routine is started by the camera.
> > > > - - name: AfTriggerCancel
> > > > - value: 2
> > > > - description: |
> > > > - The camera will cancel any active trigger and the AF routine is
> > > > - reset to its initial state.
> > > > -
> > > > - NoiseReductionMode:
> > > > type: int32_t
> > > > draft: true
> > > > @@ -507,45 +486,6 @@ controls:
> > > > The AE algorithm has started a pre-capture metering session.
> > > > \sa AePrecaptureTrigger
> > > >
> > > > - - AfState:
> > > > - type: int32_t
> > > > - draft: true
> > > > - description: |
> > > > - Control to report the current AF algorithm state. Currently identical to
> > > > - ANDROID_CONTROL_AF_STATE.
> > > > -
> > > > - Current state of the AF algorithm.
> > > > - enum:
> > > > - - name: AfStateInactive
> > > > - value: 0
> > > > - description: The AF algorithm is inactive.
> > > > - - name: AfStatePassiveScan
> > > > - value: 1
> > > > - description: |
> > > > - AF is performing a passive scan of the scene in continuous
> > > > - auto-focus mode.
> > > > - - name: AfStatePassiveFocused
> > > > - value: 2
> > > > - description: |
> > > > - AF believes the scene is in focus, but might restart scanning.
> > > > - - name: AfStateActiveScan
> > > > - value: 3
> > > > - description: |
> > > > - AF is performing a scan triggered by an AF trigger request.
> > > > - \sa AfTrigger
> > > > - - name: AfStateFocusedLock
> > > > - value: 4
> > > > - description: |
> > > > - AF believes has focused correctly and has locked focus.
> > > > - - name: AfStateNotFocusedLock
> > > > - value: 5
> > > > - description: |
> > > > - AF has not been able to focus and has locked.
> > > > - - name: AfStatePassiveUnfocused
> > > > - value: 6
> > > > - description: |
> > > > - AF has completed a passive scan without finding focus.
> > > > -
> > > > - AwbState:
> > > > type: int32_t
> > > > draft: true
> > > > @@ -690,4 +630,317 @@ controls:
> > > > value. All of the custom test patterns will be static (that is the
> > > > raw image must not vary from frame to frame).
> > > >
> > > > + - AfMode:
> > > > + type: int32_t
> > > > + draft: true
> > >
> > > Was having all these controls as draft intentional ? The previous
> > > definitions where set to drafts as they mapped to the android
> > > definition, but now that we're defining our own ones...
> >
> > Not really... I only made them "draft" because it seemed like
> > everything was "draft" at first! But happy to change that. Are you
> > saying that these definitely should *not* be draft, then?
> >
>
> Yeah, "draft" was used to import controls from Android as they are for
> sake of CTS compliance. Let's drop draft from the ones we're defining
Will do!
>
> > >
> > > > + description: |
> > > > + Control to set the mode of the AF (autofocus) algorithm. Applications
> > > > + are allowed to set a new mode, and to send additional controls for
> > > > + that new mode, in the same request.
> > > > +
> > > > + An implementation may choose not to implement all the modes.
> > > > +
> > > > + enum:
> > > > + - name: AfModeManual
> > > > + value: 0
> > > > + description: |
> > > > + The AF algorithm is in manual mode. In this mode it will never
> > > > + perform any action nor move the lens of its own accord, but an
> > > > + application can set controls to move the lens.
> > > > +
> > > > + In this mode the AfState will always report AfStateManual.
> > > > + - name: AfModeAuto
> > > > + value: 1
> > > > + description: |
> > > > + The AF algorithm is in auto mode. This means that the algorithm
> > > > + will never move the lens or change state unless the AfTrigger
> > > > + control is used. The AfTrigger control can be used to initiate a
> > > > + focus scan, the results of which will also be reported by AfState.
> > > > +
> > > > + If the autofocus algorithm is moved from AfModeAuto to another
> > > > + mode while a scan is in progress, the scan is cancelled
> > > > + immediately, without waiting for the scan to finish.
> > > > +
> > > > + When first entering this mode the AfState will report
> > > > + AfStateAuto. When a trigger control is sent, AfState will
> > > > + report AfStateAutoScanning for a period before spontaneously
> > > > + changing to AfStateAutoFocused or AfStateAutoFailed, depending on
> > > > + the outcome of the scan. It will remain in this state until
> > > > + another scan is initiated by the AfTrigger control. If a scan is
> > > > + cancelled (without changing to another mode), AfState will return
> > > > + to AfStateAuto.
> > > > + - name: AfModeContinuous
> > > > + value: 2
> > > > + description: |
> > > > + The AF algorithm is in continuous mode. This means that the lens
> > > > + can re-start a scan spontaneously at any moment, without any user
> > > > + intervention. The AfState still reports whether the algorithm is
> > > > + currently scanning or not, though the application has no ability
> > > > + to initiate or cancel scans (though it can "pause" them), nor to
> > > > + move the lens for itself.
> > > > +
> > > > + When set to AfModeContinuous, the system will immediately initiate
> > > > + a scan so AfState will report AfStateContinuousScanning, and will
> > > > + settle on one of AfStateContinuousFocused or AfStateContinuousFailed,
> > > > + depending on the scan result.
> > > > +
> > > > + - AfLensPosition:
> > > > + type: int32_t
> > > > + draft: true
> > > > + description: |
> > > > + Control to set the position of the lens to one of the following
> > > > + predefined locations. This control only has any effect when the AfMode
> > > > + is set to AfModeManual.
> > > > +
> > > > + This control is distinct from the LensPosition control, which sets the
> > > > + lens position using the lens driver's units.
> > > > + enum:
> > > > + - name: AfLensPositionMacro
> > > > + value: 0
> > > > + description: The closest focal position of the lens.
> > > > + - name: AfLensPositionHyperfocal
> > > > + value: 1
> > > > + description: Hyperfocal position.
> > > > + - name: AfLensPositionInfinity
> > > > + value: 2
> > > > + description: The furthest focal position (usually infinity).
> > >
> > > Interesting, bikeshedding on the name apart, is this some 'simplified'
> > > LensState ? Do you think there is a use case for this ? Why would
> > > anyone set the AF to auto mode then use some coarse grained control
> > > like this one ? Wouldn't it make more sense as an 'hint' to the AF
> > > algorithm to restrict the search space ?
> >
> > In the light of your comments below about the LensPosition not using
> > driver units, then maybe this could be deleted, and we could select
> > some kind of canonical range for the lens?
> >
>
> Ah wait. What I was suggesting ("An "hint" to the AF algorithm to
> restrict the search space") is basically AfRange.
I wonder if there's a case for a "custom" value for AfRange, and we
could then even have a control that sets the custom range?
>
> > The most basic use case would be following on from the capture of an
> > image after an AF scan:
> >
> > - capture image
> > - AfMode to manual
> > - Lens to hyperfocal
> > - AfMode to auto
> > - now wait for the shutter button to be pressed before you start
> > another AF scan.
>
> Naive question:
> Wouldn't setting the lens to hyperfocal render the preview unusable ?
Hyperfocal is the place where "most stuff" is normally in focus
(strictly speaking, has the greatest depth of field), so in the
absence of other information it's normally the best place to put the
lens. It's just for the preview, you'd typically run another scan for
a capture.
(Some systems run Continuous AF during preview, of course. But you
know my anxiety about CAF algorithms... in my opinion just putting the
lens to hyperfocal is way less annoying than a not-very-good CAF
implementation!)
>
> >
> > If your UI has a "macro" capture mode of some kind, maybe it's "Lens
> > to macro" instead of "Lens to hyperfocal".
>
> Why is this different than starting an AF scan (without going through
> Manual) with an AfRange that suggests in which range to scan ? Is this
> an optimization to forcefully move the lens closer to where it will
> likely end being moved to ?
If you know you're doing a macro capture it would be helpful for the
preview to put the lens somewhere where macro things are likely to be
more in focus, instead of using hyperfocal which will likely make
close objects very blurry.
>
> >
> > But as noted, if LensPosition allows us to select hyperfocal/macro
> > directly, then we can do without this one.
> >
>
> Providing my above understanding is correct, it would be a bit less intuitive
> for applications to use a numerical value instead of some easier
> "Macro"/"Hyperfocal". Of course, wrappers like your one can simplify this
> for their consumers.
True, though I'm now quite liking the "dioptre-like" canonical range
described below, where 1 = hyperfocal, 0 = infinity, large numbers =
close
>
> > >
> > > (Btw I have a use case for supplying a range of lens positions, which
> > > has been previously estimated according to some pre-defined use cases,
> > > to the AF algorithm in order restrict the AF search range to increase
> > > the speed of a scan. Do you think it's a valid use case and it's worth a
> > > control ?)
> >
> > I'd imagined doing this kind of thing with the AfSpeed and AfRange
> > controls and by having the AF algorithm tuned for the module. I'm not
> > convinced about passing in lens positions from outside the camera
> > system. Where would they come from? Shouldn't the camera system know
> > what positions to use?
> >
>
> A possible use case, for controlled environments with low latency
> requirements, is to run tests beforehand and for each 'scene' define
> what is the ideal lens movement range. When the application starts an
> AF scan it would tell the AF algorithm about the 'preferred' range.
>
> I understand this works backwards, as the lens position should be the
> result of the AF algorithm run not one of its inputs :)
Indeed, but as noted above, maybe there's an argument for an
AfRangeCustom which could be user-settable? We've added "Custom"
values for other controls in some places, IIRC.
>
> > In the Raspberry Pi world we would of course store this kind of
> > information in the tuning file. I've noticed some discussion elsewhere
> > about supplying configuration to the IPAs - it seems like this might
> > be a case in point? I think that for real camera systems, in the end,
> > there is no escape from "configuration" (aka. "tuning").
> >
> > >
> > > > +
> > > > + - AfRange:
> > > > + type: int32_t
> > > > + draft: true
> > > > + description: |
> > > > + Control to set the range of focus distances that is scanned. An
> > > > + implementation may choose not to implement all the options here.
> > >
> > > Only valid in Auto and Continuous mode, right ?
> >
> > Generally I've been expecting that you could change settings like this
> > in Manual mode too, though it obviously can't have any effect until
> > you go back to Auto/Continuous. Does that sound reasonable?
>
> I would rather assume thaat if a control is not valid in the current
> context is simply ignored and not cached to be re-applied when the
> context change ?
Possibly, though I'm slightly nervous about how you'd do things like:
- You're in Manual mode, but want to go to Auto and set the Range to Normal.
- You're in Auto mode but were using a different range. You want to
reset it to Normal before switching to Manual.
Can you send the AfRange change and the new mode at the same time in
both cases or not? Or do you need two separate requests to send the
changes? It's not clear to me exactly what would work. If you say that
the AfRange always gets updated, then the behaviour is obvious, and
you can send both AfRange and AfMode in the same request which would
be easier to use.
>
> >
> > >
> > > (This makes me think that we could easily model Continuous by adding a
> > > TriggerContinuous when in Auto mode. With a TriggerPause we could get rid of
> > > AfPause... well, it's quite a redesign so I'll stop here unless
> > > someone thinks it's a really good idea :)
> >
> > Actually I'm quite strongly against anything like this. I think Auto
> > and Continuous should be quite separate. The reason is that Auto is
> > (relatively) easy to implement, you do a scan over some range and
> > choose the best spot. But Continuous AF is an absolute nightmare to do
> > well. You have all these problems:
> >
> > - The algo doesn't realise it needs to rescan and so seems to be
> > "stuck". (Watch people waving their hands in front of the lens to try
> > and unstick it.) So you add ever more tests and arbitrary thresholds
> > on different statistics to tell if a rescan is needed, and then the
> > whole thing becomes untunable.
> > - The algo rescans too often. A car in the distance goes by and the
> > whole image see-saws in and out while a rescan happens, ending back at
> > the same place. Yet more tests, thresholds...
> > - You can never make all your tests and thresholds work at the same
> > time but in different conditions. So you make them variable depending
> > on things like exposure and gain. Now you are truly doomed.
> > - The algo is generally not robust to camera shake, which frequently
> > causes it to end up "failing" with a completely out-of-focus image.
> > And then you can't get it to rescan... cue more hand-waving in front
> > of the lens. You implement more special tests and timeouts... At this
> > point you will wish for the earth to swallow you up.
> > - When the algo rescans it starts off going the wrong way causing
> > extra see-sawing.
> > - You try to solve the last problem by doing really small initial
> > steps. Only then it ends up statistically unstable.
> >
> > From the above you may detect that we have been there and have several
> > T-shirts. Speaking for myself, I have no plans to implement Continuous
> > AF without PDAF sensors, as PDAF is quite a big help.
> >
>
> Yes, it feels like you know what you're talking about :)
>
> I understand you would like be cautious with CAF, but if one doesn't
> want to implement CAF can't she simply not report TriggerContinuous ?
>
> Anyway, that was just an idea thrown in the mixer, don't worry about
> it.
>
> > >
> > > > + enum:
> > > > + - name: AfRangeNormal
> > > > + value: 0
> > > > + description: |
> > > > + A wide range of focus distances is scanned, all the way from
> > > > + infinity down to close distances, though depending on the
> > > > + implementation, possibly not including the very closest macro
> > > > + positions.
> > > > + - name: AfRangeMacro
> > > > + value: 1
> > > > + description: Only close distances are scanned.
> > > > + - name: AfRangeFull
> > > > + value: 2
> > > > + description: |
> > > > + The full range of focus distances is scanned just as with
> > > > + AfRangeNormal but this time including the very closest macro
> > > > + positions.
> > > > +
> > > > + - AfSpeed:
> > > > + type: int32_t
> > > > + draft: true
> > > > + description: |
> > > > + Control that determines whether the AF algorithm is to move the lens
> > > > + as quickly as possible or more steadily. For example, during video
> > > > + recording it may be desirable not to move the lens too abruptly, but
> > > > + when in a preview mode (waiting for a still capture) it may be
> > > > + helpful to move the lens as quickly as is reasonably possible.
> > > > + enum:
> > > > + - name: AfSpeedNormal
> > > > + value: 0
> > > > + description: Move the lens at its usual speed.
> > > > + - name: AfSpeedFast
> > > > + value: 1
> > > > + description: Move the lens more quickly.
> > > > +
> > > > + - AfWindows:
> > > > + type: Rectangle
> > > > + draft: true
> > > > + description: |
> > > > + Sets the focus windows used by the AF algorithm. The units used express
> > > > + a proportion of the ScalerCrop control (or if unavailable, of the entire
> > > > + image), as u0.16 format numbers.
> > > > +
> > > > + In order to be activated, a rectangle must be programmed with non-zero
> > > > + width and height. If no rectangles are programmed in this way, then the
> > > > + system will choose its own single default window in the centre of the
> > > > + image.
> > > > +
> > > > + The details of how the windows are used are platform dependent. We note
> > > > + that when there is more than one AF window, a typical implementation
> > > > + might find the optimal focus position for each one and finally select
> > > > + the window closest to the camera.
> > > > +
> > > > + size: [platform dependent]
> > >
> > > I think this should just be
> > >
> > > size: [n]
> >
> > OK!
> >
> > > > +
> > > > + - AfTrigger:
> > > > + type: int32_t
> > > > + draft: true
> > > > + description: |
> > > > + This control starts an autofocus scan when AfMode is set to AfModeAuto,
> > > > + and can also be used to terminate a scan early.
> > > > +
> > > > + It is ignored if AfMode is set to AfModeManual or AfModeContinuous.
> > > > +
> > > > + enum:
> > > > + - name: AfTriggerStart
> > > > + value: 0
> > > > + description: Start an AF scan. Ignored if a scan is in progress.
> > > > + - name: AfTriggerCancel
> > > > + value: 1
> > > > + description: Cancel an AF scan. This does not cause the lens to move
> > > > + anywhere else. Ignored if no scan is in progress.
> > > > +
> > > > + - AfPause:
> > > > + type: int32_t
> > > > + draft: true
> > > > + description: |
> > > > + This control has no effect except when in continuous autofocus mode
> > > > + (AfModeContinuous). It can be used to pause any lens movements while
> > > > + (for example) images are captured. The algorithm remains inactive
> > > > + until it is instructed to resume.
> > > > +
> > > > + enum:
> > > > + - name: AfPauseImmediate
> > > > + value: 0
> > > > + description: |
> > > > + Pause the continuous autofocus algorithm immediately, whether or
> > > > + not any kind of scan is underway. If the AfState was previously
> > > > + reporting AfStateContinuousScanning it will now change to
> > > > + AfStateContinuousScanningPaused, and similarly for
> > > > + AfStateContinuousFocused and AfStateContinuousFailed.
> > > > + - name AfPauseDeferred
> > > > + value: 1
> > > > + description: |
> > > > + Pause the continuous autofocus algorithm as soon as it is no longer
> > > > + scanning. If the AfState is currently reporting
> > > > + AfStateContinuousFocused is will change to
> > > > + AfStateContinuousFocusedPaused, and similarly in the case of
> > > > + AfStateContinuousFailed.
> > > > +
> > > > + If AfState reports AfStateContinuousScanning it will change to
> > > > + AfStateContinuousScanningPausing, and then move to one of
> > > > + AfStateContinuousFocusedPaused or AfStateContinuousFailedPaused
> > > > + when the scan completes.
> > > > + - name: AfPauseResume
> > > > + value: 2
> > > > + description: |
> > > > + Resume continous autofocus operation. The algorithm starts again
> > > > + from exactly where it left off, and the AfState will drop the
> > > > + 'Paused' or 'Pausing' part of the state.
> > > > +
> > > > + - LensPosition:
> > > > + type: int32_t
> > > > + draft: true
> > > > + description: |
> > > > + Acts as a control to instruct the lens to move to a particular position
> > > > + and also reports back the position of the lens for each frame.
> > > > +
> > > > + The units are determined by the lens driver.
> > >
> > > Ugh, we cannot do this in the control definition. Driver specificities
> > > cannot surface to the controls definition, which should be platform
> > > independent.
> > >
> > > I think we'll have to establish a numerical range (go ahead if you
> > > have any idea) and translate to the actual driver values in the
> > > CameraLens class ?
> >
> > I wasn't aware of this, but maybe it's a good thing. The question is
>
> Well, isn't it the same with "Brightness", for example ?
Indeed, or ExposureTime where we explicitly want microseconds, not driver units.
>
> > what kind of range to use, and how do you know what limits apply for
> > any given camera module? How about a dioptre-like scale, normalised
> > for the hyperfocal distance H. It would go like this:
> >
> > The value n would mean being focused at distance H/n (n does not have
> > to be an integer). So:
> >
> > The value 1 would mean hyperfocal position (focal distance H/1).
> > The value 0 would mean infinity (focal distance H/0).
> > Larger values would get progressively closer.
> >
> > Setting a control value outside the allowed range will clamp it to
> > that end of the range.
>
> I sincerely don't know enough about this to tell if it's a good idea
> or not. Hence, it seems like a good idea to me :)
>
> >
> > In the Raspberry Pi world it would be sufficient for the tuning file
> > to know the allowable range of values. More generally, I could imagine
> > an application might want to say "focus at precisely 1m" in which case
> > it would have to know the value of H from a property or something,
> > though providing such information would be optional (some lenses
> > simply aren't precisely calibrated). Perhaps an application might want
> > to get the allowable range of (maybe approximate) focal distances too?
> >
> > >
> > > > +
> > > > + The LensPosition control is ignored unless the AfMode is set to
> > > > + AfModeManual.
> > > > +
> > > > + Note that this control is distinct from AfLensPosition, which allows
> > > > + the lens to be moved to its macro/hyperfocal/infinity position, rather
> > > > + than using lens driver units.
> > > > +
> > > > + - AfState:
> > > > + type: int32_t
> > > > + draft: true
> > > > + description: |
> > > > + Reports the current state of the AF algorithm. The possible state changes
> > > > + are described below, although we add that the following general state
> > > > + changes are also allowed.
> > >
> > > wow, there is a lot of states here :)
> >
> > Did you follow the link to the state transition diagram? I would hope
> > that should go some way to explaining it! Here's the link again for
> > convenience:
> >
> > https://docs.google.com/drawings/d/1rU2nS82dEC8Y2eY0lKPkQQhd67gKLOl7PvSeUcL1j_E/edit?usp=sharing
> >
>
> Missed it. Thanks for providing it.
>
> > >
> > > Let me copy them here
> > >
> > > - name: AfStateManual
> > > - name: AfStateAuto
> > > - name: AfStateAutoScanning
> > > - name: AfStateAutoFocused
> > > - name: AfStateAutoFailed
> > > - name: AfStateContinuousScanning
> > > - name: AfStateContinuousScanningPausing
> > > - name: AfStateContinuousScanningPaused
> > > - name: AfStateContinuousFocused
> > > - name: AfStateContinuousFocusedPaused
> > > - name: AfStateContinuousFailed
> > > - name: AfStateContinuousFailedPaused
> >
> > I "multiplied out" all the different parts of the state to make a
> > single long list. Actually I think that makes it more Android-like.
> > But we could reduce this number if, for example, we're happy to report
> > the state of AfMode separately on every frame. I don't mind - do we
> > have a general preference for this kind of thing?
> >
>
> Good question.
>
> More generally we need to establish a policy for metadata. For
> controls we know that controls should be attached to a request only
> when their value changes. If a control is not supplied with new
> values, then the old (or default) one will stay valid.
>
> This implies the PH/IPA state machine is then stateful and 'remembers'
> what was the last value a control was set to.
>
> What about metadata instead ? Can we assume the same stateful-ness in
> applications too, or is it a burden we cannot impose on them ?
>
> Han-lin, do you know how does that work with Android ? Does it report
> metadata even when their values do not change or do they assume that,
> like controls, a metadata is reported only when its value has changed.
>
> What would the preference be for the RPi applications ?
I don't think I have a very strong preference. I'm happy with either:
- Report everything in the AfState on every frame.
- Report the state in multiple metadata items (say, AfState and
AfMode) on every frame.
I'm less keen on having applications have to remember if they sent
something and then wait for the request where it actually happened, it
seems to me that would simply add annoying boilerplate code to all
applications. What do you think?
>
> > >
> > > (please note that AfStateAuto seems mis-aligned, unless I messed up
> > > something while cutting&paste. Please check the compiled doc to see
> > > how it looks like...)
> >
> > I will check.
> >
> > >
> > > So we have three macro groups
> > >
> > > - Manual
> > >
> > > - Auto
> > > - Scanning
> > > - Focus
> > > - Failed
> > >
> > > - Continuous
> > >
> > > - Scanning
> > > - Pausing
> > > - Paused
> > > - Focused
> > > - Paused
> > > - Failed
> > > - Paused
> > >
> > > The first group is AfMode (Manual, Auto, Continuous) and can be
> > > deduced by inspecting the AfMode metadata.
> > >
> > > The second group is the actual state (Scannng, Focused, Failed)
> > >
> > > The third group is specific to continuous mode as it accepts a Pause
> > > trigger.
> > >
> > > I understand why you have this, as expressing "The AfState is set to
> > > $something when in AfModeManual" etc etc might get cumbersome, but I
> > > think it might be worth a try
> > >
> > > 1) Manual mode (the bad)
> > > As we all know there is a delay between when a lens is instructed
> > > to move and when it reaches its final state.
> > >
> > > In Manual mode the lens is moved with LensPosition and
> > > AfLensPosition.
> > >
> > > To know the current state:
> > > - With LensPosition metadata can be inspected until it doesn't reach the
> > > desired value.
> > > - With AfLensPosition it is not clear to me how to know when the lens
> > > as reched one of the pre-defined positions.
> >
> > It sounds like we're deleting AfLensPosition.
> >
> > >
> > > Is it so wrong to report either (Scanning,Focus,Failed) even for
> > > Manual mode ?
> >
> > I don't think it's wrong. If we have a single long list of states then
> > it just makes for more states. If we don't then it's harmless, though
> > I'm not aware of a use for it.
> >
> > >
> > > 2) AutoMode (the good)
> > > This seems the easy one, as (Scanning,Focus,Failed) are already
> > > there
> > >
> > > 3) Continous (the ugly)
> > > Continuous mode can be paused. That's a shame :)
> > > Would that be that bad keeping a Paused or Pausing state which can
> > > only be returned for Continuous mode.
> >
> > The whole paused/pausing thing rather complicates things, but it does
> > seem to me that it's necessary. If you were in Continuous mode, you
> > would rather the lens didn't set off again just as you want to capture
> > a sequence of pictures.
> >
> > >
> > > All in all, can we reduce the number of states to
> > >
> > > - Scanning
> > >
> > > The AF algorithm is scanning and has not yet reached a
> > > stable result. When running in Auto mode an AF scan is
> > > started by using the AfTrigger control. When in Manual mode
> > > and a new lens position is requested either by sending a
> > > LensPosition or AfLensPosition control, the AF state will
> > > report "Scanning" until the lens doesn't reach the desired
> > > position. Similarly, when running in continuous mode and the
> > > AF algorithm decides to start a scan spontaneously the
> > > AfState will report Scanning until a new stable state is
> > > reached.
> > >
> > > - Focus
> > >
> > > The AF algorithm has reached a stable state and the image is
> > > now focused. When running in Manual mode the lens has
> > > reached the position requested using either LensPosition and
> > > AfLensPosition.
> > >
> > > - Failed
> > >
> > > The AF algorithm has completed a scan and without finding a
> > > good focus position. When running in manual mode, the camera
> > > has failed moving the lens to the desired position.
> > >
> > > - Paused (copying your below text, please see comments below)
> > >
> > > The AF algorithm is in continuous mode (AfModeContinuous) and
> > > was scanning when AfPauseImmediate was sent. The scan will not
> > > complete and the algorithm will remain in this state.
> > >
> > > - Pausing
> > >
> > > The AF algorithm is in continuous mode (AfModeContinuous) and
> > > was scanning when AfPauseDeferred was sent. The scan will complete
> > > at which point the algorithm moves to the
> > > AfStateContinuousFocusedPaused or AfStateContinuousFailedPaused
> > > state.
> >
> > I think we could get by if we report the following for every frame:
> >
> > AfMode - Manual, Auto or Continuous
> > AfState - Scanning, Focused or Failed
> > AfPause - Resume (which means it's running), Immediate or Deferred
> > (these states only apply during Continuous; the Deferred value can
> > only occur when also Scanning).
> >
> > Or we could wrap AfState and AfPause together:
> >
> > AfState - Scanning, Focused. Failed, ScanningPausing, ScanningPaused,
> > FocusedPaused, FailedPaused.
> >
> > That makes some of the interaction between pausing and scanning
> > clearer, but then it's more states again. I don't know...
> >
>
> Let's clarify what the policy on metadata is first.
Agree!
>
> > >
> > >
> > > > +
> > > > + In any of the AfStateManual or AfStateContinuous states, the AfMode
> > > > + may be set to AfModeAuto and the algorithm will move to the
> > > > + AfStateAuto state. If AfTriggerStart is sent at the same time
> > > > + then the algorithm will move to AfStateAutoScanning state.
> > > > +
> > > > + In any of the AfStateAuto or AfStateManual states, the AfMode may
> > > > + be set to AfModeContinuous and the algorithm will move to
> > > > + AfStateContinuousScanning.
> > > > +
> > > > + In any of the AfStateAuto or AfStateContinuous states, the AfMode may
> > > > + be set to AfModeManual and the algorithm will move to
> > > > + AfStateManual. The lens will not be moved and will be left where
> > > > + it was at that moment.
> > > > +
> > > > + enum:
> > > > + - name: AfStateManual
> > > > + value: 0
> > > > + description: |
> > > > + The AF algorithm is in manual mode (AfModeManual). The LensPosition
> > > > + and AfLensPosition controls can be used directly to move the lens.
> > > > + - name: AfStateAuto
> > > > + value: 1
> > > > + description: |
> > > > + The AF algorithm is in auto mode (AfModeAuto), and has either just
> > > > + been moved into that state, or a scan that was in progress has been
> > > > + cancelled.
> > > > + - name: AfStateAutoScanning
> > > > + value: 2
> > > > + description: |
> > > > + The AF algorithm is in auto mode (AfModeAuto), and a scan has been
> > > > + started using the AfTrigger control. The scan can be cancelled by
> > > > + sending AfTriggerCancel at which point the algorithm will either
> > > > + move back to AfStateAuto or, if the scan actually completes
> > > > + before the cancel request is processed, to one of
> > > > + AfStateAutoFocused or AfStateAutoFailed.
> > > > + - name: AfStateAutoFocused
> > > > + value: 3
> > > > + description: |
> > > > + The AF algorithm is in auto mode (AfModeAuto), and a scan has
> > > > + completed with the result that the algorithm believes the image is
> > > > + now in focus.
> > > > + - name: AfStateAutoFailed
> > > > + value: 4
> > > > + description: |
> > > > + The AF algorithm is in auto mode (AfModeAuto), and a scan has
> > > > + completed with the result that the algorithm did not find a good
> > > > + focus position.
> > > > + - name: AfStateContinuousScanning
> > > > + value: 5
> > > > + description: |
> > > > + The AF algorithm is in continuous mode (AfModeContinuous) and
> > > > + is currently scanning for a good focus position. This occurs when
> > > > + the mode is first set to AfModeContinuous, or may happen
> > > > + spontaneously when the algorithm believes a re-scan is needed.
> > > > + - name: AfStateContinuousScanningPausing
> > > > + value: 6
> > > > + description: |
> > > > + The AF algorithm is in continuous mode (AfModeContinuous) and
> > > > + was scanning when AfPauseDeferred was sent. The scan will complete
> > > > + at which point the algorithm moves to the
> > > > + AfStateContinuousFocusedPaused or AfStateContinuousFailedPaused
> > > > + state.
> > > > + - name: AfStateContinuousScanningPaused
> > > > + value: 7
> > > > + description: |
> > > > + The AF algorithm is in continuous mode (AfModeContinuous) and
> > > > + was scanning when AfPauseImmediate was sent. The scan will not
> > >
> > > Why can't the algorithm be paused when it has reached a stable state
> > > but only when scanning ? Won't a pause 'freeze' the algorithm in its
> > > current state no matter what it is ? I understand with
> > > "AfPauseDeferred" that it has to wait for AF to be locked before
> > > stopping it, but with "Immediate" doesn't "pause" apply to all states ?
> >
> > Wasn't sure I understood this. It can be paused once it's reached
> > Focused or Failed, and then both PauseImmediate and PauseDeferred will
> > cause it to report FocusedPaused or FailedPaused. Was that the
> > question?
>
> You're right, I have probably missed FocusedPaused and FailedPaused.
> :)
>
> Thanks
> j
Thanks for all the discussion!
David
>
> >
> > Anyway, thanks very much!
> >
> > David
> >
> > >
> > > Thanks
> > > j
> > >
> > > > + complete and the algorithm will remain in this state. The scan
> > > > + may be resumed by sending AfPauseResume.
> > > > + - name: AfStateContinuousFocused
> > > > + value: 8
> > > > + description: |
> > > > + The AF algorithm is in continuous mode (AfModeContinuous) and
> > > > + a scan has completed with the algorithm believing it has found a
> > > > + good focus position.
> > > > + - name: AfStateContinuousFocusedPaused
> > > > + value: 9
> > > > + description: |
> > > > + The AF algorithm was in the AfStateContinuousFocused state and
> > > > + has been paused (by either AfPauseImmediate or AfPauseDeferred),
> > > > + or it was in the AfStateContinuousScanningPausing state and the
> > > > + scan has completed successfully. The algorithm will now remain
> > > > + in this state, and may be resumed by sending AfPauseResume.
> > > > + - name: AfStateContinuousFailed
> > > > + value: 10
> > > > + description: |
> > > > + The AF algorithm is in continuous mode (AfModeContinuous) and
> > > > + a scan has completed without finding a good focus position.
> > > > + - name: AfStateContinuousFailedPaused
> > > > + value: 11
> > > > + description: |
> > > > + The AF algorithm was in the AfStateContinuousFailed state and
> > > > + has been paused (by either AfPauseImmediate or AfPauseDeferred),
> > > > + or it was in the AfStateContinuousScanningPausing state and the
> > > > + scan has completed unsuccessfully. The algorithm will now remain
> > > > + in this state, and may be resumed by sending AfPauseResume.
> > > > +
> > > > ...
> > > > --
> > > > 2.30.2
> > > >
More information about the libcamera-devel
mailing list