[libcamera-devel] [PATCH v2 1/1] libcamera: controls: Controls for driving AF (autofocus) algorithms
Hanlin Chen
hanlinchen at chromium.org
Thu Apr 7 13:31:07 CEST 2022
Hi David and Jacopo,
On Thu, Apr 7, 2022 at 5:05 PM David Plowman <david.plowman at raspberrypi.com>
wrote:
> Hi Jacopo
>
> Thanks for the reply!
>
> On Wed, 6 Apr 2022 at 17:34, Jacopo Mondi <jacopo at jmondi.org> wrote:
> >
> > Hi David,
> > + Han-lin for an Android question on metadata
> >
> > On Wed, Apr 06, 2022 at 02:05:20PM +0100, David Plowman wrote:
> > > Hi Jacopo
> > >
> > > Thanks for taking the time to read all this!
> > >
> > > On Tue, 5 Apr 2022 at 21:36, Jacopo Mondi <jacopo at jmondi.org> wrote:
> > > >
> > > > Hi David,
> > > > sorry for being late :)
> > > >
> > > > On Thu, Mar 31, 2022 at 04:17:47PM +0100, David Plowman via
> libcamera-devel wrote:
> > > > > This patch describes a series of controls that allow applications
> to
> > > > > drive AF algorithms:
> > > > >
> > > > > AfMode - manual, auto or continuous
> > > > > AfLensPosition - set lens position to macro/hyperfocal/infinity
> > > > > AfRange - full, macro or normal
> > > > > AfSpeed - fast or slow
> > > > > AfWindows - AF window locations
> > > > > AfTrigger - start (trigger an AF scan) or cancel
> > > > > AfPause - pause continuous AF
> > > > > LensPosition - position of lens from lens driver
> > > > > AfState - reports the mode, whether scanning/success/failure
> > > > > ---
> > > > > src/libcamera/control_ids.yaml | 373
> +++++++++++++++++++++++++++------
> > > > > 1 file changed, 313 insertions(+), 60 deletions(-)
> > > > >
> > > > > diff --git a/src/libcamera/control_ids.yaml
> b/src/libcamera/control_ids.yaml
> > > > > index 9d4638ae..23607368 100644
> > > > > --- a/src/libcamera/control_ids.yaml
> > > > > +++ b/src/libcamera/control_ids.yaml
> > > > > @@ -406,27 +406,6 @@ controls:
> > > > > The camera will cancel any active or completed
> metering sequence.
> > > > > The AE algorithm is reset to its initial state.
> > > > >
> > > > > - - AfTrigger:
> > > > > - type: int32_t
> > > > > - draft: true
> > > > > - description: |
> > > > > - Control for AF trigger. Currently identical to
> > > > > - ANDROID_CONTROL_AF_TRIGGER.
> > > > > -
> > > > > - Whether the camera device will trigger autofocus for this
> request.
> > > > > - enum:
> > > > > - - name: AfTriggerIdle
> > > > > - value: 0
> > > > > - description: The trigger is idle.
> > > > > - - name: AfTriggerStart
> > > > > - value: 1
> > > > > - description: The AF routine is started by the camera.
> > > > > - - name: AfTriggerCancel
> > > > > - value: 2
> > > > > - description: |
> > > > > - The camera will cancel any active trigger and the AF
> routine is
> > > > > - reset to its initial state.
> > > > > -
> > > > > - NoiseReductionMode:
> > > > > type: int32_t
> > > > > draft: true
> > > > > @@ -507,45 +486,6 @@ controls:
> > > > > The AE algorithm has started a pre-capture metering
> session.
> > > > > \sa AePrecaptureTrigger
> > > > >
> > > > > - - AfState:
> > > > > - type: int32_t
> > > > > - draft: true
> > > > > - description: |
> > > > > - Control to report the current AF algorithm state.
> Currently identical to
> > > > > - ANDROID_CONTROL_AF_STATE.
> > > > > -
> > > > > - Current state of the AF algorithm.
> > > > > - enum:
> > > > > - - name: AfStateInactive
> > > > > - value: 0
> > > > > - description: The AF algorithm is inactive.
> > > > > - - name: AfStatePassiveScan
> > > > > - value: 1
> > > > > - description: |
> > > > > - AF is performing a passive scan of the scene in
> continuous
> > > > > - auto-focus mode.
> > > > > - - name: AfStatePassiveFocused
> > > > > - value: 2
> > > > > - description: |
> > > > > - AF believes the scene is in focus, but might restart
> scanning.
> > > > > - - name: AfStateActiveScan
> > > > > - value: 3
> > > > > - description: |
> > > > > - AF is performing a scan triggered by an AF trigger
> request.
> > > > > - \sa AfTrigger
> > > > > - - name: AfStateFocusedLock
> > > > > - value: 4
> > > > > - description: |
> > > > > - AF believes has focused correctly and has locked
> focus.
> > > > > - - name: AfStateNotFocusedLock
> > > > > - value: 5
> > > > > - description: |
> > > > > - AF has not been able to focus and has locked.
> > > > > - - name: AfStatePassiveUnfocused
> > > > > - value: 6
> > > > > - description: |
> > > > > - AF has completed a passive scan without finding focus.
> > > > > -
> > > > > - AwbState:
> > > > > type: int32_t
> > > > > draft: true
> > > > > @@ -690,4 +630,317 @@ controls:
> > > > > value. All of the custom test patterns will be static
> (that is the
> > > > > raw image must not vary from frame to frame).
> > > > >
> > > > > + - AfMode:
> > > > > + type: int32_t
> > > > > + draft: true
> > > >
> > > > Was having all these controls as draft intentional ? The previous
> > > > definitions where set to drafts as they mapped to the android
> > > > definition, but now that we're defining our own ones...
> > >
> > > Not really... I only made them "draft" because it seemed like
> > > everything was "draft" at first! But happy to change that. Are you
> > > saying that these definitely should *not* be draft, then?
> > >
> >
> > Yeah, "draft" was used to import controls from Android as they are for
> > sake of CTS compliance. Let's drop draft from the ones we're defining
>
> Will do!
>
> >
> > > >
> > > > > + description: |
> > > > > + Control to set the mode of the AF (autofocus) algorithm.
> Applications
> > > > > + are allowed to set a new mode, and to send additional
> controls for
> > > > > + that new mode, in the same request.
> > > > > +
> > > > > + An implementation may choose not to implement all the
> modes.
> > > > > +
> > > > > + enum:
> > > > > + - name: AfModeManual
> > > > > + value: 0
> > > > > + description: |
> > > > > + The AF algorithm is in manual mode. In this mode it
> will never
> > > > > + perform any action nor move the lens of its own
> accord, but an
> > > > > + application can set controls to move the lens.
> > > > > +
> > > > > + In this mode the AfState will always report
> AfStateManual.
> > > > > + - name: AfModeAuto
> > > > > + value: 1
> > > > > + description: |
> > > > > + The AF algorithm is in auto mode. This means that the
> algorithm
> > > > > + will never move the lens or change state unless the
> AfTrigger
> > > > > + control is used. The AfTrigger control can be used to
> initiate a
> > > > > + focus scan, the results of which will also be
> reported by AfState.
> > > > > +
> > > > > + If the autofocus algorithm is moved from AfModeAuto
> to another
> > > > > + mode while a scan is in progress, the scan is
> cancelled
> > > > > + immediately, without waiting for the scan to finish.
> > > > > +
> > > > > + When first entering this mode the AfState will report
> > > > > + AfStateAuto. When a trigger control is sent, AfState
> will
> > > > > + report AfStateAutoScanning for a period before
> spontaneously
> > > > > + changing to AfStateAutoFocused or AfStateAutoFailed,
> depending on
> > > > > + the outcome of the scan. It will remain in this state
> until
> > > > > + another scan is initiated by the AfTrigger control.
> If a scan is
> > > > > + cancelled (without changing to another mode), AfState
> will return
> > > > > + to AfStateAuto.
> > > > > + - name: AfModeContinuous
> > > > > + value: 2
> > > > > + description: |
> > > > > + The AF algorithm is in continuous mode. This means
> that the lens
> > > > > + can re-start a scan spontaneously at any moment,
> without any user
> > > > > + intervention. The AfState still reports whether the
> algorithm is
> > > > > + currently scanning or not, though the application has
> no ability
> > > > > + to initiate or cancel scans (though it can "pause"
> them), nor to
> > > > > + move the lens for itself.
> > > > > +
> > > > > + When set to AfModeContinuous, the system will
> immediately initiate
> > > > > + a scan so AfState will report
> AfStateContinuousScanning, and will
> > > > > + settle on one of AfStateContinuousFocused or
> AfStateContinuousFailed,
> > > > > + depending on the scan result.
> > > > > +
> > > > > + - AfLensPosition:
> > > > > + type: int32_t
> > > > > + draft: true
> > > > > + description: |
> > > > > + Control to set the position of the lens to one of the
> following
> > > > > + predefined locations. This control only has any effect
> when the AfMode
> > > > > + is set to AfModeManual.
> > > > > +
> > > > > + This control is distinct from the LensPosition control,
> which sets the
> > > > > + lens position using the lens driver's units.
> > > > > + enum:
> > > > > + - name: AfLensPositionMacro
> > > > > + value: 0
> > > > > + description: The closest focal position of the lens.
> > > > > + - name: AfLensPositionHyperfocal
> > > > > + value: 1
> > > > > + description: Hyperfocal position.
> > > > > + - name: AfLensPositionInfinity
> > > > > + value: 2
> > > > > + description: The furthest focal position (usually
> infinity).
> > > >
> > > > Interesting, bikeshedding on the name apart, is this some
> 'simplified'
> > > > LensState ? Do you think there is a use case for this ? Why would
> > > > anyone set the AF to auto mode then use some coarse grained control
> > > > like this one ? Wouldn't it make more sense as an 'hint' to the AF
> > > > algorithm to restrict the search space ?
> > >
> > > In the light of your comments below about the LensPosition not using
> > > driver units, then maybe this could be deleted, and we could select
> > > some kind of canonical range for the lens?
> > >
> >
> > Ah wait. What I was suggesting ("An "hint" to the AF algorithm to
> > restrict the search space") is basically AfRange.
>
> I wonder if there's a case for a "custom" value for AfRange, and we
> could then even have a control that sets the custom range?
>
> >
> > > The most basic use case would be following on from the capture of an
> > > image after an AF scan:
> > >
> > > - capture image
> > > - AfMode to manual
> > > - Lens to hyperfocal
> > > - AfMode to auto
> > > - now wait for the shutter button to be pressed before you start
> > > another AF scan.
> >
> > Naive question:
> > Wouldn't setting the lens to hyperfocal render the preview unusable ?
>
> Hyperfocal is the place where "most stuff" is normally in focus
> (strictly speaking, has the greatest depth of field), so in the
> absence of other information it's normally the best place to put the
> lens. It's just for the preview, you'd typically run another scan for
> a capture.
>
> (Some systems run Continuous AF during preview, of course. But you
> know my anxiety about CAF algorithms... in my opinion just putting the
> lens to hyperfocal is way less annoying than a not-very-good CAF
> implementation!)
>
> >
> > >
> > > If your UI has a "macro" capture mode of some kind, maybe it's "Lens
> > > to macro" instead of "Lens to hyperfocal".
> >
> > Why is this different than starting an AF scan (without going through
> > Manual) with an AfRange that suggests in which range to scan ? Is this
> > an optimization to forcefully move the lens closer to where it will
> > likely end being moved to ?
>
> If you know you're doing a macro capture it would be helpful for the
> preview to put the lens somewhere where macro things are likely to be
> more in focus, instead of using hyperfocal which will likely make
> close objects very blurry.
>
> >
> > >
> > > But as noted, if LensPosition allows us to select hyperfocal/macro
> > > directly, then we can do without this one.
> > >
> >
> > Providing my above understanding is correct, it would be a bit less
> intuitive
> > for applications to use a numerical value instead of some easier
> > "Macro"/"Hyperfocal". Of course, wrappers like your one can simplify this
> > for their consumers.
>
> True, though I'm now quite liking the "dioptre-like" canonical range
> described below, where 1 = hyperfocal, 0 = infinity, large numbers =
> close
>
> >
> > > >
> > > > (Btw I have a use case for supplying a range of lens positions, which
> > > > has been previously estimated according to some pre-defined use
> cases,
> > > > to the AF algorithm in order restrict the AF search range to increase
> > > > the speed of a scan. Do you think it's a valid use case and it's
> worth a
> > > > control ?)
> > >
> > > I'd imagined doing this kind of thing with the AfSpeed and AfRange
> > > controls and by having the AF algorithm tuned for the module. I'm not
> > > convinced about passing in lens positions from outside the camera
> > > system. Where would they come from? Shouldn't the camera system know
> > > what positions to use?
> > >
> >
> > A possible use case, for controlled environments with low latency
> > requirements, is to run tests beforehand and for each 'scene' define
> > what is the ideal lens movement range. When the application starts an
> > AF scan it would tell the AF algorithm about the 'preferred' range.
> >
> > I understand this works backwards, as the lens position should be the
> > result of the AF algorithm run not one of its inputs :)
>
> Indeed, but as noted above, maybe there's an argument for an
> AfRangeCustom which could be user-settable? We've added "Custom"
> values for other controls in some places, IIRC.
>
> >
> > > In the Raspberry Pi world we would of course store this kind of
> > > information in the tuning file. I've noticed some discussion elsewhere
> > > about supplying configuration to the IPAs - it seems like this might
> > > be a case in point? I think that for real camera systems, in the end,
> > > there is no escape from "configuration" (aka. "tuning").
> > >
> > > >
> > > > > +
> > > > > + - AfRange:
> > > > > + type: int32_t
> > > > > + draft: true
> > > > > + description: |
> > > > > + Control to set the range of focus distances that is
> scanned. An
> > > > > + implementation may choose not to implement all the
> options here.
> > > >
> > > > Only valid in Auto and Continuous mode, right ?
> > >
> > > Generally I've been expecting that you could change settings like this
> > > in Manual mode too, though it obviously can't have any effect until
> > > you go back to Auto/Continuous. Does that sound reasonable?
> >
> > I would rather assume thaat if a control is not valid in the current
> > context is simply ignored and not cached to be re-applied when the
> > context change ?
>
> Possibly, though I'm slightly nervous about how you'd do things like:
>
> - You're in Manual mode, but want to go to Auto and set the Range to
> Normal.
> - You're in Auto mode but were using a different range. You want to
> reset it to Normal before switching to Manual.
>
> Can you send the AfRange change and the new mode at the same time in
> both cases or not? Or do you need two separate requests to send the
> changes? It's not clear to me exactly what would work. If you say that
> the AfRange always gets updated, then the behaviour is obvious, and
> you can send both AfRange and AfMode in the same request which would
> be easier to use.
>
> >
> > >
> > > >
> > > > (This makes me think that we could easily model Continuous by adding
> a
> > > > TriggerContinuous when in Auto mode. With a TriggerPause we could
> get rid of
> > > > AfPause... well, it's quite a redesign so I'll stop here unless
> > > > someone thinks it's a really good idea :)
> > >
> > > Actually I'm quite strongly against anything like this. I think Auto
> > > and Continuous should be quite separate. The reason is that Auto is
> > > (relatively) easy to implement, you do a scan over some range and
> > > choose the best spot. But Continuous AF is an absolute nightmare to do
> > > well. You have all these problems:
> > >
> > > - The algo doesn't realise it needs to rescan and so seems to be
> > > "stuck". (Watch people waving their hands in front of the lens to try
> > > and unstick it.) So you add ever more tests and arbitrary thresholds
> > > on different statistics to tell if a rescan is needed, and then the
> > > whole thing becomes untunable.
> > > - The algo rescans too often. A car in the distance goes by and the
> > > whole image see-saws in and out while a rescan happens, ending back at
> > > the same place. Yet more tests, thresholds...
> > > - You can never make all your tests and thresholds work at the same
> > > time but in different conditions. So you make them variable depending
> > > on things like exposure and gain. Now you are truly doomed.
> > > - The algo is generally not robust to camera shake, which frequently
> > > causes it to end up "failing" with a completely out-of-focus image.
> > > And then you can't get it to rescan... cue more hand-waving in front
> > > of the lens. You implement more special tests and timeouts... At this
> > > point you will wish for the earth to swallow you up.
> > > - When the algo rescans it starts off going the wrong way causing
> > > extra see-sawing.
> > > - You try to solve the last problem by doing really small initial
> > > steps. Only then it ends up statistically unstable.
> > >
> > > From the above you may detect that we have been there and have several
> > > T-shirts. Speaking for myself, I have no plans to implement Continuous
> > > AF without PDAF sensors, as PDAF is quite a big help.
> > >
> >
> > Yes, it feels like you know what you're talking about :)
> >
> > I understand you would like be cautious with CAF, but if one doesn't
> > want to implement CAF can't she simply not report TriggerContinuous ?
> >
> > Anyway, that was just an idea thrown in the mixer, don't worry about
> > it.
> >
> > > >
> > > > > + enum:
> > > > > + - name: AfRangeNormal
> > > > > + value: 0
> > > > > + description: |
> > > > > + A wide range of focus distances is scanned, all the
> way from
> > > > > + infinity down to close distances, though depending on
> the
> > > > > + implementation, possibly not including the very
> closest macro
> > > > > + positions.
> > > > > + - name: AfRangeMacro
> > > > > + value: 1
> > > > > + description: Only close distances are scanned.
> > > > > + - name: AfRangeFull
> > > > > + value: 2
> > > > > + description: |
> > > > > + The full range of focus distances is scanned just as
> with
> > > > > + AfRangeNormal but this time including the very
> closest macro
> > > > > + positions.
> > > > > +
> > > > > + - AfSpeed:
> > > > > + type: int32_t
> > > > > + draft: true
> > > > > + description: |
> > > > > + Control that determines whether the AF algorithm is to
> move the lens
> > > > > + as quickly as possible or more steadily. For example,
> during video
> > > > > + recording it may be desirable not to move the lens too
> abruptly, but
> > > > > + when in a preview mode (waiting for a still capture) it
> may be
> > > > > + helpful to move the lens as quickly as is reasonably
> possible.
> > > > > + enum:
> > > > > + - name: AfSpeedNormal
> > > > > + value: 0
> > > > > + description: Move the lens at its usual speed.
> > > > > + - name: AfSpeedFast
> > > > > + value: 1
> > > > > + description: Move the lens more quickly.
> > > > > +
> > > > > + - AfWindows:
> > > > > + type: Rectangle
> > > > > + draft: true
> > > > > + description: |
> > > > > + Sets the focus windows used by the AF algorithm. The
> units used express
> > > > > + a proportion of the ScalerCrop control (or if
> unavailable, of the entire
> > > > > + image), as u0.16 format numbers.
> > > > > +
> > > > > + In order to be activated, a rectangle must be programmed
> with non-zero
> > > > > + width and height. If no rectangles are programmed in this
> way, then the
> > > > > + system will choose its own single default window in the
> centre of the
> > > > > + image.
> > > > > +
> > > > > + The details of how the windows are used are platform
> dependent. We note
> > > > > + that when there is more than one AF window, a typical
> implementation
> > > > > + might find the optimal focus position for each one and
> finally select
> > > > > + the window closest to the camera.
> > > > > +
> > > > > + size: [platform dependent]
> > > >
> > > > I think this should just be
> > > >
> > > > size: [n]
> > >
> > > OK!
> > >
> > > > > +
> > > > > + - AfTrigger:
> > > > > + type: int32_t
> > > > > + draft: true
> > > > > + description: |
> > > > > + This control starts an autofocus scan when AfMode is set
> to AfModeAuto,
> > > > > + and can also be used to terminate a scan early.
> > > > > +
> > > > > + It is ignored if AfMode is set to AfModeManual or
> AfModeContinuous.
> > > > > +
> > > > > + enum:
> > > > > + - name: AfTriggerStart
> > > > > + value: 0
> > > > > + description: Start an AF scan. Ignored if a scan is in
> progress.
> > > > > + - name: AfTriggerCancel
> > > > > + value: 1
> > > > > + description: Cancel an AF scan. This does not cause the
> lens to move
> > > > > + anywhere else. Ignored if no scan is in progress.
> > > > > +
> > > > > + - AfPause:
> > > > > + type: int32_t
> > > > > + draft: true
> > > > > + description: |
> > > > > + This control has no effect except when in continuous
> autofocus mode
> > > > > + (AfModeContinuous). It can be used to pause any lens
> movements while
> > > > > + (for example) images are captured. The algorithm remains
> inactive
> > > > > + until it is instructed to resume.
> > > > > +
> > > > > + enum:
> > > > > + - name: AfPauseImmediate
> > > > > + value: 0
> > > > > + description: |
> > > > > + Pause the continuous autofocus algorithm immediately,
> whether or
> > > > > + not any kind of scan is underway. If the AfState was
> previously
> > > > > + reporting AfStateContinuousScanning it will now
> change to
> > > > > + AfStateContinuousScanningPaused, and similarly for
> > > > > + AfStateContinuousFocused and AfStateContinuousFailed.
> > > > > + - name AfPauseDeferred
> > > > > + value: 1
> > > > > + description: |
> > > > > + Pause the continuous autofocus algorithm as soon as
> it is no longer
> > > > > + scanning. If the AfState is currently reporting
> > > > > + AfStateContinuousFocused is will change to
> > > > > + AfStateContinuousFocusedPaused, and similarly in the
> case of
> > > > > + AfStateContinuousFailed.
> > > > > +
> > > > > + If AfState reports AfStateContinuousScanning it will
> change to
> > > > > + AfStateContinuousScanningPausing, and then move to
> one of
> > > > > + AfStateContinuousFocusedPaused or
> AfStateContinuousFailedPaused
> > > > > + when the scan completes.
> > > > > + - name: AfPauseResume
> > > > > + value: 2
> > > > > + description: |
> > > > > + Resume continous autofocus operation. The algorithm
> starts again
> > > > > + from exactly where it left off, and the AfState will
> drop the
> > > > > + 'Paused' or 'Pausing' part of the state.
> > > > > +
> > > > > + - LensPosition:
> > > > > + type: int32_t
> > > > > + draft: true
> > > > > + description: |
> > > > > + Acts as a control to instruct the lens to move to a
> particular position
> > > > > + and also reports back the position of the lens for each
> frame.
> > > > > +
> > > > > + The units are determined by the lens driver.
> > > >
> > > > Ugh, we cannot do this in the control definition. Driver
> specificities
> > > > cannot surface to the controls definition, which should be platform
> > > > independent.
> > > >
> > > > I think we'll have to establish a numerical range (go ahead if you
> > > > have any idea) and translate to the actual driver values in the
> > > > CameraLens class ?
> > >
> > > I wasn't aware of this, but maybe it's a good thing. The question is
> >
> > Well, isn't it the same with "Brightness", for example ?
>
> Indeed, or ExposureTime where we explicitly want microseconds, not driver
> units.
>
> >
> > > what kind of range to use, and how do you know what limits apply for
> > > any given camera module? How about a dioptre-like scale, normalised
> > > for the hyperfocal distance H. It would go like this:
> > >
> > > The value n would mean being focused at distance H/n (n does not have
> > > to be an integer). So:
> > >
> > > The value 1 would mean hyperfocal position (focal distance H/1).
> > > The value 0 would mean infinity (focal distance H/0).
> > > Larger values would get progressively closer.
> > >
> > > Setting a control value outside the allowed range will clamp it to
> > > that end of the range.
> >
> > I sincerely don't know enough about this to tell if it's a good idea
> > or not. Hence, it seems like a good idea to me :)
> >
> > >
> > > In the Raspberry Pi world it would be sufficient for the tuning file
> > > to know the allowable range of values. More generally, I could imagine
> > > an application might want to say "focus at precisely 1m" in which case
> > > it would have to know the value of H from a property or something,
> > > though providing such information would be optional (some lenses
> > > simply aren't precisely calibrated). Perhaps an application might want
> > > to get the allowable range of (maybe approximate) focal distances too?
> > >
> > > >
> > > > > +
> > > > > + The LensPosition control is ignored unless the AfMode is
> set to
> > > > > + AfModeManual.
> > > > > +
> > > > > + Note that this control is distinct from AfLensPosition,
> which allows
> > > > > + the lens to be moved to its macro/hyperfocal/infinity
> position, rather
> > > > > + than using lens driver units.
> > > > > +
> > > > > + - AfState:
> > > > > + type: int32_t
> > > > > + draft: true
> > > > > + description: |
> > > > > + Reports the current state of the AF algorithm. The
> possible state changes
> > > > > + are described below, although we add that the following
> general state
> > > > > + changes are also allowed.
> > > >
> > > > wow, there is a lot of states here :)
> > >
> > > Did you follow the link to the state transition diagram? I would hope
> > > that should go some way to explaining it! Here's the link again for
> > > convenience:
> > >
> > >
> https://docs.google.com/drawings/d/1rU2nS82dEC8Y2eY0lKPkQQhd67gKLOl7PvSeUcL1j_E/edit?usp=sharing
> > >
> >
> > Missed it. Thanks for providing it.
> >
> > > >
> > > > Let me copy them here
> > > >
> > > > - name: AfStateManual
> > > > - name: AfStateAuto
> > > > - name: AfStateAutoScanning
> > > > - name: AfStateAutoFocused
> > > > - name: AfStateAutoFailed
> > > > - name: AfStateContinuousScanning
> > > > - name: AfStateContinuousScanningPausing
> > > > - name: AfStateContinuousScanningPaused
> > > > - name: AfStateContinuousFocused
> > > > - name: AfStateContinuousFocusedPaused
> > > > - name: AfStateContinuousFailed
> > > > - name: AfStateContinuousFailedPaused
> > >
> > > I "multiplied out" all the different parts of the state to make a
> > > single long list. Actually I think that makes it more Android-like.
> > > But we could reduce this number if, for example, we're happy to report
> > > the state of AfMode separately on every frame. I don't mind - do we
> > > have a general preference for this kind of thing?
> > >
> >
> > Good question.
> >
> > More generally we need to establish a policy for metadata. For
> > controls we know that controls should be attached to a request only
> > when their value changes. If a control is not supplied with new
> > values, then the old (or default) one will stay valid.
> >
> > This implies the PH/IPA state machine is then stateful and 'remembers'
> > what was the last value a control was set to.
> >
> > What about metadata instead ? Can we assume the same stateful-ness in
> > applications too, or is it a burden we cannot impose on them ?
> >
> > Han-lin, do you know how does that work with Android ? Does it report
> > metadata even when their values do not change or do they assume that,
> > like controls, a metadata is reported only when its value has changed.
> >
>
I think Android metadata doesn't have the assumption, and is supposed to
return with each frame.
In fact, with partial results supported, Android doesn't enforce a strict
order of returning metadata among requests.
It's more like each metadata is independent of the previous one.
> What would the preference be for the RPi applications ?
>
> I don't think I have a very strong preference. I'm happy with either:
>
> - Report everything in the AfState on every frame.
> - Report the state in multiple metadata items (say, AfState and
> AfMode) on every frame.
>
> I'm less keen on having applications have to remember if they sent
> something and then wait for the request where it actually happened, it
> seems to me that would simply add annoying boilerplate code to all
> applications. What do you think?
>
> > > >
> > > > (please note that AfStateAuto seems mis-aligned, unless I messed up
> > > > something while cutting&paste. Please check the compiled doc to see
> > > > how it looks like...)
> > >
> > > I will check.
> > >
> > > >
> > > > So we have three macro groups
> > > >
> > > > - Manual
> > > >
> > > > - Auto
> > > > - Scanning
> > > > - Focus
> > > > - Failed
> > > >
> > > > - Continuous
> > > >
> > > > - Scanning
> > > > - Pausing
> > > > - Paused
> > > > - Focused
> > > > - Paused
> > > > - Failed
> > > > - Paused
> > > >
> > > > The first group is AfMode (Manual, Auto, Continuous) and can be
> > > > deduced by inspecting the AfMode metadata.
> > > >
> > > > The second group is the actual state (Scannng, Focused, Failed)
> > > >
> > > > The third group is specific to continuous mode as it accepts a Pause
> > > > trigger.
> > > >
> > > > I understand why you have this, as expressing "The AfState is set to
> > > > $something when in AfModeManual" etc etc might get cumbersome, but I
> > > > think it might be worth a try
> > > >
> > > > 1) Manual mode (the bad)
> > > > As we all know there is a delay between when a lens is instructed
> > > > to move and when it reaches its final state.
> > > >
> > > > In Manual mode the lens is moved with LensPosition and
> > > > AfLensPosition.
> > > >
> > > > To know the current state:
> > > > - With LensPosition metadata can be inspected until it doesn't
> reach the
> > > > desired value.
> > > > - With AfLensPosition it is not clear to me how to know when the
> lens
> > > > as reched one of the pre-defined positions.
> > >
> > > It sounds like we're deleting AfLensPosition.
> > >
> > > >
> > > > Is it so wrong to report either (Scanning,Focus,Failed) even for
> > > > Manual mode ?
> > >
> > > I don't think it's wrong. If we have a single long list of states then
> > > it just makes for more states. If we don't then it's harmless, though
> > > I'm not aware of a use for it.
> > >
> > > >
> > > > 2) AutoMode (the good)
> > > > This seems the easy one, as (Scanning,Focus,Failed) are already
> > > > there
> > > >
> > > > 3) Continous (the ugly)
> > > > Continuous mode can be paused. That's a shame :)
> > > > Would that be that bad keeping a Paused or Pausing state which can
> > > > only be returned for Continuous mode.
> > >
> > > The whole paused/pausing thing rather complicates things, but it does
> > > seem to me that it's necessary. If you were in Continuous mode, you
> > > would rather the lens didn't set off again just as you want to capture
> > > a sequence of pictures.
> > >
> > > >
> > > > All in all, can we reduce the number of states to
> > > >
> > > > - Scanning
> > > >
> > > > The AF algorithm is scanning and has not yet reached a
> > > > stable result. When running in Auto mode an AF scan is
> > > > started by using the AfTrigger control. When in Manual mode
> > > > and a new lens position is requested either by sending a
> > > > LensPosition or AfLensPosition control, the AF state will
> > > > report "Scanning" until the lens doesn't reach the desired
> > > > position. Similarly, when running in continuous mode and
> the
> > > > AF algorithm decides to start a scan spontaneously the
> > > > AfState will report Scanning until a new stable state is
> > > > reached.
> > > >
> > > > - Focus
> > > >
> > > > The AF algorithm has reached a stable state and the image
> is
> > > > now focused. When running in Manual mode the lens has
> > > > reached the position requested using either LensPosition
> and
> > > > AfLensPosition.
> > > >
> > > > - Failed
> > > >
> > > > The AF algorithm has completed a scan and without finding a
> > > > good focus position. When running in manual mode, the
> camera
> > > > has failed moving the lens to the desired position.
> > > >
> > > > - Paused (copying your below text, please see comments below)
> > > >
> > > > The AF algorithm is in continuous mode (AfModeContinuous)
> and
> > > > was scanning when AfPauseImmediate was sent. The scan will
> not
> > > > complete and the algorithm will remain in this state.
> > > >
> > > > - Pausing
> > > >
> > > > The AF algorithm is in continuous mode (AfModeContinuous)
> and
> > > > was scanning when AfPauseDeferred was sent. The scan will
> complete
> > > > at which point the algorithm moves to the
> > > > AfStateContinuousFocusedPaused or
> AfStateContinuousFailedPaused
> > > > state.
> > >
> > > I think we could get by if we report the following for every frame:
> > >
> > > AfMode - Manual, Auto or Continuous
> > > AfState - Scanning, Focused or Failed
> > > AfPause - Resume (which means it's running), Immediate or Deferred
> > > (these states only apply during Continuous; the Deferred value can
> > > only occur when also Scanning).
> > >
> > > Or we could wrap AfState and AfPause together:
> > >
> > > AfState - Scanning, Focused. Failed, ScanningPausing, ScanningPaused,
> > > FocusedPaused, FailedPaused.
> > >
> > > That makes some of the interaction between pausing and scanning
> > > clearer, but then it's more states again. I don't know...
> > >
> >
> > Let's clarify what the policy on metadata is first.
>
> Agree!
>
> >
> > > >
> > > >
> > > > > +
> > > > > + In any of the AfStateManual or AfStateContinuous states,
> the AfMode
> > > > > + may be set to AfModeAuto and the algorithm will move to
> the
> > > > > + AfStateAuto state. If AfTriggerStart is sent at the same
> time
> > > > > + then the algorithm will move to AfStateAutoScanning state.
> > > > > +
> > > > > + In any of the AfStateAuto or AfStateManual states, the
> AfMode may
> > > > > + be set to AfModeContinuous and the algorithm will move to
> > > > > + AfStateContinuousScanning.
> > > > > +
> > > > > + In any of the AfStateAuto or AfStateContinuous states,
> the AfMode may
> > > > > + be set to AfModeManual and the algorithm will move to
> > > > > + AfStateManual. The lens will not be moved and will be
> left where
> > > > > + it was at that moment.
> > > > > +
> > > > > + enum:
> > > > > + - name: AfStateManual
> > > > > + value: 0
> > > > > + description: |
> > > > > + The AF algorithm is in manual mode (AfModeManual).
> The LensPosition
> > > > > + and AfLensPosition controls can be used directly to
> move the lens.
> > > > > + - name: AfStateAuto
> > > > > + value: 1
> > > > > + description: |
> > > > > + The AF algorithm is in auto mode (AfModeAuto), and
> has either just
> > > > > + been moved into that state, or a scan that was in
> progress has been
> > > > > + cancelled.
> > > > > + - name: AfStateAutoScanning
> > > > > + value: 2
> > > > > + description: |
> > > > > + The AF algorithm is in auto mode (AfModeAuto), and a
> scan has been
> > > > > + started using the AfTrigger control. The scan can be
> cancelled by
> > > > > + sending AfTriggerCancel at which point the algorithm
> will either
> > > > > + move back to AfStateAuto or, if the scan actually
> completes
> > > > > + before the cancel request is processed, to one of
> > > > > + AfStateAutoFocused or AfStateAutoFailed.
> > > > > + - name: AfStateAutoFocused
> > > > > + value: 3
> > > > > + description: |
> > > > > + The AF algorithm is in auto mode (AfModeAuto), and a
> scan has
> > > > > + completed with the result that the algorithm believes
> the image is
> > > > > + now in focus.
> > > > > + - name: AfStateAutoFailed
> > > > > + value: 4
> > > > > + description: |
> > > > > + The AF algorithm is in auto mode (AfModeAuto), and a
> scan has
> > > > > + completed with the result that the algorithm did not
> find a good
> > > > > + focus position.
> > > > > + - name: AfStateContinuousScanning
> > > > > + value: 5
> > > > > + description: |
> > > > > + The AF algorithm is in continuous mode
> (AfModeContinuous) and
> > > > > + is currently scanning for a good focus position. This
> occurs when
> > > > > + the mode is first set to AfModeContinuous, or may
> happen
> > > > > + spontaneously when the algorithm believes a re-scan
> is needed.
> > > > > + - name: AfStateContinuousScanningPausing
> > > > > + value: 6
> > > > > + description: |
> > > > > + The AF algorithm is in continuous mode
> (AfModeContinuous) and
> > > > > + was scanning when AfPauseDeferred was sent. The scan
> will complete
> > > > > + at which point the algorithm moves to the
> > > > > + AfStateContinuousFocusedPaused or
> AfStateContinuousFailedPaused
> > > > > + state.
> > > > > + - name: AfStateContinuousScanningPaused
> > > > > + value: 7
> > > > > + description: |
> > > > > + The AF algorithm is in continuous mode
> (AfModeContinuous) and
> > > > > + was scanning when AfPauseImmediate was sent. The scan
> will not
> > > >
> > > > Why can't the algorithm be paused when it has reached a stable state
> > > > but only when scanning ? Won't a pause 'freeze' the algorithm in its
> > > > current state no matter what it is ? I understand with
> > > > "AfPauseDeferred" that it has to wait for AF to be locked before
> > > > stopping it, but with "Immediate" doesn't "pause" apply to all
> states ?
> > >
> > > Wasn't sure I understood this. It can be paused once it's reached
> > > Focused or Failed, and then both PauseImmediate and PauseDeferred will
> > > cause it to report FocusedPaused or FailedPaused. Was that the
> > > question?
> >
> > You're right, I have probably missed FocusedPaused and FailedPaused.
> > :)
> >
> > Thanks
> > j
>
> Thanks for all the discussion!
>
> David
>
> >
> > >
> > > Anyway, thanks very much!
> > >
> > > David
> > >
> > > >
> > > > Thanks
> > > > j
> > > >
> > > > > + complete and the algorithm will remain in this state.
> The scan
> > > > > + may be resumed by sending AfPauseResume.
> > > > > + - name: AfStateContinuousFocused
> > > > > + value: 8
> > > > > + description: |
> > > > > + The AF algorithm is in continuous mode
> (AfModeContinuous) and
> > > > > + a scan has completed with the algorithm believing it
> has found a
> > > > > + good focus position.
> > > > > + - name: AfStateContinuousFocusedPaused
> > > > > + value: 9
> > > > > + description: |
> > > > > + The AF algorithm was in the AfStateContinuousFocused
> state and
> > > > > + has been paused (by either AfPauseImmediate or
> AfPauseDeferred),
> > > > > + or it was in the AfStateContinuousScanningPausing
> state and the
> > > > > + scan has completed successfully. The algorithm will
> now remain
> > > > > + in this state, and may be resumed by sending
> AfPauseResume.
> > > > > + - name: AfStateContinuousFailed
> > > > > + value: 10
> > > > > + description: |
> > > > > + The AF algorithm is in continuous mode
> (AfModeContinuous) and
> > > > > + a scan has completed without finding a good focus
> position.
> > > > > + - name: AfStateContinuousFailedPaused
> > > > > + value: 11
> > > > > + description: |
> > > > > + The AF algorithm was in the AfStateContinuousFailed
> state and
> > > > > + has been paused (by either AfPauseImmediate or
> AfPauseDeferred),
> > > > > + or it was in the AfStateContinuousScanningPausing
> state and the
> > > > > + scan has completed unsuccessfully. The algorithm will
> now remain
> > > > > + in this state, and may be resumed by sending
> AfPauseResume.
> > > > > +
> > > > > ...
> > > > > --
> > > > > 2.30.2
> > > > >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.libcamera.org/pipermail/libcamera-devel/attachments/20220407/9fd70a09/attachment-0001.htm>
More information about the libcamera-devel
mailing list