[libcamera-devel] [RFC PATCH 1/1] libcamera: controls: Controls for driving AF (autofocus) algorithms
David Plowman
david.plowman at raspberrypi.com
Mon Jan 17 16:48:55 CET 2022
Hi again everyone
Thanks for the latest replies and all the discussions. I think the best
thing is probably if I get on and produce a v2 of this patch. I think the
main differences will be: a "pause" control for CAF, and I might add an
"off" state in which the lens can be moved. Then we can resume the
discussion once more!
Best regards
David
On Mon, 17 Jan 2022 at 10:13, Hanlin Chen <hanlinchen at chromium.org> wrote:
> The mail is re-sent by another account to avoid DMARC rejection. Sorry
> for the inconvenience.
>
> Hi David,
> Thanks for the comments.
>
> On Fri, Jan 14, 2022 at 1:54 AM David Plowman
> <david.plowman at raspberrypi.com> wrote:
> >
> > Hi Hanlin
> >
> > Sorry for taking some time to get back to this!
> No worries. We all work asynchronously :)
>
> > From one of your other emails, and checking the Android documentation,
> > I see that "Trigger" does mean something in CAF mode though, as you
> > say, a "pause" control captures the meaning a bit better. I think
> > Android uses "trigger start" to mean:
> >
> > - In "video" CAF mode: pause immediately, whatever is happening.
> > - In "picture" CAF mode: pause when any current (passive) scan is
> finished.
> >
> > Perhaps a pause control would have 3 values: "pause immediately",
> > "pause when ready" and "resume".
> I moved the comment up, so I can comment on it earlier.
> Yes, I think it is more clear, although the algorithm may need to
> implement it for both video and picture CAF.
> I suppose it's fine?
> >
> > On Tue, 21 Dec 2021 at 10:21, Hanlin Chen <hanlinchen at chromium.org>
> wrote:
> > >
> > > Hi David, thanks for the comments.
> > >
> > > On Fri, Dec 17, 2021 at 4:51 PM David Plowman
> > > <david.plowman at raspberrypi.com> wrote:
> > > >
> > > > Hi again
> > > >
> > > > Thanks for the comments!
> > > >
> > > > On Thu, 16 Dec 2021 at 11:18, Hanlin Chen <hanlinchen at chromium.org>
> wrote:
> > > > >
> > > > > Hi David, thanks for this work!
> > > > >
> > > > > On Mon, Dec 13, 2021 at 11:22 PM David Plowman
> > > > > <david.plowman at raspberrypi.com> wrote:
> > > > > >
> > > > > > This patch describes a series of controls that allow
> applications to
> > > > > > drive AF algorithms:
> > > > > >
> > > > > > AfMode - auto or continuous
> > > > > > AfRange - full, macro or normal
> > > > > > AfSpeed - fast or slow
> > > > > > AfMethod - single or multi-spot
> > > > > > AfWindow - AF window locations
> > > > > > AfTrigger - start (trigger an AF scan) or cancel
> > > > > > LensPosition - position of lens from lens driver
> > > > > > AfState - reset, scanning, focused or failed
> > > > > > ---
> > > > > > src/libcamera/control_ids.yaml | 227
> ++++++++++++++++++++++++---------
> > > > > > 1 file changed, 167 insertions(+), 60 deletions(-)
> > > > > >
> > > > > > diff --git a/src/libcamera/control_ids.yaml
> b/src/libcamera/control_ids.yaml
> > > > > > index 9d4638ae..e579f7b7 100644
> > > > > > --- a/src/libcamera/control_ids.yaml
> > > > > > +++ b/src/libcamera/control_ids.yaml
> > > > > > @@ -406,27 +406,6 @@ controls:
> > > > > > The camera will cancel any active or completed
> metering sequence.
> > > > > > The AE algorithm is reset to its initial state.
> > > > > >
> > > > > > - - AfTrigger:
> > > > > > - type: int32_t
> > > > > > - draft: true
> > > > > > - description: |
> > > > > > - Control for AF trigger. Currently identical to
> > > > > > - ANDROID_CONTROL_AF_TRIGGER.
> > > > > > -
> > > > > > - Whether the camera device will trigger autofocus for
> this request.
> > > > > > - enum:
> > > > > > - - name: AfTriggerIdle
> > > > > > - value: 0
> > > > > > - description: The trigger is idle.
> > > > > > - - name: AfTriggerStart
> > > > > > - value: 1
> > > > > > - description: The AF routine is started by the camera.
> > > > > > - - name: AfTriggerCancel
> > > > > > - value: 2
> > > > > > - description: |
> > > > > > - The camera will cancel any active trigger and the
> AF routine is
> > > > > > - reset to its initial state.
> > > > > > -
> > > > > > - NoiseReductionMode:
> > > > > > type: int32_t
> > > > > > draft: true
> > > > > > @@ -507,45 +486,6 @@ controls:
> > > > > > The AE algorithm has started a pre-capture metering
> session.
> > > > > > \sa AePrecaptureTrigger
> > > > > >
> > > > > > - - AfState:
> > > > > > - type: int32_t
> > > > > > - draft: true
> > > > > > - description: |
> > > > > > - Control to report the current AF algorithm state.
> Currently identical to
> > > > > > - ANDROID_CONTROL_AF_STATE.
> > > > > > -
> > > > > > - Current state of the AF algorithm.
> > > > > > - enum:
> > > > > > - - name: AfStateInactive
> > > > > > - value: 0
> > > > > > - description: The AF algorithm is inactive.
> > > > > > - - name: AfStatePassiveScan
> > > > > > - value: 1
> > > > > > - description: |
> > > > > > - AF is performing a passive scan of the scene in
> continuous
> > > > > > - auto-focus mode.
> > > > > > - - name: AfStatePassiveFocused
> > > > > > - value: 2
> > > > > > - description: |
> > > > > > - AF believes the scene is in focus, but might
> restart scanning.
> > > > > > - - name: AfStateActiveScan
> > > > > > - value: 3
> > > > > > - description: |
> > > > > > - AF is performing a scan triggered by an AF trigger
> request.
> > > > > > - \sa AfTrigger
> > > > > > - - name: AfStateFocusedLock
> > > > > > - value: 4
> > > > > > - description: |
> > > > > > - AF believes has focused correctly and has locked
> focus.
> > > > > > - - name: AfStateNotFocusedLock
> > > > > > - value: 5
> > > > > > - description: |
> > > > > > - AF has not been able to focus and has locked.
> > > > > > - - name: AfStatePassiveUnfocused
> > > > > > - value: 6
> > > > > > - description: |
> > > > > > - AF has completed a passive scan without finding
> focus.
> > > > > > -
> > > > > > - AwbState:
> > > > > > type: int32_t
> > > > > > draft: true
> > > > > > @@ -690,4 +630,171 @@ controls:
> > > > > > value. All of the custom test patterns will be
> static (that is the
> > > > > > raw image must not vary from frame to frame).
> > > > > >
> > > > > > + - AfMode:
> > > > > > + type: int32_t
> > > > > > + draft: true
> > > > > > + description: |
> > > > > > + Control to set the mode of the AF (autofocus) algorithm.
> > > > > > + enum:
> > > > > > + - name: AfModeAuto
> > > > > > + value: 0
> > > > > > + description: |
> > > > > > + The AF algorithm is in auto mode. This means that
> the algorithm
> > > > > > + will never move the lens or change state unless the
> AfTrigger
> > > > > > + control is used. The AfTrigger control can be used
> to initiate a
> > > > > > + focus scan, the results of which will be reported
> by the AfState.
> > > > > > +
> > > > > > + In this mode, an application can move the lens for
> itself using
> > > > > > + the LensPosition control. If the algorithm was
> scanning when the
> > > > > > + lens is moved in this way, then the scan is
> implicitly cancelled.
> > > > >
> > > > > I would suggest splitting the Auto mode into AfModeOnDemand and
> > > > > AfModeManual (The naming could be re-considered).
> > > > > The reason is that the sub-controls for each AfModes could be
> distinct.
> > > > > 1. AfModeManual:
> > > > > LensPosition is valid only for the mode, and AfState is always
> AfStateReset.
> > > > > 2. AfModeOnDemand
> > > > > AfTrigger and AfRange are valid only for the mode, and AfState
> reports
> > > > > accordingly. The scan is implicitly canceled when changing from the
> > > > > mode.
> > > >
> > > > I'm a little bit nervous about having a mode where the lens position
> > > > control doesn't work. In my earlier email I described performing an
> AF
> > > > scan, capturing an image, then moving the lens back to hyperfocal. It
> > > > seems inconvenient to have to put the AF algorithm into a separate
> > > > mode in order to move the lens, only to put it back in the mode where
> > > > a scan is once again allowed.
> > > >
> > > > I don't think anything is lost by doing it this way - if an
> > > > application doesn't want to perform an AF scan it should simply not
> > > > request one.
> > > I agreed that it does cover the controls from Android, i.e., the
> > > Android request can be translated to the API.
> > > My concern is from another direction: translation from libcamera to
> > > Android Camera3.
> > > From the Android to libcamera:
> > > ANDROID_CONTROL_AF_MODE_OFF + ANDROID_LENS_FOCUS_DISTANCE ==>
> > > AfModeAuto + lensPosition
> > > ANDROID_CONTROL_AF_MODE_CONTINUOUS_VIDEO +
> > > ANDROID_CONTROL_AE_PRECAPTURE_TRIGGER_START ==> AfModeAuto +
> > > AfTrigger
> >
> > I don't quite understand how an AE_PRECAPTURE_TRIGGER_START affects
> > the AF state machine, isn't that something we'd send to the AE
> > algorithm (if it had such a control)? And then we'd wait for the AE
> > state to say it was finished?
>
> Ah... It's a copy error, I simply mean AF_TRIGGER here.
> >
> > > From libcamera to Android:
> > > AfModeAuto + lensPostion (Assuming current position is reported by
> > > libcamera) ==> ??
> > > It could be CAF, AUTO or OFF mode.
> >
> > As I understand it, Android only lets you set the lens position when
> > AF mode is "OFF". So in this case we'd have to set the Android AF to
> > "OFF" along with the lens position, right?
> >
> > If the next thing libcamera wants to do is something other than move
> > the lens, then we'll have to set Android back to the state
> > corresponding to libcamera's. Does that work?
> >
> The context here is that I seem to assume that if we don't have
> "pause" in CAF, we need to mimic it by changing Auto with
> lensPosition.
> Thus, when translating back to Android, it's not clear that it's Auto
> or CAF with mimicking "pause".
> Maybe my assumption is wrong?
>
> However, if we have "pause", then the problem disappears:
> with lensPosition == OFF, without == AUTO.
>
> > > States from libcamera Android:
> > > AfModeAuto + AfTrigger + focused ==>
> > > ANDROID_CONTROL_AF_STATE_FOCUSED_LOCKED? or
> > > ANDROID_CONTROL_AF_STATE_PASSIVE_FOCUSED?
> >
> > If there was a trigger, then I believe the "focused" state needs to
> > correspond to AF_STATE_FOCUSED_LOCKED. In fact, is it the case that in
> > auto mode, Android can never report AF_STATE_PASSIVE_FOCUSED?
> Same above.
> Yes, if it's AF_MODE_AUTO, it never report AF_STATE_PASSIVE_FOCUSED.
> >
> > >
> > > For the Android adapter layer, above might be workarounded by caching
> > > the requests and do reference and find out current context.
> > > A more subtle case might arise when integrating other platforms.
> > > For example, Qualcomm closed 3A lib follows the modes and states
> > > according to Android (Partly true for Intel and MTK).
> > > In the case, the translation flow will like:
> > > Android API -> libcamera API -> Android like API for 3A lib ->
> > > libcamera API -> Android API
> > > <Android> <libcamera> <IPA>
> > > <libcamera> <Android>
> > > It's hard to predict how to work around all corner cases.
> > > This makes me believe that if we could have a clear one to one mapping
> > > between Android and libcamera API, it could be beneficial later.
> > > However, it will make the API more verbose.
> > >
> > > Another perspective is that Android defines required state machine
> transition:
> > > https://source.android.com/devices/camera/camera3_3Amodes
> > >
> > > For IPA to implement the state transition satisfying Android's
> > > requirement, the IPA should know exactly how the libcamera states
> > > mapping back to the Android states.
> > > Above is not satisfying because we will be somehow restricted by
> > > Android's definition ;-|, However, I think it could be beneficial if
> > > the API cam map libcamera states/modes to all Android states/modes
> > > directly without extra context/call order information.
> >
> > I'm really in two minds about the question of an "off" mode in
> > libcamera. Either we make libcamera application code handle the
> > transition to "off" and back to "auto", or a libcamera-to-Android
> > translation layer would have to do it. Does anyone else out there feel
> > strongly?
>
> I suppose we could align it with AE (or let AE align with AF :P). AE
> has a similar case for AnalogueGainMode and ExposureTimeMode.
> https://patchwork.libcamera.org/patch/15179/
> Where Paul seems to use "Auto" and "Disabled" for the case.
> cc'ed Paul.
>
> >
> > >
> > > >
> > > > I think AfRange applies to CAF as well. I see no reason why you might
> > > > not want to restrict the range of searches for just the same reasons
> > > > as you might do with "regular" AF.
> > > Agreed.
> > > >
> > > > > 3. AfModeContinuous
> > > > > AfSpeed and "pause" are valid only for the mode, and AfState
> reports
> > > > > accordingly. The scan is implicitly canceled when changing from the
> > > > > mode.
> > > >
> > > > I'm not sure AfSpeed is *necessarily* only for CAF. I think we've
> done
> > > > AF systems in the past where even regular AF has a "I really want to
> > > > take the picture quickly" mode. But it could be a platform choice?
> > > Agreed.
> > > >
> > > > >
> > > > > > + - name: AfModeContinuous
> > > > > > + value: 1
> > > > > > + description: |
> > > > > > + The AF algorithm is in continuous mode. This means
> that the lens
> > > > > > + can re-start a scan spontaneously at any moment,
> without any user
> > > > > > + intervention. The AfState still reports whether the
> algorithm is
> > > > > > + currently scanning or not, though the application
> has no ability
> > > > > > + to initiate or cancel scans, nor move the lens for
> itself.
> > > > > > +
> > > > > > + The mode can be set to AfModeAuto which has the
> effect of
> > > > > > + "pausing" any continuous AF activity, after which
> it could then
> > > > > > + be moved back to AfModeContinuous to resume
> operation.
> > > > > > +
> > > > > > + - AfRange:
> > > > > > + type: int32_t
> > > > > > + draft: true
> > > > > > + description: |
> > > > > > + Control to set the range of focus distances that is
> scanned.
> > > > > > + enum:
> > > > > > + - name: AfRangeFull
> > > > > > + value: 0
> > > > > > + description: The full range of focus distances is
> scanned.
> > > > > > + - name: AfRangeMacro
> > > > > > + value: 1
> > > > > > + description: Only close distances are scanned.
> > > > > > + - name: AfRangeNormal
> > > > > > + value: 2
> > > > > > + description: |
> > > > > > + The full range of focus distances is scanned except
> for some of
> > > > > > + the closest macro positions.
> > > > > > +
> > > > > > + - AfSpeed:
> > > > > > + type: int32_t
> > > > > > + draft: true
> > > > > > + description: |
> > > > > > + Control that determines whether the AF algorithm is to
> move the lens
> > > > > > + as quickly as possible or more steadily. For example,
> during video
> > > > > > + recording it may be desirable not to move the lens too
> abruptly, but
> > > > > > + when in a preview mode (waiting for a still capture) it
> may be
> > > > > > + helpful to move the lens as quickly as is reasonably
> possible.
> > > > > > + enum:
> > > > > > + - name: AfSpeedFast
> > > > > > + value: 0
> > > > > > + description: Move the lens quickly.
> > > > > > + - name: AfSpeedSlow
> > > > > > + value: 0
> > > > > > + description: Move the lens more steadily.
> > > > > How about
> > > > > s/Fast/Aggressively
> > > > > s/Slow/Steadily
> > > >
> > > > I agree the choice is tricky here. I think I would regard what I
> > > > called "Slow" to be the "usual" choice, so perhaps "AfSpeedNormal" is
> > > > better (and maybe make that value 0). I still prefer "AfModeFast" to
> > > > "AfModeAggressive" - "aggressive" sounds just a bit too...
> aggressive?
> > > > :)
> > > LGTM.
> > > >
> > > > But thanks for all the discussion. Glad we're making progress on all
> this!
> > > >
> > > > Best regards
> > > > David
> > > >
> > > > > > +
> > > > > > + - AfMethod:
> > > > > > + type: int32_t
> > > > > > + draft: true
> > > > > > + description: |
> > > > > > + Control whether the AF algorithm uses a single window
> in the image to
> > > > > > + determine the best focus position, or multiple windows
> simultaneously.
> > > > > > + enum:
> > > > > > + - name: AfMethodSingle
> > > > > > + value: 0
> > > > > > + description: |
> > > > > > + A single window within the image, defaulting to the
> centre, is used
> > > > > > + to select the best focus distance.
> > > > > > + - name: AfMethodMultiSpot
> > > > > > + value: 0
> > > > > > + description: |
> > > > > > + Multiple windows within the image are used to
> select the best focus
> > > > > > + distance. The best focus distance is found for each
> one of the
> > > > > > + windows, and then the distance that is closest to
> the camera is
> > > > > > + selected.
> > > > > > +
> > > > > > + - AfWindow:
> > > > > > + type: Rectangle
> > > > > > + draft: true
> > > > > > + description: |
> > > > > > + Sets the focus windows used by the AF algorithm. The
> units used express
> > > > > > + a proportion of the ScalerCrop control (or if
> unavailable, of the entire
> > > > > > + image), as u0.16 format numbers.
> > > > > > +
> > > > > > + In order to be activated, a rectangle must be
> programmed with non-zero
> > > > > > + width and height. If no rectangles are programmed in
> this way, then the
> > > > > > + system will choose its own single default window in
> the centre of the
> > > > > > + image.
> > > > > > +
> > > > > > + If AfMethod is set to AfMethodSingle, then only the
> first Rectangle in
> > > > > > + this list is used (or the system default one if it is
> unprogrammed).
> > > > > > +
> > > > > > + If AfMethod is set to AfMethodMultiSpot then all the
> valid Rectangles in
> > > > > > + this list are used. The size of the control indicates
> how many such
> > > > > > + windows can be programmed and will vary between
> different platforms.
> > > > > > +
> > > > > > + size: [platform dependent]
> > > > > > +
> > > > > > + - AfTrigger:
> > > > > > + type: int32_t
> > > > > > + draft: true
> > > > > > + description: |
> > > > > > + This control starts an autofocus scan when AfMode is
> set to AfModeAuto,
> > > > > > + and can also be used to terminate a scan early.
> > > > > > +
> > > > > > + It is ignored if AfMode is set to AfModeContinuous.
> > > > > > +
> > > > > > + enum:
> > > > > > + - name: AfTriggerStart
> > > > > > + value: 0
> > > > > > + description: Start an AF scan. Ignored if a scan is
> in progress.
> > > > > > + - name: AfTriggerCancel
> > > > > > + value: 1
> > > > > > + description: Cancel an AF scan. Ingored if no scan is
> in progress.
> >
> > From one of your other emails, and checking the Android documentation,
> > I see that "Trigger" does mean something in CAF mode though, as you
> > say, a "pause" control captures the meaning a bit better. I think
> > Android uses "trigger start" to mean:
> >
> > - In "video" CAF mode: pause immediately, whatever is happening.
> > - In "picture" CAF mode: pause when any current (passive) scan is
> finished.
> >
> > Perhaps a pause control would have 3 values: "pause immediately",
> > "pause when ready" and "resume".
> >
> > Thanks and best regards
> >
> > David
> >
> > > > > > +
> > > > > > + - LensPosition:
> > > > > > + type: int32_t
> > > > > > + draft: true
> > > > > > + description: |
> > > > > > + Acts as a control to instruct the lens to move to a
> particular position
> > > > > > + and also reports back the position of the lens for
> each frame.
> > > > > > +
> > > > > > + The units are determined by the lens driver.
> > > > > > +
> > > > > > + If the LensPosition control is used while an AF scan
> is in progress,
> > > > > > + the scan is implicitly cancelled and the lens is then
> moved to the
> > > > > > + desired location. It is ignored if AfMode is set to
> AfModeContinuous.
> > > > > > +
> > > > > > + - AfState:
> > > > > > + type: int32_t
> > > > > > + draft: true
> > > > > > + description: |
> > > > > > + Reports the current state of the AF algorithm.
> > > > > > + enum:
> > > > > > + - name: AfStateReset
> > > > > > + value: 0
> > > > > > + description: |
> > > > > > + The AF algorithm reports this state when:
> > > > > > + * The camera system has just been started.
> > > > > > + * A scan has been cancelled.
> > > > > > + * The lens has been moved by the LensPosition
> control.
> > > > > > + - name: AfStateScanning
> > > > > > + value: 1
> > > > > > + description: |
> > > > > > + AF is performing a scan. This state can be
> entered spontaneously
> > > > > > + if AfMode is set to AfModeContinuous, otherwise
> it requires the
> > > > > > + application to use the AfTrigger control to start
> the scan.
> > > > > > + - name: AfStateFocused
> > > > > > + value: 2
> > > > > > + description: |
> > > > > > + An AF scan has been performed and the algorithm
> believes the
> > > > > > + scene is in focus.
> > > > > > + - name: AfStateFailed
> > > > > > + value: 3
> > > > > > + description: |
> > > > > > + An AF scan has been performed but the algorithm
> has not been able
> > > > > > + to find the best focus position.
> > > > > > +
> > > > > > ...
> > > > > > --
> > > > > > 2.30.2
> > > > > >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.libcamera.org/pipermail/libcamera-devel/attachments/20220117/9f545f86/attachment-0001.htm>
More information about the libcamera-devel
mailing list