<div dir="ltr"><div dir="ltr">Hi David and Jacopo,</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Apr 7, 2022 at 5:05 PM David Plowman <<a href="mailto:david.plowman@raspberrypi.com">david.plowman@raspberrypi.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi Jacopo<br>
<br>
Thanks for the reply!<br>
<br>
On Wed, 6 Apr 2022 at 17:34, Jacopo Mondi <<a href="mailto:jacopo@jmondi.org" target="_blank">jacopo@jmondi.org</a>> wrote:<br>
><br>
> Hi David,<br>
> + Han-lin for an Android question on metadata<br>
><br>
> On Wed, Apr 06, 2022 at 02:05:20PM +0100, David Plowman wrote:<br>
> > Hi Jacopo<br>
> ><br>
> > Thanks for taking the time to read all this!<br>
> ><br>
> > On Tue, 5 Apr 2022 at 21:36, Jacopo Mondi <<a href="mailto:jacopo@jmondi.org" target="_blank">jacopo@jmondi.org</a>> wrote:<br>
> > ><br>
> > > Hi David,<br>
> > > sorry for being late :)<br>
> > ><br>
> > > On Thu, Mar 31, 2022 at 04:17:47PM +0100, David Plowman via libcamera-devel wrote:<br>
> > > > This patch describes a series of controls that allow applications to<br>
> > > > drive AF algorithms:<br>
> > > ><br>
> > > > AfMode - manual, auto or continuous<br>
> > > > AfLensPosition - set lens position to macro/hyperfocal/infinity<br>
> > > > AfRange - full, macro or normal<br>
> > > > AfSpeed - fast or slow<br>
> > > > AfWindows - AF window locations<br>
> > > > AfTrigger - start (trigger an AF scan) or cancel<br>
> > > > AfPause - pause continuous AF<br>
> > > > LensPosition - position of lens from lens driver<br>
> > > > AfState - reports the mode, whether scanning/success/failure<br>
> > > > ---<br>
> > > > src/libcamera/control_ids.yaml | 373 +++++++++++++++++++++++++++------<br>
> > > > 1 file changed, 313 insertions(+), 60 deletions(-)<br>
> > > ><br>
> > > > diff --git a/src/libcamera/control_ids.yaml b/src/libcamera/control_ids.yaml<br>
> > > > index 9d4638ae..23607368 100644<br>
> > > > --- a/src/libcamera/control_ids.yaml<br>
> > > > +++ b/src/libcamera/control_ids.yaml<br>
> > > > @@ -406,27 +406,6 @@ controls:<br>
> > > > The camera will cancel any active or completed metering sequence.<br>
> > > > The AE algorithm is reset to its initial state.<br>
> > > ><br>
> > > > - - AfTrigger:<br>
> > > > - type: int32_t<br>
> > > > - draft: true<br>
> > > > - description: |<br>
> > > > - Control for AF trigger. Currently identical to<br>
> > > > - ANDROID_CONTROL_AF_TRIGGER.<br>
> > > > -<br>
> > > > - Whether the camera device will trigger autofocus for this request.<br>
> > > > - enum:<br>
> > > > - - name: AfTriggerIdle<br>
> > > > - value: 0<br>
> > > > - description: The trigger is idle.<br>
> > > > - - name: AfTriggerStart<br>
> > > > - value: 1<br>
> > > > - description: The AF routine is started by the camera.<br>
> > > > - - name: AfTriggerCancel<br>
> > > > - value: 2<br>
> > > > - description: |<br>
> > > > - The camera will cancel any active trigger and the AF routine is<br>
> > > > - reset to its initial state.<br>
> > > > -<br>
> > > > - NoiseReductionMode:<br>
> > > > type: int32_t<br>
> > > > draft: true<br>
> > > > @@ -507,45 +486,6 @@ controls:<br>
> > > > The AE algorithm has started a pre-capture metering session.<br>
> > > > \sa AePrecaptureTrigger<br>
> > > ><br>
> > > > - - AfState:<br>
> > > > - type: int32_t<br>
> > > > - draft: true<br>
> > > > - description: |<br>
> > > > - Control to report the current AF algorithm state. Currently identical to<br>
> > > > - ANDROID_CONTROL_AF_STATE.<br>
> > > > -<br>
> > > > - Current state of the AF algorithm.<br>
> > > > - enum:<br>
> > > > - - name: AfStateInactive<br>
> > > > - value: 0<br>
> > > > - description: The AF algorithm is inactive.<br>
> > > > - - name: AfStatePassiveScan<br>
> > > > - value: 1<br>
> > > > - description: |<br>
> > > > - AF is performing a passive scan of the scene in continuous<br>
> > > > - auto-focus mode.<br>
> > > > - - name: AfStatePassiveFocused<br>
> > > > - value: 2<br>
> > > > - description: |<br>
> > > > - AF believes the scene is in focus, but might restart scanning.<br>
> > > > - - name: AfStateActiveScan<br>
> > > > - value: 3<br>
> > > > - description: |<br>
> > > > - AF is performing a scan triggered by an AF trigger request.<br>
> > > > - \sa AfTrigger<br>
> > > > - - name: AfStateFocusedLock<br>
> > > > - value: 4<br>
> > > > - description: |<br>
> > > > - AF believes has focused correctly and has locked focus.<br>
> > > > - - name: AfStateNotFocusedLock<br>
> > > > - value: 5<br>
> > > > - description: |<br>
> > > > - AF has not been able to focus and has locked.<br>
> > > > - - name: AfStatePassiveUnfocused<br>
> > > > - value: 6<br>
> > > > - description: |<br>
> > > > - AF has completed a passive scan without finding focus.<br>
> > > > -<br>
> > > > - AwbState:<br>
> > > > type: int32_t<br>
> > > > draft: true<br>
> > > > @@ -690,4 +630,317 @@ controls:<br>
> > > > value. All of the custom test patterns will be static (that is the<br>
> > > > raw image must not vary from frame to frame).<br>
> > > ><br>
> > > > + - AfMode:<br>
> > > > + type: int32_t<br>
> > > > + draft: true<br>
> > ><br>
> > > Was having all these controls as draft intentional ? The previous<br>
> > > definitions where set to drafts as they mapped to the android<br>
> > > definition, but now that we're defining our own ones...<br>
> ><br>
> > Not really... I only made them "draft" because it seemed like<br>
> > everything was "draft" at first! But happy to change that. Are you<br>
> > saying that these definitely should *not* be draft, then?<br>
> ><br>
><br>
> Yeah, "draft" was used to import controls from Android as they are for<br>
> sake of CTS compliance. Let's drop draft from the ones we're defining<br>
<br>
Will do!<br>
<br>
><br>
> > ><br>
> > > > + description: |<br>
> > > > + Control to set the mode of the AF (autofocus) algorithm. Applications<br>
> > > > + are allowed to set a new mode, and to send additional controls for<br>
> > > > + that new mode, in the same request.<br>
> > > > +<br>
> > > > + An implementation may choose not to implement all the modes.<br>
> > > > +<br>
> > > > + enum:<br>
> > > > + - name: AfModeManual<br>
> > > > + value: 0<br>
> > > > + description: |<br>
> > > > + The AF algorithm is in manual mode. In this mode it will never<br>
> > > > + perform any action nor move the lens of its own accord, but an<br>
> > > > + application can set controls to move the lens.<br>
> > > > +<br>
> > > > + In this mode the AfState will always report AfStateManual.<br>
> > > > + - name: AfModeAuto<br>
> > > > + value: 1<br>
> > > > + description: |<br>
> > > > + The AF algorithm is in auto mode. This means that the algorithm<br>
> > > > + will never move the lens or change state unless the AfTrigger<br>
> > > > + control is used. The AfTrigger control can be used to initiate a<br>
> > > > + focus scan, the results of which will also be reported by AfState.<br>
> > > > +<br>
> > > > + If the autofocus algorithm is moved from AfModeAuto to another<br>
> > > > + mode while a scan is in progress, the scan is cancelled<br>
> > > > + immediately, without waiting for the scan to finish.<br>
> > > > +<br>
> > > > + When first entering this mode the AfState will report<br>
> > > > + AfStateAuto. When a trigger control is sent, AfState will<br>
> > > > + report AfStateAutoScanning for a period before spontaneously<br>
> > > > + changing to AfStateAutoFocused or AfStateAutoFailed, depending on<br>
> > > > + the outcome of the scan. It will remain in this state until<br>
> > > > + another scan is initiated by the AfTrigger control. If a scan is<br>
> > > > + cancelled (without changing to another mode), AfState will return<br>
> > > > + to AfStateAuto.<br>
> > > > + - name: AfModeContinuous<br>
> > > > + value: 2<br>
> > > > + description: |<br>
> > > > + The AF algorithm is in continuous mode. This means that the lens<br>
> > > > + can re-start a scan spontaneously at any moment, without any user<br>
> > > > + intervention. The AfState still reports whether the algorithm is<br>
> > > > + currently scanning or not, though the application has no ability<br>
> > > > + to initiate or cancel scans (though it can "pause" them), nor to<br>
> > > > + move the lens for itself.<br>
> > > > +<br>
> > > > + When set to AfModeContinuous, the system will immediately initiate<br>
> > > > + a scan so AfState will report AfStateContinuousScanning, and will<br>
> > > > + settle on one of AfStateContinuousFocused or AfStateContinuousFailed,<br>
> > > > + depending on the scan result.<br>
> > > > +<br>
> > > > + - AfLensPosition:<br>
> > > > + type: int32_t<br>
> > > > + draft: true<br>
> > > > + description: |<br>
> > > > + Control to set the position of the lens to one of the following<br>
> > > > + predefined locations. This control only has any effect when the AfMode<br>
> > > > + is set to AfModeManual.<br>
> > > > +<br>
> > > > + This control is distinct from the LensPosition control, which sets the<br>
> > > > + lens position using the lens driver's units.<br>
> > > > + enum:<br>
> > > > + - name: AfLensPositionMacro<br>
> > > > + value: 0<br>
> > > > + description: The closest focal position of the lens.<br>
> > > > + - name: AfLensPositionHyperfocal<br>
> > > > + value: 1<br>
> > > > + description: Hyperfocal position.<br>
> > > > + - name: AfLensPositionInfinity<br>
> > > > + value: 2<br>
> > > > + description: The furthest focal position (usually infinity).<br>
> > ><br>
> > > Interesting, bikeshedding on the name apart, is this some 'simplified'<br>
> > > LensState ? Do you think there is a use case for this ? Why would<br>
> > > anyone set the AF to auto mode then use some coarse grained control<br>
> > > like this one ? Wouldn't it make more sense as an 'hint' to the AF<br>
> > > algorithm to restrict the search space ?<br>
> ><br>
> > In the light of your comments below about the LensPosition not using<br>
> > driver units, then maybe this could be deleted, and we could select<br>
> > some kind of canonical range for the lens?<br>
> ><br>
><br>
> Ah wait. What I was suggesting ("An "hint" to the AF algorithm to<br>
> restrict the search space") is basically AfRange.<br>
<br>
I wonder if there's a case for a "custom" value for AfRange, and we<br>
could then even have a control that sets the custom range?<br>
<br>
><br>
> > The most basic use case would be following on from the capture of an<br>
> > image after an AF scan:<br>
> ><br>
> > - capture image<br>
> > - AfMode to manual<br>
> > - Lens to hyperfocal<br>
> > - AfMode to auto<br>
> > - now wait for the shutter button to be pressed before you start<br>
> > another AF scan.<br>
><br>
> Naive question:<br>
> Wouldn't setting the lens to hyperfocal render the preview unusable ?<br>
<br>
Hyperfocal is the place where "most stuff" is normally in focus<br>
(strictly speaking, has the greatest depth of field), so in the<br>
absence of other information it's normally the best place to put the<br>
lens. It's just for the preview, you'd typically run another scan for<br>
a capture.<br>
<br>
(Some systems run Continuous AF during preview, of course. But you<br>
know my anxiety about CAF algorithms... in my opinion just putting the<br>
lens to hyperfocal is way less annoying than a not-very-good CAF<br>
implementation!)<br>
<br>
><br>
> ><br>
> > If your UI has a "macro" capture mode of some kind, maybe it's "Lens<br>
> > to macro" instead of "Lens to hyperfocal".<br>
><br>
> Why is this different than starting an AF scan (without going through<br>
> Manual) with an AfRange that suggests in which range to scan ? Is this<br>
> an optimization to forcefully move the lens closer to where it will<br>
> likely end being moved to ?<br>
<br>
If you know you're doing a macro capture it would be helpful for the<br>
preview to put the lens somewhere where macro things are likely to be<br>
more in focus, instead of using hyperfocal which will likely make<br>
close objects very blurry.<br>
<br>
><br>
> ><br>
> > But as noted, if LensPosition allows us to select hyperfocal/macro<br>
> > directly, then we can do without this one.<br>
> ><br>
><br>
> Providing my above understanding is correct, it would be a bit less intuitive<br>
> for applications to use a numerical value instead of some easier<br>
> "Macro"/"Hyperfocal". Of course, wrappers like your one can simplify this<br>
> for their consumers.<br>
<br>
True, though I'm now quite liking the "dioptre-like" canonical range<br>
described below, where 1 = hyperfocal, 0 = infinity, large numbers =<br>
close<br>
<br>
><br>
> > ><br>
> > > (Btw I have a use case for supplying a range of lens positions, which<br>
> > > has been previously estimated according to some pre-defined use cases,<br>
> > > to the AF algorithm in order restrict the AF search range to increase<br>
> > > the speed of a scan. Do you think it's a valid use case and it's worth a<br>
> > > control ?)<br>
> ><br>
> > I'd imagined doing this kind of thing with the AfSpeed and AfRange<br>
> > controls and by having the AF algorithm tuned for the module. I'm not<br>
> > convinced about passing in lens positions from outside the camera<br>
> > system. Where would they come from? Shouldn't the camera system know<br>
> > what positions to use?<br>
> ><br>
><br>
> A possible use case, for controlled environments with low latency<br>
> requirements, is to run tests beforehand and for each 'scene' define<br>
> what is the ideal lens movement range. When the application starts an<br>
> AF scan it would tell the AF algorithm about the 'preferred' range.<br>
><br>
> I understand this works backwards, as the lens position should be the<br>
> result of the AF algorithm run not one of its inputs :)<br>
<br>
Indeed, but as noted above, maybe there's an argument for an<br>
AfRangeCustom which could be user-settable? We've added "Custom"<br>
values for other controls in some places, IIRC.<br>
<br>
><br>
> > In the Raspberry Pi world we would of course store this kind of<br>
> > information in the tuning file. I've noticed some discussion elsewhere<br>
> > about supplying configuration to the IPAs - it seems like this might<br>
> > be a case in point? I think that for real camera systems, in the end,<br>
> > there is no escape from "configuration" (aka. "tuning").<br>
> ><br>
> > ><br>
> > > > +<br>
> > > > + - AfRange:<br>
> > > > + type: int32_t<br>
> > > > + draft: true<br>
> > > > + description: |<br>
> > > > + Control to set the range of focus distances that is scanned. An<br>
> > > > + implementation may choose not to implement all the options here.<br>
> > ><br>
> > > Only valid in Auto and Continuous mode, right ?<br>
> ><br>
> > Generally I've been expecting that you could change settings like this<br>
> > in Manual mode too, though it obviously can't have any effect until<br>
> > you go back to Auto/Continuous. Does that sound reasonable?<br>
><br>
> I would rather assume thaat if a control is not valid in the current<br>
> context is simply ignored and not cached to be re-applied when the<br>
> context change ?<br>
<br>
Possibly, though I'm slightly nervous about how you'd do things like:<br>
<br>
- You're in Manual mode, but want to go to Auto and set the Range to Normal.<br>
- You're in Auto mode but were using a different range. You want to<br>
reset it to Normal before switching to Manual.<br>
<br>
Can you send the AfRange change and the new mode at the same time in<br>
both cases or not? Or do you need two separate requests to send the<br>
changes? It's not clear to me exactly what would work. If you say that<br>
the AfRange always gets updated, then the behaviour is obvious, and<br>
you can send both AfRange and AfMode in the same request which would<br>
be easier to use.<br>
<br>
><br>
> ><br>
> > ><br>
> > > (This makes me think that we could easily model Continuous by adding a<br>
> > > TriggerContinuous when in Auto mode. With a TriggerPause we could get rid of<br>
> > > AfPause... well, it's quite a redesign so I'll stop here unless<br>
> > > someone thinks it's a really good idea :)<br>
> ><br>
> > Actually I'm quite strongly against anything like this. I think Auto<br>
> > and Continuous should be quite separate. The reason is that Auto is<br>
> > (relatively) easy to implement, you do a scan over some range and<br>
> > choose the best spot. But Continuous AF is an absolute nightmare to do<br>
> > well. You have all these problems:<br>
> ><br>
> > - The algo doesn't realise it needs to rescan and so seems to be<br>
> > "stuck". (Watch people waving their hands in front of the lens to try<br>
> > and unstick it.) So you add ever more tests and arbitrary thresholds<br>
> > on different statistics to tell if a rescan is needed, and then the<br>
> > whole thing becomes untunable.<br>
> > - The algo rescans too often. A car in the distance goes by and the<br>
> > whole image see-saws in and out while a rescan happens, ending back at<br>
> > the same place. Yet more tests, thresholds...<br>
> > - You can never make all your tests and thresholds work at the same<br>
> > time but in different conditions. So you make them variable depending<br>
> > on things like exposure and gain. Now you are truly doomed.<br>
> > - The algo is generally not robust to camera shake, which frequently<br>
> > causes it to end up "failing" with a completely out-of-focus image.<br>
> > And then you can't get it to rescan... cue more hand-waving in front<br>
> > of the lens. You implement more special tests and timeouts... At this<br>
> > point you will wish for the earth to swallow you up.<br>
> > - When the algo rescans it starts off going the wrong way causing<br>
> > extra see-sawing.<br>
> > - You try to solve the last problem by doing really small initial<br>
> > steps. Only then it ends up statistically unstable.<br>
> ><br>
> > From the above you may detect that we have been there and have several<br>
> > T-shirts. Speaking for myself, I have no plans to implement Continuous<br>
> > AF without PDAF sensors, as PDAF is quite a big help.<br>
> ><br>
><br>
> Yes, it feels like you know what you're talking about :)<br>
><br>
> I understand you would like be cautious with CAF, but if one doesn't<br>
> want to implement CAF can't she simply not report TriggerContinuous ?<br>
><br>
> Anyway, that was just an idea thrown in the mixer, don't worry about<br>
> it.<br>
><br>
> > ><br>
> > > > + enum:<br>
> > > > + - name: AfRangeNormal<br>
> > > > + value: 0<br>
> > > > + description: |<br>
> > > > + A wide range of focus distances is scanned, all the way from<br>
> > > > + infinity down to close distances, though depending on the<br>
> > > > + implementation, possibly not including the very closest macro<br>
> > > > + positions.<br>
> > > > + - name: AfRangeMacro<br>
> > > > + value: 1<br>
> > > > + description: Only close distances are scanned.<br>
> > > > + - name: AfRangeFull<br>
> > > > + value: 2<br>
> > > > + description: |<br>
> > > > + The full range of focus distances is scanned just as with<br>
> > > > + AfRangeNormal but this time including the very closest macro<br>
> > > > + positions.<br>
> > > > +<br>
> > > > + - AfSpeed:<br>
> > > > + type: int32_t<br>
> > > > + draft: true<br>
> > > > + description: |<br>
> > > > + Control that determines whether the AF algorithm is to move the lens<br>
> > > > + as quickly as possible or more steadily. For example, during video<br>
> > > > + recording it may be desirable not to move the lens too abruptly, but<br>
> > > > + when in a preview mode (waiting for a still capture) it may be<br>
> > > > + helpful to move the lens as quickly as is reasonably possible.<br>
> > > > + enum:<br>
> > > > + - name: AfSpeedNormal<br>
> > > > + value: 0<br>
> > > > + description: Move the lens at its usual speed.<br>
> > > > + - name: AfSpeedFast<br>
> > > > + value: 1<br>
> > > > + description: Move the lens more quickly.<br>
> > > > +<br>
> > > > + - AfWindows:<br>
> > > > + type: Rectangle<br>
> > > > + draft: true<br>
> > > > + description: |<br>
> > > > + Sets the focus windows used by the AF algorithm. The units used express<br>
> > > > + a proportion of the ScalerCrop control (or if unavailable, of the entire<br>
> > > > + image), as u0.16 format numbers.<br>
> > > > +<br>
> > > > + In order to be activated, a rectangle must be programmed with non-zero<br>
> > > > + width and height. If no rectangles are programmed in this way, then the<br>
> > > > + system will choose its own single default window in the centre of the<br>
> > > > + image.<br>
> > > > +<br>
> > > > + The details of how the windows are used are platform dependent. We note<br>
> > > > + that when there is more than one AF window, a typical implementation<br>
> > > > + might find the optimal focus position for each one and finally select<br>
> > > > + the window closest to the camera.<br>
> > > > +<br>
> > > > + size: [platform dependent]<br>
> > ><br>
> > > I think this should just be<br>
> > ><br>
> > > size: [n]<br>
> ><br>
> > OK!<br>
> ><br>
> > > > +<br>
> > > > + - AfTrigger:<br>
> > > > + type: int32_t<br>
> > > > + draft: true<br>
> > > > + description: |<br>
> > > > + This control starts an autofocus scan when AfMode is set to AfModeAuto,<br>
> > > > + and can also be used to terminate a scan early.<br>
> > > > +<br>
> > > > + It is ignored if AfMode is set to AfModeManual or AfModeContinuous.<br>
> > > > +<br>
> > > > + enum:<br>
> > > > + - name: AfTriggerStart<br>
> > > > + value: 0<br>
> > > > + description: Start an AF scan. Ignored if a scan is in progress.<br>
> > > > + - name: AfTriggerCancel<br>
> > > > + value: 1<br>
> > > > + description: Cancel an AF scan. This does not cause the lens to move<br>
> > > > + anywhere else. Ignored if no scan is in progress.<br>
> > > > +<br>
> > > > + - AfPause:<br>
> > > > + type: int32_t<br>
> > > > + draft: true<br>
> > > > + description: |<br>
> > > > + This control has no effect except when in continuous autofocus mode<br>
> > > > + (AfModeContinuous). It can be used to pause any lens movements while<br>
> > > > + (for example) images are captured. The algorithm remains inactive<br>
> > > > + until it is instructed to resume.<br>
> > > > +<br>
> > > > + enum:<br>
> > > > + - name: AfPauseImmediate<br>
> > > > + value: 0<br>
> > > > + description: |<br>
> > > > + Pause the continuous autofocus algorithm immediately, whether or<br>
> > > > + not any kind of scan is underway. If the AfState was previously<br>
> > > > + reporting AfStateContinuousScanning it will now change to<br>
> > > > + AfStateContinuousScanningPaused, and similarly for<br>
> > > > + AfStateContinuousFocused and AfStateContinuousFailed.<br>
> > > > + - name AfPauseDeferred<br>
> > > > + value: 1<br>
> > > > + description: |<br>
> > > > + Pause the continuous autofocus algorithm as soon as it is no longer<br>
> > > > + scanning. If the AfState is currently reporting<br>
> > > > + AfStateContinuousFocused is will change to<br>
> > > > + AfStateContinuousFocusedPaused, and similarly in the case of<br>
> > > > + AfStateContinuousFailed.<br>
> > > > +<br>
> > > > + If AfState reports AfStateContinuousScanning it will change to<br>
> > > > + AfStateContinuousScanningPausing, and then move to one of<br>
> > > > + AfStateContinuousFocusedPaused or AfStateContinuousFailedPaused<br>
> > > > + when the scan completes.<br>
> > > > + - name: AfPauseResume<br>
> > > > + value: 2<br>
> > > > + description: |<br>
> > > > + Resume continous autofocus operation. The algorithm starts again<br>
> > > > + from exactly where it left off, and the AfState will drop the<br>
> > > > + 'Paused' or 'Pausing' part of the state.<br>
> > > > +<br>
> > > > + - LensPosition:<br>
> > > > + type: int32_t<br>
> > > > + draft: true<br>
> > > > + description: |<br>
> > > > + Acts as a control to instruct the lens to move to a particular position<br>
> > > > + and also reports back the position of the lens for each frame.<br>
> > > > +<br>
> > > > + The units are determined by the lens driver.<br>
> > ><br>
> > > Ugh, we cannot do this in the control definition. Driver specificities<br>
> > > cannot surface to the controls definition, which should be platform<br>
> > > independent.<br>
> > ><br>
> > > I think we'll have to establish a numerical range (go ahead if you<br>
> > > have any idea) and translate to the actual driver values in the<br>
> > > CameraLens class ?<br>
> ><br>
> > I wasn't aware of this, but maybe it's a good thing. The question is<br>
><br>
> Well, isn't it the same with "Brightness", for example ?<br>
<br>
Indeed, or ExposureTime where we explicitly want microseconds, not driver units.<br>
<br>
><br>
> > what kind of range to use, and how do you know what limits apply for<br>
> > any given camera module? How about a dioptre-like scale, normalised<br>
> > for the hyperfocal distance H. It would go like this:<br>
> ><br>
> > The value n would mean being focused at distance H/n (n does not have<br>
> > to be an integer). So:<br>
> ><br>
> > The value 1 would mean hyperfocal position (focal distance H/1).<br>
> > The value 0 would mean infinity (focal distance H/0).<br>
> > Larger values would get progressively closer.<br>
> ><br>
> > Setting a control value outside the allowed range will clamp it to<br>
> > that end of the range.<br>
><br>
> I sincerely don't know enough about this to tell if it's a good idea<br>
> or not. Hence, it seems like a good idea to me :)<br>
><br>
> ><br>
> > In the Raspberry Pi world it would be sufficient for the tuning file<br>
> > to know the allowable range of values. More generally, I could imagine<br>
> > an application might want to say "focus at precisely 1m" in which case<br>
> > it would have to know the value of H from a property or something,<br>
> > though providing such information would be optional (some lenses<br>
> > simply aren't precisely calibrated). Perhaps an application might want<br>
> > to get the allowable range of (maybe approximate) focal distances too?<br>
> ><br>
> > ><br>
> > > > +<br>
> > > > + The LensPosition control is ignored unless the AfMode is set to<br>
> > > > + AfModeManual.<br>
> > > > +<br>
> > > > + Note that this control is distinct from AfLensPosition, which allows<br>
> > > > + the lens to be moved to its macro/hyperfocal/infinity position, rather<br>
> > > > + than using lens driver units.<br>
> > > > +<br>
> > > > + - AfState:<br>
> > > > + type: int32_t<br>
> > > > + draft: true<br>
> > > > + description: |<br>
> > > > + Reports the current state of the AF algorithm. The possible state changes<br>
> > > > + are described below, although we add that the following general state<br>
> > > > + changes are also allowed.<br>
> > ><br>
> > > wow, there is a lot of states here :)<br>
> ><br>
> > Did you follow the link to the state transition diagram? I would hope<br>
> > that should go some way to explaining it! Here's the link again for<br>
> > convenience:<br>
> ><br>
> > <a href="https://docs.google.com/drawings/d/1rU2nS82dEC8Y2eY0lKPkQQhd67gKLOl7PvSeUcL1j_E/edit?usp=sharing" rel="noreferrer" target="_blank">https://docs.google.com/drawings/d/1rU2nS82dEC8Y2eY0lKPkQQhd67gKLOl7PvSeUcL1j_E/edit?usp=sharing</a><br>
> ><br>
><br>
> Missed it. Thanks for providing it.<br>
><br>
> > ><br>
> > > Let me copy them here<br>
> > ><br>
> > > - name: AfStateManual<br>
> > > - name: AfStateAuto<br>
> > > - name: AfStateAutoScanning<br>
> > > - name: AfStateAutoFocused<br>
> > > - name: AfStateAutoFailed<br>
> > > - name: AfStateContinuousScanning<br>
> > > - name: AfStateContinuousScanningPausing<br>
> > > - name: AfStateContinuousScanningPaused<br>
> > > - name: AfStateContinuousFocused<br>
> > > - name: AfStateContinuousFocusedPaused<br>
> > > - name: AfStateContinuousFailed<br>
> > > - name: AfStateContinuousFailedPaused<br>
> ><br>
> > I "multiplied out" all the different parts of the state to make a<br>
> > single long list. Actually I think that makes it more Android-like.<br>
> > But we could reduce this number if, for example, we're happy to report<br>
> > the state of AfMode separately on every frame. I don't mind - do we<br>
> > have a general preference for this kind of thing?<br>
> ><br>
><br>
> Good question.<br>
><br>
> More generally we need to establish a policy for metadata. For<br>
> controls we know that controls should be attached to a request only<br>
> when their value changes. If a control is not supplied with new<br>
> values, then the old (or default) one will stay valid.<br>
><br>
> This implies the PH/IPA state machine is then stateful and 'remembers'<br>
> what was the last value a control was set to.<br>
><br>
> What about metadata instead ? Can we assume the same stateful-ness in<br>
> applications too, or is it a burden we cannot impose on them ?<br>
><br>
> Han-lin, do you know how does that work with Android ? Does it report<br>
> metadata even when their values do not change or do they assume that,<br>
> like controls, a metadata is reported only when its value has changed.<br>
><br></blockquote><div><br></div><div><div>I think Android metadata doesn't have the assumption, and is supposed to return with each frame.</div><div>In fact, with partial results supported, Android doesn't enforce a strict order of returning metadata among requests.</div></div><div>It's more like each metadata is independent of the previous one. <br></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
> What would the preference be for the RPi applications ?<br>
<br>
I don't think I have a very strong preference. I'm happy with either:<br>
<br>
- Report everything in the AfState on every frame.<br>
- Report the state in multiple metadata items (say, AfState and<br>
AfMode) on every frame.<br>
<br>
I'm less keen on having applications have to remember if they sent<br>
something and then wait for the request where it actually happened, it<br>
seems to me that would simply add annoying boilerplate code to all<br>
applications. What do you think? </blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
><br>
> > ><br>
> > > (please note that AfStateAuto seems mis-aligned, unless I messed up<br>
> > > something while cutting&paste. Please check the compiled doc to see<br>
> > > how it looks like...)<br>
> ><br>
> > I will check.<br>
> ><br>
> > ><br>
> > > So we have three macro groups<br>
> > ><br>
> > > - Manual<br>
> > ><br>
> > > - Auto<br>
> > > - Scanning<br>
> > > - Focus<br>
> > > - Failed<br>
> > ><br>
> > > - Continuous<br>
> > ><br>
> > > - Scanning<br>
> > > - Pausing<br>
> > > - Paused<br>
> > > - Focused<br>
> > > - Paused<br>
> > > - Failed<br>
> > > - Paused<br>
> > ><br>
> > > The first group is AfMode (Manual, Auto, Continuous) and can be<br>
> > > deduced by inspecting the AfMode metadata.<br>
> > ><br>
> > > The second group is the actual state (Scannng, Focused, Failed)<br>
> > ><br>
> > > The third group is specific to continuous mode as it accepts a Pause<br>
> > > trigger.<br>
> > ><br>
> > > I understand why you have this, as expressing "The AfState is set to<br>
> > > $something when in AfModeManual" etc etc might get cumbersome, but I<br>
> > > think it might be worth a try<br>
> > ><br>
> > > 1) Manual mode (the bad)<br>
> > > As we all know there is a delay between when a lens is instructed<br>
> > > to move and when it reaches its final state.<br>
> > ><br>
> > > In Manual mode the lens is moved with LensPosition and<br>
> > > AfLensPosition.<br>
> > ><br>
> > > To know the current state:<br>
> > > - With LensPosition metadata can be inspected until it doesn't reach the<br>
> > > desired value.<br>
> > > - With AfLensPosition it is not clear to me how to know when the lens<br>
> > > as reched one of the pre-defined positions.<br>
> ><br>
> > It sounds like we're deleting AfLensPosition.<br>
> ><br>
> > ><br>
> > > Is it so wrong to report either (Scanning,Focus,Failed) even for<br>
> > > Manual mode ?<br>
> ><br>
> > I don't think it's wrong. If we have a single long list of states then<br>
> > it just makes for more states. If we don't then it's harmless, though<br>
> > I'm not aware of a use for it.<br>
> ><br>
> > ><br>
> > > 2) AutoMode (the good)<br>
> > > This seems the easy one, as (Scanning,Focus,Failed) are already<br>
> > > there<br>
> > ><br>
> > > 3) Continous (the ugly)<br>
> > > Continuous mode can be paused. That's a shame :)<br>
> > > Would that be that bad keeping a Paused or Pausing state which can<br>
> > > only be returned for Continuous mode.<br>
> ><br>
> > The whole paused/pausing thing rather complicates things, but it does<br>
> > seem to me that it's necessary. If you were in Continuous mode, you<br>
> > would rather the lens didn't set off again just as you want to capture<br>
> > a sequence of pictures.<br>
> ><br>
> > ><br>
> > > All in all, can we reduce the number of states to<br>
> > ><br>
> > > - Scanning<br>
> > ><br>
> > > The AF algorithm is scanning and has not yet reached a<br>
> > > stable result. When running in Auto mode an AF scan is<br>
> > > started by using the AfTrigger control. When in Manual mode<br>
> > > and a new lens position is requested either by sending a<br>
> > > LensPosition or AfLensPosition control, the AF state will<br>
> > > report "Scanning" until the lens doesn't reach the desired<br>
> > > position. Similarly, when running in continuous mode and the<br>
> > > AF algorithm decides to start a scan spontaneously the<br>
> > > AfState will report Scanning until a new stable state is<br>
> > > reached.<br>
> > ><br>
> > > - Focus<br>
> > ><br>
> > > The AF algorithm has reached a stable state and the image is<br>
> > > now focused. When running in Manual mode the lens has<br>
> > > reached the position requested using either LensPosition and<br>
> > > AfLensPosition.<br>
> > ><br>
> > > - Failed<br>
> > ><br>
> > > The AF algorithm has completed a scan and without finding a<br>
> > > good focus position. When running in manual mode, the camera<br>
> > > has failed moving the lens to the desired position.<br>
> > ><br>
> > > - Paused (copying your below text, please see comments below)<br>
> > ><br>
> > > The AF algorithm is in continuous mode (AfModeContinuous) and<br>
> > > was scanning when AfPauseImmediate was sent. The scan will not<br>
> > > complete and the algorithm will remain in this state.<br>
> > ><br>
> > > - Pausing<br>
> > ><br>
> > > The AF algorithm is in continuous mode (AfModeContinuous) and<br>
> > > was scanning when AfPauseDeferred was sent. The scan will complete<br>
> > > at which point the algorithm moves to the<br>
> > > AfStateContinuousFocusedPaused or AfStateContinuousFailedPaused<br>
> > > state.<br>
> ><br>
> > I think we could get by if we report the following for every frame:<br>
> ><br>
> > AfMode - Manual, Auto or Continuous<br>
> > AfState - Scanning, Focused or Failed<br>
> > AfPause - Resume (which means it's running), Immediate or Deferred<br>
> > (these states only apply during Continuous; the Deferred value can<br>
> > only occur when also Scanning).<br>
> ><br>
> > Or we could wrap AfState and AfPause together:<br>
> ><br>
> > AfState - Scanning, Focused. Failed, ScanningPausing, ScanningPaused,<br>
> > FocusedPaused, FailedPaused.<br>
> ><br>
> > That makes some of the interaction between pausing and scanning<br>
> > clearer, but then it's more states again. I don't know...<br>
> ><br>
><br>
> Let's clarify what the policy on metadata is first.<br>
<br>
Agree!<br>
<br>
><br>
> > ><br>
> > ><br>
> > > > +<br>
> > > > + In any of the AfStateManual or AfStateContinuous states, the AfMode<br>
> > > > + may be set to AfModeAuto and the algorithm will move to the<br>
> > > > + AfStateAuto state. If AfTriggerStart is sent at the same time<br>
> > > > + then the algorithm will move to AfStateAutoScanning state.<br>
> > > > +<br>
> > > > + In any of the AfStateAuto or AfStateManual states, the AfMode may<br>
> > > > + be set to AfModeContinuous and the algorithm will move to<br>
> > > > + AfStateContinuousScanning.<br>
> > > > +<br>
> > > > + In any of the AfStateAuto or AfStateContinuous states, the AfMode may<br>
> > > > + be set to AfModeManual and the algorithm will move to<br>
> > > > + AfStateManual. The lens will not be moved and will be left where<br>
> > > > + it was at that moment.<br>
> > > > +<br>
> > > > + enum:<br>
> > > > + - name: AfStateManual<br>
> > > > + value: 0<br>
> > > > + description: |<br>
> > > > + The AF algorithm is in manual mode (AfModeManual). The LensPosition<br>
> > > > + and AfLensPosition controls can be used directly to move the lens.<br>
> > > > + - name: AfStateAuto<br>
> > > > + value: 1<br>
> > > > + description: |<br>
> > > > + The AF algorithm is in auto mode (AfModeAuto), and has either just<br>
> > > > + been moved into that state, or a scan that was in progress has been<br>
> > > > + cancelled.<br>
> > > > + - name: AfStateAutoScanning<br>
> > > > + value: 2<br>
> > > > + description: |<br>
> > > > + The AF algorithm is in auto mode (AfModeAuto), and a scan has been<br>
> > > > + started using the AfTrigger control. The scan can be cancelled by<br>
> > > > + sending AfTriggerCancel at which point the algorithm will either<br>
> > > > + move back to AfStateAuto or, if the scan actually completes<br>
> > > > + before the cancel request is processed, to one of<br>
> > > > + AfStateAutoFocused or AfStateAutoFailed.<br>
> > > > + - name: AfStateAutoFocused<br>
> > > > + value: 3<br>
> > > > + description: |<br>
> > > > + The AF algorithm is in auto mode (AfModeAuto), and a scan has<br>
> > > > + completed with the result that the algorithm believes the image is<br>
> > > > + now in focus.<br>
> > > > + - name: AfStateAutoFailed<br>
> > > > + value: 4<br>
> > > > + description: |<br>
> > > > + The AF algorithm is in auto mode (AfModeAuto), and a scan has<br>
> > > > + completed with the result that the algorithm did not find a good<br>
> > > > + focus position.<br>
> > > > + - name: AfStateContinuousScanning<br>
> > > > + value: 5<br>
> > > > + description: |<br>
> > > > + The AF algorithm is in continuous mode (AfModeContinuous) and<br>
> > > > + is currently scanning for a good focus position. This occurs when<br>
> > > > + the mode is first set to AfModeContinuous, or may happen<br>
> > > > + spontaneously when the algorithm believes a re-scan is needed.<br>
> > > > + - name: AfStateContinuousScanningPausing<br>
> > > > + value: 6<br>
> > > > + description: |<br>
> > > > + The AF algorithm is in continuous mode (AfModeContinuous) and<br>
> > > > + was scanning when AfPauseDeferred was sent. The scan will complete<br>
> > > > + at which point the algorithm moves to the<br>
> > > > + AfStateContinuousFocusedPaused or AfStateContinuousFailedPaused<br>
> > > > + state.<br>
> > > > + - name: AfStateContinuousScanningPaused<br>
> > > > + value: 7<br>
> > > > + description: |<br>
> > > > + The AF algorithm is in continuous mode (AfModeContinuous) and<br>
> > > > + was scanning when AfPauseImmediate was sent. The scan will not<br>
> > ><br>
> > > Why can't the algorithm be paused when it has reached a stable state<br>
> > > but only when scanning ? Won't a pause 'freeze' the algorithm in its<br>
> > > current state no matter what it is ? I understand with<br>
> > > "AfPauseDeferred" that it has to wait for AF to be locked before<br>
> > > stopping it, but with "Immediate" doesn't "pause" apply to all states ?<br>
> ><br>
> > Wasn't sure I understood this. It can be paused once it's reached<br>
> > Focused or Failed, and then both PauseImmediate and PauseDeferred will<br>
> > cause it to report FocusedPaused or FailedPaused. Was that the<br>
> > question?<br>
><br>
> You're right, I have probably missed FocusedPaused and FailedPaused.<br>
> :)<br>
><br>
> Thanks<br>
> j<br>
<br>
Thanks for all the discussion!<br>
<br>
David<br>
<br>
><br>
> ><br>
> > Anyway, thanks very much!<br>
> ><br>
> > David<br>
> ><br>
> > ><br>
> > > Thanks<br>
> > > j<br>
> > ><br>
> > > > + complete and the algorithm will remain in this state. The scan<br>
> > > > + may be resumed by sending AfPauseResume.<br>
> > > > + - name: AfStateContinuousFocused<br>
> > > > + value: 8<br>
> > > > + description: |<br>
> > > > + The AF algorithm is in continuous mode (AfModeContinuous) and<br>
> > > > + a scan has completed with the algorithm believing it has found a<br>
> > > > + good focus position.<br>
> > > > + - name: AfStateContinuousFocusedPaused<br>
> > > > + value: 9<br>
> > > > + description: |<br>
> > > > + The AF algorithm was in the AfStateContinuousFocused state and<br>
> > > > + has been paused (by either AfPauseImmediate or AfPauseDeferred),<br>
> > > > + or it was in the AfStateContinuousScanningPausing state and the<br>
> > > > + scan has completed successfully. The algorithm will now remain<br>
> > > > + in this state, and may be resumed by sending AfPauseResume.<br>
> > > > + - name: AfStateContinuousFailed<br>
> > > > + value: 10<br>
> > > > + description: |<br>
> > > > + The AF algorithm is in continuous mode (AfModeContinuous) and<br>
> > > > + a scan has completed without finding a good focus position.<br>
> > > > + - name: AfStateContinuousFailedPaused<br>
> > > > + value: 11<br>
> > > > + description: |<br>
> > > > + The AF algorithm was in the AfStateContinuousFailed state and<br>
> > > > + has been paused (by either AfPauseImmediate or AfPauseDeferred),<br>
> > > > + or it was in the AfStateContinuousScanningPausing state and the<br>
> > > > + scan has completed unsuccessfully. The algorithm will now remain<br>
> > > > + in this state, and may be resumed by sending AfPauseResume.<br>
> > > > +<br>
> > > > ...<br>
> > > > --<br>
> > > > 2.30.2<br>
> > > ><br>
</blockquote></div></div>