[libcamera-devel] How to handle sensors with per-mode sensitivities

Tue May 11 16:12:29 CEST 2021

Hi David,

On 11/05/2021 10:21, David Plowman wrote:
> Hi everyone
> 
> I'd like to kick off a discussion about sensors with "per-mode
> sensitivities". What I mean by this is that one readout mode (say, a
> 2x2 binned mode) might yield an image with twice the signal level of
> another mode (the full resolution readout, for example). And indeed, I
> have just such a sensor on my desk!

Does 'twice the signal level' mean that the pixel values are distinctly
different (i.e. green 32 becomes green 64 or such) or that the signal to
noise is much better, so you have a 'cleaner' image?

Or perhaps given some of the options to vote on - both of those are a
little related.

(edit: Learning about stops has probably explained this a bit more to me)

> How do we handle this? Let's look at an example, and perhaps we can
> make the discussion more fun by having a vote on the outcomes!

I hope there's doughnuts for the winners ;-)

  < Be right back, off to buy a doughnut :-D >

> Use case: Camera running a preview in its 2x2 binned mode, with the
> usual "auto" algorithms. We query the exposure and gain, and for the
> sake of argument, let's say we obtain 20ms and 1.5x gain. Now we
> switch to full resolution capture. We also query the exposure and gain
> of the capture image. The full resolution mode - as in my case -
> produces exactly half the signal level of the binned mode.
> 
> What do we expect to happen? Here are 3 alternatives:

I think I need to learn what the definition of a stop is here. I've
heard of f-stop, but I thought that was to do with lenses, so I assume
this is a similar measure on exposure?

For any other readers, I've just read

http://www.john-rowell.com/blog/2017/3/27/what-is-a-stop-of-light

"""
> A stop of light is not a fundamental unit, but a method of measuring 
> the increase or decrease of light in a photographic exposure. 1
> light stop is either double (+ 1-stop) or half (- 1-stop). So,
> irrespective of your initial exposure, increasing it by 1-stop means
> twice as much light, and decreasing it by 1-stop means half the
> amount of light. So far, so good right? Well, it gets a little more
> complex when you think of more than 1-stop. For example, if you
> increased the light by 2-stops, it is actually 4x the light (2x2),
> and 3-stops is 8x."""

So hopefully now I'm on the same wavelength .. I mean ... ;-)

And now I (hopefully) understand that in A. below, because the exposure
time has remained constant, as has the gain, but the same amount of
light which was previously binned down by 2x2, is now 'spread' across
2x2 more pixels, so the 'signal' is lower.

(The same amount of measured light is split across more pixels.. if I
interpret correctly.)

I can see how this can make a big difference.

> A. The capture is also 20ms and 1.5x gain. But it is under-exposed by
> a whole stop.

This sounds to me like what happens if nothing else changes, because the
same measured amount of light is now shared across more pixels?

> B. The capture is also 20ms and 1.5x gain. But it is not
> under-exposed, it has the same brightness as the preview did.

I'm not sure how or if this is possible? Is there another factor at play
which can adjust things to make sure the two captures are equivalent?

> C. The capture again is not under-exposed. But the exposure time and
> gain have changed and are, between them, exactly double what the
> preview reported. For example, we might have 30ms and 2x.

Ok so - I think what I would expect the user experience to be - is
probably C, but I don't know if it should be the exposure or gain should
be adjusted... (or both).

> Cast your votes!

Given the framing of your question:

"How do we handle this?"
 - I think my vote is C.

I believe the user expects the image to stay constant brightness
(I'm ready to be shouted down, or proven wrong here). Of course the IPA
will have made adjustments to either the exposure or gain as part of the
mode change, and those values will be reported through the metadata.
(This is assuming we're in auto modes

I think A is what would happen if we don't account for the binning mode
change though, which might be what would happen if there were manual
ISO/Exposure/Gain values set ... ?

> As another related thought experiment, you might consider what happens
> if, after querying the preview exposure and gain, we then set these as
> fixed values, disabling the "auto" algorithm. What do we expect now?

Aha - you're one step ahead of me ;-)

> Solution 1
> 
> Option A seems pretty unhelpful to me, but currently, the only other
> solution that I can implement is B. In the full resolution mode I
> double all the gain values under the hood, so the true gain is
> actually 3x, even though no one else knows this.

Oh - I see - so for Option B, what it means is that the IPA made changes
- but reported back that there were no changes.

That ... doesn't sound good to me ?

> In practice this actually works quite well, but there are some
> drawbacks. Firstly, we are lying (somewhat). Secondly, it slightly
> subverts our camera tuning assumptions, though I think this is easily
> surmountable.  Finally, it gives us no way to do a full resolution
> capture with a true gain of 1x.

Indeed, I think we somehow need to be honest from our IPA of what gains
and exposures are used in the metadata.

> Solution 2
> 
> An alternative solution might be one where the AEC/AGC in particular
> understands per-mode sensitivities. In this case, the best answer
> might be C. When the mode-switch happens, the AEC/AGC spots the change
> in sensitivity, works out the previous total exposure, doubles it, and
> now reapportions the exposure and gain according to the exposure
> profile.

Ok, that seems to be what I was considering with my vote on option C.

> My second thought experiment, where we fix the preview exposure and
> gain, is still unclear. It would seem weird to change them if the
> application specifically fixed them. Should the application have been
> aware of the change in sensitivity and contrive to have doubled the values?
> That's starting to feel quite burdensome.

Indeed, the application would somehow then have to be aware of the light
sensitivity change because of changing the binning mode or such.

If we assume a machine vision use case where we might expect the
application to fix the mode, exposure, and gain etc ... I wonder if it
would also need/want to know things like the sensitivity. Does that mean
therefore that we need to report any binning in the metadata? And if so
- is that enough? or is there other data we need to report?

> I've wondered sometimes whether setting exposure and gain separately
> is actually a rather bad idea. Wouldn't it be useful if applications
> could set the *total exposure*, and then have the AEC/AGC choose
> according to the current exposure profile? From there it's only a
> small step to having a means of setting the total exposure but
> allowing for changing mode sensitivities - perhaps we call this
> "normalised total exposure"? That seems relatively elegant to me,
> though it makes work for us (well, certainly for me!).

Is this the sort of area where a helper function could assist getting
the desired result, while still providing the low level controls to
applications if needed? Or I guess not specifically as the operation
requires running the IPA algorithms. But perhaps if the algorithms run
even in manual mode, and are able to report what they 'would' set, that
gives a way to get the expected results from the IPA while still letting
the helpers/app decide whether or not to apply them? I fear that's
sounding too complicated though :S eeep.

> But what do folks think? And are there other solutions?

I'm not sure of other solutions at the moment, but lets see what other
votes we get ;-)

--
Kieran

> 
> Thanks
> 
> David
> _______________________________________________
> libcamera-devel mailing list
> libcamera-devel at lists.libcamera.org
> https://lists.libcamera.org/listinfo/libcamera-devel
> 

-- 
Regards
--
Kieran