[libcamera-devel] How to handle sensors with per-mode sensitivities

[David Plowman]

Hi everyone

I'd like to kick off a discussion about sensors with "per-mode
sensitivities". What I mean by this is that one readout mode (say, a
2x2 binned mode) might yield an image with twice the signal level of
another mode (the full resolution readout, for example). And indeed, I
have just such a sensor on my desk!

How do we handle this? Let's look at an example, and perhaps we can
make the discussion more fun by having a vote on the outcomes!

Use case: Camera running a preview in its 2x2 binned mode, with the
usual "auto" algorithms. We query the exposure and gain, and for the
sake of argument, let's say we obtain 20ms and 1.5x gain. Now we
switch to full resolution capture. We also query the exposure and gain
of the capture image. The full resolution mode - as in my case -
produces exactly half the signal level of the binned mode.

What do we expect to happen? Here are 3 alternatives:

A. The capture is also 20ms and 1.5x gain. But it is under-exposed by
a whole stop.

B. The capture is also 20ms and 1.5x gain. But it is not
under-exposed, it has the same brightness as the preview did.

C. The capture again is not under-exposed. But the exposure time and
gain have changed and are, between them, exactly double what the
preview reported. For example, we might have 30ms and 2x.

Cast your votes!

As another related thought experiment, you might consider what happens
if, after querying the preview exposure and gain, we then set these as
fixed values, disabling the "auto" algorithm. What do we expect now?

Solution 1

Option A seems pretty unhelpful to me, but currently, the only other
solution that I can implement is B. In the full resolution mode I
double all the gain values under the hood, so the true gain is
actually 3x, even though no one else knows this.

In practice this actually works quite well, but there are some
drawbacks. Firstly, we are lying (somewhat). Secondly, it slightly
subverts our camera tuning assumptions, though I think this is easily
surmountable.  Finally, it gives us no way to do a full resolution
capture with a true gain of 1x.

Solution 2

An alternative solution might be one where the AEC/AGC in particular
understands per-mode sensitivities. In this case, the best answer
might be C. When the mode-switch happens, the AEC/AGC spots the change
in sensitivity, works out the previous total exposure, doubles it, and
now reapportions the exposure and gain according to the exposure
profile.

My second thought experiment, where we fix the preview exposure and
gain, is still unclear. It would seem weird to change them if the
application specifically fixed them. Should the application have been
aware of the change in sensitivity and contrive to have doubled the values?
That's starting to feel quite burdensome.

I've wondered sometimes whether setting exposure and gain separately
is actually a rather bad idea. Wouldn't it be useful if applications
could set the *total exposure*, and then have the AEC/AGC choose
according to the current exposure profile? From there it's only a
small step to having a means of setting the total exposure but
allowing for changing mode sensitivities - perhaps we call this
"normalised total exposure"? That seems relatively elegant to me,
though it makes work for us (well, certainly for me!).

But what do folks think? And are there other solutions?

Thanks

David