[libcamera-devel] reporting of gain and analogue gain

[David Plowman]

Hi Laurent

Thanks for the comments! More discussion below...

On Mon, 12 Oct 2020 at 00:44, Laurent Pinchart
<laurent.pinchart at ideasonboard.com> wrote:
>
> Hi David,
>
> On Sat, Oct 10, 2020 at 09:57:16AM +0100, David Plowman wrote:
> > On Sat, 10 Oct 2020 at 02:43, Laurent Pinchart wrote:
> > > On Wed, Oct 07, 2020 at 05:15:15PM +0100, David Plowman wrote:
> > > > On Wed, 7 Oct 2020 at 15:58, Kieran Bingham wrote:
> > > > > On 07/10/2020 15:41, David Plowman wrote:
> > > > > > Hi everyone
> > > > > >
> > > > > > Whilst doing some maintenance on the Raspberry Pi AGC I noticed a bit
> > > > > > of an oversight - we have no way to report the total gain being
> > > > > > applied to an image. We have controls to report:
> > > > > >
> > > > > > * the exposure time of the sensor
> > > > > > * the analogue gain applied in the sensor
> > > > > > * red and blue colour gains applied by the AWB
> > > > > >
> > > > > > but nothing for any (global) digital gain demanded by the AGC. The
> > > > > > consequence is that when I capture a jpeg I always report an ISO value
> > > > > > based purely on the analogue gain which may be incorrect (when a
> > > > > > higher gain is required than the sensor allows then we make up the
> > > > > > difference with digital gain).
> > >
> > > Just to make sure there's no misunderstanding, that's the digital gain
> > > applied in the sensor, right ?
> >
> > Sorry, I wasn't clear about that. No, I was referring to the gain
> > applied by the ISP.
>
> I'm glad I asked a stupid question then :-)
>
> Should we consider the sensor digital gain too in this discussion, even
> if not used for Raspberry Pi ?

Yes, we probably should, though we'd certainly like it to be easy to
continue to ignore it! I guess we could say something like:

* A pipeline handler may, or may not, expose a "sensor digital gain" control.
* A pipeline handler may, or may not, use the sensor's digital gain,
if it has one.
* If a pipeline handler uses the sensor's digital gain, it may, or may
not, report the "sensor digital gain" explicitly.
* If a pipeline handler uses, but does not report, the sensor's
digital gain then it "should" fold the value into the reported
analogue gain.

That's all a bit vague, though!

>
> > > > > > Any thoughts on this? Obviously there could be a digital gain control...
> > > > >
> > > > > I assume the IPA would be then responsible for setting this? Is it
> > > > > read-only?
> > > >
> > > > Certainly in the Raspberry Pi implementation it would be a read-only
> > > > control which the IPA sets for you. You could imagine some
> > > > implementations would allow you to set it (the old proprietary stack
> > > > of ours does, which I think is a source of confusion) but we're trying
> > > > to keep the behaviour simple.
> > > >
> > > > > Would we expect an application to be able to control both the analogue
> > > > > gain and the digital gain manually? (When AGC is disabled I guess?)
> > > > >
> > > > > Or would they only see an overall 'gain' control?
> > > >
> > > > The AGC algorithm we've implemented actually treats "gain" as a single
> > > > thing. If the analogue gain goes high enough that's what you'll get.
> > > > If you want to go higher, the analogue gain will max out and the
> > > > remainder will be digital gain.
> > > >
> > > > > I'm suspecting that indeed, it would be exposed as a separate control,
> > > > > because it will expose a feature that the ISP is capable of delivering.
> > > >
> > > > Yes, probably. I think maybe I feel another patch set coming on.
> > > > Perhaps I'll get the other AGC ones out of the way first while we wait
> > > > to see if anyone else has any opinions...
> > >
> > > I can see two options, either reporting the analog and digital gains as
> > > separate controls, or as a combined gain control. The latter would allow
> > > abstracting different sensor architectures more easily, at the expense
> > > of not exposing the analog/digital split.
> > >
> > > Are there use cases on the application side that would benefit from
> > > knowing the analog/digital gain split ? And are there use cases for
> > > applying digital gain without maxing the analog gain out first ?
> >
> > The discussion probably changes a bit since I was referring to the
> > digital gain in the ISP. Dealing with that one first, I think you
> > could expect that an application would want to distinguish between the
> > two. For example, when recording the "ISO" for a raw file you'd use
> > the analogue gain, but when recording it for a jpeg you'd have to take
> > both into account.
>
> Agreed. And if we take sensor digital gain into account too, that would
> be recorded in the ISO value for both DNG and JPEG.
>
> > I can imagine that some AGC implementations would let you set the
> > digital gain in the ISP explicitly (indeed the existing Broadcom
> > camera stack does!) but I think that's often been a source of
> > confusion for us and our users and so, as part of my obsessive mission
> > to make the IPAs simpler, I've combined the two together.
>
> Another stupid question, when you say you've combined the two together,
> that's the ISP digital gain and ?

The Raspberry PI behaviour is currently that you can ask for what is
effectively a total gain. If the requested gain can be achieved purely
with analogue gain, that's what you get. You only get digital gain
when the analogue gain has maxed out. (Some of our users have actually
expressed that this is the behaviour they expect.)

>
> > (There'd be nothing to stop us limiting the analogue gain if we wanted
> > to or - here's an entertaining idea - splitting the two gains up in
> > the exposure profile so you could ramp them up separately, but
> > overall, I'm disinclined to add this unless there's a requirement.)
> >
> > As you say, digital gain in the sensor is a thing too, though I can't
> > say that we've ever used it much (maybe ever?) ourselves. It's
> > certainly more reasonable to sweep that into the analogue gain value,
> > I'd have thought, as it would apply to raw files too. Exposing that to
> > application control is perfectly reasonable, I guess, though I think
> > in Pi-world we would want to "opt out" of extra complexity like that!
>
> Do we also need to take the ColourGains property into account ? What are
> all the gain stages you have in your hardware pipeline ?

So the pipeline in the Pi only has one stage where all these gains are
applied. There is a single gain value for each of red, green and blue.
The interface to these three numbers is split in two so that we can
define the colour gains (red and blue, which map onto
V4L2_CID_RED/BLUE_BALANCE) and a "global" gain separately.

The three gains applied by the hardware block are then exactly:
red_gain*global_gain, global_gain (for green), and
blue_gain*global_gain. These numbers are always chosen so that the
smallest actual gain ever applied is 1 (though they will all get
larger as more digital gain is demanded).

Personally I think the handling of the colour gains seems mostly OK to
me, though I would quite like V4L2_CID_GREEN_BALANCE if it were
possible, as this would simplify the AGC somewhat (I could spend less
time worrying about what happens when either the red or blue colour
gains go less than one).

I hope that helps a bit!?

Best regards
David

>
> --
> Regards,
>
> Laurent Pinchart