[libcamera-devel] reporting of gain and analogue gain

[Laurent Pinchart]

Hi David,

On Sat, Oct 10, 2020 at 09:57:16AM +0100, David Plowman wrote:
> On Sat, 10 Oct 2020 at 02:43, Laurent Pinchart wrote:
> > On Wed, Oct 07, 2020 at 05:15:15PM +0100, David Plowman wrote:
> > > On Wed, 7 Oct 2020 at 15:58, Kieran Bingham wrote:
> > > > On 07/10/2020 15:41, David Plowman wrote:
> > > > > Hi everyone
> > > > >
> > > > > Whilst doing some maintenance on the Raspberry Pi AGC I noticed a bit
> > > > > of an oversight - we have no way to report the total gain being
> > > > > applied to an image. We have controls to report:
> > > > >
> > > > > * the exposure time of the sensor
> > > > > * the analogue gain applied in the sensor
> > > > > * red and blue colour gains applied by the AWB
> > > > >
> > > > > but nothing for any (global) digital gain demanded by the AGC. The
> > > > > consequence is that when I capture a jpeg I always report an ISO value
> > > > > based purely on the analogue gain which may be incorrect (when a
> > > > > higher gain is required than the sensor allows then we make up the
> > > > > difference with digital gain).
> >
> > Just to make sure there's no misunderstanding, that's the digital gain
> > applied in the sensor, right ?
> 
> Sorry, I wasn't clear about that. No, I was referring to the gain
> applied by the ISP.

I'm glad I asked a stupid question then :-)

Should we consider the sensor digital gain too in this discussion, even
if not used for Raspberry Pi ?

> > > > > Any thoughts on this? Obviously there could be a digital gain control...
> > > >
> > > > I assume the IPA would be then responsible for setting this? Is it
> > > > read-only?
> > >
> > > Certainly in the Raspberry Pi implementation it would be a read-only
> > > control which the IPA sets for you. You could imagine some
> > > implementations would allow you to set it (the old proprietary stack
> > > of ours does, which I think is a source of confusion) but we're trying
> > > to keep the behaviour simple.
> > >
> > > > Would we expect an application to be able to control both the analogue
> > > > gain and the digital gain manually? (When AGC is disabled I guess?)
> > > >
> > > > Or would they only see an overall 'gain' control?
> > >
> > > The AGC algorithm we've implemented actually treats "gain" as a single
> > > thing. If the analogue gain goes high enough that's what you'll get.
> > > If you want to go higher, the analogue gain will max out and the
> > > remainder will be digital gain.
> > >
> > > > I'm suspecting that indeed, it would be exposed as a separate control,
> > > > because it will expose a feature that the ISP is capable of delivering.
> > >
> > > Yes, probably. I think maybe I feel another patch set coming on.
> > > Perhaps I'll get the other AGC ones out of the way first while we wait
> > > to see if anyone else has any opinions...
> >
> > I can see two options, either reporting the analog and digital gains as
> > separate controls, or as a combined gain control. The latter would allow
> > abstracting different sensor architectures more easily, at the expense
> > of not exposing the analog/digital split.
> >
> > Are there use cases on the application side that would benefit from
> > knowing the analog/digital gain split ? And are there use cases for
> > applying digital gain without maxing the analog gain out first ?
> 
> The discussion probably changes a bit since I was referring to the
> digital gain in the ISP. Dealing with that one first, I think you
> could expect that an application would want to distinguish between the
> two. For example, when recording the "ISO" for a raw file you'd use
> the analogue gain, but when recording it for a jpeg you'd have to take
> both into account.

Agreed. And if we take sensor digital gain into account too, that would
be recorded in the ISO value for both DNG and JPEG.

> I can imagine that some AGC implementations would let you set the
> digital gain in the ISP explicitly (indeed the existing Broadcom
> camera stack does!) but I think that's often been a source of
> confusion for us and our users and so, as part of my obsessive mission
> to make the IPAs simpler, I've combined the two together.

Another stupid question, when you say you've combined the two together,
that's the ISP digital gain and ?

> (There'd be nothing to stop us limiting the analogue gain if we wanted
> to or - here's an entertaining idea - splitting the two gains up in
> the exposure profile so you could ramp them up separately, but
> overall, I'm disinclined to add this unless there's a requirement.)
> 
> As you say, digital gain in the sensor is a thing too, though I can't
> say that we've ever used it much (maybe ever?) ourselves. It's
> certainly more reasonable to sweep that into the analogue gain value,
> I'd have thought, as it would apply to raw files too. Exposing that to
> application control is perfectly reasonable, I guess, though I think
> in Pi-world we would want to "opt out" of extra complexity like that!

Do we also need to take the ColourGains property into account ? What are
all the gain stages you have in your hardware pipeline ?

-- 
Regards,

Laurent Pinchart