ISP on imx8mp with libcamera: how is the sensor involved?

[Martin Kepplinger]

Am Donnerstag, dem 10.04.2025 um 20:08 +0300 schrieb Laurent Pinchart:
> Hi Martin,
> 
> On Thu, Apr 10, 2025 at 08:00:18AM +0000, Martin Kepplinger wrote:
> > hi Laurent, Jacobo and all,
> > 
> > I'm planning to test libcamera on imx8mp (latest mainline v6.12-
> > based
> > kernel without any nxp-stuff, where ISI works fine) when changing
> > the
> > dts to cam->mipi->ISP and I have one high-level conceptual
> > question:
> > 
> > I short: How is or can the sensor itself be involved during the
> > running
> > IPA?
> > 
> > Simplified: When the libcamera ISP algorithm would result in, say,
> > "gain too low, let's increase", shouldn't it do that at the sensor
> > itself? (Let's say it's a "stupid" sensor hardware module that can
> > only
> > set absolte values, without an ISP on it).
> 
> Correct, that's how auto-exposure needs to work, updating the
> sensor's
> analog gain and exposure time based on the statistics extracted from
> the
> scene.
> 
> > The above is at least not what I find when reading libcamera docs.
> > It
> > looks like the IPA is indeed restricted to ISP-settings and never
> > changes sensor-controls...
> > 
> > When looking at what NXP intends to do with their kernels, they
> > require
> > to patch the sensor kernel drivers to support an ISP-API so that
> > the
> > sensor itself is involved... (of course require lots of other
> > libraries
> > and whatnot that I hope to be able to avoid using).
> > 
> > Do you know what I mean and can you point me in the right
> > direction?
> 
> libcamera controls both the ISP and the sensor. Compared to
> closed-source camera stacks that may handle this through
> communication
> between the sensor driver and the ISP driver in the kernel, we
> control
> the sensor directly from userspace.
> 
> In a nutshell, the IPA module calculates the sensor analog gain and
> exposure time in the AGC algorithm. For i.MX8MP, the calculation
> happens
> in the Agc::process() function, in src/ipa/rkisp1/algorithms/agc.cpp,
> and the values are then stored in the frame context in the
> Agc::prepare() function:
> 
>         /* Populate exposure and gain in auto mode */
>         if (frameContext.agc.autoExposureEnabled)
>                 frameContext.agc.exposure = activeAutoExposure;
>         if (frameContext.agc.autoGainEnabled)
>                 frameContext.agc.gain = activeAutoGain;
> 
> Then, the IPARkISP1::setControls() function in
> src/ipa/rkisp1/rkisp1.cpp
> converts those values to V4L2 control values, and calls
> setSensorControls.emit(). This communicate the sensor control values
> to
> the pipeline handler through the IPA interface (defined for rkisp1 in
> include/libcamera/ipa/rkisp1.mojom if you're curious).
> 
> The signal is received by the pipeline handler and processed by the
> RkISP1CameraData::setSensorControls() function, which queues the
> values
> to the "delayed control" helper (very briefly, the helper handles the
> fact that gain and exposure time may get processed internally by the
> sensor with different delays, for instance one of the two being
> delayed
> by one frame, and the other one by two frames). At the next start of
> frame, DelayedControls::applyControls() is called by the pipeline
> handler, and the function then sets the V4L2 controls on the sensor.
> 

That sounds awesome and I should've simply read some code, even if it's
C++ :) I'm looking forward to testing. I hope that this clarification
can be valuable for others as well.

thank you,

                          martin