ISP on imx8mp with libcamera: how is the sensor involved?

[Laurent Pinchart]

Hi Martin,

On Thu, Apr 10, 2025 at 08:00:18AM +0000, Martin Kepplinger wrote:
> hi Laurent, Jacobo and all,
> 
> I'm planning to test libcamera on imx8mp (latest mainline v6.12-based
> kernel without any nxp-stuff, where ISI works fine) when changing the
> dts to cam->mipi->ISP and I have one high-level conceptual question:
> 
> I short: How is or can the sensor itself be involved during the running
> IPA?
> 
> Simplified: When the libcamera ISP algorithm would result in, say,
> "gain too low, let's increase", shouldn't it do that at the sensor
> itself? (Let's say it's a "stupid" sensor hardware module that can only
> set absolte values, without an ISP on it).

Correct, that's how auto-exposure needs to work, updating the sensor's
analog gain and exposure time based on the statistics extracted from the
scene.

> The above is at least not what I find when reading libcamera docs. It
> looks like the IPA is indeed restricted to ISP-settings and never
> changes sensor-controls...
> 
> When looking at what NXP intends to do with their kernels, they require
> to patch the sensor kernel drivers to support an ISP-API so that the
> sensor itself is involved... (of course require lots of other libraries
> and whatnot that I hope to be able to avoid using).
> 
> Do you know what I mean and can you point me in the right direction?

libcamera controls both the ISP and the sensor. Compared to
closed-source camera stacks that may handle this through communication
between the sensor driver and the ISP driver in the kernel, we control
the sensor directly from userspace.

In a nutshell, the IPA module calculates the sensor analog gain and
exposure time in the AGC algorithm. For i.MX8MP, the calculation happens
in the Agc::process() function, in src/ipa/rkisp1/algorithms/agc.cpp,
and the values are then stored in the frame context in the
Agc::prepare() function:

	/* Populate exposure and gain in auto mode */
	if (frameContext.agc.autoExposureEnabled)
		frameContext.agc.exposure = activeAutoExposure;
	if (frameContext.agc.autoGainEnabled)
		frameContext.agc.gain = activeAutoGain;

Then, the IPARkISP1::setControls() function in src/ipa/rkisp1/rkisp1.cpp
converts those values to V4L2 control values, and calls
setSensorControls.emit(). This communicate the sensor control values to
the pipeline handler through the IPA interface (defined for rkisp1 in
include/libcamera/ipa/rkisp1.mojom if you're curious).

The signal is received by the pipeline handler and processed by the
RkISP1CameraData::setSensorControls() function, which queues the values
to the "delayed control" helper (very briefly, the helper handles the
fact that gain and exposure time may get processed internally by the
sensor with different delays, for instance one of the two being delayed
by one frame, and the other one by two frames). At the next start of
frame, DelayedControls::applyControls() is called by the pipeline
handler, and the function then sets the V4L2 controls on the sensor.

-- 
Regards,

Laurent Pinchart