[libcamera-devel] [PATCH v5 2/2] pipeline: rkisp1: Fix sensor ISP format mismatch

[Sebastian Fricke]

This patch fixes a mismatch of image formats during the pipeline
creation of the RkISP1. The mismatch happens because the current code
does not check if the configured format exceeds the maximum viable
resolution of the ISP.

Make sure to use a sensor format resolution that is smaller or equal to
the maximum allowed resolution for the RkISP1. The maximum resolution
is defined within the `rkisp1-common.h` file as:
define RKISP1_ISP_MAX_WIDTH 4032
define RKISP1_ISP_MAX_HEIGHT 3024

Enumerate the frame-sizes of the ISP entity and compare the maximum with
the configured resolution.

This means that some camera sensors can never operate with their maximum
resolution, for example on my OV13850 camera sensor, there are two
possible resolutions: 4224x3136 & 2112x1568, the first of those two will
never be picked as it surpasses the maximum of the ISP.

Signed-off-by: Sebastian Fricke <sebastian.fricke at posteo.net>
---
 src/libcamera/pipeline/rkisp1/rkisp1.cpp | 35 +++++++++++++++++++++---
 1 file changed, 31 insertions(+), 4 deletions(-)

diff --git a/src/libcamera/pipeline/rkisp1/rkisp1.cpp b/src/libcamera/pipeline/rkisp1/rkisp1.cpp
index 50eaa6a4..56a406c1 100644
--- a/src/libcamera/pipeline/rkisp1/rkisp1.cpp
+++ b/src/libcamera/pipeline/rkisp1/rkisp1.cpp
@@ -474,10 +474,37 @@ CameraConfiguration::Status RkISP1CameraConfiguration::validate()
 		return Invalid;
 	}
 
-	/* Select the sensor format. */
-	Size maxSize;
+	/* Get the ISP resolution limits */
+	V4L2Subdevice::Formats ispFormats = data_->isp_->formats(0);
+	if (ispFormats.empty()) {
+		LOG(RkISP1, Error) << "Unable to fetch ISP formats.";
+		return Invalid;
+	}
+	/*
+	 * The maximum resolution is identical for all media bus codes on
+	 * the RkISP1 isp entity. Therefore take the first available resolution.
+	 */
+	Size ispMaximum = ispFormats.begin()->second[0].max;
+
+	/*
+	 * Select the sensor format, use either the best fit to the configured
+	 * format or a specific sensor format, when getFormat would choose a
+	 * resolution that surpasses the ISP maximum.
+	 */
+	Size maxSensorSize;
+	for (const Size &size : sensor->sizes()) {
+		if (size.width > ispMaximum.width ||
+		    size.height > ispMaximum.height)
+			continue;
+		maxSensorSize = std::max(maxSensorSize, size);
+	}
+	Size maxConfigSize;
 	for (const StreamConfiguration &cfg : config_)
-		maxSize = std::max(maxSize, cfg.size);
+		maxConfigSize = std::max(maxConfigSize, cfg.size);
+
+	if (maxConfigSize.height <= maxSensorSize.height &&
+	    maxConfigSize.width <= maxSensorSize.width)
+		maxSensorSize = maxConfigSize;
 
 	sensorFormat_ = sensor->getFormat({ MEDIA_BUS_FMT_SBGGR12_1X12,
 					    MEDIA_BUS_FMT_SGBRG12_1X12,
@@ -491,7 +518,7 @@ CameraConfiguration::Status RkISP1CameraConfiguration::validate()
 					    MEDIA_BUS_FMT_SGBRG8_1X8,
 					    MEDIA_BUS_FMT_SGRBG8_1X8,
 					    MEDIA_BUS_FMT_SRGGB8_1X8 },
-					  maxSize);
+					  maxSensorSize);
 	if (sensorFormat_.size.isNull())
 		sensorFormat_.size = sensor->resolution();
 
-- 
2.25.1