[libcamera-devel] [RFC PATCH 2/3] ipu3: cio2: Replicate CameraSensor::getFormats() to a member function

[Jacopo Mondi]

On Fri, Jul 30, 2021 at 01:58:27PM +0530, Umang Jain wrote:
> This prepares a base to introduce custom selection of sensor format
> based on platform(soraka or nautilus) constraints. The changes in
> selection policy will be introduced in a subsequent patch.
>
> No functional changes in this patch.
>
> Signed-off-by: Umang Jain <umang.jain at ideasonboard.com>
> ---
>  src/libcamera/pipeline/ipu3/cio2.cpp | 101 ++++++++++++++++++++++++++-
>  src/libcamera/pipeline/ipu3/cio2.h   |   5 ++
>  2 files changed, 104 insertions(+), 2 deletions(-)
>
> diff --git a/src/libcamera/pipeline/ipu3/cio2.cpp b/src/libcamera/pipeline/ipu3/cio2.cpp
> index 1bcd580e..aef88afd 100644
> --- a/src/libcamera/pipeline/ipu3/cio2.cpp
> +++ b/src/libcamera/pipeline/ipu3/cio2.cpp
> @@ -7,6 +7,8 @@
>
>  #include "cio2.h"
>
> +#include <float.h>
> +

#include <math.h>
#include <limits>

>  #include <linux/media-bus-format.h>
>
>  #include <libcamera/formats.h>
> @@ -120,6 +122,14 @@ int CIO2Device::init(const MediaDevice *media, unsigned int index)
>  	if (ret)
>  		return ret;
>
> +	formats_ = sensor_->formats();
> +	if (formats_.empty()) {
> +		LOG(IPU3, Error)
> +			<< "Sensor " << sensor_->entity()->name()
> +			<< "has empty sensor format list";
> +		return -EINVAL;
> +	}
> +

in CameraSensor

	/* Enumerate, sort and cache media bus codes and sizes. */
	formats_ = subdev_->formats(pad_);
	if (formats_.empty()) {
		LOG(CameraSensor, Error) << "No image format found";
		return -EINVAL;
	}

>  	/*
>  	 * Make sure the sensor produces at least one format compatible with
>  	 * the CIO2 requirements.
> @@ -168,7 +178,7 @@ int CIO2Device::configure(const Size &size, V4L2DeviceFormat *outputFormat)
>  	 * the CIO2 output device.
>  	 */
>  	std::vector<unsigned int> mbusCodes = utils::map_keys(mbusCodesToPixelFormat);
> -	sensorFormat = sensor_->getFormat(mbusCodes, size);
> +	sensorFormat = getSensorFormat(mbusCodes, size);
>  	ret = sensor_->setFormat(&sensorFormat);
>  	if (ret)
>  		return ret;
> @@ -206,7 +216,7 @@ StreamConfiguration CIO2Device::generateConfiguration(Size size) const
>
>  	/* Query the sensor static information for closest match. */
>  	std::vector<unsigned int> mbusCodes = utils::map_keys(mbusCodesToPixelFormat);
> -	V4L2SubdeviceFormat sensorFormat = sensor_->getFormat(mbusCodes, size);
> +	V4L2SubdeviceFormat sensorFormat = getSensorFormat(mbusCodes, size);
>  	if (!sensorFormat.mbus_code) {
>  		LOG(IPU3, Error) << "Sensor does not support mbus code";
>  		return {};
> @@ -219,6 +229,93 @@ StreamConfiguration CIO2Device::generateConfiguration(Size size) const
>  	return cfg;
>  }
>
> +/**
> + * \brief Retrieve the best sensor format for a desired output
> + * \param[in] mbusCodes The list of acceptable media bus codes
> + * \param[in] size The desired size
> + *
> + * Media bus codes are selected from \a mbusCodes, which lists all acceptable
> + * codes in decreasing order of preference. Media bus codes supported by the
> + * sensor but not listed in \a mbusCodes are ignored. If none of the desired
> + * codes is supported, it returns an error.
> + *
> + * \a size indicates the desired size at the output of the sensor. This method
> + * selects the best media bus code and size supported by the sensor according
> + * to the following criteria.
> + *
> + * - The desired \a size shall fit in the sensor output size to avoid the need
> + *   to up-scale.
> + * - The sensor output size shall match the desired aspect ratio to avoid the
> + *   need to crop the field of view.
> + * - The sensor output size shall be as small as possible to lower the required
> + *   bandwidth.
> + * - The desired \a size shall be supported by one of the media bus code listed
> + *   in \a mbusCodes.
> + *
> + * When multiple media bus codes can produce the same size, the code at the
> + * lowest position in \a mbusCodes is selected.
> + *
> + * The use of this method is optional, as the above criteria may not match the
> + * needs of all pipeline handlers. Pipeline handlers may implement custom
> + * sensor format selection when needed.
> + *
> + * The returned sensor output format is guaranteed to be acceptable by the
> + * setFormat() method without any modification.
> + *
> + * \return The best sensor output format matching the desired media bus codes
> + * and size on success, or an empty format otherwise.
> + */
> +V4L2SubdeviceFormat CIO2Device::getSensorFormat(const std::vector<unsigned int> &mbusCodes,
> +						const Size &size) const
> +{
> +	unsigned int desiredArea = size.width * size.height;
> +	unsigned int bestArea = UINT_MAX;
> +	float desiredRatio = static_cast<float>(size.width) / size.height;
> +	float bestRatio = FLT_MAX;
> +	const Size *bestSize = nullptr;
> +	uint32_t bestCode = 0;
> +
> +	for (unsigned int code : mbusCodes) {
> +		const auto formats = formats_.find(code);
> +		if (formats == formats_.end())
> +			continue;
> +
> +		for (const SizeRange &range : formats->second) {
> +			const Size &sz = range.max;
> +
> +			if (sz.width < size.width || sz.height < size.height)
> +				continue;
> +
> +			float ratio = static_cast<float>(sz.width) / sz.height;
> +			float ratioDiff = fabsf(ratio - desiredRatio);
> +			unsigned int area = sz.width * sz.height;
> +			unsigned int areaDiff = area - desiredArea;
> +
> +			if (ratioDiff > bestRatio)
> +				continue;
> +
> +			if (ratioDiff < bestRatio || areaDiff < bestArea) {
> +				bestRatio = ratioDiff;
> +				bestArea = areaDiff;
> +				bestSize = &sz;
> +				bestCode = code;
> +			}
> +		}
> +	}
> +
> +	if (!bestSize) {
> +		LOG(IPU3, Debug) << "No supported format or size found";
> +		return {};
> +	}
> +
> +	V4L2SubdeviceFormat format{
> +		.mbus_code = bestCode,
> +		.size = *bestSize,
> +	};
> +
> +	return format;
> +}
> +

I think a verbatim copy of the function implemented in CameraSensor is
a good idea to highlight changes on top that happen in the subsequent
patches

I'm always a bit hesitant about using 'get' in functions that perform
computations, as a getter to me is just a field accessor. But that might
just be me.

Reviewed-by: Jacopo Mondi <jacopo at jmondi.org>

Thanks
   j


>  int CIO2Device::exportBuffers(unsigned int count,
>  			      std::vector<std::unique_ptr<FrameBuffer>> *buffers)
>  {
> diff --git a/src/libcamera/pipeline/ipu3/cio2.h b/src/libcamera/pipeline/ipu3/cio2.h
> index f28e9f1d..f3f12706 100644
> --- a/src/libcamera/pipeline/ipu3/cio2.h
> +++ b/src/libcamera/pipeline/ipu3/cio2.h
> @@ -45,6 +45,9 @@ public:
>  	int exportBuffers(unsigned int count,
>  			  std::vector<std::unique_ptr<FrameBuffer>> *buffers);
>
> +	V4L2SubdeviceFormat getSensorFormat(const std::vector<unsigned int> &mbusCodes,
> +					    const Size &size) const;
> +
>  	int start();
>  	int stop();
>
> @@ -67,6 +70,8 @@ private:
>  	std::unique_ptr<V4L2Subdevice> csi2_;
>  	std::unique_ptr<V4L2VideoDevice> output_;
>
> +	V4L2Subdevice::Formats formats_;
> +
>  	std::vector<std::unique_ptr<FrameBuffer>> buffers_;
>  	std::queue<FrameBuffer *> availableBuffers_;
>  };
> --
> 2.31.0
>