[libcamera-devel] [PATCH v6 3/3] android: camera_device: Reorder configurations before requesting

[Jacopo Mondi]

Hi Hiro,

On Fri, Dec 11, 2020 at 06:03:15AM +0000, Hirokazu Honda wrote:
> This reorders Camera3Configs before executing
> CameraConfiguration::validate() to make it easier for the Camera
> to satisfy the Android framework request.
>
> Signed-off-by: Hirokazu Honda <hiroh at chromium.org>
> Reviewed-by: Jacopo Mondi <jacopo at jmondi.org>
> Reviewed-by: Kieran Bingham <kieran.bingham at ideasonboard.com>
> Reviewed-by: Umang Jain <email at uajain.com>
> ---
>  src/android/camera_device.cpp | 110 +++++++++++++++++++++++++++++++++-
>  1 file changed, 108 insertions(+), 2 deletions(-)
>
> diff --git a/src/android/camera_device.cpp b/src/android/camera_device.cpp
> index 5e245298..a2c005af 100644
> --- a/src/android/camera_device.cpp
> +++ b/src/android/camera_device.cpp
> @@ -27,6 +27,8 @@
>
>  using namespace libcamera;
>
> +LOG_DECLARE_CATEGORY(HAL)
> +
>  namespace {
>
>  /*
> @@ -144,9 +146,112 @@ struct Camera3StreamConfig {
>  	std::vector<Camera3Stream> streams;
>  	StreamConfiguration config;
>  };
> -} /* namespace */
>
> -LOG_DECLARE_CATEGORY(HAL)
> +/*
> + * Reorder the configurations so that libcamera::Camera can accept them as much
> + * as possible. The sort rule is as follows.
> + * 1.) The configuration for NV12 request whose resolution is the largest.
> + * 2.) The configuration for JPEG request.
> + * 3.) Others. Larger resolutions and different formats are put earlier.
> + */
> +void sortCamera3StreamConfigs(std::vector<Camera3StreamConfig> &unsortedConfigs,
> +			      const camera3_stream_t *jpegStream)
> +{
> +	const Camera3StreamConfig *jpegConfig = nullptr;
> +
> +	std::map<PixelFormat, std::vector<const Camera3StreamConfig *>> formatToConfigs;
> +	for (const auto &streamConfig : unsortedConfigs) {
> +		if (jpegStream && !jpegConfig) {
> +			const auto &streams = streamConfig.streams;
> +			if (std::find_if(streams.begin(), streams.end(),
> +					 [jpegStream] (const auto& stream) {

const auto &stream

> +						 return stream.stream == jpegStream;
> +					 }) != streams.end()) {
> +				jpegConfig = &streamConfig;
> +				continue;
> +			}
> +		}
> +
> +                formatToConfigs[streamConfig.config.pixelFormat].push_back(&streamConfig);

Weird indentation

> +	}
> +	if (jpegStream && !jpegConfig)
> +		LOG(HAL, Fatal) << "No Camera3StreamConfig is found for JPEG";
> +
> +	for (auto &[format, streamConfigs] : formatToConfigs) {
> +		/* Sorted by resolution. Smaller is put first. */
> +		std::sort(streamConfigs.begin(), streamConfigs.end(),
> +			  [](const auto *streamConfigA, const auto *streamConfigB) {
> +				const Size &sizeA = streamConfigA->config.size;
> +				const Size &sizeB = streamConfigB->config.size;
> +				return sizeA < sizeB;
> +			  });
> +	}
> +
> +	std::vector<Camera3StreamConfig> sortedConfigs;
> +	sortedConfigs.reserve(unsortedConfigs.size());
> +
> +	/*
> +	 * NV12 is the most prioritized format. Put the configuration with NV12
> +	 * and the largest resolution first.
> +	 */
> +	const auto nv12It = formatToConfigs.find(formats::NV12);
> +	if (nv12It != formatToConfigs.end()) {
> +		auto &nv12Configs = nv12It->second;
> +		const Size &nv12LargestSize = nv12Configs.back()->config.size;

I would here declare a variable for
                auto nv12Largest = nv12Configs.back();
> +
> +		/*
> +		 * If JPEG will be created from NV12 and the size is larger than
> +		 * the largest NV12 configurations, then put the NV12
> +		 * configuration for JPEG first.
> +		 */
> +		if (jpegConfig && jpegConfig->config.pixelFormat == formats::NV12) {
and move
		const Size &nv12LargestSize = nv12Largest->config.size;
here

> +			const Size &nv12SizeForJpeg = jpegConfig->config.size;
> +
> +			if (nv12LargestSize < nv12SizeForJpeg) {
> +				LOG(HAL, Debug) << "Insert " << jpegConfig->config.toString();
> +				sortedConfigs.push_back(std::move(*jpegConfig));
> +				jpegConfig = nullptr;
> +			}
> +		}

So that

> +
> +		LOG(HAL, Debug) << "Insert " << nv12Configs.back()->config.toString();
		LOG(HAL, Debug) << "Insert " << nv12Largest->config.toString();

> +		sortedConfigs.push_back(*nv12Configs.back());
		sortedConfigs.push_back(*nv12Largest);

That's minor stuff though, feel free to ignore.

> +		nv12Configs.pop_back();
> +
> +		if (nv12Configs.empty())
> +			formatToConfigs.erase(nv12It);
> +	}
> +
> +	/* If the configuration for JPEG is there, then put it. */
> +	if (jpegConfig) {
> +		LOG(HAL, Debug) << "Insert " << jpegConfig->config.toString();
> +		sortedConfigs.push_back(std::move(*jpegConfig));
> +		jpegConfig = nullptr;
> +	}
> +
> +	/*
> +	 * Put configurations with different formats and larger resolutions
> +	 * earlier.
> +	 */
> +	while (!formatToConfigs.empty()) {
> +		for (auto it = formatToConfigs.begin(); it != formatToConfigs.end();) {
> +			auto& configs = it->second;

auto &configs;

Could you please make sure to run checkstyle.py before submitting ?
The simplest way to make sure is to copy utils/hooks/post-commit in
.git/hooks

> +			LOG(HAL, Debug) << "Insert " << configs.back()->config.toString();
> +			sortedConfigs.push_back(*configs.back());
> +			configs.pop_back();
> +
> +			if (configs.empty())
> +				it = formatToConfigs.erase(it);

What about moving this after the 'configs' variable declaration ? To
make sure we don't try to access back() on an empty vector ? (which
should not happen, but...)

The resulting code would be also nicer to read

			auto &configs = it->second;
			if (configs.empty()) {
				it = formatToConfigs.erase(it);
                                continue;
                        }

			LOG(HAL, Debug) << "Insert " << configs.back()->config.toString();
			sortedConfigs.push_back(*configs.back());
			configs.pop_back();
                        it++;
Also a minor.

With checkstyle.py warnings addressed (there's also one warning that
seems a false positive to me) and the here proposed suggestions
optionally considered:
Reviewed-by: Jacopo Mondi <jacopo at jmondi.org>

Thanks
   j

> +			else
> +				it++;
> +		}
> +	}
> +
> +	ASSERT(sortedConfigs.size() == unsortedConfigs.size());
> +
> +	unsortedConfigs = sortedConfigs;
> +}
> +} /* namespace */
>
>  MappedCamera3Buffer::MappedCamera3Buffer(const buffer_handle_t camera3buffer,
>  					 int flags)
> @@ -1356,6 +1461,7 @@ int CameraDevice::configureStreams(camera3_stream_configuration_t *stream_list)
>  		streamConfigs[index].streams.push_back({ jpegStream, type });
>  	}
>
> +	sortCamera3StreamConfigs(streamConfigs, jpegStream);
>  	for (const auto &streamConfig : streamConfigs) {
>  		config_->addConfiguration(streamConfig.config);
>
> --
> 2.29.2.576.ga3fc446d84-goog