[libcamera-devel] [RFC PATCH 0/4] Digital zoom

[David Plowman]

Hi everyone

I wanted to return to the question of digital zoom. Rather than simply
writing a long email I've included an implementation that shows where
I am. It's changed in the light of exchanges with Jacopo and others,
and I would say it's "for discussion" at this point. Principally:

1. I'd like users to have access to CameraSensorInfo::outputSize, and
I'm using Camera::properties() to get it. Unfortunately it changes
with every camera mode, so will we have to do this differently?

2. The terminology is a bit more Android-like. Because, why not?

3. I'm no longer enforcing any particular method of cropping. But to
make life easier for applications I've added some geometry helpers. To
understand the rationale behind these I've included some example code
further down that I think an application might use.

This work is currently made up of 4 commits:

1. The SensorOutputSize property.

2. The SensorCrop control.

3. The geometry helper functions.

4. Implementation in the Raspberry Pi pipeline handler.

Example Application Code

Assume we have:

camera - pointer to the Camera object
streamCfg - reference to the StreamConfiguration of the output image
request - pointer to a Request object
zoomFactor - floating point zoom factor, >= 1
dx - amount in pixels to pan left (-ve) or right (+ve)
dy - amount in pixels to pan up (-ve) or down (+ve)

You'll want to understand the helper methods:

Size Size::aspectRatioDown(const Size &ar) const
Returns a letter/pillarboxed version of this size.

Size operator/(const Size &size, float f)
Downscale a Size object by f.

Rectangle Size::centre(const Rectangle &region, int offsetX = 0, int offsetY = 0) const
Returns a Rectangle of this size centred at the same point as region,
optionally with offsets.

Rectangle Rectangle::clamp(const Rectangle &boundary) const
Translates the rectangle so that no part of it lies outside boundary.

1. Basic zooming into the centre

Size area = camera->properties().get(properties::SensorOutputSize);
Size size = area.aspectRatioDown(streamCfg.size) / zoomFactor;
Rectangle crop = size.centre(Rectangle(area));
request->controls.set(controls::SensorCrop, crop);

(I'm hoping this first one is easy enough to understand!)

2. Zooming anywhere in the sensor output (even into parts of the image that
were not intially visible)

Size area = camera->properties().get(properties::SensorOutputSize);
Size size = area.aspectRatioDown(streamCfg.size) / zoomFactor;
Rectangle crop = size.centre(Rectangle(area), dx, dy).clamp(Rectangle(area));
request->controls.set(controls::SensorCrop, crop);

(As before, but we have offsets dx and dy to enable panning, so we 
must also clamp the result to be sure it's within the sensor area.)

3. Zooming as before, but not allowing you to pan into anything that
wasn't initially visible

Size area = camera->properties().get(properties::SensorOutputSize);
Rectangle maxCrop = area.aspectRatioDown(streamCfg.size).centre(area);
Size size = maxCrop.size() / zoomFactor;
Rectangle crop = size.centre(maxCrop, dx, dy).clamp(maxCrop);
request->controls.set(controls::SensorCrop, crop);

(The subtle difference here is that we clamp the result against maxCrop,
not the full sensor area.)

4. Android

This time we're given the "crop" (a Rectangle) that Android has requested. I
think Android only requires us to sort out the aspect ratio.

Size size = crop.size().aspectRatioDown(streamCfg.size);
request->controls.set(controls::SensorCrop, size.centre(crop));


That's as far as I've got. Thoughts and comments welcome!

Best regards
David

David Plowman (4):
  libcamera: Add SensorOutputSize property
  libcamera: Add SensorCrop control
  libcamera: Add geometry helper functions
  libcamera: pipeline: raspberrypi: Implementation of digital zoom

 include/libcamera/geometry.h                  |  18 +++
 include/libcamera/ipa/raspberrypi.h           |   1 +
 src/ipa/raspberrypi/raspberrypi.cpp           |   7 +
 src/libcamera/camera_sensor.cpp               |   3 +
 src/libcamera/control_ids.yaml                |   9 ++
 src/libcamera/geometry.cpp                    | 129 ++++++++++++++++++
 .../pipeline/raspberrypi/raspberrypi.cpp      |  41 ++++++
 src/libcamera/property_ids.yaml               |   6 +
 8 files changed, 214 insertions(+)

-- 
2.20.1