[libcamera-devel] [RFC PATCH 0/4] Digital zoom

[Jacopo Mondi]

Hi David,

On Mon, Sep 21, 2020 at 07:24:45AM +0100, David Plowman wrote:
> Hi again everyone
>
> I was wondering if anyone else had any thoughts on this. In spite of
> the long email which is perhaps a bit off-putting I don't think
> there's actually very much complicated stuff going on. Anyway, I'm
> keen to try and stop this discussion from getting marooned on the
> backlog for too long if we can!

Sorry sorry for the late reply. We also left quite some discussion
pending on the previous version, let's try to catch up here.

>
> Thanks again
> David
>
> On Mon, 7 Sep 2020 at 17:44, David Plowman
> <david.plowman at raspberrypi.com> wrote:
> >
> > Hi everyone
> >
> > I wanted to return to the question of digital zoom. Rather than simply
> > writing a long email I've included an implementation that shows where
> > I am. It's changed in the light of exchanges with Jacopo and others,
> > and I would say it's "for discussion" at this point. Principally:
> >
> > 1. I'd like users to have access to CameraSensorInfo::outputSize, and
> > I'm using Camera::properties() to get it. Unfortunately it changes
> > with every camera mode, so will we have to do this differently?
> >

I'll reply to this in 1/4. I think it's not optimal, but we should not
block this feature any longer. Although, I'm afraid that application
might be forced to change the way they retrieve the SensorCrop (from
Camera to CameraConfiguration). How fast do yuo expect this will be
adopted? Do you plan to build anything on top ?

> > 2. The terminology is a bit more Android-like. Because, why not?
> >

:)

> > 3. I'm no longer enforcing any particular method of cropping. But to
> > make life easier for applications I've added some geometry helpers. To
> > understand the rationale behind these I've included some example code
> > further down that I think an application might use.

Nice!

> >
> > This work is currently made up of 4 commits:
> >
> > 1. The SensorOutputSize property.
> >
> > 2. The SensorCrop control.
> >
> > 3. The geometry helper functions.
> >
> > 4. Implementation in the Raspberry Pi pipeline handler.
> >

Repluies in individual patches

> > Example Application Code
> >
> > Assume we have:
> >
> > camera - pointer to the Camera object
> > streamCfg - reference to the StreamConfiguration of the output image
> > request - pointer to a Request object
> > zoomFactor - floating point zoom factor, >= 1
> > dx - amount in pixels to pan left (-ve) or right (+ve)
> > dy - amount in pixels to pan up (-ve) or down (+ve)
> >
> > You'll want to understand the helper methods:
> >
> > Size Size::aspectRatioDown(const Size &ar) const
> > Returns a letter/pillarboxed version of this size.
> >
> > Size operator/(const Size &size, float f)
> > Downscale a Size object by f.
> >
> > Rectangle Size::centre(const Rectangle &region, int offsetX = 0, int offsetY = 0) const
> > Returns a Rectangle of this size centred at the same point as region,
> > optionally with offsets.
> >
> > Rectangle Rectangle::clamp(const Rectangle &boundary) const
> > Translates the rectangle so that no part of it lies outside boundary.
> >
> > 1. Basic zooming into the centre
> >
> > Size area = camera->properties().get(properties::SensorOutputSize);
> > Size size = area.aspectRatioDown(streamCfg.size) / zoomFactor;
> > Rectangle crop = size.centre(Rectangle(area));
> > request->controls.set(controls::SensorCrop, crop);
> >
> > (I'm hoping this first one is easy enough to understand!)
> >
> > 2. Zooming anywhere in the sensor output (even into parts of the image that
> > were not intially visible)
> >
> > Size area = camera->properties().get(properties::SensorOutputSize);
> > Size size = area.aspectRatioDown(streamCfg.size) / zoomFactor;
> > Rectangle crop = size.centre(Rectangle(area), dx, dy).clamp(Rectangle(area));
> > request->controls.set(controls::SensorCrop, crop);
> >
> > (As before, but we have offsets dx and dy to enable panning, so we
> > must also clamp the result to be sure it's within the sensor area.)
> >
> > 3. Zooming as before, but not allowing you to pan into anything that
> > wasn't initially visible
> >
> > Size area = camera->properties().get(properties::SensorOutputSize);
> > Rectangle maxCrop = area.aspectRatioDown(streamCfg.size).centre(area);
> > Size size = maxCrop.size() / zoomFactor;
> > Rectangle crop = size.centre(maxCrop, dx, dy).clamp(maxCrop);
> > request->controls.set(controls::SensorCrop, crop);
> >
> > (The subtle difference here is that we clamp the result against maxCrop,
> > not the full sensor area.)

Thanks! very informative!

> >
> > 4. Android
> >
> > This time we're given the "crop" (a Rectangle) that Android has requested. I
> > think Android only requires us to sort out the aspect ratio.
> >
> > Size size = crop.size().aspectRatioDown(streamCfg.size);
> > request->controls.set(controls::SensorCrop, size.centre(crop));

We'll here have to deal with the fact Android specifies its sensor
crop with the full size sensor pixel array as reference, not the actual
sensor output frame. This shall be handled in the HAL though.

> >
> >
> > That's as far as I've got. Thoughts and comments welcome!
> >
> > Best regards
> > David
> >
> > David Plowman (4):
> >   libcamera: Add SensorOutputSize property
> >   libcamera: Add SensorCrop control
> >   libcamera: Add geometry helper functions
> >   libcamera: pipeline: raspberrypi: Implementation of digital zoom
> >
> >  include/libcamera/geometry.h                  |  18 +++
> >  include/libcamera/ipa/raspberrypi.h           |   1 +
> >  src/ipa/raspberrypi/raspberrypi.cpp           |   7 +
> >  src/libcamera/camera_sensor.cpp               |   3 +
> >  src/libcamera/control_ids.yaml                |   9 ++
> >  src/libcamera/geometry.cpp                    | 129 ++++++++++++++++++
> >  .../pipeline/raspberrypi/raspberrypi.cpp      |  41 ++++++
> >  src/libcamera/property_ids.yaml               |   6 +
> >  8 files changed, 214 insertions(+)
> >
> > --
> > 2.20.1
> >
> _______________________________________________
> libcamera-devel mailing list
> libcamera-devel at lists.libcamera.org
> https://lists.libcamera.org/listinfo/libcamera-devel