[libcamera-devel] [PATCH v1 00/14] Raspberry Pi: Camera Module 3 support

[Laurent Pinchart]

Hi Naush,

On Mon, Jan 23, 2023 at 09:11:31AM +0000, Naushir Patuck wrote:
> On Sun, 22 Jan 2023 at 19:12, Laurent Pinchart wrote:
> > On Thu, Jan 19, 2023 at 10:45:30AM +0000, Naushir Patuck via libcamera-devel wrote:
> > > Hi,
> > >
> > > This series adds support for the Raspberry Pi Camera Module 3 (IMX708).
> > > A new hybrid PDAF/CDAF autofocus algorithm has been implemented to drive the
> > > autofocus mechanism available on this module.  All libcamera focus related
> > > controls are now hanled by the IPA.
> >
> > It's nice to see support for this camera module, as well as PDAF :-) I
> > haven't tested the IMX708 personally, but Kieran told me how good he
> > thinks it is.
> >
> > We have a rule in libcamera that support for new hardware can be merged
> > only if the corresponding kernel drivers are on their way to upstream. I
> > haven't seen an imx708 driver posted to the linux-media mailing list.
> > Did I miss something ?
> >
> 
> We are in the process of getting the kernel driver ready to be posted on the
> linux-media list.  Hope to get it sent this week!
> 
> We need to de-feature some bits of the driver - specifically variant detection
> and embedded/pdaf/HDR stats metadata streams.  We are not entirely sure if HDR
> mode support is also something that may/not be accepted as-is.

Embedded data and PDAF depend on the V4L2 streams API. The good news is
that the API just got merged in the linux-media tree :-) The bad news is
that a piece is still missing in order to properly support camera
sensors. Tomi, Sakari and I are working on it, so dropping embedded data
and PDAF support for now is fine.

Regarding HDR, I suppose you need to drop sensor-side histogram support
for the same reason, but I'm not sure about the rest of the HDR-related
features. In particular, if there are any HDR controls, they should be
included in the driver already.

As for variant detection, someone has to make a proposal, so I think
this is a great time to bite the bullet :-) I'll do my best to help, but
I can't drive this development given that I'm already busy with the
streams API as mentioned above.

> > > Camera Module 3 is available either with an IR or without an IR (NoIR) filter,
> > > and a normal or wide angle lens.  This makes a total of 4 possible variants.
> > > Detection of the variant is done through an OTP on the module, and signalled
> > > back to userland through changing the suffix on the device entity name. As such,
> > > we have 4 different tuning files and 4 duplicate instances of camera helpers and
> > > sensor properties entires.
> > >
> > > There is a single HDR sensor mode available in the sensor.  Because of current
> > > limitations, this needs to be switched on/off externally before running
> > > libcamera:
> > >
> > > v4l2-ctl --set-ctrl wide_dynamic_range=[0|1] -d /dev/v4l-subdev0
> > >
> > > in order for the CameraSensor class to cache the correct sensor mode. In our
> > > libcamera-apps, you can simply use the --hdr argument to switch this on.  This
> > > is only a temporary measure, and we need to work on a more permanent solution by
> > > adding HDR pixel formats into V4L2.
> > >
> > > Naushir Patuck (7):
> > >   pipeline: ipa: raspberrypi: Check if lens actuator is available
> > >   pipeline: ipa: raspberrypi: Remove unused streamConfig
> > >   pipeline: ipa: raspberrypi: Replace entityControls
> > >   pipeline: ipa: raspberrypi: Validate lens controls
> > >   ipa: raspberrypi: Reorder header file inclusion
> > >   ipa: raspberrypi: Include autofocus controls in the IPA ControlInfoMap
> > >   ipa: raspberrypi: Add lens position to DeviceStatus
> > >
> > > Nick Hollinghurst (7):
> > >   ipa: mojom: raspberrypi: Add setLensControls() function
> > >   ipa: raspberrypi: Add autofocus algorithm interface headers
> > >   ipa: raspberrypi: Handle autofocus controls
> > >   ipa: raspberrypi: Handle autofocus algorithm results
> > >   ipa: raspberrypi: First version of autofocus algorithm using PDAF
> > >   libcamera: camera_sensor: Add Sony IMX708 sensor properties
> > >   ipa: raspberrypi: Add support for the Sony IMX708 sensor
> > >
> > >  include/libcamera/ipa/raspberrypi.mojom       |  10 +-
> > >  src/ipa/raspberrypi/cam_helper_imx708.cpp     | 350 ++++++++
> > >  src/ipa/raspberrypi/controller/af_algorithm.h |  76 ++
> > >  src/ipa/raspberrypi/controller/af_status.h    |  35 +
> > >  .../raspberrypi/controller/device_status.h    |   4 +-
> > >  src/ipa/raspberrypi/controller/pdaf_data.h    |  21 +
> > >  src/ipa/raspberrypi/controller/rpi/af.cpp     | 755 ++++++++++++++++++
> > >  src/ipa/raspberrypi/controller/rpi/af.h       | 153 ++++
> > >  src/ipa/raspberrypi/data/imx708.json          | 559 +++++++++++++
> > >  src/ipa/raspberrypi/data/imx708_noir.json     | 559 +++++++++++++
> > >  src/ipa/raspberrypi/data/imx708_wide.json     | 462 +++++++++++
> > >  .../raspberrypi/data/imx708_wide_noir.json    | 462 +++++++++++
> > >  src/ipa/raspberrypi/data/meson.build          |   4 +
> > >  src/ipa/raspberrypi/meson.build               |   2 +
> > >  src/ipa/raspberrypi/raspberrypi.cpp           | 267 ++++++-
> > >  src/libcamera/camera_sensor_properties.cpp    |  16 +
> > >  .../pipeline/raspberrypi/raspberrypi.cpp      |  35 +-
> > >  17 files changed, 3729 insertions(+), 41 deletions(-)
> > >  create mode 100644 src/ipa/raspberrypi/cam_helper_imx708.cpp
> > >  create mode 100644 src/ipa/raspberrypi/controller/af_algorithm.h
> > >  create mode 100644 src/ipa/raspberrypi/controller/af_status.h
> > >  create mode 100644 src/ipa/raspberrypi/controller/pdaf_data.h
> > >  create mode 100644 src/ipa/raspberrypi/controller/rpi/af.cpp
> > >  create mode 100644 src/ipa/raspberrypi/controller/rpi/af.h
> > >  create mode 100644 src/ipa/raspberrypi/data/imx708.json
> > >  create mode 100644 src/ipa/raspberrypi/data/imx708_noir.json
> > >  create mode 100644 src/ipa/raspberrypi/data/imx708_wide.json
> > >  create mode 100644 src/ipa/raspberrypi/data/imx708_wide_noir.json

-- 
Regards,

Laurent Pinchart