[libcamera-devel] [PATCH v1 00/14] Raspberry Pi: Camera Module 3 support

[Laurent Pinchart]

Hi Naush,

On Mon, Jan 23, 2023 at 12:33:51PM +0000, Naushir Patuck wrote:
> On Mon, 23 Jan 2023 at 09:55, Laurent Pinchart wrote:
> > On Mon, Jan 23, 2023 at 09:11:31AM +0000, Naushir Patuck wrote:
> > > On Sun, 22 Jan 2023 at 19:12, Laurent Pinchart wrote:
> > > > On Thu, Jan 19, 2023 at 10:45:30AM +0000, Naushir Patuck via libcamera-devel wrote:
> > > > > Hi,
> > > > >
> > > > > This series adds support for the Raspberry Pi Camera Module 3 (IMX708).
> > > > > A new hybrid PDAF/CDAF autofocus algorithm has been implemented to drive the
> > > > > autofocus mechanism available on this module.  All libcamera focus related
> > > > > controls are now hanled by the IPA.
> > > >
> > > > It's nice to see support for this camera module, as well as PDAF :-) I
> > > > haven't tested the IMX708 personally, but Kieran told me how good he
> > > > thinks it is.
> > > >
> > > > We have a rule in libcamera that support for new hardware can be merged
> > > > only if the corresponding kernel drivers are on their way to upstream. I
> > > > haven't seen an imx708 driver posted to the linux-media mailing list.
> > > > Did I miss something ?
> > >
> > > We are in the process of getting the kernel driver ready to be posted on the
> > > linux-media list.  Hope to get it sent this week!
> > >
> > > We need to de-feature some bits of the driver - specifically variant detection
> > > and embedded/pdaf/HDR stats metadata streams.  We are not entirely sure if HDR
> > > mode support is also something that may/not be accepted as-is.
> >
> > Embedded data and PDAF depend on the V4L2 streams API. The good news is
> > that the API just got merged in the linux-media tree :-) The bad news is
> > that a piece is still missing in order to properly support camera
> > sensors. Tomi, Sakari and I are working on it, so dropping embedded data
> > and PDAF support for now is fine.
> >
> > Regarding HDR, I suppose you need to drop sensor-side histogram support
> > for the same reason, but I'm not sure about the rest of the HDR-related
> > features. In particular, if there are any HDR controls, they should be
> > included in the driver already.
> 
> Agreed, we will leave the HDR control in-place as is.   This does eventually
> want to be replaced by a proper V4L2 format type, we will propose something
> for that.

Thank you.

> Of course doing any sort of metering without the HDR metadata on the sensor is
> going to be very difficult!

Yes, of course :-) We'll get that fixed. There's momentum with the V4L2
streams API, so code will get merged in mainline to support the needs of
the IMX708 for embedded data, PDAF and HDR.

We actually want to use Unicam and a camera sensor that produces
multiple streams as one of the platforms to validate the fitness of the
next pieces of the API during development. I was initially considering
using the IMX477 for that, but the IMX708 could also be a good
candidate.

> > As for variant detection, someone has to make a proposal, so I think
> > this is a great time to bite the bullet :-) I'll do my best to help, but
> > I can't drive this development given that I'm already busy with the
> > streams API as mentioned above.
> >
> > > > > Camera Module 3 is available either with an IR or without an IR (NoIR) filter,
> > > > > and a normal or wide angle lens.  This makes a total of 4 possible variants.
> > > > > Detection of the variant is done through an OTP on the module, and signalled
> > > > > back to userland through changing the suffix on the device entity name. As such,
> > > > > we have 4 different tuning files and 4 duplicate instances of camera helpers and
> > > > > sensor properties entires.
> > > > >
> > > > > There is a single HDR sensor mode available in the sensor.  Because of current
> > > > > limitations, this needs to be switched on/off externally before running
> > > > > libcamera:
> > > > >
> > > > > v4l2-ctl --set-ctrl wide_dynamic_range=[0|1] -d /dev/v4l-subdev0
> > > > >
> > > > > in order for the CameraSensor class to cache the correct sensor mode. In our
> > > > > libcamera-apps, you can simply use the --hdr argument to switch this on.  This
> > > > > is only a temporary measure, and we need to work on a more permanent solution by
> > > > > adding HDR pixel formats into V4L2.
> > > > >
> > > > > Naushir Patuck (7):
> > > > >   pipeline: ipa: raspberrypi: Check if lens actuator is available
> > > > >   pipeline: ipa: raspberrypi: Remove unused streamConfig
> > > > >   pipeline: ipa: raspberrypi: Replace entityControls
> > > > >   pipeline: ipa: raspberrypi: Validate lens controls
> > > > >   ipa: raspberrypi: Reorder header file inclusion
> > > > >   ipa: raspberrypi: Include autofocus controls in the IPA ControlInfoMap
> > > > >   ipa: raspberrypi: Add lens position to DeviceStatus
> > > > >
> > > > > Nick Hollinghurst (7):
> > > > >   ipa: mojom: raspberrypi: Add setLensControls() function
> > > > >   ipa: raspberrypi: Add autofocus algorithm interface headers
> > > > >   ipa: raspberrypi: Handle autofocus controls
> > > > >   ipa: raspberrypi: Handle autofocus algorithm results
> > > > >   ipa: raspberrypi: First version of autofocus algorithm using PDAF
> > > > >   libcamera: camera_sensor: Add Sony IMX708 sensor properties
> > > > >   ipa: raspberrypi: Add support for the Sony IMX708 sensor
> > > > >
> > > > >  include/libcamera/ipa/raspberrypi.mojom       |  10 +-
> > > > >  src/ipa/raspberrypi/cam_helper_imx708.cpp     | 350 ++++++++
> > > > >  src/ipa/raspberrypi/controller/af_algorithm.h |  76 ++
> > > > >  src/ipa/raspberrypi/controller/af_status.h    |  35 +
> > > > >  .../raspberrypi/controller/device_status.h    |   4 +-
> > > > >  src/ipa/raspberrypi/controller/pdaf_data.h    |  21 +
> > > > >  src/ipa/raspberrypi/controller/rpi/af.cpp     | 755 ++++++++++++++++++
> > > > >  src/ipa/raspberrypi/controller/rpi/af.h       | 153 ++++
> > > > >  src/ipa/raspberrypi/data/imx708.json          | 559 +++++++++++++
> > > > >  src/ipa/raspberrypi/data/imx708_noir.json     | 559 +++++++++++++
> > > > >  src/ipa/raspberrypi/data/imx708_wide.json     | 462 +++++++++++
> > > > >  .../raspberrypi/data/imx708_wide_noir.json    | 462 +++++++++++
> > > > >  src/ipa/raspberrypi/data/meson.build          |   4 +
> > > > >  src/ipa/raspberrypi/meson.build               |   2 +
> > > > >  src/ipa/raspberrypi/raspberrypi.cpp           | 267 ++++++-
> > > > >  src/libcamera/camera_sensor_properties.cpp    |  16 +
> > > > >  .../pipeline/raspberrypi/raspberrypi.cpp      |  35 +-
> > > > >  17 files changed, 3729 insertions(+), 41 deletions(-)
> > > > >  create mode 100644 src/ipa/raspberrypi/cam_helper_imx708.cpp
> > > > >  create mode 100644 src/ipa/raspberrypi/controller/af_algorithm.h
> > > > >  create mode 100644 src/ipa/raspberrypi/controller/af_status.h
> > > > >  create mode 100644 src/ipa/raspberrypi/controller/pdaf_data.h
> > > > >  create mode 100644 src/ipa/raspberrypi/controller/rpi/af.cpp
> > > > >  create mode 100644 src/ipa/raspberrypi/controller/rpi/af.h
> > > > >  create mode 100644 src/ipa/raspberrypi/data/imx708.json
> > > > >  create mode 100644 src/ipa/raspberrypi/data/imx708_noir.json
> > > > >  create mode 100644 src/ipa/raspberrypi/data/imx708_wide.json
> > > > >  create mode 100644 src/ipa/raspberrypi/data/imx708_wide_noir.json

-- 
Regards,

Laurent Pinchart