[libcamera-devel] [PATCH v2 3/5] libcamera: camera_sensor: Rename controls() method

[Laurent Pinchart]

Hi Jacopo,

Thank you for the patch.

On Wed, Dec 30, 2020 at 10:47:08AM +0100, Jacopo Mondi wrote:
> On Tue, Dec 29, 2020 at 06:10:08PM +0100, Niklas Söderlund wrote:
> > On 2020-12-28 17:55:58 +0100, Jacopo Mondi wrote:
> > > The CameraSensor::controls() methods returns the information relative to
> > > the V4L2 controls registered by the sensor sub-device driver. Its
> > > current use is to inform the IPA module of two pipelines (RkISP1 and
> > > VIMC) about the V4L2 controls limits.
> > >
> > > The CameraSensor class has a controls_ field, which is instead the
> > > ControlInfoMap of libcamera controls registered by the CameraSensor
> > > class and meant to be exported as Camera controls.
> > >
> > > To prepare to register libcamera controls in the CameraSensor::controls_
> > > info map, and remove any ambiguity on the intended usage of
> > > CameraSensor::controls(), rename the method in
> > > CameraSensor::subdevControls() and update its users in the code base.
> > >
> > > Reviewed-by: Paul Elder <paul.elder at ideasonboard.com>
> > > Signed-off-by: Jacopo Mondi <jacopo at jmondi.org>
> >
> > The patch in itself is good but I'm not sure I understand the direction
> > we are taking with CameraSensor. I understand we should not expose any
> > V4L2 controls to applications but from the pipeline to core helpers (and
> > IPA) do we really wish to use libcamera controls? I can't see that we
> > will have core helpers or IPA that would work with both V4L2 and
> > something new. In my view such interfaces should be kept as close to the
> > drivers it interacts with, in this case V4L2.
> 
> This patch does not question that. The class has a map of libcamera
> control info, created by inspecting the v4l2 controls of the subdev or
> the sensor database. These should be libcamera cotrols, otherwise each
> pipeline handler should make the translation itself, which is not
> correct in my opinion. The field is called controls_ and it seems
> natural to retrieve it through a controls() method.

I disagree here. The camera sensor is internal, the controls it exposes
must not be copied verbatim by the pipeline handler to the controls
exposed by the camera class. I'm OK using libcamera::controls right now
to avoid growing this series, but we'll need to switch to
sensor-specific controls sooner than latter. The exposure time is a good
example, we will possibly need to exposure it as a coarse exposure time
expressed as a number of lines and a fine exposure time expressed as a
number of pixels, instead of a single exposure time, due to how
ill-defined the latter is, especially with the existing V4L2 controls
related to the exposure time (EXPOSURE, HBLANK, VBLANK, PIXEL_RATE and
of course the active analog crop rectangle - see my review of previous
patches in the series). I'm not sure what will be needed exactly, but
the CameraSensor class should certainly not expose anything directly to
the public API.

> If we need to provide the IPA the raw V4L2 controls limits (and I
> think we should keep doing so) they can be accessed with the here
> re-named subdevControls() method.

I don't think we should provide V4L2 controls to the IPA. We should
provide a single set of sensor-specific controls, that will provide a
good abstraction of how a camera sensor operates, without being tied to
a particular kernel API (V4L2 in this case). We can have both controls()
and subdevControls() in the interim, but that's really buying technical
debt, which is growing quite large and worries me. We need to at least
be on the same page regarding the direction we want to take. Could you
please add appropriate \todo comments ?

> It's just about naming, really.
> 
> > I might be missing some use-case where mixing libcamera and V4L2
> > controls in helpers and lowlevel interfaces are desired but I can't
> > really see it. At first I thought maybe we would have some controls that
> > simply could be passed from application to CameraSensor but for all
> > cases I can think of I can see the possibility for an IPA to be involved
> > and then I get confused as we mix and match between the two already too
> > much in my taste ;-)
> 
> It's more about the fact the CameraSensor has to register
> libcamera::ControlInfoMap created collecting information from the
> sensor db, the v4l2 kernel interface (and possibily other methods).
> 
> The ControlInfoMap should be made available to pipeline handlers, so
> that they can use its content and, in addition to the
> pipeline-specific controls (as it happens in the last patch of the
> series) register the libcamera::Camera ControlInfoMap.
> 
> Alternatively, each pipeline should make the translation between v4l2
> controls and libcamera controls, inspect the sensor db etc etc.

Pipeline handlers should translate between sensor-specific controls and
libcamera public controls.

> > Maybe the right solution is to move even further in the direction taken
> > here with the goal to not expose V4L2 controls outside the V4L2 classes.
> > In either case I would like to understand what we aim for.
> 
> I think the IPA-pipeline interface should keep using v4l2 controls, at
> least for now.

For now as in this patch series, yes. This should however be as short
term as possible, it's quite urgent to fix this.

> This patch only renames the method used to access them from the
> CameraSensor.
> 
> > For this rename change alone,
> >
> > Reviewed-by: Niklas Söderlund <niklas.soderlund at ragnatech.se>
> >
> > > ---
> > >  include/libcamera/internal/camera_sensor.h | 2 +-
> > >  src/libcamera/camera_sensor.cpp            | 6 +++---
> > >  src/libcamera/pipeline/rkisp1/rkisp1.cpp   | 2 +-
> > >  src/libcamera/pipeline/vimc/vimc.cpp       | 4 ++--
> > >  4 files changed, 7 insertions(+), 7 deletions(-)
> > >
> > > diff --git a/include/libcamera/internal/camera_sensor.h b/include/libcamera/internal/camera_sensor.h
> > > index 0357b2a630f7..841c7f4bef0f 100644
> > > --- a/include/libcamera/internal/camera_sensor.h
> > > +++ b/include/libcamera/internal/camera_sensor.h
> > > @@ -57,7 +57,7 @@ public:
> > >  				      const Size &size) const;
> > >  	int setFormat(V4L2SubdeviceFormat *format);
> > >
> > > -	const ControlInfoMap &controls() const;
> > > +	const ControlInfoMap &subdevControls() const;
> > >  	ControlList getControls(const std::vector<uint32_t> &ids);
> > >  	int setControls(ControlList *ctrls);
> > >
> > > diff --git a/src/libcamera/camera_sensor.cpp b/src/libcamera/camera_sensor.cpp
> > > index 18dc6d723e27..337d73c524bf 100644
> > > --- a/src/libcamera/camera_sensor.cpp
> > > +++ b/src/libcamera/camera_sensor.cpp
> > > @@ -617,10 +617,10 @@ int CameraSensor::setFormat(V4L2SubdeviceFormat *format)
> > >  }
> > >
> > >  /**
> > > - * \brief Retrieve the supported V4L2 controls and their information
> > > - * \return A map of the V4L2 controls supported by the sensor
> > > + * \brief Retrieve the controls supported by the V4L2 subdev and their information
> > > + * \return A map of the V4L2 controls supported by the video subdevice
> > >   */
> > > -const ControlInfoMap &CameraSensor::controls() const
> > > +const ControlInfoMap &CameraSensor::subdevControls() const
> > >  {
> > >  	return subdev_->controls();
> > >  }
> > > diff --git a/src/libcamera/pipeline/rkisp1/rkisp1.cpp b/src/libcamera/pipeline/rkisp1/rkisp1.cpp
> > > index 5ce37febc1d7..628d1f39bfbd 100644
> > > --- a/src/libcamera/pipeline/rkisp1/rkisp1.cpp
> > > +++ b/src/libcamera/pipeline/rkisp1/rkisp1.cpp
> > > @@ -906,7 +906,7 @@ int PipelineHandlerRkISP1::start(Camera *camera, [[maybe_unused]] ControlList *c
> > >  	}
> > >
> > >  	std::map<unsigned int, const ControlInfoMap &> entityControls;
> > > -	entityControls.emplace(0, data->sensor_->controls());
> > > +	entityControls.emplace(0, data->sensor_->subdevControls());
> > >
> > >  	IPAOperationData ipaConfig;
> > >  	data->ipa_->configure(sensorInfo, streamConfig, entityControls,
> > > diff --git a/src/libcamera/pipeline/vimc/vimc.cpp b/src/libcamera/pipeline/vimc/vimc.cpp
> > > index 36325ffbbd7d..d5f6e8a453ff 100644
> > > --- a/src/libcamera/pipeline/vimc/vimc.cpp
> > > +++ b/src/libcamera/pipeline/vimc/vimc.cpp
> > > @@ -338,7 +338,7 @@ void PipelineHandlerVimc::stop(Camera *camera)
> > >
> > >  int PipelineHandlerVimc::processControls(VimcCameraData *data, Request *request)
> > >  {
> > > -	ControlList controls(data->sensor_->controls());
> > > +	ControlList controls(data->sensor_->subdevControls());
> > >
> > >  	for (auto it : request->controls()) {
> > >  		unsigned int id = it.first;
> > > @@ -480,7 +480,7 @@ int VimcCameraData::init()
> > >  		return -ENODEV;
> > >
> > >  	/* Initialise the supported controls. */
> > > -	const ControlInfoMap &controls = sensor_->controls();
> > > +	const ControlInfoMap &controls = sensor_->subdevControls();
> > >  	ControlInfoMap::Map ctrls;
> > >
> > >  	for (const auto &ctrl : controls) {

-- 
Regards,

Laurent Pinchart