[libcamera-devel] [PATCH 00/10] Concurrent camera support in simple pipeline handler

[Laurent Pinchart]

Hello,

Another day, another patch series that has been sitting in my tree and
that I'd like to stop rebasing.

The series reworks the simple pipeline handler to correctly support
streaming from multiple cameras concurrently. The reason I haven't
posted it yet is that the libcamera core gets doesn't support this
feature yet. It's easy to work around by removing some checks for
testing, but will take more time to fix correctly.

The simple pipeline handler already supports registering multiple
cameras, and relies on the libcamera core blocking concurrent access to
different cameras backed by the same media device. It thus doesn't
implement internally mutual exclusion between cameras that share
hardware resources, but, based on the assumption that a single camera
will be used at a time, implements camera operation around a global
active camera pointer.

Patch 01/10 is the only libcamera core patch. It extends the MediaEntity
class to expose the type of userspace interface that the entity
implements. Patches 02/10 to 06/10 then rework the way the pipeline
handler stores data about pipelines and accesses video devices and
subdevs.

Patch 02/10 extends the SimpleCameraData entity representation to store
more information about graph traversal, and patch 04/10 stores the
entity corresponding to the video node with all other pipeline entities
in SimpleCameraData. Patches 03/10, 05/10 and 06/10 generalize handling
of video devices and subdev by opening them all together at match time
and storing them in a single container in the SimplePipelineHandler
class.

With this done, path 07/10 adds an entity reservation mechanism in the
SimplePipelineHandler class, to handle mutual exclusion when multiple
cameras use the same entities. With this change, the simple pipeline
handler handles concurrent access to cameras internally, without
depending on mutual exclusion implemented by the libcamera core anymore.
This doesn't result in any change of behaviour, but paves the way to
concurrent usage of multiple cameras that don't share entities once the
restriction in the libcamera core will be lifted.

Patch 08/10 continues in that direction by moving the converter from the
SimplePipelineHandler class to the SimpleCameraData class. This allows
usage of the converter by multiple cameras concurrently. Patch 09/10
then moves the bufferReady handler to the SimpleCameraData class as
well, to handle buffer completion in the correct context without relying
on the global active camera pointer.

Finally, patch 10/10 removes the active camera pointer as it's not used
anymore.

This series has only been partially tested, as the libcamera core
doesn't correctly support concurrent streaming from cameras belonging to
the same pipeline handler. The code was tested with a single camera and
didn't introduce any regression, and was also tested on i.MX8MP with a
few hacks in the libcamera core to lift the restrictions and a few
out-of-tree patches to support the i.MX8MP (those are not ready for
submission yet). I would thus understand a reluctance to get this merged
already, even if I believe that the absence of regression, combined with
the cleaner (in my opinion) implementation of the simple pipeline
handler are good enough reasons by themselves to already merge this.

Laurent Pinchart (10):
  libcamera: media_object: Expose entity type
  libcamera: pipeline: simple: Add sink and source pads to entity data
  libcamera: pipeline: simple: Delay opening of video device until
    init()
  libcamera: pipeline: simple: Store video node entity in camera data
  libcamera: pipeline: simple: Store all entity devices in common map
  libcamera: pipeline: simple: Open all video devices at match() time
  libcamera: pipeline: simple: Add pipeline pad reservation mechanism
  libcamera: pipeline: simple: Move converter to SimpleCameraData
  libcamera: pipeline: simple: Move bufferReady handler to
    SimpleCameraData
  libcamera: pipeline: simple: Remove
    SimplePipelineHandler::activeCamera_

 include/libcamera/internal/media_object.h |  11 +-
 src/libcamera/media_device.cpp            |   9 +-
 src/libcamera/media_object.cpp            |  50 ++-
 src/libcamera/pipeline/simple/simple.cpp  | 506 ++++++++++++++--------
 4 files changed, 377 insertions(+), 199 deletions(-)

-- 
Regards,

Laurent Pinchart