[libcamera-devel] [PATCH 00/10] Concurrent camera support in simple pipeline handler

[Martin Kepplinger]

Am Sonntag, dem 08.08.2021 um 18:07 +0300 schrieb Laurent Pinchart:
> Hi Martin,
> 
> Would you be able to test this series on the Librem5 ? I'd like to
> make
> sure it doesn't cause any regression.

hi Laurent, could you send me the patches directly and tell me where
they apply?

thanks,
                          martin

> 
> On Fri, Aug 06, 2021 at 01:24:26AM +0300, Laurent Pinchart wrote:
> > Hello,
> > 
> > Another day, another patch series that has been sitting in my tree
> > and
> > that I'd like to stop rebasing.
> > 
> > The series reworks the simple pipeline handler to correctly support
> > streaming from multiple cameras concurrently. The reason I haven't
> > posted it yet is that the libcamera core gets doesn't support this
> > feature yet. It's easy to work around by removing some checks for
> > testing, but will take more time to fix correctly.
> > 
> > The simple pipeline handler already supports registering multiple
> > cameras, and relies on the libcamera core blocking concurrent
> > access to
> > different cameras backed by the same media device. It thus doesn't
> > implement internally mutual exclusion between cameras that share
> > hardware resources, but, based on the assumption that a single
> > camera
> > will be used at a time, implements camera operation around a global
> > active camera pointer.
> > 
> > Patch 01/10 is the only libcamera core patch. It extends the
> > MediaEntity
> > class to expose the type of userspace interface that the entity
> > implements. Patches 02/10 to 06/10 then rework the way the pipeline
> > handler stores data about pipelines and accesses video devices and
> > subdevs.
> > 
> > Patch 02/10 extends the SimpleCameraData entity representation to
> > store
> > more information about graph traversal, and patch 04/10 stores the
> > entity corresponding to the video node with all other pipeline
> > entities
> > in SimpleCameraData. Patches 03/10, 05/10 and 06/10 generalize
> > handling
> > of video devices and subdev by opening them all together at match
> > time
> > and storing them in a single container in the SimplePipelineHandler
> > class.
> > 
> > With this done, path 07/10 adds an entity reservation mechanism in
> > the
> > SimplePipelineHandler class, to handle mutual exclusion when
> > multiple
> > cameras use the same entities. With this change, the simple
> > pipeline
> > handler handles concurrent access to cameras internally, without
> > depending on mutual exclusion implemented by the libcamera core
> > anymore.
> > This doesn't result in any change of behaviour, but paves the way
> > to
> > concurrent usage of multiple cameras that don't share entities once
> > the
> > restriction in the libcamera core will be lifted.
> > 
> > Patch 08/10 continues in that direction by moving the converter
> > from the
> > SimplePipelineHandler class to the SimpleCameraData class. This
> > allows
> > usage of the converter by multiple cameras concurrently. Patch
> > 09/10
> > then moves the bufferReady handler to the SimpleCameraData class as
> > well, to handle buffer completion in the correct context without
> > relying
> > on the global active camera pointer.
> > 
> > Finally, patch 10/10 removes the active camera pointer as it's not
> > used
> > anymore.
> > 
> > This series has only been partially tested, as the libcamera core
> > doesn't correctly support concurrent streaming from cameras
> > belonging to
> > the same pipeline handler. The code was tested with a single camera
> > and
> > didn't introduce any regression, and was also tested on i.MX8MP
> > with a
> > few hacks in the libcamera core to lift the restrictions and a few
> > out-of-tree patches to support the i.MX8MP (those are not ready for
> > submission yet). I would thus understand a reluctance to get this
> > merged
> > already, even if I believe that the absence of regression, combined
> > with
> > the cleaner (in my opinion) implementation of the simple pipeline
> > handler are good enough reasons by themselves to already merge
> > this.
> > 
> > Laurent Pinchart (10):
> >   libcamera: media_object: Expose entity type
> >   libcamera: pipeline: simple: Add sink and source pads to entity
> > data
> >   libcamera: pipeline: simple: Delay opening of video device until
> >     init()
> >   libcamera: pipeline: simple: Store video node entity in camera
> > data
> >   libcamera: pipeline: simple: Store all entity devices in common
> > map
> >   libcamera: pipeline: simple: Open all video devices at match()
> > time
> >   libcamera: pipeline: simple: Add pipeline pad reservation
> > mechanism
> >   libcamera: pipeline: simple: Move converter to SimpleCameraData
> >   libcamera: pipeline: simple: Move bufferReady handler to
> >     SimpleCameraData
> >   libcamera: pipeline: simple: Remove
> >     SimplePipelineHandler::activeCamera_
> > 
> >  include/libcamera/internal/media_object.h |  11 +-
> >  src/libcamera/media_device.cpp            |   9 +-
> >  src/libcamera/media_object.cpp            |  50 ++-
> >  src/libcamera/pipeline/simple/simple.cpp  | 506 ++++++++++++++----
> > ----
> >  4 files changed, 377 insertions(+), 199 deletions(-)
>