[libcamera-devel] [PATCH v2 00/13] Initial libcamera threading model

[Laurent Pinchart]

Hello,

This series defines and implements an initial version of the threading
model.

libcamera currently relies on the application providing an event loop.
This cumbersome when the direct user of libcamera is not an application
but a framework, as no event loop is available in that case. The Android
camera HAL and the V4L2 adaptation layers had to create an internal
thread to make an event loop available to libcamera, and handle
synchronization between calls from the upper framework and the internal
thread. A GStreamer element would require a similar implementation,
resulting in lots of code duplication.

Furthermore, relying on an application event loop may block libcamera
from processing events if the application performs long blocking
operations. This is not compatible with the real-time constraints of
pipeline handlers and IPAs.

For this reason, creating and managing threads internally to libcamera
is a better option. This requires defining a threading model, as all of
a sudden multiple contexts of execution will be available when only one
context existed before.

Some of the preparatory work from v1 has been merged already, but the
series still starts with 5 preparatory patches (some of them being split
from the larger "libcamera: camera_manager: Run the camera manager in a
thread" patch). The most notable change there is the move of private
data members of the CameraManager and Camera classes to private classes,
helping with ABI breakage avoidance in future work.

The next two patches consist solely of documentation. Patch 06/13
defines the threading model and patch 07/13 documents it in core
classes. Patch 08/13 then makes signal connection and disconnection
thread-safe to support multi-threaded contexts.

Patch 09/13 is the most disruptive change, as it creates an thread for
the CameraManager. At the moment this is the only thread created by
libcamera, and all pipeline handlers and camera are bound to it. This
model is likely too simple and will need to be extended with more
threads in the future to avoid low-priority blocking operations
interfering with real-time camera constraints. The change is however a
step in the right direction, as it brings consideration of multiple
threads to all of libcamera.

Patches 10/13 to 11/13 then document and implement the threading model
in the Camera and PipelineHandler classes. Note that patch 09/13 breaks
operation of libcamera, and patch 10/13 fixes that. This is caused by
the threading model implementation being split in two patches. I have
decided to organize the changes this way to keep changes reviewable, but
I'm open to squashing the two patches together, especially now that
09/13 has become smaller.

Patches 12/13 and 13/13 finally simplify the V4L2 compatibility and
Android camera HAL layers by removing the internal threads, as libcamera
doesn't require an externally-provided event loop anymore. I have tested
12/13 with yavta, but 13/13 is currently only compile-tested.

Laurent Pinchart (13):
  libcamera: Fix documentation of buffer allocation/export functions
  libcamera: camera_manager: Move private data members to private
    implementation
  libcamera: camera_manager: Return a copy of the vector from cameras()
  libcamera: camera: Move private data members to private implementation
  libcamera: camera: Centralize state checks in Private class
  libcamera: Define the threading model
  libcamera: Document thread-safety attributes of core classes
  libcamera: signal: Make connection and disconnection thread-safe
  libcamera: camera_manager: Run the camera manager in a thread
  libcamera: camera: Implement the threading model
  libcamera: pipeline_handler: Document the threading model
  v4l2: Remove internal thread
  android: Remove internal thread

 Documentation/Doxyfile.in                |   5 +-
 include/libcamera/camera.h               |  28 +-
 include/libcamera/camera_manager.h       |  13 +-
 src/android/camera3_hal.cpp              |   8 +-
 src/android/camera_device.cpp            |  48 +--
 src/android/camera_device.h              |  16 +-
 src/android/camera_hal_manager.cpp       |  78 ++---
 src/android/camera_hal_manager.h         |  17 +-
 src/android/camera_ops.cpp               |  96 ++++++
 src/android/camera_ops.h                 |  15 +
 src/android/camera_proxy.cpp             | 180 -----------
 src/android/camera_proxy.h               |  42 ---
 src/android/meson.build                  |   2 +-
 src/libcamera/camera.cpp                 | 362 ++++++++++++++++-------
 src/libcamera/camera_manager.cpp         | 298 ++++++++++++++-----
 src/libcamera/device_enumerator.cpp      |   2 +
 src/libcamera/event_notifier.cpp         |   2 +
 src/libcamera/framebuffer_allocator.cpp  |  50 +---
 src/libcamera/include/pipeline_handler.h |   4 +-
 src/libcamera/ipc_unixsocket.cpp         |   2 +
 src/libcamera/object.cpp                 |  10 +-
 src/libcamera/pipeline_handler.cpp       |  44 ++-
 src/libcamera/signal.cpp                 |  33 +++
 src/libcamera/thread.cpp                 |  86 ++++++
 src/libcamera/timer.cpp                  |  20 +-
 src/libcamera/v4l2_videodevice.cpp       |   5 +-
 src/v4l2/v4l2_camera.h                   |   2 +-
 src/v4l2/v4l2_camera_proxy.cpp           |  43 +--
 src/v4l2/v4l2_compat_manager.cpp         |  48 +--
 src/v4l2/v4l2_compat_manager.h           |  13 +-
 30 files changed, 906 insertions(+), 666 deletions(-)
 create mode 100644 src/android/camera_ops.cpp
 create mode 100644 src/android/camera_ops.h
 delete mode 100644 src/android/camera_proxy.cpp
 delete mode 100644 src/android/camera_proxy.h

-- 
Regards,

Laurent Pinchart