[libcamera-devel] [RFC PATCH 00/17] IPA isolation example, with IPC, fully plumbed
Paul Elder
paul.elder at ideasonboard.com
Wed Aug 26 13:09:09 CEST 2020
This patchset implements IPA isolation, and is fully plumbed and
runnable on the raspberrypi pipeline handler. This patchset is not
organized as a proper patchset, but rather such that it is easier to
understand what components there are, and how they will fit together.
To restate the problem, we have two goals:
- to be able to run IPAs isolated in a separate process
- the isolation must be transparent to both the pipeline handler and
the IPA
During design of the IPC mechanism, we realized that we could support
both custom fuctions and custom data structures, which would be a lot
nicer than the tedious manual de/serialization that we had before with
IPAOperationData. This patchset shows how this can be achieved.
The architecture of the whole thing is as follows:
pipeline handler -> IPAProxyRPi --thread--> IPARPi
\
\-> IPAIPC --IPA--> IPAProxyRPiWorker -> IPARPi
The pipeline handler author will write an IPA interface definition, in
mojo format. This will declare what functions the IPA shall implement,
what callbacks the IPA can signal, and what custom data structures can
be passed (in addition to primitives and libcamera data structures). An
example of this can be seen in 1/17. This IPA interface will become the
IPAInterface specific to the pipeline handler. In the example in this
RFC, that will be IPARPiInterface. In the diagram above, IPAProxyRPi and
IPARPi both implement IPARPiInterface.
A compiler will generate a header for the custom data structures, as
well as a header for the IPA interface. This can also be seen in 1/17.
Patches 2/17 and 3/17 show how the pipeline author and IPA author would
expect to use the IPA interface and data structures defined in the mojo
file. I think it is quite straightforward and transparent.
Patch 4/17 shows the hand-written IPADataSerializer for built-in and
libcamera types, as well as the IPADataSerializer for the custom data
types, which will be generated. These will be used by the proxies, which
can be seen in 5/17.
Each pipeline shall have its own proxy. This is to allow custom data
types and functions. The proxy will be generated from some proxy
template, and crossed with the IPA interface as defined in the mojo
file. The proxy decides whether to call the IPA function directly, or
via IPC. Thus, all IPA modules will be wrapped in an IPAProxy. The proxy
is also responsible for de/serializing the data, before sending it
through the IPC mechanism. The IPC mechanism is designed to be
swap-outabble, and simply exposes an async and synchronous version of
send(), and a callback signal. These can be seen in 6/17, along with an
implementation using UnixSocket.
7/17 enables this whole infrastructure in meson. Note that this only
implements IPC for raspberrypi, so make sure to disable the other
pipeline handlers. This can be done like:
meson -Dpipelines=raspberrypi build
Also, to test isolation, the following needs to be applied:
diff --git a/src/libcamera/ipa_manager.cpp
b/src/libcamera/ipa_manager.cpp
index 26458153..be4fd6db 100644
--- a/src/libcamera/ipa_manager.cpp
+++ b/src/libcamera/ipa_manager.cpp
@@ -292,7 +292,7 @@ std::unique_ptr<IPAProxy>
IPAManager::createIPA(PipelineHandler *pipe,
}
std::unique_ptr<IPAProxy> proxy =
- pf->create(m, !self_->isSignatureValid(m));
+ pf->create(m, true);
if (!proxy->isValid()) {
LOG(IPAManager, Error) << "Failed to load proxy";
return nullptr;
})
And don't forget to set permission on the dma heap:
sudo chmod 766 /dev/dma_heap/linux,cma
8/17 is a collection of tests that I implmented during debug. If there
is any value in them, I'll polish them and merge them. Otherwise I'll
discard them.
The rest of the patches are all small tweaks in other parts of libcamera
to get this whole IPC infrastructure to run.
Paul Elder (17):
IPA: IPC: raspberrypi: Add data definition and generated header
libcamera: pipeline: raspberrypi: Use generated IPARPiInterface
libcamera: IPA: raspberrypi: Use generated IPARPiInterface
IPA: IPC: add IPADataSerializer
IPA: IPC: add IPAProxyRPi and IPAProxyRPiWorker
IPA: IPC: add IPAIPC
meson: Enable IPC
tests: IPC: Add various tests for the IPC framework
libcamera: ProcessManager: make ProcessManager lifetime explicitly
managed
libcamera: IPAModule: Replace ipa_context with IPAInterface
libcamera: IPAProxy: Remove stop() override
libcamera: IPAProxy: Add isolate parameter to create()
libcamera: PipelineHandler: Remove IPA from base class
libcamera: IPAInterface: Remove all functions from IPAInterface
libcamera: IPAInterface: make ipaCreate return IPAInterface
libcamera: IPAManager: Fetch IPAProxy corresponding to pipeline
libcamera: IPAManager: add isolation flag to proxy creation
.../libcamera/internal/ipa_data_serializer.h | 816 ++++++++++++++++++
include/libcamera/internal/ipa_ipc.h | 73 ++
.../libcamera/internal/ipa_ipc_unixsocket.h | 116 +++
include/libcamera/internal/ipa_module.h | 4 +-
include/libcamera/internal/ipa_proxy.h | 24 +-
.../internal/ipa_proxy_raspberrypi.h | 117 +++
include/libcamera/internal/pipeline_handler.h | 1 -
include/libcamera/internal/process.h | 27 +
include/libcamera/ipa/ipa_interface.h | 23 +-
include/libcamera/ipa/raspberrypi.h | 61 +-
include/libcamera/ipa/raspberrypi.mojom | 97 +++
.../libcamera/ipa/raspberrypi_serializer.h | 487 +++++++++++
include/libcamera/ipa/raspberrypi_wrapper.h | 136 +++
src/ipa/libipa/meson.build | 4 +-
src/ipa/raspberrypi/raspberrypi.cpp | 107 ++-
src/libcamera/camera_manager.cpp | 2 +
src/libcamera/ipa_data_serializer.cpp | 29 +
src/libcamera/ipa_ipc.cpp | 58 ++
src/libcamera/ipa_ipc_unixsocket.cpp | 177 ++++
src/libcamera/ipa_manager.cpp | 7 +-
src/libcamera/ipa_module.cpp | 16 +-
src/libcamera/meson.build | 5 +-
.../pipeline/raspberrypi/raspberrypi.cpp | 125 +--
src/libcamera/process.cpp | 46 +-
src/libcamera/proxy/ipa_proxy_raspberrypi.cpp | 381 ++++++++
src/libcamera/proxy/meson.build | 5 +-
.../worker/ipa_proxy_raspberrypi_worker.cpp | 355 ++++++++
src/libcamera/proxy/worker/meson.build | 3 +-
test/ipa/meson.build | 4 +-
test/ipc/meson.build | 3 +-
test/ipc/unixsocket_fds.cpp | 284 ++++++
.../control_serialization_ipa.cpp | 85 ++
test/serialization/meson.build | 5 +-
.../rpi_action_serialization.cpp | 141 +++
.../rpi_config_serialization.cpp | 162 ++++
35 files changed, 3778 insertions(+), 208 deletions(-)
create mode 100644 include/libcamera/internal/ipa_data_serializer.h
create mode 100644 include/libcamera/internal/ipa_ipc.h
create mode 100644 include/libcamera/internal/ipa_ipc_unixsocket.h
create mode 100644 include/libcamera/internal/ipa_proxy_raspberrypi.h
create mode 100644 include/libcamera/ipa/raspberrypi.mojom
create mode 100644 include/libcamera/ipa/raspberrypi_serializer.h
create mode 100644 include/libcamera/ipa/raspberrypi_wrapper.h
create mode 100644 src/libcamera/ipa_data_serializer.cpp
create mode 100644 src/libcamera/ipa_ipc.cpp
create mode 100644 src/libcamera/ipa_ipc_unixsocket.cpp
create mode 100644 src/libcamera/proxy/ipa_proxy_raspberrypi.cpp
create mode 100644 src/libcamera/proxy/worker/ipa_proxy_raspberrypi_worker.cpp
create mode 100644 test/ipc/unixsocket_fds.cpp
create mode 100644 test/serialization/control_serialization_ipa.cpp
create mode 100644 test/serialization/rpi_action_serialization.cpp
create mode 100644 test/serialization/rpi_config_serialization.cpp
--
2.27.0
More information about the libcamera-devel
mailing list