[libcamera-devel] [PATCH 00/23] IPA isolation implementation

Paul Elder paul.elder at ideasonboard.com
Tue Sep 15 16:20:15 CEST 2020 

This patchset implements IPA isolation, and is fully plumbed and
runnable on the raspberrypi pipeline handler. This also includes code
generators for the IPA proxies, and headers and serializers for custom
data structures and functions. Unlike the RFC, this patch series is
structures like a normal patch series, in order that should be merged.
The only exception is patches 9/23-22/23, which must be squashed to
avoid bisection breakage, and are only kept separate to ease review.

To see samples of generated code (without running this), see the RFC
"IPA isolation example, with IPC, fully plumbed".

To restate the problem, we have two goals:
- to be able to run IPAs isolated in a separate process
- the isolation must be transparent to both the pipeline handler and
  the IPA
During design of the IPC mechanism, we realized that we could support
both custom fuctions and custom data structures, which would be a lot
nicer than the tedious manual de/serialization that we had before with
IPAOperationData.

The architecture of the whole thing is as follows:

pipeline handler -> IPAProxyRPi --thread--> IPARPi
                                \
                                 \-> IPAIPC --IPC--> IPAProxyRPiWorker -> IPARPi

Where the pipeline author defines the interface between the pipeline
handler and the IPA, as well as any data structures. Based on this,
headers, de/serializers, and the proxy (including the worker) will be
generated.

Patche 2/23 imports mojo directly from the chromium repos, with some
big things pruned. Patches 3/23 to 8/23 add the meat of this IPC and
custom IPA interface mechanisms. Patches 9/23 to 22/23 are modifications
in the rest of libcamera to make the whole mechanism work, and must be
squashed together to avoid bisection breakage.

Patch 17/23 may be of interest, as it shows how one would write the
interface and data definition. We may need to have some rules, such as
start(), stop(), and init() are required, or something along those
lines.

Patch 18/23 is also noteworthy, as it shows how one would use the
interface that has been defined. Since the headers and proxies are
autogenerated, there isn't a patch where the generated code can be seen,
however.

Paul Elder (23):
  libcamera: ProcessManager: make ProcessManager lifetime explicitly
    managed
  utils: ipc: import mojo
  utils: ipc: add templates for code generation for IPC mechanism
  utils: ipc: add generator script
  meson: ipa, proxy: Generate headers and proxy with mojo
  libcamera: Add IPADataSerializer
  libcamera: Add IPAIPC
  libcamera: Add IPAIPC implementation based on unix socket
  libcamera: IPAModule: Replace ipa_context with IPAInterface
  libcamera: IPAProxy: Remove stop() override
  libcamera: IPAProxy: Add isolate parameter to create()
  libcamera: PipelineHandler: Remove IPA from base class
  libcamera: IPAInterface: Remove all functions from IPAInterface
  libcamera: IPAInterface: make ipaCreate return IPAInterface
  libcamera: IPAManager: Fetch IPAProxy corresponding to pipeline
  libcamera: IPAManager: add isolation flag to proxy creation
  ipa: raspberrypi: Add mojom data definition file
  libcamera: pipeline, ipa: raspberrypi: Use new data definition
  libcamera: IPAProxy: Remove registration mechanism
  libcamera: proxy: Remove IPAProxyLinux and IPAProxyThread
  libcamera: IPAManager: Make createIPA return proxy directly
  ipa: remove libipa
  libcamera: PipelineHandler: Move printing pipeline names to
    CameraManager

 .../libcamera/internal/ipa_data_serializer.h  |  876 ++++++++
 include/libcamera/internal/ipa_ipc.h          |   41 +
 .../libcamera/internal/ipa_ipc_unixsocket.h   |  115 +
 include/libcamera/internal/ipa_manager.h      |   31 +-
 include/libcamera/internal/ipa_module.h       |    4 +-
 include/libcamera/internal/ipa_proxy.h        |   31 -
 include/libcamera/internal/pipeline_handler.h |    1 -
 include/libcamera/internal/process.h          |   27 +
 include/libcamera/ipa/ipa_interface.h         |   23 +-
 include/libcamera/ipa/meson.build             |   89 +
 include/libcamera/ipa/raspberrypi.h           |   55 +-
 include/libcamera/ipa/raspberrypi.mojom       |  110 +
 src/ipa/libipa/ipa_interface_wrapper.cpp      |  285 ---
 src/ipa/libipa/ipa_interface_wrapper.h        |   61 -
 src/ipa/libipa/meson.build                    |   15 -
 src/ipa/meson.build                           |    2 -
 src/ipa/raspberrypi/meson.build               |    2 -
 src/ipa/raspberrypi/raspberrypi.cpp           |  110 +-
 src/libcamera/camera_manager.cpp              |    4 +
 src/libcamera/ipa_data_serializer.cpp         |   29 +
 src/libcamera/ipa_ipc.cpp                     |   36 +
 src/libcamera/ipa_ipc_unixsocket.cpp          |  175 ++
 src/libcamera/ipa_manager.cpp                 |   47 +-
 src/libcamera/ipa_module.cpp                  |   16 +-
 src/libcamera/ipa_proxy.cpp                   |   85 -
 src/libcamera/meson.build                     |    4 +-
 .../pipeline/raspberrypi/raspberrypi.cpp      |  119 +-
 src/libcamera/pipeline_handler.cpp            |    3 -
 src/libcamera/process.cpp                     |   46 +-
 src/libcamera/proxy/ipa_proxy_linux.cpp       |  103 -
 src/libcamera/proxy/ipa_proxy_thread.cpp      |  172 --
 src/libcamera/proxy/meson.build               |   24 +-
 .../proxy/worker/ipa_proxy_linux_worker.cpp   |   90 -
 src/libcamera/proxy/worker/meson.build        |   28 +-
 utils/ipc/generate.py                         |   17 +
 .../module_generated.h.tmpl                   |   95 +
 .../module_ipa_proxy.cpp.tmpl                 |  219 ++
 .../module_ipa_proxy.h.tmpl                   |  108 +
 .../module_ipa_proxy_worker.cpp.tmpl          |  167 ++
 .../module_serializer.h.tmpl                  |   43 +
 .../libcamera_templates/proxy_functions.tmpl  |  180 ++
 .../libcamera_templates/serializer.tmpl       |  262 +++
 .../generators/mojom_libcamera_generator.py   |  403 ++++
 utils/ipc/mojo/public/LICENSE                 |   27 +
 utils/ipc/mojo/public/tools/.style.yapf       |    6 +
 utils/ipc/mojo/public/tools/BUILD.gn          |   18 +
 utils/ipc/mojo/public/tools/bindings/BUILD.gn |  108 +
 .../ipc/mojo/public/tools/bindings/README.md  |  816 +++++++
 .../chromium_bindings_configuration.gni       |   51 +
 .../tools/bindings/compile_typescript.py      |   27 +
 .../tools/bindings/concatenate-files.py       |   54 +
 ...concatenate_and_replace_closure_exports.py |   73 +
 .../bindings/format_typemap_generator_args.py |   36 +
 .../tools/bindings/gen_data_files_list.py     |   52 +
 .../tools/bindings/generate_type_mappings.py  |  187 ++
 .../ipc/mojo/public/tools/bindings/mojom.gni  | 1941 +++++++++++++++++
 .../bindings/mojom_bindings_generator.py      |  390 ++++
 .../mojom_bindings_generator_unittest.py      |   62 +
 .../tools/bindings/mojom_types_downgrader.py  |  119 +
 .../tools/bindings/validate_typemap_config.py |   57 +
 utils/ipc/mojo/public/tools/mojom/README.md   |   14 +
 .../mojom/check_stable_mojom_compatibility.py |  170 ++
 ...eck_stable_mojom_compatibility_unittest.py |  260 +++
 .../mojo/public/tools/mojom/const_unittest.py |   90 +
 .../mojo/public/tools/mojom/enum_unittest.py  |   92 +
 .../mojo/public/tools/mojom/mojom/BUILD.gn    |   43 +
 .../mojo/public/tools/mojom/mojom/__init__.py |    0
 .../mojo/public/tools/mojom/mojom/error.py    |   28 +
 .../mojo/public/tools/mojom/mojom/fileutil.py |   45 +
 .../tools/mojom/mojom/fileutil_unittest.py    |   40 +
 .../tools/mojom/mojom/generate/__init__.py    |    0
 .../mojom/mojom/generate/constant_resolver.py |   93 +
 .../tools/mojom/mojom/generate/generator.py   |  325 +++
 .../mojom/generate/generator_unittest.py      |   74 +
 .../tools/mojom/mojom/generate/module.py      | 1635 ++++++++++++++
 .../mojom/mojom/generate/module_unittest.py   |   31 +
 .../public/tools/mojom/mojom/generate/pack.py |  258 +++
 .../mojom/mojom/generate/pack_unittest.py     |  225 ++
 .../mojom/mojom/generate/template_expander.py |   83 +
 .../tools/mojom/mojom/generate/translate.py   |  854 ++++++++
 .../mojom/generate/translate_unittest.py      |   73 +
 .../tools/mojom/mojom/parse/__init__.py       |    0
 .../public/tools/mojom/mojom/parse/ast.py     |  427 ++++
 .../tools/mojom/mojom/parse/ast_unittest.py   |  121 +
 .../mojom/mojom/parse/conditional_features.py |   82 +
 .../parse/conditional_features_unittest.py    |  233 ++
 .../public/tools/mojom/mojom/parse/lexer.py   |  251 +++
 .../tools/mojom/mojom/parse/lexer_unittest.py |  198 ++
 .../public/tools/mojom/mojom/parse/parser.py  |  488 +++++
 .../mojom/mojom/parse/parser_unittest.py      | 1390 ++++++++++++
 .../mojo/public/tools/mojom/mojom_parser.py   |  361 +++
 .../tools/mojom/mojom_parser_test_case.py     |   73 +
 .../tools/mojom/mojom_parser_unittest.py      |  171 ++
 .../tools/mojom/stable_attribute_unittest.py  |  127 ++
 .../mojom/version_compatibility_unittest.py   |  397 ++++
 .../public/tools/run_all_python_unittests.py  |   28 +
 utils/ipc/tools/diagnosis/crbug_1001171.py    |   51 +
 97 files changed, 16107 insertions(+), 1101 deletions(-)
 create mode 100644 include/libcamera/internal/ipa_data_serializer.h
 create mode 100644 include/libcamera/internal/ipa_ipc.h
 create mode 100644 include/libcamera/internal/ipa_ipc_unixsocket.h
 create mode 100644 include/libcamera/ipa/raspberrypi.mojom
 delete mode 100644 src/ipa/libipa/ipa_interface_wrapper.cpp
 delete mode 100644 src/ipa/libipa/ipa_interface_wrapper.h
 delete mode 100644 src/ipa/libipa/meson.build
 create mode 100644 src/libcamera/ipa_data_serializer.cpp
 create mode 100644 src/libcamera/ipa_ipc.cpp
 create mode 100644 src/libcamera/ipa_ipc_unixsocket.cpp
 delete mode 100644 src/libcamera/proxy/ipa_proxy_linux.cpp
 delete mode 100644 src/libcamera/proxy/ipa_proxy_thread.cpp
 delete mode 100644 src/libcamera/proxy/worker/ipa_proxy_linux_worker.cpp
 create mode 100755 utils/ipc/generate.py
 create mode 100644 utils/ipc/generators/libcamera_templates/module_generated.h.tmpl
 create mode 100644 utils/ipc/generators/libcamera_templates/module_ipa_proxy.cpp.tmpl
 create mode 100644 utils/ipc/generators/libcamera_templates/module_ipa_proxy.h.tmpl
 create mode 100644 utils/ipc/generators/libcamera_templates/module_ipa_proxy_worker.cpp.tmpl
 create mode 100644 utils/ipc/generators/libcamera_templates/module_serializer.h.tmpl
 create mode 100644 utils/ipc/generators/libcamera_templates/proxy_functions.tmpl
 create mode 100644 utils/ipc/generators/libcamera_templates/serializer.tmpl
 create mode 100644 utils/ipc/generators/mojom_libcamera_generator.py
 create mode 100644 utils/ipc/mojo/public/LICENSE
 create mode 100644 utils/ipc/mojo/public/tools/.style.yapf
 create mode 100644 utils/ipc/mojo/public/tools/BUILD.gn
 create mode 100644 utils/ipc/mojo/public/tools/bindings/BUILD.gn
 create mode 100644 utils/ipc/mojo/public/tools/bindings/README.md
 create mode 100644 utils/ipc/mojo/public/tools/bindings/chromium_bindings_configuration.gni
 create mode 100644 utils/ipc/mojo/public/tools/bindings/compile_typescript.py
 create mode 100755 utils/ipc/mojo/public/tools/bindings/concatenate-files.py
 create mode 100755 utils/ipc/mojo/public/tools/bindings/concatenate_and_replace_closure_exports.py
 create mode 100755 utils/ipc/mojo/public/tools/bindings/format_typemap_generator_args.py
 create mode 100644 utils/ipc/mojo/public/tools/bindings/gen_data_files_list.py
 create mode 100755 utils/ipc/mojo/public/tools/bindings/generate_type_mappings.py
 create mode 100644 utils/ipc/mojo/public/tools/bindings/mojom.gni
 create mode 100755 utils/ipc/mojo/public/tools/bindings/mojom_bindings_generator.py
 create mode 100644 utils/ipc/mojo/public/tools/bindings/mojom_bindings_generator_unittest.py
 create mode 100755 utils/ipc/mojo/public/tools/bindings/mojom_types_downgrader.py
 create mode 100755 utils/ipc/mojo/public/tools/bindings/validate_typemap_config.py
 create mode 100644 utils/ipc/mojo/public/tools/mojom/README.md
 create mode 100755 utils/ipc/mojo/public/tools/mojom/check_stable_mojom_compatibility.py
 create mode 100755 utils/ipc/mojo/public/tools/mojom/check_stable_mojom_compatibility_unittest.py
 create mode 100644 utils/ipc/mojo/public/tools/mojom/const_unittest.py
 create mode 100644 utils/ipc/mojo/public/tools/mojom/enum_unittest.py
 create mode 100644 utils/ipc/mojo/public/tools/mojom/mojom/BUILD.gn
 create mode 100644 utils/ipc/mojo/public/tools/mojom/mojom/__init__.py
 create mode 100644 utils/ipc/mojo/public/tools/mojom/mojom/error.py
 create mode 100644 utils/ipc/mojo/public/tools/mojom/mojom/fileutil.py
 create mode 100644 utils/ipc/mojo/public/tools/mojom/mojom/fileutil_unittest.py
 create mode 100644 utils/ipc/mojo/public/tools/mojom/mojom/generate/__init__.py
 create mode 100644 utils/ipc/mojo/public/tools/mojom/mojom/generate/constant_resolver.py
 create mode 100644 utils/ipc/mojo/public/tools/mojom/mojom/generate/generator.py
 create mode 100644 utils/ipc/mojo/public/tools/mojom/mojom/generate/generator_unittest.py
 create mode 100644 utils/ipc/mojo/public/tools/mojom/mojom/generate/module.py
 create mode 100644 utils/ipc/mojo/public/tools/mojom/mojom/generate/module_unittest.py
 create mode 100644 utils/ipc/mojo/public/tools/mojom/mojom/generate/pack.py
 create mode 100644 utils/ipc/mojo/public/tools/mojom/mojom/generate/pack_unittest.py
 create mode 100644 utils/ipc/mojo/public/tools/mojom/mojom/generate/template_expander.py
 create mode 100644 utils/ipc/mojo/public/tools/mojom/mojom/generate/translate.py
 create mode 100644 utils/ipc/mojo/public/tools/mojom/mojom/generate/translate_unittest.py
 create mode 100644 utils/ipc/mojo/public/tools/mojom/mojom/parse/__init__.py
 create mode 100644 utils/ipc/mojo/public/tools/mojom/mojom/parse/ast.py
 create mode 100644 utils/ipc/mojo/public/tools/mojom/mojom/parse/ast_unittest.py
 create mode 100644 utils/ipc/mojo/public/tools/mojom/mojom/parse/conditional_features.py
 create mode 100644 utils/ipc/mojo/public/tools/mojom/mojom/parse/conditional_features_unittest.py
 create mode 100644 utils/ipc/mojo/public/tools/mojom/mojom/parse/lexer.py
 create mode 100644 utils/ipc/mojo/public/tools/mojom/mojom/parse/lexer_unittest.py
 create mode 100644 utils/ipc/mojo/public/tools/mojom/mojom/parse/parser.py
 create mode 100644 utils/ipc/mojo/public/tools/mojom/mojom/parse/parser_unittest.py
 create mode 100755 utils/ipc/mojo/public/tools/mojom/mojom_parser.py
 create mode 100644 utils/ipc/mojo/public/tools/mojom/mojom_parser_test_case.py
 create mode 100644 utils/ipc/mojo/public/tools/mojom/mojom_parser_unittest.py
 create mode 100644 utils/ipc/mojo/public/tools/mojom/stable_attribute_unittest.py
 create mode 100644 utils/ipc/mojo/public/tools/mojom/version_compatibility_unittest.py
 create mode 100755 utils/ipc/mojo/public/tools/run_all_python_unittests.py
 create mode 100644 utils/ipc/tools/diagnosis/crbug_1001171.py

-- 
2.27.0

More information about the libcamera-devel mailing list