[libcamera-devel] [PATCH 00/38] IPA isolation implementation
paul.elder at ideasonboard.com
paul.elder at ideasonboard.com
Tue Sep 22 15:42:49 CEST 2020
On Tue, Sep 22, 2020 at 10:34:59PM +0900, Paul Elder wrote:
> This patch series depends on the patch series "Unify utils locations".
>
> This patchset implements IPA isolation, and is fully plumbed and
> runnable on all pipeline handlers. This also includes code
> generators for the IPA proxies, and headers and serializers for custom
> data structures and functions. Unlike the RFC, this patch series is
> structured like a normal patch series, in order that should be merged.
> The only exception is patches 16/38-36/38, which must be squashed to
> avoid bisection breakage, and are only kept separate to ease review.
Hello all,
Im so sorry, I forgot to put the v2 tag on this whole series.
Paul
> To see samples of generated code (without running this), see the RFC
> "IPA isolation example, with IPC, fully plumbed".
>
> To restate the problem, we have two goals:
> - to be able to run IPAs isolated in a separate process
> - the isolation must be transparent to both the pipeline handler and
> the IPA
> During design of the IPC mechanism, we realized that we could support
> both custom fuctions and custom data structures, which would be a lot
> nicer than the tedious manual de/serialization that we had before with
> IPAOperationData.
>
> The architecture of the whole thing is as follows:
>
> pipeline handler -> IPAProxyRPi --thread--> IPARPi
> \
> \-> IPAIPC --IPC--> IPAProxyRPiWorker -> IPARPi
>
> Where the pipeline author defines the interface between the pipeline
> handler and the IPA, as well as any data structures. Based on this,
> headers, de/serializers, and the proxy (including the worker) will be
> generated.
>
> Patches 1/38~04/38 will be fast tracked, and I plan to merge them
> separately from the rest of this series after they are approved in this
> round.
>
> Patch 6/38 imports mojo directly from the chromium repos, with some
> big things pruned. Patches 8/38 to 15/38 add the meat of this IPC and
> custom IPA interface mechanisms. Patches 16/38 to 36/38 are modifications
> in the rest of libcamera to make the whole mechanism work, and must be
> squashed together to avoid bisection breakage. Patches 37/38 and 38/38
> add tests for the serializer and IPC components of this series.
>
> Patch 27/38 may be of interest, as it shows how one would write the
> interface and data definition. In v2 I have customized the raspberrypi
> mojom file to take advantage of the freedom in definition the IPA
> interface and data structures (as opposed to v1, where it was
> practically a direct translation of IPAOperationData).
>
> In v2 I have added some rules to the mojom file, such as start(),
> stop(), and init() are required, along with the main and callback
> interfaces. These are documented in core.mojom in patch 26/38 and
> enforced in the code generator in patch 08/38. core.mojom is a new
> addition in v2, and is where common mojom "structs" (which represent
> libcamera objects) and IPA interface definition documentaion are.
>
> Patch 28/38 is also noteworthy, as it shows how one would use the
> interface that has been defined. Since the headers and proxies are
> autogenerated, there isn't a patch where the generated code can be seen,
> however.
>
> Patches 29/38 and 30/38 also define mojom files and use them, but are
> for vimc and rkisp1, respectively. These will also give an idea on how
> the mojom definition looks and how it will be used.
>
> Changes in v2:
> - upgrade documentation
> - fix documentation compiling
> - plumb the new IPC system through the other pipelines
> - fix the issue where editing jinja templates wouldn't trigger
> recompile
> - enforce IPA interface rules in code generator
> - fix previously-untriggered code generation bugs
> - add de/serializers for all primitive types, string, and const
> ControlList
> - customized the RPi IPA interface
> - add licenses
> - add tests
>
> Paul Elder (38):
> Documentation: coding-style: Document global variable guidelines
> libcamera: ProcessManager: make ProcessManager lifetime explicitly
> managed
> libcamera: PipelineHandler: Move printing pipeline names to
> CameraManager
> libcamera: process: forward-declare EventNotifier
> libcamera: meson: Add internal headers to sources
> utils: ipc: import mojo
> libcamera: Update dep5 to specify license for mojo
> utils: ipc: add templates for code generation for IPC mechanism
> utils: ipc: add generator script
> Documentation: skip generating documentation for generated code
> meson: ipa, proxy: Generate headers and proxy with mojo
> ipa: raspberrypi: meson: Add dependency on generated headers
> libcamera: Add IPADataSerializer
> libcamera: Add IPAIPC
> libcamera: Add IPAIPC implementation based on unix socket
> libcamera: IPAModule: Replace ipa_context with IPAInterface
> libcamera: ipa_context_wrapper: Remove ipa_context_wrapper
> libcamera: IPAProxy: Remove stop() override
> libcamera: IPAProxy: Add isolate parameter to create()
> libcamera: PipelineHandler: Remove IPA from base class
> libcamera: IPAInterface: Remove all functions from IPAInterface
> libcamera: IPAInterface: make ipaCreate return IPAInterface
> libcamera: IPAInterface: remove ipa_context and functions from
> documentation
> libcamera: IPAManager: Fetch IPAProxy corresponding to pipeline
> libcamera: IPAManager: add isolation flag to proxy creation
> ipa: Add core.mojom
> ipa: raspberrypi: Add mojom data definition file
> libcamera: pipeline, ipa: raspberrypi: Use new data definition
> libcamera: pipeline, ipa: vimc: Support the new IPC mechanism
> libcamera: pipeline, ipa: rkisp1: Support the new IPC mechanism
> libcamera: IPAProxy: Remove registration mechanism
> libcamera: proxy: Remove IPAProxyLinux and IPAProxyThread
> libcamera: IPAManager: Make createIPA return proxy directly
> ipa: remove libipa
> tests: ipa_interface_test: Update to use new createIPA
> tests: Remove IPA wrappers test
> tests: Add IPADataSerializer test
> tests: Add test for IPAIPCUnixSocket
>
> .reuse/dep5 | 6 +
> Documentation/Doxyfile.in | 7 +-
> Documentation/coding-style.rst | 15 +
> Documentation/meson.build | 2 -
> .../libcamera/internal/ipa_context_wrapper.h | 52 -
> .../libcamera/internal/ipa_data_serializer.h | 1220 +++++++++++
> include/libcamera/internal/ipa_ipc.h | 41 +
> .../libcamera/internal/ipa_ipc_unixsocket.h | 115 +
> include/libcamera/internal/ipa_manager.h | 31 +-
> include/libcamera/internal/ipa_module.h | 4 +-
> include/libcamera/internal/ipa_proxy.h | 31 -
> include/libcamera/internal/meson.build | 1 -
> include/libcamera/internal/pipeline_handler.h | 1 -
> include/libcamera/internal/process.h | 29 +
> include/libcamera/ipa/core.mojom | 74 +
> include/libcamera/ipa/ipa_interface.h | 126 +-
> include/libcamera/ipa/meson.build | 89 +
> include/libcamera/ipa/raspberrypi.h | 55 +-
> include/libcamera/ipa/raspberrypi.mojom | 157 ++
> include/libcamera/ipa/rkisp1.h | 14 +-
> include/libcamera/ipa/rkisp1.mojom | 42 +
> include/libcamera/ipa/{ipa_vimc.h => vimc.h} | 4 +
> include/libcamera/ipa/vimc.mojom | 12 +
> src/ipa/libipa/ipa_interface_wrapper.cpp | 285 ---
> src/ipa/libipa/ipa_interface_wrapper.h | 61 -
> src/ipa/libipa/meson.build | 15 -
> src/ipa/meson.build | 2 -
> src/ipa/raspberrypi/meson.build | 4 +-
> src/ipa/raspberrypi/raspberrypi.cpp | 154 +-
> src/ipa/rkisp1/meson.build | 5 +-
> src/ipa/rkisp1/rkisp1.cpp | 55 +-
> src/ipa/vimc/meson.build | 5 +-
> src/ipa/vimc/vimc.cpp | 20 +-
> src/libcamera/camera_manager.cpp | 5 +
> src/libcamera/ipa_context_wrapper.cpp | 297 ---
> src/libcamera/ipa_data_serializer.cpp | 154 ++
> src/libcamera/ipa_interface.cpp | 517 +----
> src/libcamera/ipa_ipc.cpp | 110 +
> src/libcamera/ipa_ipc_unixsocket.cpp | 175 ++
> src/libcamera/ipa_manager.cpp | 47 +-
> src/libcamera/ipa_module.cpp | 10 +-
> src/libcamera/ipa_proxy.cpp | 101 -
> src/libcamera/meson.build | 6 +-
> .../pipeline/raspberrypi/raspberrypi.cpp | 170 +-
> src/libcamera/pipeline/rkisp1/rkisp1.cpp | 44 +-
> src/libcamera/pipeline/vimc/vimc.cpp | 8 +-
> src/libcamera/pipeline_handler.cpp | 11 -
> src/libcamera/process.cpp | 46 +-
> src/libcamera/proxy/ipa_proxy_linux.cpp | 103 -
> src/libcamera/proxy/ipa_proxy_thread.cpp | 172 --
> src/libcamera/proxy/meson.build | 19 +-
> .../proxy/worker/ipa_proxy_linux_worker.cpp | 90 -
> src/libcamera/proxy/worker/meson.build | 22 +-
> test/ipa/ipa_interface_test.cpp | 7 +-
> test/ipa/ipa_wrappers_test.cpp | 452 ----
> test/ipa/meson.build | 5 +-
> test/ipc/meson.build | 3 +-
> test/ipc/unixsocket_ipc.cpp | 238 ++
> test/log/log_process.cpp | 2 +
> test/process/process_test.cpp | 2 +
> .../ipa_data_serializer_test.cpp | 463 ++++
> test/serialization/meson.build | 3 +-
> utils/ipc/generate.py | 21 +
> .../libcamera_templates/meson.build | 11 +
> .../module_generated.h.tmpl | 96 +
> .../module_ipa_proxy.cpp.tmpl | 222 ++
> .../module_ipa_proxy.h.tmpl | 107 +
> .../module_ipa_proxy_worker.cpp.tmpl | 178 ++
> .../module_serializer.h.tmpl | 44 +
> .../libcamera_templates/proxy_functions.tmpl | 185 ++
> .../libcamera_templates/serializer.tmpl | 268 +++
> utils/ipc/generators/meson.build | 3 +
> .../generators/mojom_libcamera_generator.py | 452 ++++
> utils/ipc/meson.build | 14 +
> utils/ipc/mojo/public/LICENSE | 27 +
> utils/ipc/mojo/public/tools/.style.yapf | 6 +
> utils/ipc/mojo/public/tools/BUILD.gn | 18 +
> utils/ipc/mojo/public/tools/bindings/BUILD.gn | 108 +
> .../ipc/mojo/public/tools/bindings/README.md | 816 +++++++
> .../chromium_bindings_configuration.gni | 51 +
> .../tools/bindings/compile_typescript.py | 27 +
> .../tools/bindings/concatenate-files.py | 54 +
> ...concatenate_and_replace_closure_exports.py | 73 +
> .../bindings/format_typemap_generator_args.py | 36 +
> .../tools/bindings/gen_data_files_list.py | 52 +
> .../tools/bindings/generate_type_mappings.py | 187 ++
> .../ipc/mojo/public/tools/bindings/mojom.gni | 1941 +++++++++++++++++
> .../bindings/mojom_bindings_generator.py | 390 ++++
> .../mojom_bindings_generator_unittest.py | 62 +
> .../tools/bindings/mojom_types_downgrader.py | 119 +
> .../tools/bindings/validate_typemap_config.py | 57 +
> utils/ipc/mojo/public/tools/mojom/README.md | 14 +
> .../mojom/check_stable_mojom_compatibility.py | 170 ++
> ...eck_stable_mojom_compatibility_unittest.py | 260 +++
> .../mojo/public/tools/mojom/const_unittest.py | 90 +
> .../mojo/public/tools/mojom/enum_unittest.py | 92 +
> .../mojo/public/tools/mojom/mojom/BUILD.gn | 43 +
> .../mojo/public/tools/mojom/mojom/__init__.py | 0
> .../mojo/public/tools/mojom/mojom/error.py | 28 +
> .../mojo/public/tools/mojom/mojom/fileutil.py | 45 +
> .../tools/mojom/mojom/fileutil_unittest.py | 40 +
> .../tools/mojom/mojom/generate/__init__.py | 0
> .../mojom/mojom/generate/constant_resolver.py | 93 +
> .../tools/mojom/mojom/generate/generator.py | 325 +++
> .../mojom/generate/generator_unittest.py | 74 +
> .../tools/mojom/mojom/generate/module.py | 1635 ++++++++++++++
> .../mojom/mojom/generate/module_unittest.py | 31 +
> .../public/tools/mojom/mojom/generate/pack.py | 258 +++
> .../mojom/mojom/generate/pack_unittest.py | 225 ++
> .../mojom/mojom/generate/template_expander.py | 83 +
> .../tools/mojom/mojom/generate/translate.py | 854 ++++++++
> .../mojom/generate/translate_unittest.py | 73 +
> .../tools/mojom/mojom/parse/__init__.py | 0
> .../public/tools/mojom/mojom/parse/ast.py | 427 ++++
> .../tools/mojom/mojom/parse/ast_unittest.py | 121 +
> .../mojom/mojom/parse/conditional_features.py | 82 +
> .../parse/conditional_features_unittest.py | 233 ++
> .../public/tools/mojom/mojom/parse/lexer.py | 251 +++
> .../tools/mojom/mojom/parse/lexer_unittest.py | 198 ++
> .../public/tools/mojom/mojom/parse/parser.py | 488 +++++
> .../mojom/mojom/parse/parser_unittest.py | 1390 ++++++++++++
> .../mojo/public/tools/mojom/mojom_parser.py | 361 +++
> .../tools/mojom/mojom_parser_test_case.py | 73 +
> .../tools/mojom/mojom_parser_unittest.py | 171 ++
> .../tools/mojom/stable_attribute_unittest.py | 127 ++
> .../mojom/version_compatibility_unittest.py | 397 ++++
> .../public/tools/run_all_python_unittests.py | 28 +
> utils/ipc/tools/diagnosis/crbug_1001171.py | 51 +
> utils/meson.build | 1 +
> 129 files changed, 17795 insertions(+), 2678 deletions(-)
> delete mode 100644 include/libcamera/internal/ipa_context_wrapper.h
> create mode 100644 include/libcamera/internal/ipa_data_serializer.h
> create mode 100644 include/libcamera/internal/ipa_ipc.h
> create mode 100644 include/libcamera/internal/ipa_ipc_unixsocket.h
> create mode 100644 include/libcamera/ipa/core.mojom
> create mode 100644 include/libcamera/ipa/raspberrypi.mojom
> create mode 100644 include/libcamera/ipa/rkisp1.mojom
> rename include/libcamera/ipa/{ipa_vimc.h => vimc.h} (87%)
> create mode 100644 include/libcamera/ipa/vimc.mojom
> delete mode 100644 src/ipa/libipa/ipa_interface_wrapper.cpp
> delete mode 100644 src/ipa/libipa/ipa_interface_wrapper.h
> delete mode 100644 src/ipa/libipa/meson.build
> delete mode 100644 src/libcamera/ipa_context_wrapper.cpp
> create mode 100644 src/libcamera/ipa_data_serializer.cpp
> create mode 100644 src/libcamera/ipa_ipc.cpp
> create mode 100644 src/libcamera/ipa_ipc_unixsocket.cpp
> delete mode 100644 src/libcamera/proxy/ipa_proxy_linux.cpp
> delete mode 100644 src/libcamera/proxy/ipa_proxy_thread.cpp
> delete mode 100644 src/libcamera/proxy/worker/ipa_proxy_linux_worker.cpp
> delete mode 100644 test/ipa/ipa_wrappers_test.cpp
> create mode 100644 test/ipc/unixsocket_ipc.cpp
> create mode 100644 test/serialization/ipa_data_serializer_test.cpp
> create mode 100755 utils/ipc/generate.py
> create mode 100644 utils/ipc/generators/libcamera_templates/meson.build
> create mode 100644 utils/ipc/generators/libcamera_templates/module_generated.h.tmpl
> create mode 100644 utils/ipc/generators/libcamera_templates/module_ipa_proxy.cpp.tmpl
> create mode 100644 utils/ipc/generators/libcamera_templates/module_ipa_proxy.h.tmpl
> create mode 100644 utils/ipc/generators/libcamera_templates/module_ipa_proxy_worker.cpp.tmpl
> create mode 100644 utils/ipc/generators/libcamera_templates/module_serializer.h.tmpl
> create mode 100644 utils/ipc/generators/libcamera_templates/proxy_functions.tmpl
> create mode 100644 utils/ipc/generators/libcamera_templates/serializer.tmpl
> create mode 100644 utils/ipc/generators/meson.build
> create mode 100644 utils/ipc/generators/mojom_libcamera_generator.py
> create mode 100644 utils/ipc/meson.build
> create mode 100644 utils/ipc/mojo/public/LICENSE
> create mode 100644 utils/ipc/mojo/public/tools/.style.yapf
> create mode 100644 utils/ipc/mojo/public/tools/BUILD.gn
> create mode 100644 utils/ipc/mojo/public/tools/bindings/BUILD.gn
> create mode 100644 utils/ipc/mojo/public/tools/bindings/README.md
> create mode 100644 utils/ipc/mojo/public/tools/bindings/chromium_bindings_configuration.gni
> create mode 100644 utils/ipc/mojo/public/tools/bindings/compile_typescript.py
> create mode 100755 utils/ipc/mojo/public/tools/bindings/concatenate-files.py
> create mode 100755 utils/ipc/mojo/public/tools/bindings/concatenate_and_replace_closure_exports.py
> create mode 100755 utils/ipc/mojo/public/tools/bindings/format_typemap_generator_args.py
> create mode 100644 utils/ipc/mojo/public/tools/bindings/gen_data_files_list.py
> create mode 100755 utils/ipc/mojo/public/tools/bindings/generate_type_mappings.py
> create mode 100644 utils/ipc/mojo/public/tools/bindings/mojom.gni
> create mode 100755 utils/ipc/mojo/public/tools/bindings/mojom_bindings_generator.py
> create mode 100644 utils/ipc/mojo/public/tools/bindings/mojom_bindings_generator_unittest.py
> create mode 100755 utils/ipc/mojo/public/tools/bindings/mojom_types_downgrader.py
> create mode 100755 utils/ipc/mojo/public/tools/bindings/validate_typemap_config.py
> create mode 100644 utils/ipc/mojo/public/tools/mojom/README.md
> create mode 100755 utils/ipc/mojo/public/tools/mojom/check_stable_mojom_compatibility.py
> create mode 100755 utils/ipc/mojo/public/tools/mojom/check_stable_mojom_compatibility_unittest.py
> create mode 100644 utils/ipc/mojo/public/tools/mojom/const_unittest.py
> create mode 100644 utils/ipc/mojo/public/tools/mojom/enum_unittest.py
> create mode 100644 utils/ipc/mojo/public/tools/mojom/mojom/BUILD.gn
> create mode 100644 utils/ipc/mojo/public/tools/mojom/mojom/__init__.py
> create mode 100644 utils/ipc/mojo/public/tools/mojom/mojom/error.py
> create mode 100644 utils/ipc/mojo/public/tools/mojom/mojom/fileutil.py
> create mode 100644 utils/ipc/mojo/public/tools/mojom/mojom/fileutil_unittest.py
> create mode 100644 utils/ipc/mojo/public/tools/mojom/mojom/generate/__init__.py
> create mode 100644 utils/ipc/mojo/public/tools/mojom/mojom/generate/constant_resolver.py
> create mode 100644 utils/ipc/mojo/public/tools/mojom/mojom/generate/generator.py
> create mode 100644 utils/ipc/mojo/public/tools/mojom/mojom/generate/generator_unittest.py
> create mode 100644 utils/ipc/mojo/public/tools/mojom/mojom/generate/module.py
> create mode 100644 utils/ipc/mojo/public/tools/mojom/mojom/generate/module_unittest.py
> create mode 100644 utils/ipc/mojo/public/tools/mojom/mojom/generate/pack.py
> create mode 100644 utils/ipc/mojo/public/tools/mojom/mojom/generate/pack_unittest.py
> create mode 100644 utils/ipc/mojo/public/tools/mojom/mojom/generate/template_expander.py
> create mode 100644 utils/ipc/mojo/public/tools/mojom/mojom/generate/translate.py
> create mode 100644 utils/ipc/mojo/public/tools/mojom/mojom/generate/translate_unittest.py
> create mode 100644 utils/ipc/mojo/public/tools/mojom/mojom/parse/__init__.py
> create mode 100644 utils/ipc/mojo/public/tools/mojom/mojom/parse/ast.py
> create mode 100644 utils/ipc/mojo/public/tools/mojom/mojom/parse/ast_unittest.py
> create mode 100644 utils/ipc/mojo/public/tools/mojom/mojom/parse/conditional_features.py
> create mode 100644 utils/ipc/mojo/public/tools/mojom/mojom/parse/conditional_features_unittest.py
> create mode 100644 utils/ipc/mojo/public/tools/mojom/mojom/parse/lexer.py
> create mode 100644 utils/ipc/mojo/public/tools/mojom/mojom/parse/lexer_unittest.py
> create mode 100644 utils/ipc/mojo/public/tools/mojom/mojom/parse/parser.py
> create mode 100644 utils/ipc/mojo/public/tools/mojom/mojom/parse/parser_unittest.py
> create mode 100755 utils/ipc/mojo/public/tools/mojom/mojom_parser.py
> create mode 100644 utils/ipc/mojo/public/tools/mojom/mojom_parser_test_case.py
> create mode 100644 utils/ipc/mojo/public/tools/mojom/mojom_parser_unittest.py
> create mode 100644 utils/ipc/mojo/public/tools/mojom/stable_attribute_unittest.py
> create mode 100644 utils/ipc/mojo/public/tools/mojom/version_compatibility_unittest.py
> create mode 100755 utils/ipc/mojo/public/tools/run_all_python_unittests.py
> create mode 100644 utils/ipc/tools/diagnosis/crbug_1001171.py
>
> --
> 2.27.0
>
More information about the libcamera-devel
mailing list