[libcamera-devel] Summary of IPC/IDL meeting on #libcamera on 2020/07/28

[paul.elder at ideasonboard.com]

Hello All,

Here is a summary of the meeting that we (Tomasz, Laurent, and I) had on
#libcamera yesterday, regarding IPC and the interface between pipeline
handlers and IPAs.


** Introduction of problem

- We need IPAs to be able to run isolated in a separate process
- We want this isolation to be transparent to both the PH (Pipeline Handler)
  and the IPA
  - PH and IPA shouldn't have to care whether they're isolated or not
- During design of IPC mechanism, we wondered if we could also support custom
  functions or custom function parameter data structures
  - custom functions may be too complicated, but custom data structures might work
- Paul has a working prototype for custom data structures, with direct
  calling, though (no IPC)
- PH author would define data structures in IDL, and it would be compiled into
  header for PH and IPA to import, and some helpers for the de/serializer

** Q&As

What kind of data structures might be used?

- Today we have IPAOperationData, which is opcode, vector of words, and
  vector of ControlLists
- The IPAInterface functions that use IPAOperationData today are processEvent,
  to send custom events from PH to IPA, and queueFrameAction, for IPA to PH,
  and configure
- IPAs define operations, and what data needs to be passed
- But dealing with opcodes and indexes is not nice, plus later on we have to
  pass fds, so it will become even less nice
- Without isolation we wouldn't need IDL, it would just be custom data
  structures

Do you have some examples of what data would be passed this way?

- Example in src/ipa/raspberrypi/raspberrypi.cpp:193 [1] for sender, and
  src/libcamera/pipeline/raspberrypi/raspberrypi.cpp:1156 [2] for receiver
- Sender stores data in lines 225-229 and 227-279
- if RPI_IPA_CONFIG_STAGGERED_WRITE is set, there's data in data[0:2]
- if RPI_IPA_CONFIG_SENSOR is set, there's data in controls[0]
- Clearly it can become more complex, such as "ah, bit 0 is set, so foo is in
  data[0], but 1 is set, so bar is in data[1] - oh wait, now bit 0 is not
  set, only bit 1 is, so there's no foo, but bar is now in data[0]"
- It's essentially manual serialization with nullable fields

Couldn't some kind of key/value set be used?

- It would be hard to serialize

** Control flow between IPA and PH

- The flow *must* be controlled by the PH, and not the IPA
  - For open-source IPAs it's not much of an issue, but if the author of a PH
    is also the author of a closed-source IPA, then there's a risk
  - The PH should be asking the IPA for various computations, and not the
    other way around, with the IPA sending actions to the PH to be executed
- Currently, the RPi PH is correctly doing the former; eg. PH sends
  RPI_IPA_EVENT_SIGNAL_ISP_PREPARE to ask the IPA for ISP params for next
  frame, and IPA replies with RPI_IPA_ACTION_V4L2_SET_ISP and
  RPI_IPA_ACTION_RUN_ISP/_AND_DROP_FRAME
- So we want (for the PH) to accept "I need ISP params for next frame - here
  you are" while rejecting "what should I do next? - this"
  - How can we enforce this? Can it be enforced technically or only through
    review?
    - Tomasz thinks that disallowing custom data structures is required to
      enforce technically
      - maybe a request-reply model?
    - Laurent thinks that we can try to standardize (the PH-IPA data
      structures) as much as possible and later allow more use csaes, but
      they must be taken into account for IPC design today, or else
      rewrite a lot of code later
      - maybe enforce request-reply/ies model, technically where possible,
        and through review otherwise
    - Paul thinks that technical enforcement is impossible with custom data
      structures (due to reduction to the halting problem), and that we need
      custom data structures for flexibility, cleanliness and
      future-proofing of PH-IPA API - review is the only possible enforcement
  - Might disallowing spontaneous messages from the IPA work? Only allow
    request-reply patterns
    - A non-conforming IPA could then just always reply to the first one,
      since we have uses cases where one request can require multiple replies
      - eg. IPA could have a background thread for expensive operations, and
        sensor params might be sent to PH as soon as they're ready, while ISP
        params wait a bit

** Custom data structures

Can all the requests and replies carry standardized data, or do we need
generic containers (like IPAOperationData) or custom data structures?

- key/value pair map easier to serialize?
  - Only if the values are all primitives; if they're structs or vectors or maps or
    a combination, then we have a problem
- We can use data definition in an IDL to generate serialization code
  - could be useful with or without custom data structures
  - the IDL could keep the interface in sync automatically

** General interface between IPA and libcamera

What interface to provide to close-source IPAs?

- closed-source IPAs are reluctant to link to libcamera, so we can't use C++
  API with classes that require linking to libcamera
- So we have three choices:
  - C++ API without classes that require linking to libcamera (fully defined
    in header files)
  - C API
  - binary serialized buffer (here's the data, now good luck)
    - not very nice
    - and libcamera would need to protect against potentially malformed
      binary data
  - Some kind of standard IPC, eg. mojo
- Since the IPC mechanism is completely within the IPAProxy, maybe we could
  have different IPAProxies for different IPC mechanisms?

** Links

- [1] https://source.chromium.org/chromiumos/chromiumos/codesearch/+/master:src/third_party/libcamera/src/ipa/raspberrypi/raspberrypi.cpp;l=193
- [2] https://source.chromium.org/chromiumos/chromiumos/codesearch/+/master:src/third_party/libcamera/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp;l=1156


---

Paul