[Discussion] Multiple IPAs support

[Kieran Bingham]

Quoting Cheng-Hao Yang (2024-07-24 07:36:06)
> Hi folks,
> 
> I want to discuss the support of multiple IPAs within a pipeline handler,
> before I upload patches, as I think there are different approaches. It's
> needed because some chromeos models (like intelipu7) need both cpu & gpu
> IPAs, and they might have different mojo interfaces.

That sounds like multiple distinct processing blocks within a pipeline handler.

But do you mean the IPA will run 'on the GPU'? Or that it will 'control'
processing on the GPU?

I would suspect (but probably need some block diagrams) that they should
be broken up into a more generic set of pipelines and the 'main
pipeline handler' would then construct and connect everything.


I can foresee a similar thing with the soft ISP development ... it could
be added as an extra processing pipline to existing platforms. For
instance it could be added to the IPU3 to support the IR sensor...

So some sort of generic solution or design is likely desireable here.


> Currently in the `meson.build`s, they restrict the IPA name to be the same
> as a pipeline handler name [1] [2].
> 
> Here are some approaches that I can think of:
> 
> 1. Besides the ipa that has the same name as pipeline handler name, build
> the one that has a suffix `_gpu` or so. For example, build `ipu7` &
> `ipu7_gpu` for pipeline handler `ipu7`.
> This approach limits the number of IPAs to be two though, and we might need
> to change the design again if there are other requirements in the future.

Indeed, this just sounds like a hack/shortcut to me.


> 2. Build exactly the IPAs specified in the meson option: `ipas` (or we add
> another meson option). IIUC, currently the default value of meson option
> `ipas` contains all the possible values, and it might be the case for most
> of the build configs for the current use cases.
> This approach might require developers to update their build configs
> accordingly.

If you're building distinct IPA modules - I think this is fine. Don't
worry about developers updating their build configs. That sounds like
worrying that developers would have to apply patches if they want to
change the code to me ...

It should all be covered by the build system. And -Dipas=all/auto should
'do the right thing' in what ever is designed.


> 3. Build more than one shared library in one `src/ipa/${IPA_NAME}`
> directory. We can register more than one mojom interface in a pipeline
> handler name [3], so that the build can generate multiple IPA
> proxies/workers.
> This approach doesn't need to change the meson build files or logic. I
> haven't tried it myself though. Maybe there are some missing parts.

I haven't looked into the details / implementation here - but this
sounds more aligned to the approach I would take - where for complex
processing pipelines I think there should still be a top level pipeline
handler in charge, but that it can construct multiple internal
processing pipes to manage the whole streams.

I would envisage that sometimes those internal pipes might have their
own IPA interface:


 ┌────────────────────────────────────────────────┐
 │  IPU3 Pipeline Handler                         │
 │ ┌────────────────────────────────────────────┐ │
 │ │  Pipe1 (Bayer)                             │ │
 │ │ ┌────────┐    ┌────────┐    ┌────────┐     │ │ ┌─────────┐
 │ │ │ Bayer  │    │        │    │        │     │ │ │         │
 │ │ │ Sensor ├───►│  CIO2  ├───►│  ImgU  ├─────┼─┼─► Camera1 │
 │ │ │        │    │        │    │        │     │ │ │         │
 │ │ └────────┘    └────────┘    └──┬──┬──┘     │ │ └─────────┘
 │ │                                │  │        │ │
 │ │                             ┌──▼──┴──┐     │ │
 │ │                             │        │     │ │
 │ │                             │IPU3-IPA│     │ │
 │ │                             │        │     │ │
 │ │                             └────────┘     │ │
 │ │                                            │ │
 │ └────────────────────────────────────────────┘ │
 │                                                │
 │ ┌────────────────────────────────────────────┐ │
 │ │  Pipe2 (IR)                                │ │
 │ │ ┌────────┐    ┌────────┐    ┌────────┐     │ │ ┌─────────┐
 │ │ │   IR   │    │        │    │        │     │ │ │         │
 │ │ │ Sensor ├───►│  CIO2  ├───►│Soft-ISP├─────┼─┼─► Camera2 │
 │ │ │        │    │        │    │        │     │ │ │         │
 │ │ └────────┘    └────────┘    └──┬──▲──┘     │ │ └─────────┘
 │ │                                │  │        │ │
 │ │                             ┌──▼──┴──┐     │ │
 │ │                             │        │     │ │
 │ │                             │Soft-IPA│     │ │
 │ │                             │        │     │ │
 │ │                             └────────┘     │ │
 │ │                                            │ │
 │ └────────────────────────────────────────────┘ │
 │                                                │
 └────────────────────────────────────────────────┘

I've used the IPU3 IR cam as an example above because that's hardware I
have and know and I don't currently have visibility of what design you
are facing. If it sort of matches your issue then great, if not - then
it would help to know more about the underlying pipelines you are trying
to construct.


Regards
--
Kieran

> Let me know your opinions. Thanks!
> 
> BR,
> Harvey
> 
> [1]:
> https://git.libcamera.org/libcamera/libcamera.git/tree/include/libcamera/ipa/meson.build#n88
> [2]:
> https://git.libcamera.org/libcamera/libcamera.git/tree/src/ipa/meson.build#n47
> [3]:
> https://git.libcamera.org/libcamera/libcamera.git/tree/include/libcamera/ipa/meson.build#n64