[Discussion] Multiple IPAs support

[Cheng-Hao Yang]

Hi Kieran,

Thanks for the quick response!

On Wed, Jul 24, 2024 at 3:42 PM Kieran Bingham <
kieran.bingham at ideasonboard.com> wrote:

> Quoting Cheng-Hao Yang (2024-07-24 07:36:06)
> > Hi folks,
> >
> > I want to discuss the support of multiple IPAs within a pipeline handler,
> > before I upload patches, as I think there are different approaches. It's
> > needed because some chromeos models (like intelipu7) need both cpu & gpu
> > IPAs, and they might have different mojo interfaces.
>
> That sounds like multiple distinct processing blocks within a pipeline
> handler.
>
> But do you mean the IPA will run 'on the GPU'? Or that it will 'control'
> processing on the GPU?
>
>
In CrOS, IPA runs in a sandbox. GPU IPA has access to GPU [1]. I believe it
can
send some requests to GPU hardware.

[1]:
https://source.chromium.org/chromiumos/chromiumos/codesearch/+/main:src/platform2/camera/common/init/cros-camera-gpu-algo.conf;l=29


> I would suspect (but probably need some block diagrams) that they should
> be broken up into a more generic set of pipelines and the 'main
> pipeline handler' would then construct and connect everything.
>
>
> I can foresee a similar thing with the soft ISP development ... it could
> be added as an extra processing pipline to existing platforms. For
> instance it could be added to the IPU3 to support the IR sensor...
>
> So some sort of generic solution or design is likely desireable here.
>
>
> > Currently in the `meson.build`s, they restrict the IPA name to be the
> same
> > as a pipeline handler name [1] [2].
> >
> > Here are some approaches that I can think of:
> >
> > 1. Besides the ipa that has the same name as pipeline handler name, build
> > the one that has a suffix `_gpu` or so. For example, build `ipu7` &
> > `ipu7_gpu` for pipeline handler `ipu7`.
> > This approach limits the number of IPAs to be two though, and we might
> need
> > to change the design again if there are other requirements in the future.
>
> Indeed, this just sounds like a hack/shortcut to me.
>
>
> > 2. Build exactly the IPAs specified in the meson option: `ipas` (or we
> add
> > another meson option). IIUC, currently the default value of meson option
> > `ipas` contains all the possible values, and it might be the case for
> most
> > of the build configs for the current use cases.
> > This approach might require developers to update their build configs
> > accordingly.
>
> If you're building distinct IPA modules - I think this is fine. Don't
> worry about developers updating their build configs. That sounds like
> worrying that developers would have to apply patches if they want to
> change the code to me ...
>
> It should all be covered by the build system. And -Dipas=all/auto should
> 'do the right thing' in what ever is designed.
>
>
Just to make sure we're aligned: This approach needs to update the logic
to use the meson option `ipas`. And yes, the previous logic is confusing to
me: It's like filtering out the options provided by developers by pipeline
names.


>
> > 3. Build more than one shared library in one `src/ipa/${IPA_NAME}`
> > directory. We can register more than one mojom interface in a pipeline
> > handler name [3], so that the build can generate multiple IPA
> > proxies/workers.
> > This approach doesn't need to change the meson build files or logic. I
> > haven't tried it myself though. Maybe there are some missing parts.
>
> I haven't looked into the details / implementation here - but this
> sounds more aligned to the approach I would take - where for complex
> processing pipelines I think there should still be a top level pipeline
> handler in charge, but that it can construct multiple internal
> processing pipes to manage the whole streams.
>
>
Actually I just tried myself with a POC patch:
https://chromium-review.googlesource.com/c/chromiumos/third_party/libcamera/+/5737834/1/include/libcamera/ipa/meson.build#75

Seems to be working. Just need to make the map into a list instead.


> I would envisage that sometimes those internal pipes might have their
> own IPA interface:
>
>
>  ┌────────────────────────────────────────────────┐
>  │  IPU3 Pipeline Handler                         │
>  │ ┌────────────────────────────────────────────┐ │
>  │ │  Pipe1 (Bayer)                             │ │
>  │ │ ┌────────┐    ┌────────┐    ┌────────┐     │ │ ┌─────────┐
>  │ │ │ Bayer  │    │        │    │        │     │ │ │         │
>  │ │ │ Sensor ├───►│  CIO2  ├───►│  ImgU  ├─────┼─┼─► Camera1 │
>  │ │ │        │    │        │    │        │     │ │ │         │
>  │ │ └────────┘    └────────┘    └──┬──┬──┘     │ │ └─────────┘
>  │ │                                │  │        │ │
>  │ │                             ┌──▼──┴──┐     │ │
>  │ │                             │        │     │ │
>  │ │                             │IPU3-IPA│     │ │
>  │ │                             │        │     │ │
>  │ │                             └────────┘     │ │
>  │ │                                            │ │
>  │ └────────────────────────────────────────────┘ │
>  │                                                │
>  │ ┌────────────────────────────────────────────┐ │
>  │ │  Pipe2 (IR)                                │ │
>  │ │ ┌────────┐    ┌────────┐    ┌────────┐     │ │ ┌─────────┐
>  │ │ │   IR   │    │        │    │        │     │ │ │         │
>  │ │ │ Sensor ├───►│  CIO2  ├───►│Soft-ISP├─────┼─┼─► Camera2 │
>  │ │ │        │    │        │    │        │     │ │ │         │
>  │ │ └────────┘    └────────┘    └──┬──▲──┘     │ │ └─────────┘
>  │ │                                │  │        │ │
>  │ │                             ┌──▼──┴──┐     │ │
>  │ │                             │        │     │ │
>  │ │                             │Soft-IPA│     │ │
>  │ │                             │        │     │ │
>  │ │                             └────────┘     │ │
>  │ │                                            │ │
>  │ └────────────────────────────────────────────┘ │
>  │                                                │
>  └────────────────────────────────────────────────┘
>
> I've used the IPU3 IR cam as an example above because that's hardware I
> have and know and I don't currently have visibility of what design you
> are facing. If it sort of matches your issue then great, if not - then
> it would help to know more about the underlying pipelines you are trying
> to construct.
>
>
Do you mean that we should try to use only one IPA in each pipeline handler
(, in your case, "pipe1" and "pipe2"), and there's another "main pipeline
handler"
(, which itself doesn't own an IPA) calling "pipe1" and "pipe2", depending
on
the requests?

Yating, from Intel, told me that the GPU IPA supports an optional feature
of image
processing, which will be used in a stream (that doesn't need to call CPU
IPA).
However, I believe the source is from the same sensor. Please correct me if
I'm
wrong :)


>
> Regards
> --
> Kieran
>
> > Let me know your opinions. Thanks!
> >
> > BR,
> > Harvey
> >
> > [1]:
> >
> https://git.libcamera.org/libcamera/libcamera.git/tree/include/libcamera/ipa/meson.build#n88
> > [2]:
> >
> https://git.libcamera.org/libcamera/libcamera.git/tree/src/ipa/meson.build#n47
> > [3]:
> >
> https://git.libcamera.org/libcamera/libcamera.git/tree/include/libcamera/ipa/meson.build#n64


BR,
Harvey
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.libcamera.org/pipermail/libcamera-devel/attachments/20240724/e24cd9d6/attachment.htm>