[libcamera-devel] [PATCH v3 1/1] Use tracing with perfetto in ChromeOS

[Cheng-Hao Yang]

Hi libcamera developers,

I've consulted with Google Perfetto team, and yes, we agree
that it makes more sense to have one debugging tool as the
source of truth and generate code for the other tool, to ease
the development effort in the future.

Please also share your concerns on the two approaches:

On Sat, Sep 14, 2024 at 1:14 AM Lalit Maganti <lalitm at google.com> wrote:

>
>
> On Fri, 13 Sept 2024 at 18:08, Cheng-Hao Yang <chenghaoyang at google.com>
> wrote:
>
>> Thank you Lalit for the quick and thorough response,
>>
>> Do you mind if I directly forward your messages to libcamera upstream
>> directly?
>> I'll wait for a couple of days until everyone has the chance to speak
>> out, and do a brief summary though.
>>
>
> Sure that sounds good.
>
>
> Responses inline:
>>
>> On Fri, Sep 13, 2024 at 6:19 PM Lalit Maganti <lalitm at google.com> wrote:
>>
>>> Also +Daniele Di Proietto <ddiproietto at google.com> is the C SDK better
>>> at being closer to LTTng in ethos? I know it allows e.g. defining custom
>>> protos directly in code and writing out those? Would that be a better fit
>>> for what these folks are trying to do?
>>>
>>
>> Yeah I forgot to mention that at least in CrOS, we're using cros-built
>> Perfetto C SDK in libcamera.
>>
>>
>>>
>>> On Fri, 13 Sept 2024, 10:54 Lalit Maganti, <lalitm at google.com> wrote:
>>>
>>>> Please do feel free to setup some time with us (you can book slots at
>>>> go/perfetto-office-hours
>>>> <https://goto.google.com/perfetto-office-hours>) or reach out by IM
>>>> btw to speed iteration on this up if you have more questions/want to
>>>> discuss more.
>>>>
>>>> On Fri, 13 Sept 2024 at 10:53, Lalit Maganti <lalitm at google.com> wrote:
>>>>
>>>>> Hi Harvey,
>>>>>
>>>>> Responses inline.
>>>>>
>>>>> On Fri, 13 Sept 2024 at 08:56, Cheng-Hao Yang <chenghaoyang at google.com>
>>>>> wrote:
>>>>>
>>>>>>  Hi Primiano and Perfetto developers,
>>>>>>
>>>>>> This is Harvey, working in CrOS Camera team on libcamera project
>>>>>> <https://libcamera.org/>. I need some help from you, as I'm working
>>>>>> on supporting Perfetto in libcamera upstream, which currently uses lttng as
>>>>>> the tracing tool. Libcamera upstream has the concern that Perfetto is not
>>>>>> added in the package repositories in linux distributions, and the way to
>>>>>> build it in linux doesn't match the philosophy of Linux distributions.
>>>>>>
>>>>>> We're considering supporting both tracing tools in libcamera, while
>>>>>> libcamera upstream strongly prefers to have only one source of truth, which
>>>>>> means to maintain only one version of trace macros/configs, and generate
>>>>>> macros for the other (or both of them). This email thread contains the
>>>>>> patch <https://patchwork.libcamera.org/patch/18040/> of 3rd approach
>>>>>> listed below.
>>>>>>
>>>>>> I need your help on mainly two things:
>>>>>> 1. An official reply to confirm that it is or isn't a priority to add
>>>>>> Perfetto in the package repositories in linux distributions (I assume it
>>>>>> isn't though). After providing this info, I can confirm with libcamera
>>>>>> upstream our consensus of Perfetto support and responsibilities in the
>>>>>> future.
>>>>>>
>>>>>
>>>>> It is not a priority (and historically not been a priority) *but* at
>>>>> least for Debian we already did a bunch of the work to make a package for
>>>>> distributing Perfetto (see here
>>>>> <https://source.corp.google.com/h/android/platform/superproject/main/+/main:external/perfetto/debian/;l=1?q=f:perfetto%20f:debian&sq=repo:android%2Fplatform%2Fsuperproject%2Fmain%20b:main>).
>>>>> If someone from your side was willing to do the work to submit this
>>>>> package, we'd be happy to keep it updated and maintained going forward.
>>>>>
>>>>> So for us:
>>>>>
>>>>>    1. Is someone from your side willing to take the work we've done
>>>>>    and push it to completion?
>>>>>    2. Are there any other distros apart from Debian which libcamera
>>>>>    would need us to be in?
>>>>>
>>>>> 1. Unfortunately, I don't think we have the bandwidth to help push it
>> to completion in the current stage...
>> 2. I believe there are at least Fedora and Raspberry. I'll let the
>> libcamera upstream developers correct me.
>>
>
> That's fair. I think in that case, it would be difficult for us to get it
> supported to the level that they want as well unfortunately. We're already
> resource constrained as it is and with Android and AL work, I don't think
> we'll have the bandwidth to deal with all the work for upstreaming into
> debian and all the other distros.
>
>
>>>>>
>>>>>> 2. Take a brief look at the approaches below, and check if I missed
>>>>>> something. You may focus on the 1st or 5th, which I'm currently inclined
>>>>>> with.
>>>>>>
>>>>>
>>>>>> Here's the list of approaches that we can think of:
>>>>>>
>>>>>> 1. Generate Perfetto code from lttng macros.
>>>>>>   Pros: Easy to update libcamera tracepoint macros.
>>>>>>   Cons: Add maintenance work with the additional parser and code
>>>>>> generator; hard to implement Perfetto features that lttng doesn't
>>>>>> have.
>>>>>>
>>>>>
>>>>> An option but as you say, it does not support a bunch of features in
>>>>> Perfetto which LTTNG does not have.
>>>>>
>>>>
This one, which you might prefer.


>
>>>>>
>>>>>>
>>>>>> 2. Create a new language/config from which we generate both Perfetto
>>>>>> and lttng macros.
>>>>>>   Pros: Easy to update libcamera tracepoint macros.
>>>>>>   Cons: Probably an overkill; hard to cover all features from both
>>>>>> Perfetto and lttng.
>>>>>>
>>>>>
>>>>> Yeah would not go here.
>>>>>
>>>>> 3. As this patch implements: have duplicated libcamera tracepoint
>>>>>> definitions.
>>>>>>   Pros: Easy to add libcamera tracepoint macros to cover all features
>>>>>> from Perfetto & lttng.
>>>>>>   Cons: Developer needs to update both sides when the definition of
>>>>>> libcamera tracepoints changes; Unused macro definitions might be
>>>>>> adopted in both sides, if a feature is only available in one trace
>>>>>> library.
>>>>>>
>>>>>
>>>>> Inevitably things would get out of sync, I can see why they are
>>>>> against this.
>>>>>
>>>>> 4. Add another set of libcamera tracepoints for Perfetto, or just use
>>>>>> Perfetto macros directly in libcamera's source code.
>>>>>>   Pros: Two isolated trace stacks that don't influence each other. All
>>>>>> features can be implemented easily.
>>>>>>   Cons: Duplicated trace code in source code that might lead to
>>>>>> confusion; developers also need to update both macros when a
>>>>>> new trace is needed.
>>>>>>
>>>>>
>>>>> As above, same deal.
>>>>>
>>>>>
>>>>>> 5. Add libcamera macro usages, and generate lttng macros with codegen.
>>>>>>   Pros: It supports all perfetto features.
>>>>>>   Cons: Some lttng-only features (like ctf_integer_hex and ctf_enum)
>>>>>> are not available without weird hacks.
>>>>>>
>>>>>
>>>>> If there are things like this, I think we'd be happy to work with you
>>>>> on getting those gaps resolved (if it's feasible, I haven't looked into the
>>>>> details in this particular case, it might not be).
>>>>>
>>>>
>> Yeah it might be the best case for us now.
>>
>
> Cool sounds good :)
>

Or this one, that Google prefers. We might need to find
a way to support hex & enum in this case though.


>
> --------------
>>>>>>
>>>>>> As libcamera upstream has a strong opinion, I might end up going with
>>>>>> the 1st or 5th approach. The reason to consider the 1st is that currently
>>>>>> the libcamera trace points might be better off with Perfetto counters (or
>>>>>> slices with minimum lifetime), instead of slices. Please correct me if I'm
>>>>>> wrong, but I think Perfetto slices are most useful for nested function
>>>>>> calls. However, a lot of libcamera operations are triggered by callbacks
>>>>>> (in libcamera, Signal).
>>>>>> I tried Perfetto flow yesterday, and it works on libcamera. We can
>>>>>> possibly add a hack in the potential codegen to set it up though.
>>>>>>
>>>>>
>>>>> Perfetto slices are not just for sync function calls: you can create
>>>>> synthetic sync stacks from them based on callback lifecycles: in
>>>>> Chrome/ChromeOS where everything is based on TaskRunner (thread loops) and
>>>>> callbacks, Perfetto is used very extensively and along with flows, allows
>>>>> visualizing async stuff just fine.
>>>>>
>>>>> Counters are not a good fit for visualizing latencies of things which
>>>>> given the nature of libcamera is almost certainly what you want.
>>>>>
>>>>
>> Thanks for the info! Flow is very useful indeed. I'll do some more
>> studies on the synthetic sync stacks, and check the best way to add traces
>> in libcamera. It might be very different from the current lttng ones
>> though, I believe.
>>
>
> I mean an option here is to do LTTNG -> Perfetto conversion and then add
> some custom instrumentation in the places where it really helps having
> custom Perfetto instrumentation maybe? Maybe the libcamera folk would be
> more amenable to that?
>
>
>>
>>>>> Let me know if there's anything I haven't clarified yet.
>>>>>>
>>>>>> Many thanks!
>>>>>> Harvey
>>>>>>
>>>>>> ---------- Forwarded message ---------
>>>>>> From: Laurent Pinchart <laurent.pinchart at ideasonboard.com>
>>>>>> Date: Wed, Sep 11, 2024 at 7:10 PM
>>>>>> Subject: Re: [libcamera-devel] [PATCH v3 1/1] Use tracing with
>>>>>> perfetto in ChromeOS
>>>>>> To: Cheng-Hao Yang <chenghaoyang at chromium.org>
>>>>>> Cc: Paul Elder <paul.elder at ideasonboard.com>, Chinglin Yu <
>>>>>> chinglinyu at chromium.org>, <libcamera-devel at lists.libcamera.org>,
>>>>>> Harvey Yang <chenghaoyang at google.com>
>>>>>>
>>>>>>
>>>>>> Hi Harvey,
>>>>>>
>>>>>> On Wed, Sep 11, 2024 at 06:05:18PM +0800, Cheng-Hao Yang wrote:
>>>>>> > On Tue, Sep 10, 2024 at 12:30 AM Paul Elder wrote:
>>>>>> > > Hi Harvey,
>>>>>> > >
>>>>>> > > <snip>
>>>>>> > >
>>>>>> > > >     So in lttng we have a TRACEPOINT_EVENT_CLASS, which defines
>>>>>> arguments
>>>>>> > > >     and fields and how to obtain the fields from the args. We
>>>>>> can
>>>>>> > > >     instantiate tracepoints with TRACEPOINT_EVENT_INSTANCE,
>>>>>> where we only
>>>>>> > > >     have to specify the args, and the fields are already
>>>>>> defined and come
>>>>>> > > >     from the class. This reduces duplication in the tracepoint
>>>>>> definitions.
>>>>>> > > >
>>>>>> > > >     I'm wondering from what you've implemented here that
>>>>>> perfetto doesn't
>>>>>> > > >     support this? I don't see anything in the perfetto docs
>>>>>> that suggest
>>>>>> > > >     that this is possible either. Although we can mitigate this
>>>>>> issue by
>>>>>> > > >     generating the perfetto tracepoints :) (this is another
>>>>>> reason why I
>>>>>> > > >     think we should use the lttng tracepoints definition as the
>>>>>> main one and
>>>>>> > > >     generate the perfetto ones from those)
>>>>>>
>>>>>
>>>>> The LTTNG model is very very different from the Perfetto model. For
>>>>> us, the equivalent of LTTNG "tracepoints" are defined directly in Perfetto
>>>>> source code. You wouldn't see it on the libcamera side as you use the
>>>>> macros to directly write these (they're really protos) without having a
>>>>> "definition" in the libcamera source code.
>>>>>
>>>>>
>>>>>> > > >
>>>>>> > > > I've discussed with ChromeOS Perfetto TL chinglinyu@, and we
>>>>>> found that
>>>>>> > > > there are limited features in lttng [1], and if we use the
>>>>>> lttng tracepoints
>>>>>> > > > definition as the source of truth, it's hard/weird to implement
>>>>>> some Perfetto
>>>>>> > > > concepts, like nested slices (lifetime of trace events) and
>>>>>> flows [2].
>>>>>> > >
>>>>>> > > Ah ok, I see the problem.
>>>>>> > >
>>>>>> > > > It's not impossible, as you mentioned above, that we could
>>>>>> detect the prefix
>>>>>> > > > and suffix of each libcamera/lttng macro. However, it's a weird
>>>>>> code parser
>>>>>> > > > and generator design.
>>>>>> > > >
>>>>>> > > > [1]: https://lttng.org/man/3/lttng-ust/v2.8/
>>>>>> > > > [2]: https://perfetto.dev/docs/instrumentation/track-events
>>>>>> > >
>>>>>> > > <snip>
>>>>>> > >
>>>>>> > > > I agree that in the current functionalities, it's not that hard
>>>>>> to implement
>>>>>> > > > a homemade C/python parser, and generate Perfetto code with
>>>>>> jinja2.
>>>>>> > > > However, we not only "limit" the functionalities of Perfetto
>>>>>> with lttng's
>>>>>> > > > definition, but also add another layer to be maintained.
>>>>>> > > > It's true that having duplication of libcamera tracepoint
>>>>>> definitions
>>>>>> > > > requires more maintenance work when updating the libcamera
>>>>>> > > > tracepoints, while the effort to maintain the parser and code
>>>>>> generation
>>>>>> > > > is also non-negligible.
>>>>>> > > >
>>>>>> > > > I also think that creating a new language from which we generate
>>>>>> > > > tracepoints for both is an overkill though. It seems that we
>>>>>> don't have
>>>>>> > > > a perfect way to support both lttng and Perfetto...
>>>>>> > > >
>>>>>> > > > BTW, I tried lex and yacc to parse lttng macros, and tbh it's
>>>>>> very painful
>>>>>> > > > that it's very hard to use lex to recognize C/C++ type as a
>>>>>> token... So
>>>>>> > > > yeah if we end up deciding to take this route, a homemade
>>>>>> parser from
>>>>>> > > > scratch makes more sense to me.
>>>>>> > > >
>>>>>> > > > So let me list the approaches with pros and cons:
>>>>>> > > > 1. Generate Perfetto code from lttng macros.
>>>>>> > > >   Pros: Easy to update libcamera tracepoint macros.
>>>>>> > > >   Cons: Add maintenance work with the additional parser and code
>>>>>> > > > generator; hard to implement Perfetto features that lttng
>>>>>> doesn't have.
>>>>>> > >
>>>>>> > > This makes me wonder, if Perfetto is a superset of the features of
>>>>>> > > lttng, why not go the other way; use Perfetto tracepoints as the
>>>>>> main
>>>>>> > > authority and then generate lttng tracepoints from those.
>>>>>> > >
>>>>>> > > What are your thoughts?
>>>>>> >
>>>>>> > I don't think Perfetto MACROs is a superset of lttng MACROs. If we
>>>>>> go
>>>>>> > with this approach, there are some features dropped (like
>>>>>> integer_hex
>>>>>> > and enum, and the concept of TP_ARGS), and the code would be messy.
>>>>>> > Take request_complete_buffer as the example:
>>>>>> >
>>>>>> > ```
>>>>>> > # Perfetto
>>>>>> > TRACE_EVENT("libcamera", "request_complete_buffer",
>>>>>> >       "request_private_ptr", reinterpret_cast<uintptr_t>(req),
>>>>>> >       "cookie", req->_o<libcamera::Request>()->cookie(),
>>>>>> >       "status", req->_o<libcamera::Request>()->status(),
>>>>>> >       "buffer_ptr", reinterpret_cast<uintptr_t>(buf),
>>>>>> >       "buffer_status", buf->metadata().status);
>>>>>> > ```
>>>>>> >
>>>>>> > It'd be how perfetto implements the macro. Originally lttng has
>>>>>> only two
>>>>>> > inputs: `libcamera::Request::internal *req` and
>>>>>> > `libcamera::FrameBuffer *buf`. Unless we're going to do complicated
>>>>>> > type diagnosis and simplification, we might end up with having five
>>>>>> > inputs as well.
>>>>>>
>>>>>
I take this back. As we'll most likely have a self-defined
function that takes the two inputs with the correct C/C++
types, we might still be able to maintain the input size for
lttng macros.

BR,
Harvey


> Also, I don't know how to indicate the usage of
>>>>>> > ctf_integer_hex (for `req` and `buf`) and ctf_enum (for status).
>>>>>> >
>>>>>> > I'd still prefer the 3rd or 4th approach.
>>>>>>
>>>>>> The 4th option, duplicating trace points in the libcamera sources, is
>>>>>> the one I like the least. The 3rd option is also something I don't
>>>>>> like
>>>>>> much. It means that anyone adding trace points will need to do extra
>>>>>> work to support two trace systems. Testing it locally will be hard,
>>>>>> even
>>>>>> for build testing. We could possibly handle the build testing with CI
>>>>>> (see below), even if that will be way less convenient that doing local
>>>>>> builds. Runtime testing is likely out of question.
>>>>>>
>>>>>> I'm not comfortable with the amount of extra work this will cause to
>>>>>> everybody to support a Google-only trace system. I'm willing to find a
>>>>>> compromise though, if Google could do the necessary work needed to
>>>>>> package Perfetto in distributions, I could accept extra work on our
>>>>>> side
>>>>>> too.
>>>>>>
>>>>>> > > > 2. Creating a new language from which we generate both Perfetto
>>>>>> > > > and lttng macros.
>>>>>> > > >   Pros: Easy to update libcamera tracepoint macros.
>>>>>> > > >   Cons: Probably an overkill; hard to cover all features from
>>>>>> both
>>>>>> > > > Perfetto and lttng.
>>>>>> > > >
>>>>>> > > > 3. As this patch implements: have duplicated libcamera
>>>>>> tracepoint
>>>>>> > > > definitions.
>>>>>> > > >   Pros: Easy to add libcamera tracepoint macros to cover all
>>>>>> features
>>>>>> > > > from Perfetto & lttng.
>>>>>> > > >   Cons: Developer needs to update both sides when the
>>>>>> definition of
>>>>>> > > > libcamera tracepoints changes; Unused macro definitions might be
>>>>>> > > > adopted in both sides, if a feature is only available in one
>>>>>> trace
>>>>>> > > library.
>>>>>> > > >
>>>>>> > > > 4. Add another set of libcamera tracepoints for Perfetto, or
>>>>>> just use
>>>>>> > > > Perfetto macros directly in libcamera's source code.
>>>>>> > > >   Pros: Two isolated trace stacks that don't influence each
>>>>>> other. All
>>>>>> > > > features can be implemented easily.
>>>>>> > > >   Cons: Duplicated trace code in source code that might lead to
>>>>>> > > > confusion; developers also need to update both macros when a
>>>>>> > > > new trace is needed.
>>>>>> > > >
>>>>>> > > > I think approach 3rd and 4th are easier to maintain in the long
>>>>>> > > > term. WDYT?
>>>>>> > > >
>>>>>> > > > Also, I'd like to know if you think Perfetto will be used in
>>>>>> other linux
>>>>>> > > > distributions than ChromeOS or Android.
>>>>>> > >
>>>>>> > > If Perfetto gets added to the package repositories in linux
>>>>>> > > distributions then I think it would be used :)
>>>>>> > >
>>>>>> > > Not being able to sudo apt install perfetto is a pretty big
>>>>>> blocker.
>>>>>> >
>>>>>> > Yeah I understand, while Perfetto is self-contained and simple to
>>>>>> build
>>>>>> > in linux distros.
>>>>>>
>>>>>> Do you mean by running the tools/install-build-deps script that
>>>>>> downloads toolchain binaries from google servers ? That doesn't really
>>>>>> match the philosophy of Linux distributions.
>>>>>>
>>>>>
>>>>> Yeah not accepting this makes sense I'm sympathetic to this.
>>>>>
>>>>>
>>>>>>
>>>>>> > May we consider testing it on the gitlab CI, after we
>>>>>> > somehow add the support?
>>>>>>
>>>>>> If we add support for it I certainly would like to see it being tested
>>>>>> in CI, yes. After a quick look at the Perfetto build system, it looks
>>>>>> like adding Perfetto to the CI docker containers may not be trivial.
>>>>>> Someone will need to volunteer.
>>>>>>
>>>>>> > > > Chinglin, please help amend anything I missed as well :)
>>>>>>
>>>>>> --
>>>>>> Regards,
>>>>>>
>>>>>> Laurent Pinchart
>>>>>>
>>>>>>
>>>>>> --
>>>>>> BR,
>>>>>> Harvey Yang
>>>>>>
>>>>>> --
>>>>>> You received this message because you are subscribed to the Google
>>>>>> Groups "Perfetto-dev" group.
>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>> send an email to perfetto-dev+unsubscribe at google.com.
>>>>>> To view this discussion on the web visit
>>>>>> https://groups.google.com/a/google.com/d/msgid/perfetto-dev/CAC%3DwSGWxc4W9nZOoieTMhG9h-Kh%2BwH%3D%3DiPfNwxqd0ybNntVFWQ%40mail.gmail.com
>>>>>> <https://groups.google.com/a/google.com/d/msgid/perfetto-dev/CAC%3DwSGWxc4W9nZOoieTMhG9h-Kh%2BwH%3D%3DiPfNwxqd0ybNntVFWQ%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>>>>> .
>>>>>>
>>>>>
>>
>> --
>> BR,
>> Harvey Yang
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.libcamera.org/pipermail/libcamera-devel/attachments/20240918/98a7c7ba/attachment.htm>