Help Re. Recommended Linux Distribution for libcamera Development for i.MX8

[w.robertson at cairnwater.com]

Hi Laurent,

Thank you very much!

Would it be a help for me to typeset your explanation into HTML for the 
https://libcamera.org/ website so that it could help others? It's much 
clearer and much more up-to-date than many of the explanations that are 
floating around and an enormous help to someone like me from an 
application development background with little embedded experience.

> I've attached my configuration file
> for builroot 2024.11.1 to this e-mail, you can use it as a starting
> point.

Thank you very much. I think the attachment got stripped from the email 
- would it be OK to send it again? It might be my email system - sorry 
about that.

> While
> many vendors provide their own Linux distributions, BSPs and SDKs and
> market them as easy to use, turnkey options, I adhere to the (old ?)
> school of thoughts that believes understanding what goes behind the
> scenes is important. You can then make a fully informed decision on
> which components to pick and how much manual control and customization
> you need.

Yes - I've often seen technologies which aim to allow developers to work 
without understanding what's going on behind the scenes 
seem-like-a-good-idea-at-the-time but sneak up to bite their victims on 
the backside later on in terms of unforeseen problems or unforeseen 
costs.

It's also very important for something like a wildlife camera where what 
is and isn't loaded at boot time may have to be carefully controlled.

The most common boot loader for Arm-based embedded Linux systems
> is U-Boot. It's a well established de facto standard in the industry,
> and is what NXP uses for the i.MX8MP. Alternatives exist, but I really
> wouldn't recommend them if you're not already experienced with embedded
> Linux systems.

Thank you very much - that's an enormous help to know that U-Boot is the 
de facto standard and used by NXP.

> Arm has standardized the ARMv8 secure boot flow and split it in 
> multiple
> stages, named BL1 (the boot ROM in the SoC, the only component that
> can't be updated), BL2 (second stage boot loader, implemented by U-Boot
> and named SPL), BL31 (trusted firmware, a reference implementation made
> by Arm is available at [1]), and BL33 (the last boot loader stage, 
> known
> as untrusted boot loader, and implemented by U-Boot proper). There's
> plenty of information online that describes the boot process, for
> instance [2].
> 
> The boot loader is responsible for initializing all low-level hardware
> required to load the kernel (system DRAM, permanent storage such as
> eMMC or SD cards, ...). It then loads the kernel, and passes execution
> to it. Parts of the boot loader can remain persistent in system DRAM
> after loading the kernel, and offer runtime services to the kernel
> (control of power domains, secure operations such as access to crypto
> engines, ...).
> 
> [1] https://github.com/TrustedFirmware-A/trusted-firmware-a
> [2] https://www.youtube.com/watch?v=GXFw8SV-51g

Thank you very much - that's very good to know how the steps fit 
together to finally pass control to the kernel.

> # Device tree
> 
> Unlike PCI or USB devices that can be auto-detected at runtime, most
> devices inside the SoC, or connected through non-discoverable buses 
> such
> as I2C or SPI, need to be declared to the kernel. The majority of Arm
> systems use Device Tree for that purpose.
> 
> The device tree specification ([3]) defines a textual source format and
> a compiled binary format. Device tree bindings then define (as YAML
> schemas) how each particular device is described. The bindings
> maintained as part of the Linux kernel sources ([4]). Finally, as large
> set of device tree sources describing various boards is available in 
> the
> kernel source tree as well ([5]). Vendors can also maintain their 
> device
> trees outside of the kernel repository.
> 
> Device trees are compiled to a binary DTB file and stored in media
> accessible by the boot loader. They are loaded by the boot loader as
> part of the boot process, and passed to the Linux kernel when starting
> it.
> 
> [3] https://www.devicetree.org/
> [4] 
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/devicetree/bindings
> [5] 
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/arm64/boot/dts/
> 

Thank you very much. It's a big help to know that some vendors maintain 
their DTs outside of the kernel source - I'd been thinking that if all 
possible DTs were included in the kernel repository itself it would 
become very large.

It's also a big help to me to understand that DTs have to be available 
to the boot loader as well as to the kernel.

> # Linux kernel
> 
> The kernel is responsible for controlling the hardware and providing
> services to userspace. While there is only one Linux kernel, SoC and
> board vendor typically fork it to add a set of modifications, from
> changes in the kernel core to additional device drivers. This set can 
> be
> as small as a handful of patches or as big as millions of lines,
> depending on the vendor. Vendor kernels are often referred to as BSP
> (Board Support Package) kernels.
> 
> After being loaded in system DRAM by the boot loader, the kernel gets
> executed and goes through a fairly long initialization phase. Once the
> initialization is complete, the kernel mounts the root file system,
> loads the init binary, and executes it.

Thank you. Is there usually any easy way to see all the steps that a 
vendor has doen to modify the SoC vendor's kernel or the steps the SoC 
vendor has doen to modify the original kernel? (Or just do a diff 
between a vendor's kernel and the original?)

I get the feeling that a lot of the Debix Linux distributions and Yocto 
build are based on the NXP distributions and Yocto and I was interested 
in seeing what the differences were.

I guess the more modification there is in a vendor's kernel the more 
risk there is that a developer may become tied to their SoM or their SoC 
so it's good for a developer to have a feel for this rather than going 
into it blindly?

> # Distributions and build frameworks
> 
> Linux distributions typically provide all those components in the form
> of an installable image. Depending on the type of distribution, you can
> download a ready-made binary image, or you may need to build one. My
> experience is that ready-made images, provided by distributions
> originating from desktop systems (Debian, Ubuntu, Fedora, ...) may be
> simpler to install, but more difficult to customize. I'm certainly
> biased though, I'm quite sure that someone more familiar with Debian or
> Fedora development would have less trouble compiling a custom kernel
> using the packaging infrastructure provided by those distributions. As
> I'm not such a person, I'll describe alternatives.
> 
> In addition to ready-made images, there are build automation frameworks
> that help you build a system image and a corresponding 
> cross-compilation
> environment (often referred to as SDK). Two common frameworks are
> buildroot and OpenEmbedded (often referred to as Yocto, although that's
> a misuse of the term). I'm more familiar with buildroot, and that's 
> what
> I use with my Debix board, so I will focus on that framework.
> 
> Buildroot facilitates the process of building a complete set of
> cross-compilation tools, as well as a root file system, from source. It
> supports thousands of packages and can build the boot loader, Linux
> kernel and root file system image. Building a system image with
> buildroot only requires a few simple steps:
> 
> - Cloning the builroot repository
> - Running `make menuconfig` to select configuration options
> - Running `make` to build the image
> 
> Deciding which of the thousand packages to select in the configuration
> is difficult if you lack experience. I've attached my configuration 
> file
> for builroot 2024.11.1 to this e-mail, you can use it as a starting
> point.

Thank you very much. I think building on an SDK on a more powerful 
machine makes a lot more sense than my attempts to do it natively. I 
don't have experience of building machines beyond shell scripts to build 
Docker containers so I'll start with buildroot.

It's very helpful for me to understand the similarities between 
buildroot, OpenEmbedded and Yocto - these terms often get thrown around 
without an explanation.

> # Boot flow and system images
> 
> Linux is very versatile when it comes to booting a system. Most 
> embedded
> boards come with instructions for flashing a system image on an SD 
> card,
> and storing all the components (boot loader, device tree, kernel and
> root file system) on the card. Buildroot can create images compatible
> with that boot flow, and that is possibly a good first step to get
> started. However, that's not the boot flow I use.
> 
> I find copying files to an SD card every time I make modifications to
> the kernel to be cumbersome and error-prone. Instead, a common kernel
> and userspace development workflow relies on the following boot flow:
> 
> - The boot ROM loads the boot loader from storage media (SD card or
>   eMMC)
> - The boot loader acquires an IP address through DHCP, and loads the
>   device tree and kernel over the network through TFTP
> - The kernel mounts the root file system over the network through NFS
> 
> With this workflow, all components but the boot loader are loaded over
> the network. To make modifications to the device tree, kernel, or any
> userspace component (including libcamera), I can cross-compile them on
> my development machine, and copy them to the TFTP root directory (for
> the device tree or kernel) or the NFS root directory (for kernel 
> modules
> and any userspace component). New userspace components will be
> immediately available on the target device, and running a new kernel
> only requires a reboot.
> 
> My buildroot configuration is tailored for that workflow, and only
> produces a root file system as a tarball, without building a full 
> system
> image. It should be possible to use buildroot to build a full system
> image for the Debix Model A board, but I don't believe anyone has
> contributed the default configuration files, so there would be a bit of
> work to do.

I can see that's more efficient for continuous integration than using SD 
cards.

> # Picking the right U-Boot version
> 
> Vendors often don't upstream support for their boards in U-Boot. I
> believe that the Debix Model A board is well supported in U-Boot
> 2024.10, which is the version used by buildroot 2024.11.1, but I 
> haven't
> tested that. I have instead compiled U-Boot manually as I needed to 
> make
> changes to its configuration to suit my environment.

I thought I'd found Debix's source for U-Boot:

https://github.com/debix-tech/uboot

but the last commit was 3 years ago so maybe it's somewhere else.

> Recompiling the boot loader manually can be cumbersome. An easy option
> may be to flash the system image provided by Polyhex for the Debix 
> Model
> A on an SD card to provide the boot loader, and then replacing the
> kernel and root file system.

That does sound a good way for me to get started - am I right in 
thinking that Debix describe that process here?:

https://www.debix.io/Software/blog_info/id/59.html

> That's lots of information, but I'm sure there are still be lots of 
> gaps
> you'll have to fill. I'm also pretty sure it would be possible to take 
> a
> general purpose Linux distribution such as Fedora or Ubuntu and run it
> on the Debix board, and replace custom components, but I can't help you
> with that as that's not what I do. You will need to experiment and
> gather information, slowly learning how the pieces fit together.

I think I might try to do one very slimmed-down Linux distribution with 
the minimum functionality needed to run the MIPI CSI-2 interface, ISP 
and SD card which consumes minimal resources, can boot fast, can waken 
from sleep state fast and can monitor wildlife but not much else and 
another separate much larger and a much more resource intensive general 
purpose distribution (including web server, Samba, Wi-Fi connection, 
etc.) that's booted when someone wants to interact with the system. I've 
never doen this before but hoping the U-Boot might allow the 
distribution to be chosen at boot time depending on a simple digital 
electronic input - not sure.

It is an enormous jigsaw puzzle...

Thank you very much for all your help!

Will

On 2025-01-19 16:26, Laurent Pinchart wrote:
> Hi Will,
> 
> On Sat, Jan 18, 2025 at 12:48:29PM +0000, w.robertson at cairnwater.com 
> wrote:
>> Hi all,
>> 
>> Any advice you could give on getting started using libcamera on the
>> Debix Model A would be wonderful. With fresh Ubuntu Desktop 24.10 and
>> Raspberry Pi OS images written to an SD card (Using the RPi imager) 
>> the
>> Debix Model A won't start to boot and with the Debix Ubuntu 
>> distribution
>> it boots but I get an error when I try to install the dependencies
>> needed to rebuild the kernel.
>> 
>> My Debix Model A doesn't have the 2 switches to select boot source so 
>> I
>> think it boots from SD only and doesn't have on-board eMMC
>> 
>> I suspect from the NXP i.MX documentation that with vanilla Ubuntu I'm
>> missing the BSP but I'm not sure.
>> 
>> I've been trying to follow libcamera Getting Started but it doesn't 
>> say
>> which distribution is used:
>> 
>> https://libcamera.org/getting-started.html
>> 
>> I'd be enormously grateful for any advice.
> 
> OK. Brace yourself, because it will be a bit of a ride, but it's an
> interesting one.
> 
> There are multiple options (it would be too easy otherwise), and which
> one is best depends on your use cases and personal preferences. While
> many vendors provide their own Linux distributions, BSPs and SDKs and
> market them as easy to use, turnkey options, I adhere to the (old ?)
> school of thoughts that believes understanding what goes behind the
> scenes is important. You can then make a fully informed decision on
> which components to pick and how much manual control and customization
> you need.
> 
> Very roughly speaking, an Arm Linux system needs four major components:
> 
> - A boot loader
> - A device tree (for systems that are device-tree based)
> - A kernel
> - A root file system
> 
> # Boot loader
> 
> The boot loader here is a blanket term that comprises all the software
> components that run from power up to handing control over to the Linux
> kernel. The most common boot loader for Arm-based embedded Linux 
> systems
> is U-Boot. It's a well established de facto standard in the industry,
> and is what NXP uses for the i.MX8MP. Alternatives exist, but I really
> wouldn't recommend them if you're not already experienced with embedded
> Linux systems.
> 
> Arm has standardized the ARMv8 secure boot flow and split it in 
> multiple
> stages, named BL1 (the boot ROM in the SoC, the only component that
> can't be updated), BL2 (second stage boot loader, implemented by U-Boot
> and named SPL), BL31 (trusted firmware, a reference implementation made
> by Arm is available at [1]), and BL33 (the last boot loader stage, 
> known
> as untrusted boot loader, and implemented by U-Boot proper). There's
> plenty of information online that describes the boot process, for
> instance [2].
> 
> The boot loader is responsible for initializing all low-level hardware
> required to load the kernel (system DRAM, permanent storage such as
> eMMC or SD cards, ...). It then loads the kernel, and passes execution
> to it. Parts of the boot loader can remain persistent in system DRAM
> after loading the kernel, and offer runtime services to the kernel
> (control of power domains, secure operations such as access to crypto
> engines, ...).
> 
> [1] https://github.com/TrustedFirmware-A/trusted-firmware-a
> [2] https://www.youtube.com/watch?v=GXFw8SV-51g
> 
> # Device tree
> 
> Unlike PCI or USB devices that can be auto-detected at runtime, most
> devices inside the SoC, or connected through non-discoverable buses 
> such
> as I2C or SPI, need to be declared to the kernel. The majority of Arm
> systems use Device Tree for that purpose.
> 
> The device tree specification ([3]) defines a textual source format and
> a compiled binary format. Device tree bindings then define (as YAML
> schemas) how each particular device is described. The bindings
> maintained as part of the Linux kernel sources ([4]). Finally, as large
> set of device tree sources describing various boards is available in 
> the
> kernel source tree as well ([5]). Vendors can also maintain their 
> device
> trees outside of the kernel repository.
> 
> Device trees are compiled to a binary DTB file and stored in media
> accessible by the boot loader. They are loaded by the boot loader as
> part of the boot process, and passed to the Linux kernel when starting
> it.
> 
> [3] https://www.devicetree.org/
> [4] 
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/devicetree/bindings
> [5] 
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/arm64/boot/dts/
> 
> # Linux kernel
> 
> The kernel is responsible for controlling the hardware and providing
> services to userspace. While there is only one Linux kernel, SoC and
> board vendor typically fork it to add a set of modifications, from
> changes in the kernel core to additional device drivers. This set can 
> be
> as small as a handful of patches or as big as millions of lines,
> depending on the vendor. Vendor kernels are often referred to as BSP
> (Board Support Package) kernels.
> 
> After being loaded in system DRAM by the boot loader, the kernel gets
> executed and goes through a fairly long initialization phase. Once the
> initialization is complete, the kernel mounts the root file system,
> loads the init binary, and executes it.
> 
> # Root file system
> 
> The root file system contains all the userspace components of a Linux
> system. Linux supports a large number of storage media and file system
> types, including SD cards, eMMCs, SATA or NVMe drives, or accessing the
> root file system over the network.
> 
> # Distributions and build frameworks
> 
> Linux distributions typically provide all those components in the form
> of an installable image. Depending on the type of distribution, you can
> download a ready-made binary image, or you may need to build one. My
> experience is that ready-made images, provided by distributions
> originating from desktop systems (Debian, Ubuntu, Fedora, ...) may be
> simpler to install, but more difficult to customize. I'm certainly
> biased though, I'm quite sure that someone more familiar with Debian or
> Fedora development would have less trouble compiling a custom kernel
> using the packaging infrastructure provided by those distributions. As
> I'm not such a person, I'll describe alternatives.
> 
> In addition to ready-made images, there are build automation frameworks
> that help you build a system image and a corresponding 
> cross-compilation
> environment (often referred to as SDK). Two common frameworks are
> buildroot and OpenEmbedded (often referred to as Yocto, although that's
> a misuse of the term). I'm more familiar with buildroot, and that's 
> what
> I use with my Debix board, so I will focus on that framework.
> 
> Buildroot facilitates the process of building a complete set of
> cross-compilation tools, as well as a root file system, from source. It
> supports thousands of packages and can build the boot loader, Linux
> kernel and root file system image. Building a system image with
> buildroot only requires a few simple steps:
> 
> - Cloning the builroot repository
> - Running `make menuconfig` to select configuration options
> - Running `make` to build the image
> 
> Deciding which of the thousand packages to select in the configuration
> is difficult if you lack experience. I've attached my configuration 
> file
> for builroot 2024.11.1 to this e-mail, you can use it as a starting
> point.
> 
> # Boot flow and system images
> 
> Linux is very versatile when it comes to booting a system. Most 
> embedded
> boards come with instructions for flashing a system image on an SD 
> card,
> and storing all the components (boot loader, device tree, kernel and
> root file system) on the card. Buildroot can create images compatible
> with that boot flow, and that is possibly a good first step to get
> started. However, that's not the boot flow I use.
> 
> I find copying files to an SD card every time I make modifications to
> the kernel to be cumbersome and error-prone. Instead, a common kernel
> and userspace development workflow relies on the following boot flow:
> 
> - The boot ROM loads the boot loader from storage media (SD card or
>   eMMC)
> - The boot loader acquires an IP address through DHCP, and loads the
>   device tree and kernel over the network through TFTP
> - The kernel mounts the root file system over the network through NFS
> 
> With this workflow, all components but the boot loader are loaded over
> the network. To make modifications to the device tree, kernel, or any
> userspace component (including libcamera), I can cross-compile them on
> my development machine, and copy them to the TFTP root directory (for
> the device tree or kernel) or the NFS root directory (for kernel 
> modules
> and any userspace component). New userspace components will be
> immediately available on the target device, and running a new kernel
> only requires a reboot.
> 
> My buildroot configuration is tailored for that workflow, and only
> produces a root file system as a tarball, without building a full 
> system
> image. It should be possible to use buildroot to build a full system
> image for the Debix Model A board, but I don't believe anyone has
> contributed the default configuration files, so there would be a bit of
> work to do.
> 
> # Picking the right U-Boot version
> 
> Vendors often don't upstream support for their boards in U-Boot. I
> believe that the Debix Model A board is well supported in U-Boot
> 2024.10, which is the version used by buildroot 2024.11.1, but I 
> haven't
> tested that. I have instead compiled U-Boot manually as I needed to 
> make
> changes to its configuration to suit my environment.
> 
> Recompiling the boot loader manually can be cumbersome. An easy option
> may be to flash the system image provided by Polyhex for the Debix 
> Model
> A on an SD card to provide the boot loader, and then replacing the
> kernel and root file system.
> 
> 
> That's lots of information, but I'm sure there are still be lots of 
> gaps
> you'll have to fill. I'm also pretty sure it would be possible to take 
> a
> general purpose Linux distribution such as Fedora or Ubuntu and run it
> on the Debix board, and replace custom components, but I can't help you
> with that as that's not what I do. You will need to experiment and
> gather information, slowly learning how the pieces fit together.
> 
>> On 2025-01-17 21:02, w.robertson at cairnwater.com wrote:
>> > Hi
>> >
>> > Which Linux distribution is recommended for libcamera development using
>> > the Debix Model A (NXP i.MX 8M) board?
>> >
>> > I've been trying using the Debix Model A Ubuntu distribution (Ubuntu
>> > 22.04) from https://debix.io/Software/downloadn.html but running into
>> > problems when I try to do a kernel rebuild using the procedure here
>> > https://wiki.ubuntu.com/Kernel/BuildYourOwnKernel and was wondering if
>> > I'd be better using vanilla Ubuntu or Debian?
>> >
>> > Thank you for your help!