libcamera - Development of Open Source Embedded Linux Wildlife Camera System

[w.robertson at cairnwater.com]

Hi Laurent, Jacopo and Kieran,

I was trying to think of ways that I could help out. All my current 
experience is server side Python/.Net/SQL ( www.faclair.com is one 
project) and as a climbing arborist so I don't have any kernel 
experience.

I'm compiling a list of SoMs, SiPs and SBCs that should in theory work 
with libcamera with relevant information from their technical data 
sheets and questions in emails to their manufacturers - at the moment 
this is in CritterCam's git repository but I could make this available 
for other libcamera users and testers if this would be a help?

I looked at using a μC with a MIPI CSI-2 interface.

 From a conservation perspective, the research has been very successful 
but trying to fund it from my commercial work has been financially 
disastrous - I'm moving out of my flat and into my van to save money in 
the hope that enough funding to survive might become available next year 
- if I tried to port libcamera to a μC I'd go bankrupt long before the 
work could be completed.

The need for CritterCam to outlast all currently available silicon - 
keeping up with very rapidly evolving processor and sensor technology - 
and to support multiple contributors contributing both software features 
and add-on off-the-shelf and custom electronic modules to a single open 
source repository also encouraged me to look at a μP and embedded Linux 
instead of a μC.

The only ways I can see at the moment of having any possibility of 
getting a high quality camera working on a μC would be to work together 
with Raspberry Pi or ST.

A talk by Naushir Patuck from Raspberry Pi in November 2024 mentioned 
that Raspberry Pi were considering the possibility of a μC with MIPI 
CSI-2 support:

https://www.digikey.com/en/blog/webinar-introduction-raspberry-pi-imaging-and-computer-vision-tools

ST seem to have very recently unveiled the STM32N6x7 (datasheet v1 
November 2024) and STM32N6x5 (datasheet link gives 404 error) Cortex-M55 
μC with MIPI CSI-2 support:

https://www.st.com/en/microcontrollers-microprocessors/stm32n6-series.html

> the i.MX8MP, for instance, has a Cortex M core

The STM32MP25 also includes a Cortex M33 - so the i.MX8MP and STM32MP25 
raise the theoretical possibility of having both a low-power μC solution 
and a more power hungry but much more flexible μP solution that both use 
the same silicon - avoiding the potentially costly headache of 
CritterCam maintaining seperate low power μC and high flexibility μP 
forks - Cortex M can't support CSI-2 so would be limited to simpler 
camera modules.

> I presume you would have more luck asking on the rpi forums.

Every time the need for a low power sleep or suspend mode has been 
raised on the RPi forums the answer has been a very decisive "No. Switch 
it off and reboot." - this is a massive problem that renders RPi SBCs 
effectively useless for any power sensitive application - it lead me to 
look at silicon from ST and NXP instead. I keep hoping that something 
might appear from RPi but nothing has.

> Yes, to realize and tune a camera system you need knowledge of both
the ISP, the image sensor and the optics.

> If a vendor instead provide tuning data pre-calculated for
a specific set of camera modules designed to work with their platforms
then yes, you would have a choice of pre-tuned cameras to pick from.

Thinking about this - I used to work in the physics, chemistry and 
nonlinear optics of optoelectronic materials - I don't know if this 
could be any help or not - the thought went through my mind of 
developing a small optical bench on which an image sensor and any 
associated optics could be placed then red, green, blue and IR lasers 
modulated and scanned over both it's surface and possibly a known 
reference image sensor while a computer reads data from it so that 
precise calibration and tuning data for that particular make and model 
of image sensor and any associated optics could be determined and open 
sourced? I don't know if this would be of use or not (perhaps using this 
data to train a simple NN could be a way of implementing calibration and 
tuning algorithms if NPUs or GPUs become fast enough - NPUs and GPUs 
might be good at handling the simple but highly parallel tensor 
operations required without proprietory tie-in)?

This might allow some flaws and limitations inherent in the 
manufacturing of the sensor and optics to be compensated for through 
firmware tuning and calibration?

> The implementation of
the algorithms, how they use and combine features of different IPs on
the platforms (gpu, other accelerators etc) and the tuning of the whole
camera pipeline is usually what SoC vendors compete on.

> Sensor driver manufacturers are not involved when it comes to ISP
algorithms development. Knowledge of the ISP, its design and
documentation are usually only accessible to the SoC vendors (again
apart from RPi that fully documents their ISP publicly).

Thank you ver much - it's a big help for me to understand that - so 
basically one way that businesses like ST and NXP compete with each 
other is by competing to provide the best ISP algorithms as well as the 
best silicon?

> The RPi pipeline handler implements a selection logic to pick the 
> "best"
sensor configuration given a desired output streams configuration:
https://git.libcamera.org/libcamera/libcamera.git/tree/src/libcamera/pipeline/rpi/common/pipeline_base.cpp#n937

> libcamera offers an API to override the pipeline handler choice, so I
would say your application is free to chose any configuration
offered by the sensor's driver.

Thank you very much - that's an enormous help - I'd been confused by 
that. Typically commercial wildlife cameras boast the number of pixels - 
often including nonexistent "interpolated pixels" - whereas the big 
limiting factor on the image quality is the signal to noise ratio on 
those pixels so binning may be very valuable - particularly in low light 
conditions or when processing power and storage are limited.

> Some vendors
decide to distribute their "advanced" algorithms as binaries through
different channels and support their platforms in mainline libcamera 
with
a reduced number of features in the source implementation.

Why don't they contribute their "advanced" proprietory features as 
obfuscated and compiled binary plugins for libcamera?

> Maybe that's why I've never seen a flying squirrel in Nuuksio :-(

The bad news is that their numbers have been dropping very rapidly. The 
good news is that research into Pteromys volans by Ralf Wistbacka and 
others and into Red Squirrels, Hazel Dormice and Garden Dormice by 
Goedele Verbeylen, Thomas Briner and others has identified lack of 
suitable nesting holes as a significant cause for this decline and that 
we now have very fast, precise and scalable chainsaw and rotary carving 
techniques to create nest holes for them. The other good news is that 
Goedele Verbeylen and her colleagues have brought Garden Dormice back 
from the brink of extinction in Belgium, proving that it can be done.

> If you need to
operate with really low power, keeping the Cortex A cores and DRAM
powered, even in low-power mode, may go over your power budget.

Yup. I'm keeping a close eye on reported power consumption for SoMs and 
I'll measure power consumption. I'm also trying to keep everything small 
to fit into a small amount of RAM and tentatively rejecting any board 
that uses DDR4 instead of LPDDR4.

The power available is limited by the amount of sun, size of solar 
panels, size of rechargeable batteries and battery chemistry (Li and 
LiFePO4 batteries work well in summer but only PbA batteries work below 
a few °C). It may be that tiny wind generators could be added in 
difficult conditions. Like any tree dwelling creature, CritterCam has to 
be very light, flexible and adaptable to be successful.

Simply by allowing the battery, solar panel and the μP to be in seperate 
boxes, allowing videos and photos to be optionally compressed and 
allowing WiFi or Ethernet to be switched on to download data, CritterCam 
will remove the need for conservationists and researchers to risk their 
lives on wobbly ladders changing batteries and SD cards.

An energy and power meter I ordered should arrive in a week or two and 
I'll start measuring how much power a solar panel can give in the 
typical worst case of a forest floor.

When things get really desperate an optional GSM module could be 
switched on to send a simple SMS or MQTT message to ask someone to come 
and change a low battery for a fully charged one and consider adding 
more solar panels.

> C-PHY is more recent, it can provide the same bandwidth with a lower 
> number of signals, but isn't as widely supported (I expect that to 
> improve over time).

That's good to know - C-PHY's 3 level signalling must be giving big 
headaches for silicon designers used to working with digital signals. 
Good to know that for the time being D-PHY is the best supported.

> CSI-3 was killed by Intel and Qualcomm before it reached the market

That's good to know. One less thing to keep track of. CSI-1 and CSI-3 
RIP.

Will

On 2024-12-22 10:30, Laurent Pinchart wrote:
> Hi Will,
> 
> On Fri, Dec 20, 2024 at 11:04:09AM +0000, w.robertson at cairnwater.com 
> wrote:
>> Hello Laurent and Kieran,
>> 
>> > This sounds very interesting. We rarely get contacted by people working
>> > on projects related to animals or nature. I live in Finland, so I would
>> > be happy to help the flying squirrels from Nuuksio :-)
>> 
>> That's wonderful! Dormouse species hibernate but the flying squirrels
>> can't hibernate and so must have nest holes which provide protection
>> from weather and allow them to build well insulated nests to survive 
>> the
>> winter, they also ideally need nest holes with entrances which are the
>> right size to let them in but keep the larger red squirrels out. Past
>> forestry practice was to remove dead trees and trees with holes (which
>> were seen as unsightly or unstable) and this has left a drastic 
>> shortage
>> of suitable nest holes for the flying squirrels.
> 
> Maybe that's why I've never seen a flying squirrel in Nuuksio :-(
> 
>> Most of my research is unpaid (with occasional small sponsorship here
>> and there) so resources have to be used very efficiently.
>> 
>> By developing new rotary boring and precision chainsaw carving
>> techniques we can carve nest holes entirely through a single narrow,
>> very precise entrance with a much larger nesting chamber behind.
>> 
>> > I'll try to help, in my (unfortunately limited) free time. The best
>> > option, if compatible with your needs, would be to have open
>> > discussions
>> > either on the libcamera development mailing list, or on the project's
>> > IRC channel (you can find contact information in
>> > https://libcamera.org/contributing.html). That way other developers
>> > will also be able to chime in.
>> 
>> That would be wonderful!
>> 
>> > libcamera doesn't currently support the STM32MP2, but ST is working on
>> > it. They showcased a working prototype at Embedded World in April this
>> > year, and I assume they will provide a version usable in production
>> > soon. While they haven't started upstreaming much of their code in
>> > libcamera, I believe they plan to do so. The platform is therefore
>> > interesting for low-power applications.
>> 
>> Thank you very much - I'd been unsure about that - that's very helpful
>> to know - I'll maybe try to find a way to give ST some gentle
>> encouragement in the spring - they're pushing aggressively to sell 
>> both
>> their image sensors and STM32MP2 so libcamera support should be a high
>> strategic priority for them. I get the feeling that having launched
>> STM32MP2 into both industrial control and domestic appliance markets
>> they've got a fairly heavy workload at the moment and the 
>> STM32MP257F-DK
>> dev. board is due to be launched later than planned.
>> 
>> > The i.MX8MP is much better supported in libcamera at the moment, and we
>> > are actively improving its support. There are low-cost development
>> > boards, such as the Debix Model A that we use for development (note that
>> > its camera connector is not standard, but they have a Debix I/O board
>> > that can interface the Model A camera connector to a Raspberry Pi 15
>> > pins connector).
>> 
>> Thank you very much! That's wonderful to have a good dev. board! I'd
>> been looking at i.MX8MP SoMs and SiPs but some of them still seem to 
>> be
>> in the design stage with some technical information missing from data
>> sheets. Will Debix Model A's LPDDR4 allow it to quickly enter and 
>> waken
>> from a low power sleep mode?
> 
> I haven't experienced with low power sleep and wake up time myself, so 
> I
> can't tell.
> 
> Generally speaking, it all depends on your power budget. If you need to
> operate with really low power, keeping the Cortex A cores and DRAM
> powered, even in low-power mode, may go over your power budget. Other
> options are possible, such as turning power to the Cortex A and DRAM
> off, and handling fast wake up tasks in a smaller core (the i.MX8MP, 
> for
> instance, has a Cortex M core that could be used for this). The wake up
> time gets significantly increased, as Linux will need to boot from
> scratch. You may be able to start powering up the camera sensor in
> parallel from the Cortex M core to save some time, or even drive the
> camera completely from the Cortex M, but that would require porting
> libcamera (and drivers) to whatever OS you would be running there, 
> which
> would require really significant effort.
> 
>> The 4 lane MIPI CSI seems potentially more
>> flexible than the STM32MP2's 2 lane CSI-2 interface.
> 
> 4 lanes will give you more bandwidth, but unless you target really high
> resolutions at high frame rates, that shouldn't be necessary.
> 
>> I guess the best way for me to get a Debix Model A or B is just to 
>> order
>> it from RS?
> 
> I believe so.
> 
>> (One thing that's confusing me - when modern data sheets like the 
>> Debix
>> Model A data sheet say "MIPI CSI 4-lane" I guess they mean "MIPI 
>> CSI-2"?
>> I'd heard of CSI-1 which supports only 1 lane and seems old and rarely
>> used now, of CSI-2 which supports multiple lanes and theoretically a
>> choice of C-PHY of D-PHY physical layers (though I guess many CSI-2
>> implementations maybe don't implement C-PHY because of its 
>> complexity?)
>> for the physical layer and CSI-3 which is based on UniPro and rarely
>> implemented at the moment.)
> 
> Today the terms "MIPI CSI" (or sometimes just "MIPI camera") refers to
> CSI-2. CSI-1 has retired a very long time ago, and CSI-3 was killed by
> Intel and Qualcomm before it reached the market (there were some
> prototype implementations, but nothing in mass production).
> 
> D-PHY is the most common PHY for CSI-2, and should be supported
> everywhere. C-PHY is more recent, it can provide the same bandwidth 
> with
> a lower number of signals, but isn't as widely supported (I expect that
> to improve over time). I don't think the choice of PHY matters too much
> in your case, whatever is supported by both the camera sensor and the
> SoC will be fine.
> 
>> Thank you very much for all your help!
>> 
>> Will
>> 
>> On 2024-12-19 23:17, Laurent Pinchart wrote:
>> > Hello Will,
>> >
>> > (CC'ing my colleague Kieran)
>> >
>> > On Thu, Dec 19, 2024 at 07:46:53PM +0000, w.robertson at cairnwater.com
>> > wrote:
>> >> Hi Laurent
>> >>
>> >> Thank you very much for your video "Giving Linux a Camera Stack:
>> >> libcamera's 3 Years Journey and Exciting Future" and the wonderful
>> >> work
>> >> of you and the libcamera developers in bringing such a powerful and
>> >> structured approach to such an extremely difficult and complex
>> >> problem.
>> >
>> > You're more than welcome.
>> >
>> >> I work as a climbing arborist in practice and research and software
>> >> engineer. My research is mostly voluntary and focused on new minimally
>> >> invasive techniques to carve critically needed nest holes for
>> >> endangered
>> >> tree dwelling dormouse species (Hazel Dormouse, Garden Dormouse,
>> >> Forest
>> >> Dormouse), bat species and in Finland the European Flying Squirrel
>> >> (Pteromys volans).
>> >>
>> >> To overcome problems and limitations with commercial wildlife cameras
>> >> for small mammals I started work on CritterCam which - if I can get it
>> >> to work - will be a fully open source wildlife camera system which is
>> >> vastly better, vastly more flexible, vastly more independent in the
>> >> field and significantly cheaper than anything commercially available
>> >> at
>> >> the moment. Working with embedded Linux and with several different
>> >> manufacturers for the microprocessor and image sensors for CritterCam
>> >> means that CritterCam users can never be price gouged and CritterCam
>> >> will be able to adapt to update all of its critical components with
>> >> newer alternatives well into the future.
>> >>
>> >> This is an article with some videos of Garden Dormice exploring our
>> >> carved nest holes published in Flemish yesterday by Goedele Verbeylen:
>> >>
>> >> https://www.natuurpunt.be/nieuws/boomholtes-kerven-voor-bedreigde-eikelmuizen
>> >>
>> >> This is a short summary that I wrote in English and German:
>> >>
>> >> https://drive.google.com/file/d/1jkFA-PgDr-O9-nGSgODCUFkukj63F4fc/view?usp=sharing
>> >>
>> >> We have a small website at:
>> >>
>> >> new-homes-for-old-friends.cairnwater.com
>> >
>> > This sounds very interesting. We rarely get contacted by people working
>> > on projects related to animals or nature. I live in Finland, so I would
>> > be happy to help the flying squirrels from Nuuksio :-)
>> >
>> >> My background is mainly server side so I don't have kernel experience
>> >> and I was wondering if you or another libcamera might be willing to
>> >> give
>> >> me any pointers to help me get started using supported MIPI CSI-2
>> >> cameras with embedded Linux or ANdroid?
>> >
>> > I'll try to help, in my (unfortunately limited) free time. The best
>> > option, if compatible with your needs, would be to have open
>> > discussions
>> > either on the libcamera development mailing list, or on the project's
>> > IRC channel (you can find contact information in
>> > https://libcamera.org/contributing.html). That way other developers
>> > will
>> > also be able to chime in.
>> >
>> >> I'm hoping to build a first prototype using the STM32MP257F-DK dev.
>> >> board based on the STM32MP257 μP when it becomes available in January
>> >> but I'm also looking at SoMs and SiPs based on other μPs like the NXP
>> >> i.MX 8M Plus that support a CSI-2 interface and a low power sleep mode
>> >> -
>> >> I can also do some initial testing using a Raspberry Pi SBC.
>> >
>> > libcamera doesn't currently support the STM32MP2, but ST is working on
>> > it. They showcased a working prototype at Embedded World in April this
>> > year, and I assume they will provide a version usable in production
>> > soon. While they haven't started upstreaming much of their code in
>> > libcamera, I believe they plan to do so. The platform is therefore
>> > interesting for low-power applications.
>> >
>> > The i.MX8MP is much better supported in libcamera at the moment, and we
>> > are actively improving its support. There are low-cost development
>> > boards, such as the Debix Model A that we use for development (note
>> > that
>> > its camera connector is not standard, but they have a Debix I/O board
>> > that can interface the Model A camera connector to a Raspberry Pi 15
>> > pins connector).
>> >
>> > Raspberry Pi platforms are also well supported, with the work being
>> > performed by Raspberry Pi.