libcamera - Development of Open Source Embedded Linux Wildlife Camera System

[w.robertson at cairnwater.com]

Hi Laurent, Jacopo and Kieran,

All my best wishes to you for 2025!

I put together this overview summary of CritterCam that hopefully 
explains how the engineering and economics of the situation make it 
feasible - let me know if it would be of any help to you as an example 
use case (it might be a good one because I'm not trying to sell anything 
or make a sales pitch but I am trying to take the speed and aggression 
of modern software engineering to an until-now far-too-sleepy area of 
electronics):

https://bitbucket.org/WillRobertRobertson/crittercam/src/main/README.md

The in-depth analysis behind that is in separate files (far too long to 
include in the summary) that're in a private repository for now because 
they have email addresses, etc.  (Think I'm developing a preference for 
GitHub over BitBucket...)

Also translated Goedele's most recent article to English :-)

https://new-homes-for-old-friends.cairnwater.com/carved-tree-cavities-for-endangered-dormice/

All the best for 2025!

Will

On 2024-12-22 19:22, w.robertson at cairnwater.com wrote:
> Hi Laurent, Jacopo and Kieran,
> 
> I was trying to think of ways that I could help out. All my current 
> experience is server side Python/.Net/SQL ( www.faclair.com is one 
> project) and as a climbing arborist so I don't have any kernel 
> experience.
> 
> I'm compiling a list of SoMs, SiPs and SBCs that should in theory work 
> with libcamera with relevant information from their technical data 
> sheets and questions in emails to their manufacturers - at the moment 
> this is in CritterCam's git repository but I could make this available 
> for other libcamera users and testers if this would be a help?
> 
> I looked at using a μC with a MIPI CSI-2 interface.
> 
> From a conservation perspective, the research has been very successful 
> but trying to fund it from my commercial work has been financially 
> disastrous - I'm moving out of my flat and into my van to save money in 
> the hope that enough funding to survive might become available next 
> year - if I tried to port libcamera to a μC I'd go bankrupt long before 
> the work could be completed.
> 
> The need for CritterCam to outlast all currently available silicon - 
> keeping up with very rapidly evolving processor and sensor technology - 
> and to support multiple contributors contributing both software 
> features and add-on off-the-shelf and custom electronic modules to a 
> single open source repository also encouraged me to look at a μP and 
> embedded Linux instead of a μC.
> 
> The only ways I can see at the moment of having any possibility of 
> getting a high quality camera working on a μC would be to work together 
> with Raspberry Pi or ST.
> 
> A talk by Naushir Patuck from Raspberry Pi in November 2024 mentioned 
> that Raspberry Pi were considering the possibility of a μC with MIPI 
> CSI-2 support:
> 
> https://www.digikey.com/en/blog/webinar-introduction-raspberry-pi-imaging-and-computer-vision-tools
> 
> ST seem to have very recently unveiled the STM32N6x7 (datasheet v1 
> November 2024) and STM32N6x5 (datasheet link gives 404 error) 
> Cortex-M55 μC with MIPI CSI-2 support:
> 
> https://www.st.com/en/microcontrollers-microprocessors/stm32n6-series.html
> 
>> the i.MX8MP, for instance, has a Cortex M core
> 
> The STM32MP25 also includes a Cortex M33 - so the i.MX8MP and STM32MP25 
> raise the theoretical possibility of having both a low-power μC 
> solution and a more power hungry but much more flexible μP solution 
> that both use the same silicon - avoiding the potentially costly 
> headache of CritterCam maintaining seperate low power μC and high 
> flexibility μP forks - Cortex M can't support CSI-2 so would be limited 
> to simpler camera modules.
> 
>> I presume you would have more luck asking on the rpi forums.
> 
> Every time the need for a low power sleep or suspend mode has been 
> raised on the RPi forums the answer has been a very decisive "No. 
> Switch it off and reboot." - this is a massive problem that renders RPi 
> SBCs effectively useless for any power sensitive application - it lead 
> me to look at silicon from ST and NXP instead. I keep hoping that 
> something might appear from RPi but nothing has.
> 
>> Yes, to realize and tune a camera system you need knowledge of both
> the ISP, the image sensor and the optics.
> 
>> If a vendor instead provide tuning data pre-calculated for
> a specific set of camera modules designed to work with their platforms
> then yes, you would have a choice of pre-tuned cameras to pick from.
> 
> Thinking about this - I used to work in the physics, chemistry and 
> nonlinear optics of optoelectronic materials - I don't know if this 
> could be any help or not - the thought went through my mind of 
> developing a small optical bench on which an image sensor and any 
> associated optics could be placed then red, green, blue and IR lasers 
> modulated and scanned over both it's surface and possibly a known 
> reference image sensor while a computer reads data from it so that 
> precise calibration and tuning data for that particular make and model 
> of image sensor and any associated optics could be determined and open 
> sourced? I don't know if this would be of use or not (perhaps using 
> this data to train a simple NN could be a way of implementing 
> calibration and tuning algorithms if NPUs or GPUs become fast enough - 
> NPUs and GPUs might be good at handling the simple but highly parallel 
> tensor operations required without proprietory tie-in)?
> 
> This might allow some flaws and limitations inherent in the 
> manufacturing of the sensor and optics to be compensated for through 
> firmware tuning and calibration?
> 
>> The implementation of
> the algorithms, how they use and combine features of different IPs on
> the platforms (gpu, other accelerators etc) and the tuning of the whole
> camera pipeline is usually what SoC vendors compete on.
> 
>> Sensor driver manufacturers are not involved when it comes to ISP
> algorithms development. Knowledge of the ISP, its design and
> documentation are usually only accessible to the SoC vendors (again
> apart from RPi that fully documents their ISP publicly).
> 
> Thank you ver much - it's a big help for me to understand that - so 
> basically one way that businesses like ST and NXP compete with each 
> other is by competing to provide the best ISP algorithms as well as the 
> best silicon?
> 
>> The RPi pipeline handler implements a selection logic to pick the 
>> "best"
> sensor configuration given a desired output streams configuration:
> https://git.libcamera.org/libcamera/libcamera.git/tree/src/libcamera/pipeline/rpi/common/pipeline_base.cpp#n937
> 
>> libcamera offers an API to override the pipeline handler choice, so I
> would say your application is free to chose any configuration
> offered by the sensor's driver.
> 
> Thank you very much - that's an enormous help - I'd been confused by 
> that. Typically commercial wildlife cameras boast the number of pixels 
> - often including nonexistent "interpolated pixels" - whereas the big 
> limiting factor on the image quality is the signal to noise ratio on 
> those pixels so binning may be very valuable - particularly in low 
> light conditions or when processing power and storage are limited.
> 
>> Some vendors
> decide to distribute their "advanced" algorithms as binaries through
> different channels and support their platforms in mainline libcamera 
> with
> a reduced number of features in the source implementation.
> 
> Why don't they contribute their "advanced" proprietory features as 
> obfuscated and compiled binary plugins for libcamera?
> 
>> Maybe that's why I've never seen a flying squirrel in Nuuksio :-(
> 
> The bad news is that their numbers have been dropping very rapidly. The 
> good news is that research into Pteromys volans by Ralf Wistbacka and 
> others and into Red Squirrels, Hazel Dormice and Garden Dormice by 
> Goedele Verbeylen, Thomas Briner and others has identified lack of 
> suitable nesting holes as a significant cause for this decline and that 
> we now have very fast, precise and scalable chainsaw and rotary carving 
> techniques to create nest holes for them. The other good news is that 
> Goedele Verbeylen and her colleagues have brought Garden Dormice back 
> from the brink of extinction in Belgium, proving that it can be done.
> 
>> If you need to
> operate with really low power, keeping the Cortex A cores and DRAM
> powered, even in low-power mode, may go over your power budget.
> 
> Yup. I'm keeping a close eye on reported power consumption for SoMs and 
> I'll measure power consumption. I'm also trying to keep everything 
> small to fit into a small amount of RAM and tentatively rejecting any 
> board that uses DDR4 instead of LPDDR4.
> 
> The power available is limited by the amount of sun, size of solar 
> panels, size of rechargeable batteries and battery chemistry (Li and 
> LiFePO4 batteries work well in summer but only PbA batteries work below 
> a few °C). It may be that tiny wind generators could be added in 
> difficult conditions. Like any tree dwelling creature, CritterCam has 
> to be very light, flexible and adaptable to be successful.
> 
> Simply by allowing the battery, solar panel and the μP to be in 
> seperate boxes, allowing videos and photos to be optionally compressed 
> and allowing WiFi or Ethernet to be switched on to download data, 
> CritterCam will remove the need for conservationists and researchers to 
> risk their lives on wobbly ladders changing batteries and SD cards.
> 
> An energy and power meter I ordered should arrive in a week or two and 
> I'll start measuring how much power a solar panel can give in the 
> typical worst case of a forest floor.
> 
> When things get really desperate an optional GSM module could be 
> switched on to send a simple SMS or MQTT message to ask someone to come 
> and change a low battery for a fully charged one and consider adding 
> more solar panels.
> 
>> C-PHY is more recent, it can provide the same bandwidth with a lower 
>> number of signals, but isn't as widely supported (I expect that to 
>> improve over time).
> 
> That's good to know - C-PHY's 3 level signalling must be giving big 
> headaches for silicon designers used to working with digital signals. 
> Good to know that for the time being D-PHY is the best supported.
> 
>> CSI-3 was killed by Intel and Qualcomm before it reached the market
> 
> That's good to know. One less thing to keep track of. CSI-1 and CSI-3 
> RIP.
> 
> Will
> 
> On 2024-12-22 10:30, Laurent Pinchart wrote:
>> Hi Will,
>> 
>> On Fri, Dec 20, 2024 at 11:04:09AM +0000, w.robertson at cairnwater.com 
>> wrote:
>>> Hello Laurent and Kieran,
>>> 
>>> > This sounds very interesting. We rarely get contacted by people working
>>> > on projects related to animals or nature. I live in Finland, so I would
>>> > be happy to help the flying squirrels from Nuuksio :-)
>>> 
>>> That's wonderful! Dormouse species hibernate but the flying squirrels
>>> can't hibernate and so must have nest holes which provide protection
>>> from weather and allow them to build well insulated nests to survive 
>>> the
>>> winter, they also ideally need nest holes with entrances which are 
>>> the
>>> right size to let them in but keep the larger red squirrels out. Past
>>> forestry practice was to remove dead trees and trees with holes 
>>> (which
>>> were seen as unsightly or unstable) and this has left a drastic 
>>> shortage
>>> of suitable nest holes for the flying squirrels.
>> 
>> Maybe that's why I've never seen a flying squirrel in Nuuksio :-(
>> 
>>> Most of my research is unpaid (with occasional small sponsorship here
>>> and there) so resources have to be used very efficiently.
>>> 
>>> By developing new rotary boring and precision chainsaw carving
>>> techniques we can carve nest holes entirely through a single narrow,
>>> very precise entrance with a much larger nesting chamber behind.
>>> 
>>> > I'll try to help, in my (unfortunately limited) free time. The best
>>> > option, if compatible with your needs, would be to have open
>>> > discussions
>>> > either on the libcamera development mailing list, or on the project's
>>> > IRC channel (you can find contact information in
>>> > https://libcamera.org/contributing.html). That way other developers
>>> > will also be able to chime in.
>>> 
>>> That would be wonderful!
>>> 
>>> > libcamera doesn't currently support the STM32MP2, but ST is working on
>>> > it. They showcased a working prototype at Embedded World in April this
>>> > year, and I assume they will provide a version usable in production
>>> > soon. While they haven't started upstreaming much of their code in
>>> > libcamera, I believe they plan to do so. The platform is therefore
>>> > interesting for low-power applications.
>>> 
>>> Thank you very much - I'd been unsure about that - that's very 
>>> helpful
>>> to know - I'll maybe try to find a way to give ST some gentle
>>> encouragement in the spring - they're pushing aggressively to sell 
>>> both
>>> their image sensors and STM32MP2 so libcamera support should be a 
>>> high
>>> strategic priority for them. I get the feeling that having launched
>>> STM32MP2 into both industrial control and domestic appliance markets
>>> they've got a fairly heavy workload at the moment and the 
>>> STM32MP257F-DK
>>> dev. board is due to be launched later than planned.
>>> 
>>> > The i.MX8MP is much better supported in libcamera at the moment, and we
>>> > are actively improving its support. There are low-cost development
>>> > boards, such as the Debix Model A that we use for development (note that
>>> > its camera connector is not standard, but they have a Debix I/O board
>>> > that can interface the Model A camera connector to a Raspberry Pi 15
>>> > pins connector).
>>> 
>>> Thank you very much! That's wonderful to have a good dev. board! I'd
>>> been looking at i.MX8MP SoMs and SiPs but some of them still seem to 
>>> be
>>> in the design stage with some technical information missing from data
>>> sheets. Will Debix Model A's LPDDR4 allow it to quickly enter and 
>>> waken
>>> from a low power sleep mode?
>> 
>> I haven't experienced with low power sleep and wake up time myself, so 
>> I
>> can't tell.
>> 
>> Generally speaking, it all depends on your power budget. If you need 
>> to
>> operate with really low power, keeping the Cortex A cores and DRAM
>> powered, even in low-power mode, may go over your power budget. Other
>> options are possible, such as turning power to the Cortex A and DRAM
>> off, and handling fast wake up tasks in a smaller core (the i.MX8MP, 
>> for
>> instance, has a Cortex M core that could be used for this). The wake 
>> up
>> time gets significantly increased, as Linux will need to boot from
>> scratch. You may be able to start powering up the camera sensor in
>> parallel from the Cortex M core to save some time, or even drive the
>> camera completely from the Cortex M, but that would require porting
>> libcamera (and drivers) to whatever OS you would be running there, 
>> which
>> would require really significant effort.
>> 
>>> The 4 lane MIPI CSI seems potentially more
>>> flexible than the STM32MP2's 2 lane CSI-2 interface.
>> 
>> 4 lanes will give you more bandwidth, but unless you target really 
>> high
>> resolutions at high frame rates, that shouldn't be necessary.
>> 
>>> I guess the best way for me to get a Debix Model A or B is just to 
>>> order
>>> it from RS?
>> 
>> I believe so.
>> 
>>> (One thing that's confusing me - when modern data sheets like the 
>>> Debix
>>> Model A data sheet say "MIPI CSI 4-lane" I guess they mean "MIPI 
>>> CSI-2"?
>>> I'd heard of CSI-1 which supports only 1 lane and seems old and 
>>> rarely
>>> used now, of CSI-2 which supports multiple lanes and theoretically a
>>> choice of C-PHY of D-PHY physical layers (though I guess many CSI-2
>>> implementations maybe don't implement C-PHY because of its 
>>> complexity?)
>>> for the physical layer and CSI-3 which is based on UniPro and rarely
>>> implemented at the moment.)
>> 
>> Today the terms "MIPI CSI" (or sometimes just "MIPI camera") refers to
>> CSI-2. CSI-1 has retired a very long time ago, and CSI-3 was killed by
>> Intel and Qualcomm before it reached the market (there were some
>> prototype implementations, but nothing in mass production).
>> 
>> D-PHY is the most common PHY for CSI-2, and should be supported
>> everywhere. C-PHY is more recent, it can provide the same bandwidth 
>> with
>> a lower number of signals, but isn't as widely supported (I expect 
>> that
>> to improve over time). I don't think the choice of PHY matters too 
>> much
>> in your case, whatever is supported by both the camera sensor and the
>> SoC will be fine.
>> 
>>> Thank you very much for all your help!
>>> 
>>> Will
>>> 
>>> On 2024-12-19 23:17, Laurent Pinchart wrote:
>>> > Hello Will,
>>> >
>>> > (CC'ing my colleague Kieran)
>>> >
>>> > On Thu, Dec 19, 2024 at 07:46:53PM +0000, w.robertson at cairnwater.com
>>> > wrote:
>>> >> Hi Laurent
>>> >>
>>> >> Thank you very much for your video "Giving Linux a Camera Stack:
>>> >> libcamera's 3 Years Journey and Exciting Future" and the wonderful
>>> >> work
>>> >> of you and the libcamera developers in bringing such a powerful and
>>> >> structured approach to such an extremely difficult and complex
>>> >> problem.
>>> >
>>> > You're more than welcome.
>>> >
>>> >> I work as a climbing arborist in practice and research and software
>>> >> engineer. My research is mostly voluntary and focused on new minimally
>>> >> invasive techniques to carve critically needed nest holes for
>>> >> endangered
>>> >> tree dwelling dormouse species (Hazel Dormouse, Garden Dormouse,
>>> >> Forest
>>> >> Dormouse), bat species and in Finland the European Flying Squirrel
>>> >> (Pteromys volans).
>>> >>
>>> >> To overcome problems and limitations with commercial wildlife cameras
>>> >> for small mammals I started work on CritterCam which - if I can get it
>>> >> to work - will be a fully open source wildlife camera system which is
>>> >> vastly better, vastly more flexible, vastly more independent in the
>>> >> field and significantly cheaper than anything commercially available
>>> >> at
>>> >> the moment. Working with embedded Linux and with several different
>>> >> manufacturers for the microprocessor and image sensors for CritterCam
>>> >> means that CritterCam users can never be price gouged and CritterCam
>>> >> will be able to adapt to update all of its critical components with
>>> >> newer alternatives well into the future.
>>> >>
>>> >> This is an article with some videos of Garden Dormice exploring our
>>> >> carved nest holes published in Flemish yesterday by Goedele Verbeylen:
>>> >>
>>> >> https://www.natuurpunt.be/nieuws/boomholtes-kerven-voor-bedreigde-eikelmuizen
>>> >>
>>> >> This is a short summary that I wrote in English and German:
>>> >>
>>> >> https://drive.google.com/file/d/1jkFA-PgDr-O9-nGSgODCUFkukj63F4fc/view?usp=sharing
>>> >>
>>> >> We have a small website at:
>>> >>
>>> >> new-homes-for-old-friends.cairnwater.com
>>> >
>>> > This sounds very interesting. We rarely get contacted by people working
>>> > on projects related to animals or nature. I live in Finland, so I would
>>> > be happy to help the flying squirrels from Nuuksio :-)
>>> >
>>> >> My background is mainly server side so I don't have kernel experience
>>> >> and I was wondering if you or another libcamera might be willing to
>>> >> give
>>> >> me any pointers to help me get started using supported MIPI CSI-2
>>> >> cameras with embedded Linux or ANdroid?
>>> >
>>> > I'll try to help, in my (unfortunately limited) free time. The best
>>> > option, if compatible with your needs, would be to have open
>>> > discussions
>>> > either on the libcamera development mailing list, or on the project's
>>> > IRC channel (you can find contact information in
>>> > https://libcamera.org/contributing.html). That way other developers
>>> > will
>>> > also be able to chime in.
>>> >
>>> >> I'm hoping to build a first prototype using the STM32MP257F-DK dev.
>>> >> board based on the STM32MP257 μP when it becomes available in January
>>> >> but I'm also looking at SoMs and SiPs based on other μPs like the NXP
>>> >> i.MX 8M Plus that support a CSI-2 interface and a low power sleep mode
>>> >> -
>>> >> I can also do some initial testing using a Raspberry Pi SBC.
>>> >
>>> > libcamera doesn't currently support the STM32MP2, but ST is working on
>>> > it. They showcased a working prototype at Embedded World in April this
>>> > year, and I assume they will provide a version usable in production
>>> > soon. While they haven't started upstreaming much of their code in
>>> > libcamera, I believe they plan to do so. The platform is therefore
>>> > interesting for low-power applications.
>>> >
>>> > The i.MX8MP is much better supported in libcamera at the moment, and we
>>> > are actively improving its support. There are low-cost development
>>> > boards, such as the Debix Model A that we use for development (note
>>> > that
>>> > its camera connector is not standard, but they have a Debix I/O board
>>> > that can interface the Model A camera connector to a Raspberry Pi 15
>>> > pins connector).
>>> >
>>> > Raspberry Pi platforms are also well supported, with the work being
>>> > performed by Raspberry Pi.