K3A

RaspberryPi is a nice, small device which can be used for various automation purposes, internet of things or as an advanced camera. I have been using 5 RPis as a camera with motion detection and post-processing. This application requires high reliability and it was a pretty long journey until all the issues were solved. Here I am writing the most important things I have learned so you don’t need to go through the same stuff again (my cameras were placed on a remote location, making the whole process more complicated).

(Please don’t mind my English mistakes, I wrote this article in a hurry, haven’t even re-read it myself yet.)

1. Connectivity

1.2 Power

A good power is the first important requirement! Make sure you use a reliable power supply with enough power – I found the best power supply to be the one from iPad 1/2. Non-name cheap chinese adapters may work for some time but will fail in couple of years due to bad caps or overall bad design.

Don’t forget to check the cable. Dead-cheap cables with “100% copper” from Aliexpress can sometimes work for data transfer but can’t transfer enough power. The thinner the cable is, the higher voltage drop it will cause on higher power draw. There are many calculators online which can compute the voltage drop for you. Raspberry needs at least 4.65V all the time. The closer to the 5V, the better.

Lower or unstable voltage can cause great variety of “strange” behavior, including random reboots.

1.2 Wifi

Of course, the first thing you need is proper connectivity, especially if you have your RPi on a remote location. This seemed easy – just use wifi dongle, set up wpa_supplicant and all done. No. There are many wifi dongles and most of them are very bad (poor signal quality). If you don’t have super-good signal on the site, definitely buy a dongle with external antenna! Next.. firmwares can be old, making frequent disconnections, or power saving enabled, see my response here http://raspberrypi.stackexchange.com/a/17179. It’s worth setting up a wifi checking script. This works for me – https://paste.k3a.me/view/449a6f1c .

I have a great success with older Atheros AR9271 – it is also well-supported by Linux. I am still using it and would buy it again.

2. Remote shell

Related to the Connectivity. You need to be able to connect there remotely. I set up SSH tunnel (reverse port forwarding) because none of the RPis had a public IP address. Here is the script (/usr/local/bin/sshtunnel.sh) https://paste.k3a.me/view/a7e07bb1. Rc-script for it is here (/etc/init.d/sshtunnel) https://paste.k3a.me/view/7d74f0ae. All RC scripts must be writable so make sure to do ‘chmod a+x script’ on them. It is just for an inspiration, you need to edit it for your setup. Basically after boot or wifi re-connection, RPi connects to my server and starts reverse-port-forwarding so I can access RPi from my server. RPi logs in to the server using a separate account and using SSH keys (so without a password) you can look up the relevant info online/google. I suggest you to set up SSH login from the RPi too so you can log to RPi from the server without password. That improves life a lot.

3. Bad SD card

To survive an unexpected poweroff, you need to have all filesystem mounted read-only (more about it later). But first… even if you have your FS read-only, a cheap SD card can destroy the data on the card itself. So I suggest you buy a more reliable SD card, ideally check http://elinux.org/RPi_SD_cards

4. Read-only filesystem (older SysV-based Raspian)

This is one of the most complicated things but when done properly, it’s not too much work and it won’t cause any problems.

Unless you have special requirements, there are only a few paths which needs to be writable. So let’s go through all steps required to make a truly, trouble-free read-only RPi:

4.1 Update your RPi so you have the recent software

It’s a bad idea to set it up with an old software. That may complicate the update later…

4.2 Remove unnecessary services and files

Here is the command. Of course, remove only what you don’t need. But this is what you probably won’t need on a headless RPi. You definitely don’t want cron on your read-only RPi unless you have external hardware clock source, more about it later.

apt-get remove --purge wolfram-engine triggerhappy cron anacron logrotate dbus dphys-swapfile xserver-common lightdm fake-hwclock

insserv -r x11-common
apt-get autoremove --purge

4.3 Install buysbox syslog

You won’t need normal syslog text files on a read-only filesystem, either. Install busybox syslog instad. It logs into memory and is very lightweight. You can then use logread command to read syslog ringbuffer from the memory when needed.

apt-get install busybox-syslogd
dpkg --purge rsyslog

4.4 Disable filesystem check and swap

Because the filesystem will be mounted read-only, there is nothing to be corrupted so filesystem check must be disabled. I say MUST because it MUST. If you don’t have an external HW clock and use NTP time sync only and you do a change to the filesystem and reboot, filesystem check will see it as an update from the future, denying further boot, requiring manual action on the site. To disable filesystem checks, specifying ‘fastboot’ to the kernel cmdline should be enough. You also don’t want any swapfiles. You can disable them by specifying ‘noswap’ to the cmdline. So edit /boot/cmdline.txt and append the following two at the end of the line:

fastboot noswap

You may need  to replace fsck.repair=yes with fsck.mode=skip.

4.5 Set up clock sync.

Because you uninstalled fake-hwclock (it won’t be able to store clock on a readonly filesystem), you need to install and set up NTP sync. Also clock keeping is poor on a standard RPi so you may consider updating time regularly (every hour or two should be enough).

I used ntpdate for this. apt-get install ntpdate. I added it to /etc/rc.local:

/usr/sbin/ntpdate -b cz.pool.ntp.org # change the ntp server according to your location

4.6 Update some writable paths

Now you need to update a few services which writes something. wpa_supplicant for WiFi is ok as it already writes to /tmp. DHCP lease is the major problem. Simple solution is to delete the old directory and make it as a symlink to tmp like this:

rm -rf /var/lib/dhcp/
ln -s /tmp /var/lib/dhcp

You can consider adding more symlinks from some /var subdirectories, especially run,spool and lock
rm -rf /var/run /var/spool /var/lock
ln -s /tmp /var/run 
ln -s /tmp /var/spool
ln -s /tmp /var/lock

Note that boot scripts may do chmod of /var/spool to 755, therefore making the whole /tmp 755. If that happens to you, consider not using the symlink for /var/spool and add a new tmpfs row to /etc/fstab to create a new temporary mount for /var/spool separately.  (thanks Martin).

4.7 Consider disabling some other startup scripts

insserv -r bootlogs
insserv -r sudo # if you plan to be root all the time
insserv -r alsa-utils # if you don't need alsa stuff (sound output)
insserv -r console-setup
insserv -r fake-hwclock # probably already removed at this point..

If you use alsamixer to set up volume level, make sure to do so in read-write filesystem. If won’t be able to store it on a readonly filesystem. It normally uses this path /var/lib/alsa/asound.state .

4.8 Tell the kernel to mount root filesystem read-only!

Finally getting there.. Add ” ro” at the end of your  /boot/cmdline.txt line.

4.9. Add “,ro” flag to both block devices in /etc/fstab

…to let the system know you want to mount them read-only:

proc              /proc           proc    defaults     0       0
/dev/mmcblk0p1    /boot           vfat    defaults,ro  0       2
/dev/mmcblk0p2    /               ext4    defaults,ro  0       1
tmpfs             /tmp            tmpfs   defaults     0       0

4. Read-only filesystem (SystemD-based newer Raspian)

4.1 Relocate some paths which need write access

mv /etc/resolv.conf /var/run/resolv.conf && ln -s /var/run/resolv.conf /etc/resolv.conf
mv /var/lib/dhcp /var/run/dhcp && ln -s /var/run/dhcp /var/lib/dhcp
rm -rf /var/lib/dhcpcd5 && ln -s /var/run /var/lib/dhcpcd5
rm -rf /var/db && mkdir -p /var/run/db && ln -s /var/run/db /var/db

4.2 Stop and mask  SystemD timers and services which won’t work on read-only machine

systemctl stop systemd-tmpfiles-clean.timer apt-daily.timer apt-daily-upgrade.timer
systemctl disable systemd-tmpfiles-clean.timer systemd-tmpfiles-clean apt-daily.timer apt-daily-upgrade.timer
systemctl disable dphys-swapfile && rm /var/swap
systemctl disable bluetooth cron
systemctl mask systemd-update-utmp systemd-update-utmp-runlevel systemd-rfkill systemd-rfkill.socket

Notice that I disabled bluetooth as I don’t need it. You should be able to get it work eventually but I haven’t tested and disabled it.

4.3 Remove unnecessary software

apt-get remove -y avahi-daemon triggerhappy bluez
apt-get autoremove -y

Avahi daemon is for mDNS to auto-discover services on the same network – you can eventually get it work if you need it but most people don’t need it and it probably needs write access somewhere.

Triggerhappy is hotkey daemon which is useless on headless sytem.

Bluez is userspace part of the bluetooth system. I don’t need bluetooth at all.

You can remove more like xserver-common, dbus, …

4.4 Change cmdline.txt to boot read-only

…and also skip filesystem check. Replace fsck.repair=yes with fsck.mode=skip. Append ro at the end. So your cmdline should end with :

fsck.mode=skip ro

4.5 Add “,ro” flag to both block devices in /etc/fstab

…to let the system know you want to mount them read-only:

proc              /proc           proc    defaults     0       0
/dev/mmcblk0p1    /boot           vfat    defaults,ro  0       2
/dev/mmcblk0p2    /               ext4    defaults,ro  0       1
tmpfs             /tmp            tmpfs   defaults     0       0

5. Watchdog

(I tested it on old sysv-based Raspian so information here may not be up to date)

It can be useful to set up a watchdog which can reboot your RPi in case something is unresponsive or eating CPU too much.

modprobe bcm2708_wdog #load BCM watchdog module
Add bcm2708_wdog into the /etc/modules so it gets loaded on boot.
apt-get install watchdog
EDIT: Newer Raspian uses bcm2835_wdt watchdog module instead.

Edit /etc/watchdog.conf:
Uncomment the line watchdog-device = /dev/watchdog
Uncomment the line with max-load-1

You can modify other parameters or find help online. Setting a minimum free RAM amount is a good idea. Before starting the watchdog, be prepared that you may have configured it wrongly and it will reboot immediately when you start it and may continuously reboot after each boot. So be prepared to modify your SD card on a different device if that happens.

Enable  the watchdog to start at boot and start it now:

insserv watchdog
/etc/init.d/watchdog start

During some modifications to your system (in read-write mode) later, you can consider disabling watchdog first. It rebooted my box once while I was doing some filesystem changes. Fortunately it booted fine for me, but it may not for you and may require manual, local fix.

In addition to the watchdog, you should set up reboot after a kernel panic. This is done by editing /etc/sysctl.conf. Add this line:

kernel.panic = 10

This will cause to wait 10 seconds after a kernel panic, then automatically safely reboot the box.

6. Add some cool utilities

5.1 syslog?

Newer systemd-based Raspian:

The situation is great with systemd-based raspian as journald is using RAM storage by default. You can additionally configure journald in /etc/systemd/journald.conf. Please follow the official documentation.

Older sysv-based Raspian:

It is helpful to receive syslog over the network. I think you can somehow enable sending busybox syslog over the network. I made my own script for this purpose, though. It reads output from ‘logread’ and sends it simply over the UDP to my server where it is saved to a logfile. Simple and helped a few times. Make sure to set up correct hostnames by editing /etc/hostname.

5.1 Cron?

Normal cron can’t be used unless you have an real external HW clock source. Because normally you can’t be sure that the clock got updated by NTP. If you can ensure it somehow, then fine, use cron. If not and using relative time is enough, you can make a fake cron using bash script, while loop and wait commands. Here is mine: (to be placed as /usr/local/fakecron.sh) https://paste.k3a.me/view/4515b0a4 . I use this startup script for it (to be placed to /etc/fakecron) https://paste.k3a.me/view/0a995efd.

7. Reboot!

Now it’s the best part. If you did everything correctly, it will boot just fine. If not, look at syslog (or journalctl -b) and try to find out why. You can fix the SD card in a different computer if the Pi doesn’t boot.

8. What to do next

Enjoy your reliable RPi. Good work! If you ever want to update the software, just remount the root filesystem as read-write temporarily:

mount -o remount,rw /

You may want to stop watchdog temporarily. Now run your apt-get etc stuff, modify what you need.. then mount read-only again:

mount -o remount,ro /

Camera

I don’t know why and if it is still true, but raspicam leaked memory for me, causing RPi to reboot after some time. I managed to improve it by disabling preview (-n) but still I set up RPi to reboot daily just to be sure……

Done

And that should be it. Hopefully it helped. If it all works, back up your SD card using dd (google it https://www.google.cz/search?q=backup+using+dd).

If you have some tips, write them in comments. I can also update the article to include more info.

Please understand that I am often busy, so if you get stuck, please try google first. Consider asking at http://raspberrypi.stackexchange.com. If it won’t help, write a comment here and I will try to help you if I can.

Sources

• http://blog.pi3g.com/2014/04/make-raspbian-system-read-only/
• http://blog.gegg.us/2014/03/a-raspbian-read-only-root-fs-howto/