Home > Electronics, Server stuff > How to make RaspberryPi truly read-only, reliable and trouble-free

How to make RaspberryPi truly read-only, reliable and trouble-free

December 4th, 2014 Leave a comment Go to comments

RaspberryPi is a nice, small device which can be used for various automation purposes, internet of things or as an advanced camera. I have been using 5 RPis as a camera with motion detection and post-processing. This application requires high reliability and it was a pretty long journey until all the issues were solved. Here I am writing the most important things I have learned so you don’t need to go through the same stuff again (my cameras were placed on a remote location, making the whole process more complicated).

(Please don’t mind my English mistakes, I wrote this article in a hurry, haven’t even re-read it myself yet.)

1. Connectivity

Of course, the first thing you need is proper connectivity, especially if you have your RPi on a remote location. This seemed easy – just use wifi dongle, set up wpa_supplicant and all done. No. There are many wifi dongles and most of them are very bad (poor signal quality). If you don’t have super-good signal on the site, definitely buy a dongle with external antenna! Next.. firmwares can be old, making frequent disconnections, or power saving enabled, see my response here http://raspberrypi.stackexchange.com/a/17179. It’s worth setting up a wifi checking script. This works for me – https://paste.k3a.me/view/449a6f1c .

2. Remote shell

Related to the Connectivity. You need to be able to connect there remotely. I set up SSH tunnel (reverse port forwarding) because none of the RPis had a public IP address. Here is the script (/usr/local/bin/sshtunnel.sh) https://paste.k3a.me/view/a7e07bb1. Rc-script for it is here (/etc/init.d/sshtunnel) https://paste.k3a.me/view/7d74f0ae. All RC scripts must be writable so make sure to do ‘chmod a+x script’ on them. It is just for an inspiration, you need to edit it for your setup. Basically after boot or wifi re-connection, RPi connects to my server and starts reverse-port-forwarding so I can access RPi from my server. RPi logs in to the server using a separate account and using SSH keys (so without a password) you can look up the relevant info online/google. I suggest you to set up SSH login from the RPi too so you can log to RPi from the server without password. That improves life a lot.

3. Bad SD card

To survive an unexpected poweroff, you need to have all filesystem mounted read-only (more about it later). But first… even if you have your FS read-only, a cheap SD card can destroy the data on the card itself. So I suggest you buy a more reliable SD card, ideally check http://elinux.org/RPi_SD_cards

4. Read-only filesystem

This is one of the most complicated things but when done properly, it’s not too much work and it won’t cause any problems.

Unless you have special requirements, there are only a few paths which needs to be writable. So let’s go through all steps required to make a truly, trouble-free read-only RPi:

4.1 Update your RPi so you have the recent software

It’s a bad idea to set it up with an old software. That may complicate the update later…

4.2 Remove unnecessary services and files

Here is the command. Of course, remove only what you don’t need. But this is what you probably won’t need on a headless RPi. You definitely don’t want cron on your read-only RPi unless you have external hardware clock source, more about it later.

apt-get remove --purge wolfram-engine triggerhappy cron anacron logrotate dbus dphys-swapfile xserver-common lightdm fake-hwclock
insserv -r x11-common
apt-get autoremove --purge

4.3 Install buysbox syslog

You won’t need normal syslog text files on a read-only filesystem, either. Install busybox syslog instad. It logs into memory and is very lightweight. You can then use logread command to read syslog ringbuffer from the memory when needed.

apt-get install busybox-syslogd
dpkg --purge rsyslog

4.4 Disable filesystem check and swap

Because the filesystem will be mounted read-only, there is nothing to be corrupted so filesystem check must be disabled. I say MUST because it MUST. If you don’t have an external HW clock and use NTP time sync only and you do a change to the filesystem and reboot, filesystem check will see it as an update from the future, denying further boot, requiring manual action on the site. To disable filesystem checks, specifying ‘fastboot’ to the kernel cmdline should be enough. You also don’t want any swapfiles. You can disable them by specifying ‘noswap’ to the cmdline. So edit /boot/cmdline.txt and append the following two at the end of the line:

fastboot noswap

4.5 Set up clock sync.

Because you uninstalled fake-hwclock (it won’t be able to store clock on a readonly filesystem), you need to install and set up NTP sync. Also clock keeping is poor on a standard RPi so you may consider updating time regularly (every hour or two should be enough).

I used ntpdate for this. apt-get install ntpdate. I added it to /etc/rc.local:

/usr/sbin/ntpdate -b cz.pool.ntp.org # change the ntp server according to your location

4.6 Update some writable paths

Now you need to update a few services which writes something. wpa_supplicant for WiFi is ok as it already writes to /tmp. DHCP lease is the major problem. Simple solution is to delete the old directory and make it as a symlink to tmp like this:

rm -rf /var/lib/dhcp/
ln -s /tmp /var/lib/dhcp

You can consider adding more symlinks from some /var subdirectories, especially run,spool and lock
rm -rf /var/run /var/spool /var/lock
ln -s /tmp /var/run 
ln -s /tmp /var/spool
ln -s /tmp /var/lock

4.7 Consider disabling some other startup scripts

insserv -r bootlogs
insserv -r sudo # if you plan to be root all the time
insserv -r alsa-utils # if you don't need alsa stuff (sound output)
insserv -r console-setup
insserv -r fake-hwclock # probably already removed at this point..

If you use alsamixer to set up volume level, make sure to do so in read-write filesystem. If won’t be able to store it on a readonly filesystem. It normally uses this path /var/lib/alsa/asound.state .

4.8 Tell the kernel to mount root filesystem read-only!

Finally getting there.. Add ” ro” at the end of your  /boot/cmdline.txt line.

4.9. Add “,ro” flag to both block devices in /etc/fstab

…to let the system know you want to mount them read-only:

proc              /proc           proc    defaults     0       0
/dev/mmcblk0p1    /boot           vfat    defaults,ro  0       2
/dev/mmcblk0p2    /               ext4    defaults,ro  0       1
tmpfs             /tmp            tmpfs   defaults     0       0

4. Watchdog

It is useful to set up a watchdog which can reboot your RPi in case something is unresponsive or eating CPU too much.

modprobe bcm2708_wdog #load BCM watchdog module
Add bcm2708_wdog into the /etc/modules so it gets loaded on boot.
apt-get install watchdog
EDIT: Newer Raspian uses bcm2835_wdt watchdog module instead.

Edit /etc/watchdog.conf:
Uncomment the line watchdog-device = /dev/watchdog
Uncomment the line with max-load-1

You can modify other parameters or find help online. Setting a minimum free RAM amount is a good idea. Before starting the watchdog, be prepared that you may have configured it wrongly and it will reboot immediately when you start it and may continuously reboot after each boot. So be prepared to modify your SD card on a different device if that happens.

Enable  the watchdog to start at boot and start it now:

insserv watchdog
/etc/init.d/watchdog start

During some modifications to your system (in read-write mode) later, you can consider disabling watchdog first. It rebooted my box once while I was doing some filesystem changes. Fortunately it booted fine for me, but it may not for you and may require manual, local fix.

In addition to the watchdog, you should set up reboot after a kernel panic. This is done by editing /etc/sysctl.conf. Add this line:

kernel.panic = 10

This will cause to wait 10 seconds after a kernel panic, then automatically safely reboot the box.

5. Add some cool utilities

It is helpful to receive syslog over the network. I think you can somehow enable sending busybox syslog over the network. I made my own script for this purpose, though. It reads output from ‘logread’ and sends it simply over the UDP to my server where it is saved to a logfile. Simple and helped a few times. Make sure to set up correct hostnames by editing /etc/hostname.

5.1 Cron?

Normal cron can’t be used unless you have a real external HW clock source. Because normally you can’t be sure that the clock got updated by NTP. If you can ensure it somehow, then fine, use cron. If not and using relative time is enough, you can make a fake cron using bash script, while loop and wait commands. Here is mine: (to be placed as /usr/local/fakecron.sh) https://paste.k3a.me/view/4515b0a4 . I use this startup script for it (to be placed to /etc/fakecron) https://paste.k3a.me/view/0a995efd.

6. Reboot!

Now it’s the best part. If you did everything correctly, it will boot just fine. If not, look at syslog and try to find out why. You can fix the SD card in a different computer.

7. What to do next

Enjoy your reliable RPi. Good work! If you ever want to update the software, just remount the root filesystem as read-write temporarily:

mount -o remount,rw /

You may want to stop watchdog temporarily. Now run your apt-get etc stuff, modify what you need.. then mount read-only again:

mount -o remount,ro /

Camera

I don’t know why and if it is still true, but raspicam leaked memory for me, causing RPi to reboot after some time. I managed to improve it by disabling preview (-n) but still I set up RPi to reboot daily just to be sure……

Done

And that should be it. Hopefully it helped. If it all works, back up your SD card using dd (google it https://www.google.cz/search?q=backup+using+dd).

If you have some tips, write them in comments. I can also update the article to include more info.

Please understand that I am often busy, so if you get stuck, please try google first. Consider asking at http://raspberrypi.stackexchange.com. If it won’t help, write a comment here and I will try to help you if I can.

 

Sources

  • http://blog.pi3g.com/2014/04/make-raspbian-system-read-only/
  • http://blog.gegg.us/2014/03/a-raspbian-read-only-root-fs-howto/
Categories: Electronics, Server stuff Tags:
  1. bsense
    January 5th, 2015 at 10:29 | #1

    Really, really nice advice!

    I got to some similar results by the hard and time consuming way. So I really appreciate this excelkent compilation.

    I have an important comment regarding the readonly configuration, that I am pursing by now: if i apply your suggestion, would i be able to write to GPIO pins (through wiring pi, symlinks or )

    And what about mysql database updating, would it work or shall config it to be mounted in another partition or external usb sd unit.

    Your insights will be much appreciated.

    Regards

  2. k3a
    January 5th, 2015 at 13:06 | #2

    @bsense
    Writing to GPIO will work normally as you are interacting with the kernel, for example using /sys/class/gpio/ – it is not a regular file system.

    Mysql – database software requires writes to table data files each time something changes in the database (adding row, deleting, etc). Therefore it would require some read-write filesystem. As you suggested, you can mount a separate partition just for this but be careful about possible filesystem damage on power loss due to read-write mount. Some filesystems can cope better with power loss situations than others. If smaller, temporal database is enough, you can mount tmpfs (disk from RAM memory) or just simply use sqlite file(s) in /tmp/.

    Hope if helped.

  3. bsense
    January 5th, 2015 at 14:45 | #3

    @k3a
    Thank you, master k3a.

    Totally agree with corruption issues. That’s the goal that drove me to your excellent page.

    Few considerations or questions:
    You have experience on nginx, does it require to have some additional files to be mapped to tmpfs with symlinks or it keeps all logs to var/log etc.?

    In a general way, how could I check which files are being written to for a specific app?Is there any file resources tracer? Or any tool to reveal files opened for rw (despite no writing is actually performed)?

    Finally, about partitioning, obviously the partition to be written to might be worn out, but it shouldn’t affect the rootfs as long as they are “far away” in terms of the flash bytes that are written (i guess that for a single byte to be written to, the flash memory actually rewrites a complete chunk of them). Do you have any hint or rule for that?

    If i have to mount an external sd connected to the usb port there might be some performance penalties. Some rough tests showed that it hardly reaches 3 MB/s

    Regards.

    • k3a
      March 9th, 2015 at 21:06 | #4

      Hi bsence,
      sorry for responding so late, I was very busy.
      I don’t know if nginx requires any special write locations except /var but in /var it also stores some temporary files, caches, pidfile, maybe more. Some of the paths may be configurable.

      You can list files opened by a process using lsof for example (http://www.ibm.com/developerworks/aix/library/au-lsof.html). There my be an app tracking file operations but I had never used any as it was enough for me to use strace which I can recommend you – it tracks every syscall to the kernel, displaying all arguments, result codes, errno, etc. Not only you can see it open() and stat() various files, you can also use strace to see the reason why some app fails or crashes.

      Re partitioning – you are right. The problem is the way how classic filesystems work vs how a flash memory works. What we get for an SD card is a ‘wrapper’ for block devices. Internally SD card is a flash memory (http://en.wikipedia.org/wiki/Secure_Digital#mediaviewer/File:Sd_insides.png) and such memory requires special handling. A flash memory in Linux is normally /dev/mtdblock0 and you can’t directly mount it. There are filesystems tailored for such devices like JFFS2, LogFS, UniF, among other operations they manage wear-leveling and try to prevent data loss. With SD card block device we rely on the flash controller on the SD card itself! The fact is, when it needs to change bit 0 to 1, it must erase (set to 1) the whole block. And the block size is specific for each flash memory (http://www.ehow.com/info_12215785_sd-card-block-sizes.html). It may help to partition it with 2*blocksize gabs but I can’t guarantee it is enough. Various SD cards has various flash controllers and I bought a cheap Chinese clone which somehow buffered writes or something – damaging the filesystem, while a different card wasn’t doing it. You have to try.

      Hope it helped.

  4. anezch
    February 23rd, 2015 at 03:49 | #5

    Hi @k3a,

    Thank you for writing this guide. Is there any idea on making the /etc/resolv.conf file writable? I want to make a clone of my setup, but when the system boots in read only, dhclient can’t update resolv.conf with the nameservers configuration retrieved from the dhcp server.

    Obviously this will be a problem when the system is connected to a different network as the original one. Of course I can remount root as read write and restart network manually, but it will be cool if the system just works throughout different networks without manual intervention :)

    Thanks,

  5. anezch
    February 26th, 2015 at 02:56 | #6

    Nevermind, installing package resolvconf solves this problem.

  6. tcz
    March 19th, 2015 at 14:32 | #7

    Hi, read everything here (nice work) and at http://blog.pi3g.com/2014/04/make-raspbian-system-read-only/ and if you have a min, i have a few questions:

    1- On the other website, you mention a script you wrote to remount everything as RW temporarily but this was while using unionfs. On here you tell us to just use “mount -o remount,rw /” is that correct?

    2- What are the other implications, beside cron? Can it mess with my scripts that uses date (to generate a unique filename) and sleep?

    3- When you say “you can’t be sure that the clock got updated by NTP” do you mean in case there was an error like a network problem? If we rule that problem out shouldn’t cron work just fine with frequent ntp updates? or does the clock drift too much?

    Thank you.

    • k3a
      March 20th, 2015 at 11:09 | #8

      Good questions. Here are the answers:

      1. Yes, this is correct. First I tried unionfs. It worked quite well but one time the kernel killed unionfs process due to unsufficient RAM (should not happen normally but you know). Unionfs is a user-space implementation of a filesystem so it runs as a process. That crash prevented remote ssh access. And after some time the filesystem got corrupted anyway (maybe it had got corrupted during boot before unionfs was set up?). So in the end I can recommend only the true read-only solution for this.

      2. fake-hwclock must to be uninstalled in both cases (unionfs or true readonly root). Normally it is there to store the time when you shutdown or reboot, so the next time it boots it sets the stored time immediately, before doing filesystem checks etc. When you have read-only filesystem, you can’t let it store this time so it will keep restoring some old date from the time before you switched to read-only. If you don’t disable filesystem check at boot and don’t disable fake-hwclock and then modify a single file with the correct time (after NTP), the next time you boot filesytem check will fail (timestamp from the future) preventing the boot to continue, requiring manual action. So you have to remove fake-hwclock, disable filesystem check (not needed as you have it readonly most of the time anyway) and run NTP after it boots and it 1h intervals or so. I used to run cron on a readonly system with a cron rule to reboot it daily. One day I found it in an infinite reboot loop – that was because cron also needs to store some data about runs and those couldn’t be saved. After cron was started, it thought it’s time to start the @daily job. It’s just fine and actually safer to have that fake cron shell script running instead in this case.

      3. It is related to cron problem from 2. I am actually not 100% sure why it resulted it doing @daily cron job immediately after boot but I guess there are only two possibilities – either it stored some non-ideal cron state while I had filesystem mounted RW for maintenance OR simply NTP update couldn’t finish before cron @daily occurred, rebooting the system. You can try having a normal cron installed and ensure NTP is done and finishes successfully before starting cron. But I can’t guarantee it won’t do something like I experienced due to the cron unable to store its state (=store the fact that it has already run @daily job today).

      Hope it helps.

  7. Stefan
    May 3rd, 2015 at 09:20 | #9

    @k3b
    Hi k3b,

    nice tutorial – thanks a lot for sharing it. Maybe some tip for you – busybox-syslogd has an option for setting a remote logserver: “-R HOST[:PORT]”. Check http://www.busybox.net/downloads/BusyBox.html for more information.

    BTW: Do you ever tried something like overlayfs or aufs on “your solution”?

    Cheers,
    Stefan

  8. k3a
    May 4th, 2015 at 13:00 | #10

    @Stefan
    Hey Stefan,
    good tip with -R option, thanks. :)

    I’ve only tried UnionFS-fuse. It worked fine until I got OOM (out of memory) situation and unionfs process got killed as it is implemented in the user-space. :P It may be worth trying one of the in-kernel implementations but I still like this real read-only system as it is simpler and thus potentially ‘more stable’.

  9. Pete
    May 10th, 2015 at 22:32 | #11

    Hi, k3b!

    Thanks a bunch for this, this is exactly what I need for my project.

    A quick question for you, in step 4.6 you show how to add links for R/W files to tmp. Should I do this for /var/log as well?

    Thanks!

    Pete

    • k3a
      May 11th, 2015 at 10:47 | #12

      Hi Pete,
      it depends on the software you have installed. I prefer setting /tmp as a logpath in the software config file. For some software it is not possible so setting a symlink is an acceptable solution. ;)

  10. May 14th, 2015 at 23:19 | #13

    Hi k3a,
    well, excellent article, really helped me a lot, I’ve published also on my blog truly based on your work, and added some interesting stuff like prompt indicating if read only or read write and save bash history at logout. You can take a look and grab back information if you want.
    link is http://hallard.me/raspberry-pi-read-only/

  11. Jakub
    May 20th, 2015 at 13:22 | #15

    Hello,

    As stated on https://www.kernel.org/doc/Documentation/filesystems/ext4.txt “ro” option doesn’t disable journaling. You should add “ro,noload” in fstab

    • k3a
      May 20th, 2015 at 13:28 | #16

      Hi. You are right but I think in this case it doesn’t make sense – correct me if I am wrong. After a partition is mounted read-only, journal is not used as writes are not possible. So if the ro partition is not unmounted cleanly (power loss), the journal will be empty anyway. The only special case would be when you mount it read-write for maintenance and you experience power loss. In that case it is probably even better to let it replay the journal after the next boot, before mounting read-only again.

  12. rino
    June 30th, 2015 at 18:46 | #17

    Hi, I’ve tried your approach, but there still seem to be some changes made to the sd card.
    I copied the sd card with dd and created a md5 checksum. The run the raspi again and checked the the md5 checksum again and they do not match!

    Any ideas, why there are still changes made to the sd card?

    • k3a
      June 30th, 2015 at 18:53 | #18

      Interesting. Are you sure you edited /etc/fstab and set ro for / and also edited /boot/cmdline.txt to include: fastboot noswap ro ?

      Also don’t forget other partitions – like /boot. These must be mounted ro too!
      Tmpfs (RAM) filesystems like /tmp can be mounted rw course.

      You can try running this slow command to find files modified in the last 24 hours:
      find / -type f -mtime -1 -print

      Or compare images copied by dd to find differences (for example using vbindiff).
      Maybe there is still something written/modified during booting, it so, it would be really helpful to find out what.

  13. rino
    June 30th, 2015 at 20:53 | #19

    Ok, thanks for your suggestions, i’ve tried vbindiff and it seems that only a few bytes at the very beginning are different.

    Further research let me to the idea, that the superblock is modified when inserting the sd card to the PC.

    With
    sudo tune2fs -l
    the superblock content can be displayed, and it shows, that the last mount time and mount count is modified everytime the sd card is mounted on the PC.

    I will try to disable automount and retry mounting the sd card read only on the pc to.

    I’ll post the results as soon as available.

  14. teddy
    July 16th, 2015 at 06:55 | #20

    really helpful article!

    one question about : 2. Remote shell
    how many connections a server can handle?
    is vpn a better idea than this method?

    • k3a
      July 18th, 2015 at 21:08 | #21

      It creates reverse port forwarding so when connecting from the remote “master” server you can make as many connections as you need to these RPis – it is only limited by system resources / kernel settings on both sides. VPN should work too, it’s matter of preference. I consider VPN to be unnecessarily complicated for me as I have my server with SSH daemon usable and I am logged in there almost all the time. So from my server I can SSH into any of my online RPis like this: ssh -p 2203 root@localhost where 2203 is one of the forwarded ports. In this case the local port 2203 is “wired” to the port 22 on a remote RPi with private IP address.

  15. hwleijp
    July 22nd, 2015 at 01:33 | #22

    Very cool. What is the version of your pi ? Have you tried for RaspPi2 with version 8?

  16. teddy
    July 22nd, 2015 at 07:56 | #23

    seems a reasonable choice, got another option for some situation, really thanks!

  17. teddy
    July 22nd, 2015 at 08:02 | #24

    ro Mount filesystem read only. Note that ext4 will
    replay the journal (and thus write to the
    partition) even when mounted “read only”. The
    mount options “ro,noload” can be used to prevent
    writes to the filesystem.

  18. Kai
    July 23rd, 2015 at 17:46 | #25

    Hi k3a,

    Thank you for writing this detailed tutorial. It’s really helpful and I successfully get the file system read-only. But I have a few questions that may apply only to my setup:

    I partitioned the SD card to have 4 partitions in total:
    1. boot – read-only
    2. / – read-only
    3. /opt – read-write
    4. /var/log – read-write

    Q1. I can’t use wifi dongles to connect to Internet, but Ethernet works fine. Any way we can get WiFi working?
    Details: after mounting the file systems read-only and reboot, the /etc/network/interfaces file cannot be created because it’s under /etc/network/ directory and it is read-only. I have a static wpa_supplicant.conf file with all the APs in /opt/ and created a symlink for /etc/wpa_supplicant/wpa_supplicant.conf. WiFi works fine before I mounted file systems read-only so my WiFi dongle is working fine.

    Q2. I didn’t remove fake-hwclock, and I have a script written to synchronize the time with our server, so I assume in this case cron will work fine?

    Thanks,
    Kai

  19. k3a
    July 25th, 2015 at 17:01 | #26

    @Kai
    Q1 – /etc/network/interfaces remains constant for me so I let it be created before making the filesystem read-only and never had a problem with it. I am using Raspian though. This is my /etc/network/interfaces content https://paste.k3a.me/view/raw/e2c1dc61. You may need to update your startup script(s) if you are not using Raspian or make a symlink /etc/network/interfaces to point to a file on some read-write partition (e.g. /tmp).

    Q2 – cron may work as long as you are not rebooting the machine often. Even then, worst case it may do some jobs multiple times until the time is synchronised. It was bad for me because I had a reboot cron job, so just after it finished booting it triggered another reboot. :P fake-hwclock is just useless on a read-only system as it can’t write a clock timestamp during shutdown – it will just fail. So it’s safe to uninstall it, especially when you have your time sync set up.

    Hope it helps,
    K3A

  20. k3a
    July 25th, 2015 at 17:03 | #27

    @teddy
    Yes, but as long as the filesystem was mounted read-only the journal should be empty. So the next time it get mounted, it will replay the empty journal, not doing any writes. Or am I wrong?

  21. k3a
    July 25th, 2015 at 17:05 | #28

    @hwleijp
    I am using Pi1 A and B

  22. July 27th, 2015 at 09:55 | #29

    I tried the above and the system boots with the file system ro.

    So far so good, but the directoried that were linked to /tmp don’t seem to be, though I don’t recall seeing any error reports when I did the ln -s commands.

    By the way, is there any reason why apache won’t work on a system with /root ro? Similarly, could X11 and remote desktop access work? Both did, neither do now.

    Cheers, J/.

    • k3a
      July 27th, 2015 at 10:49 | #30

      If there is a tmpfs mounted to /tmp then directories symlinked to it must work (e.g. ln -s /tmp /var/run). It is possible that other software like X11 or remote desktop is trying to write into different directories which are read-only. You need to check syslog and/or logs created by that software to find out why they won’t work.

  23. Kai
    July 29th, 2015 at 22:16 | #31

    @k3a
    Thanks for your reply! I got it working. There are two things that were incorrectly set.
    1. /tmp was mounted under /, so symlinks to /tmp didn’t work
    2. I also have to symlink /etc/resolv.conf to /tmp/resolv.conf, otherwise DNS won’t work.
    Now everything is working and hopefully the SD card will be much more reliable than before!

    Kai

  24. PSS
    November 1st, 2015 at 10:54 | #32

    Hi,
    I am running OSMC on Pi 2 as an HTPC. A DS1307 real time clock is connected to the Pi, as the device is usually not connected to the network.
    I want to configure the device for direct a power off, just like a TV (without a proper shut down).
    I have a few questions on this:
    1. If make it a read only system, will I be able to directly pull the power plug without damaging the file system.
    2. In case it is a read only system, will it not save any settings that I change while running KODI (Language settings, WiFi password etc).
    3. I followed the steps mentioned by you on my OSMC in order to make it read-only. And since I am not good at Pi software, I followed the steps to the letter. But it did not work. The OSMC did not boot after making the changes.

    Could you please guide me. I don’t want to add any hardware/UPS etc to shut it down safely.

    • k3a
      November 4th, 2015 at 13:54 | #33

      1. yes
      2. it won’t because the filesystem is simply not writable. If you need to change some files, you have to do something special, like using a network filesystem (nfs) for storing these files. For temporary files, you can use /tmp or a new tmpfs filesystem. Content stored there will be lost after power off though as data from these filesystems are stored in RAM.
      3. You have a system with sound and graphical output so you shouldn’t uninstall these packages: alsa-utils, x11-common, dbus. Without uninstalling these it should work but it would help to see the boot log and errors shown there. If you can’t see anything graphically, you need to attach a UART serial interface (http://www.instructables.com/id/Connect-the-Raspberry-Pi-to-network-using-UART/?ALLSTEPS may help)

      Hope it helped.

  25. PSS
    November 5th, 2015 at 18:28 | #34

    @k3a
    Hi
    Thanks a lot for the help.
    I’ll do it again and will follow the step three mentioned in your reply.
    In case of failure, I think I’ll be able to provide the logs as the whole boot sequence is displayed on screen.
    And thanks again for the response. If I succeed this time, your article would save me a lot of effort and money that I was going to spend on additional, space consuming hardware.

  26. ivan
    November 19th, 2015 at 10:59 | #35

    Hello

    thanks for your post, I follow it and works fine but after final reboot I can’t no more access via ssh???

    Any trick to find out what’s wrong with my PI?

    • k3a
      November 22nd, 2015 at 21:17 | #36

      Can you see the boot log on screen? What distribution do you use? Raspian?

  27. franz
    December 13th, 2015 at 19:56 | #37

    Hi k3a

    thanks for this really nice story.
    It works for me generally. However I made a strange observation.
    After bootup the /tmp directory has restricted access rights and some of my startup applications do not run because they want to write a log which I ln-ked to /tmp

    drwxr-xr-x 11 root root 360 Dec 13 19:53 tmp

    Even I specify mode=777 in fstab, this does not help.

    tmpfs /tmp tmpfs rw,mode=777 0 0

    I am running the newest Raspbian Jessie. Maybe this issue has something to do with systemd which processes fstab ?

    • k3a
      December 13th, 2015 at 19:58 | #38

      That’s strange. What are the outputs of `mount` command and `stat /tmp` command?

  28. franz
    December 13th, 2015 at 20:52 | #39

    Here you are:
    root@pibplus:/home/pi# mount
    /dev/mmcblk0p2 on / type ext4 (ro,relatime,data=ordered)
    devtmpfs on /dev type devtmpfs (rw,relatime,size=242612k,nr_inodes=60653,mode=755)
    sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
    proc on /proc type proc (rw,relatime)
    tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
    devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
    tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)
    tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
    tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
    cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd)
    cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
    cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
    cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
    cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
    cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
    cgroup on /sys/fs/cgroup/net_cls type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls)
    systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=22,pgrp=1,timeout=300,minproto=5,maxproto=5,direct)
    debugfs on /sys/kernel/debug type debugfs (rw,relatime)
    mqueue on /dev/mqueue type mqueue (rw,relatime)
    configfs on /sys/kernel/config type configfs (rw,relatime)
    tmpfs on /tmp type tmpfs (rw,relatime,mode=777)
    /dev/mmcblk0p1 on /boot type vfat (ro,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,errors=remount-ro)
    root@pibplus:/home/pi# stat /tmp
    File: ‘/tmp’
    Size: 320 Blocks: 0 IO Block: 4096 directory
    Device: 1dh/29d Inode: 5375 Links: 10
    Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)
    Access: 1970-01-01 01:00:07.670000000 +0100
    Modify: 1970-01-01 01:01:41.700000000 +0100
    Change: 1970-01-01 01:01:41.700000000 +0100
    Birth: –
    root@pibplus:/home/pi#

  29. k3a
    December 13th, 2015 at 20:59 | #40

    @franz
    Hmm, this looks good. What exact error are you getting trying to write? Can you try writing something manually under the same user the process is running? e.g. `echo test > /tmp/test` or `su username -c ‘echo test > /tmp/test’`

  30. franz
    December 13th, 2015 at 21:06 | #41

    pi@pibplus ~ $ echo test > /tmp/test
    -bash: /tmp/test: Keine Berechtigung

    Sorry, thats German, but it means as you would expect: no permission (to write)
    I can change the permissions manually of course or in /rc.local with chmod 777 /tmp and then all is fine.
    But I would need the permissions right DURING startup, not at the end.
    I think the systemd doesn’t understand the mode=777

  31. k3a
    December 13th, 2015 at 21:23 | #42

    @franz
    Yep, sorry missed it’s 755 (I am working with electronics at the moment). You are right, this thread seems related although they haven’t figured out the problem :( https://bbs.archlinux.org/viewtopic.php?id=195596 Maybe it is really a bug and modifying some systemd unit file(s) may fix it. I will keep searching and report if I find something.

  32. k3a
    December 13th, 2015 at 21:31 | #43

    @franz
    It seems to be specified in /usr/lib/systemd/system/tmp.mount on my Arch linux. Under [mount] I have this line “Options=mode=1777,strictatime”. Maybe you can edit the system file or disable the system and create a new .mount file in /etc/systemd. For your reference here is the complete content of my /usr/lib/systemd/system/tmp.mount: https://paste.k3a.me/view/b15438be . Has it helped?

  33. franz
    December 13th, 2015 at 21:36 | #44

    ok, thanks, will do the same

  34. franz
    December 13th, 2015 at 21:39 | #45

    Oh, the file on my raspi looks the same. I also tried to edit the Options as 777 and deleted the other parameter but no change, still write permissions missing.

  35. k3a
    December 13th, 2015 at 22:04 | #46

    @franz
    Have you rebooted after changing the file? You can also look at `systemctl status tmp.mount`. My output says ‘Process: 273 ExecMount=/usr/bin/mount tmpfs /tmp -n -t tmpfs -o mode=1777,strictatime (code=exited, status=0/SUCCESS)’.

    There must be something else mounting it wrongly. Maybe even try something crazy like `grep -r 755 /usr/lib/systemd /etc`. This will search for 755 inside every file in specified directories.

  36. franz
    December 14th, 2015 at 19:44 | #47

    Yes, rebooted all the time of course.
    This is my output here.
    Its not looking as it should be.

    root@pibplus:/home/pi# systemctl status tmp.mount
    â tmp.mount – /tmp
    Loaded: loaded (/etc/fstab; enabled)
    Active: active (mounted) since Mon 2015-12-14 20:24:19 CET; 1min 55s ago
    Where: /tmp
    What: tmpfs
    Docs: man:fstab(5)
    man:systemd-fstab-generator(8)
    Process: 143 ExecMount=/bin/mount -n tmpfs /tmp -t tmpfs -o rw,mode=1777 (code=exited, status=0/SUCCESS)

    Dec 14 20:24:19 pibplus systemd[1]: Mounted /tmp.
    root@pibplus:/home/pi# ls -l /
    drwxr-xr-x 10 root root 340 Dec 14 20:25 tmp

  37. Alex
    February 28th, 2016 at 23:45 | #48

    Mostly worked for me, but am getting a couple of errors during boot up. The one that seems most serious is, I think, related to dhcpcd:

    pi@ospi:/etc/network/run $ systemctl status dhcpd.service
    Failed to get D-Bus connection: No such file or directory
    pi@ospi:/etc/network/run $ sudo systemctl status dhcpd.service
    ● dhcpd.service
    Loaded: not-found (Reason: No such file or directory)
    Active: inactive (dead)

    One significant piece of fallout is that DNS lookup does not work. Where should I look to try to run this down. Please don’t assume that I’ve checked the places that would be obvious.

    Thanks.

  38. Alex
    March 1st, 2016 at 07:11 | #49

    @Alex
    Nevermind. I think I’ve fixed it. It was really a systemd problem requiring adjustment of the associated service file so that it wrote the PID file to /tmp.

    Thanks

  39. Ian
    March 6th, 2016 at 05:24 | #50

    Hi,

    I found this to be very helpful, and I still use it as reference. I do have a question, do you think it would be possible to extend this so that the file system is contained in a loopback file on the FAT formatted boot partition? Where I work I’m receiving resistance to additional Raspberry Pis because the Windows centric technicians don’t understand how the file system is ‘hidden’ in the ext3/4 partition. Using a loopback file might allow all files necessary to boot a Pi to be on a standard FAT formatted SD card where the user simply needs to copy the necessary files, even on a Windows box.

    Thanks

  40. k3a
    March 13th, 2016 at 19:04 | #51

    @Ian
    It should be possible but would require a custom initramfs image/script. You would edit initramfs line in config.txt on the FAT boot partition to point to a new initramfs. And that initramfs, in addition to existing stuff, would mount a loopback filesystem image as root (using commands like mount built-in into that initramfs). It’s more advanced operation and I never did this before so it’s your turn, I guess.. ;) but it should be possible, linux is highly flexible (unlike Windows).

  41. mjs
    April 9th, 2016 at 10:50 | #52

    Hi,
    I use my Rasberry as a NAS server. Is it recommend to set read-only mode then? I tried do it but sth came wrong and my computers don’t see rasberry now. I set everything the same like in that tutorial https://hallard.me/raspberry-pi-read-only/ but there are some differences between that and yours. Now I don’t know if I should reset everything and stay in read-write mode or try to do set read-only once again with your tips..

  42. k3a
    April 11th, 2016 at 15:20 | #53

    @mjs
    Hi. If you have external disk(s) used for storing data, then it makes sense to make you SDcard partitions read-only to improve SD reliability. It is difficult to say what a problem is in your case without seeing logs and actual filesystem. If you want to spend more time, you can try using some serial console, or try the whole process again. If you think it’s too difficult for you then it’s probably not worth it. From my experience, it also highly depends on SD card quality. Today sdcards are very cheap but also very unreliable. This is a great read if you want to finally know why sd cards are so bad nowadays http://bunniefoo.com/bunnie/sdcard-30c3-pub.pdf looks like the best idea is puchasing an sdcard which has a datasheet, ideally SLC, made with military or industrial specs. Consumer-grade cards from lower-end price spectrum are terrible!

  43. Andrew
    May 25th, 2016 at 07:06 | #54

    Thank you for this, it was very helpful.
    Unfortunately, I was using it to protect an installation that was running video-looper on startup and now video-looper will no longer autorun. do you know what I might have removed or changed that would interfere, and is it possible to use this program with a read only card?

  44. mefi
    May 27th, 2016 at 08:58 | #55

    Thanks for this. It really gives the stability to micro SD card that I need.

    I have one question that may not be really about this topic, but in some way it is. I developed a Qt application which is basically a dummy application. It receives messages and images over UDP sockets, and shows them on the screen. The problem I have is that when I am in read-only mode application just aborts after about half a second or so. When it is run in read-write mode it works great.

    After removing chunks of code and testing it in read-only mode I found out that if application don’t process received messages, which i really have to since that’s the only functionality application has, it doesn’t get aborted. I am not that good with Linux to be honest, but what seems logical to me is that when I read that data in application it needs to be stored somewhere, and that somewhere is in read-only part of memory.

    Does what I wrote make any sense? If yes which parts of file system need to be mounted as tmpfs to fix it? If not what is actually happening and is it possible to fix it somehow?

    By the way I am using Raspbian Jessie Lite if that means something to you.

    Thanks in advance

  45. Tomas
    May 27th, 2016 at 13:46 | #56

    @Alex
    Hi. Can you please post how you fixed it – I have the same problem with latest jessie/systems in ro ? Thanks. Tomas

  46. k3a
    May 27th, 2016 at 22:53 | #57

    @mefi
    It doesn’t appear to be a socket problem. A temporary memory used for received data you mentioned is owned by the kernel and resides always in RAM. There must be something else, maybe Qt itself trying to write something somewhere. To debug it, I suggest using strace command (strace path/to/your/program). It will show you last syscalls before it aborts.

  47. Hahn
    June 3rd, 2016 at 08:21 | #58

    Just a great helb to get started with ro :-)

    Is this kompatible with apache?

    I tried to build a RPI – SVN-Server with:
    sd-card = ro
    2xUSB-stick as btrfs raid1 mounted to /media/usb_raid1

    I folled jeremy’s intruction to install svn + apache for svn webaccess:
    https://www.jeremymorgan.com/tutorials/raspberry-pi/raspberry-pi-how-to-svn-server/

    BUT: unfortunately apache doesn’t want to start … because of missing dbus?
    Is there a way to use application that usually expact dbus?

    cu,
    Hahn

    • k3a
      June 3rd, 2016 at 20:37 | #59

      This is quite difficult to help as it is an application-specific problem. You will have to look into log files and potentially use strace tool to find out why it won’t start. It should be possible to configure dbus to use the tmpfs location for writes.

  48. kxb3292
    June 6th, 2016 at 04:42 | #60

    Hello,

    I have a setup of raspberry pi 3 with raspbian jessie installed. On top of it I have installed kodi v16.
    I have been able to switch read only/read write mode using this tutorial.

    However, when i put the system in read only mode and try to launch kodi I get the following message:
    “””
    Could not init logging classes. Permission errors on ~/.kodi (/home/pi/.kodi/temp/)
    ERROR: Unable to create application. Exiting
    Couldn’t get a file descriptor referring to the console
    “””
    Is there a way to make kodi write its logs to a different location (like RAM?) or make it stop writing all logs all together?

    I would really appreciate any help!

    Thank you so much,
    kxb3292

    • k3a
      June 7th, 2016 at 14:18 | #61

      It doesn’t seem like Kodi allows you to specify a different location for ~/.kodi or its subfolders but you can always make a symlink.
      It is best to keep most read-only files where it expects them and just make symlinks to folders where it needs to write during a normal working state.
      I think it always needs to write some log file.

      So something like this may be enough:
      rm -rf /home/pi/.kodi/temp # remove old temp folder
      ln -s /tmp /home/pi/.kodi/temp # link Kodi’s temp to /tmp

  49. frggu
    October 7th, 2016 at 11:04 | #62

    Hello,
    this is how to get samba server working on a read-only filesystem:

    #go into rw mode, install samba:
    rw
    sudo apt-get install –reinstall samba samba-common

    #Kill samba:
    sudo service smbd stop
    sudo service nmbd stop

    #Copy over the samba folder to your home folder and delete old folders:
    cp -R /var/lib/samba /home/user/samba
    sudo rm -r /var/lib/samba
    sudo rm -r /var/cache/samba
    sudo rm -r /var/log/samba

    #create a startup script
    sudo nano /etc/init.d/fixsamba.h

    #with content:
    #this copies the samba folder from our home folder to the temporary fs on bootup and creates
    #necessary folders for samba to work with on the temporary fs

    #!/bin/bash
    mkdir /tmp/samba
    cp -R /home/user/samba /tmp/samba/lib-samba
    mkdir /tmp/samba/cache-samba
    mkdir /tmp/samba/log-samba
    sudo ln -s /tmp/samba/log-samba /var/log/samba

    #Execute this once, make executable, set startup:
    sudo chmod 755 /etc/init.d/fixsamba.h
    sudo /etc/init.d/fixsamba.sh
    sudo update-rc.d fixsamba.sh defaults

    #Create some more symlinks
    sudo ln -s /tmp/lib-samba /var/lib/samba
    sudo ln -s /tmp/cache-samba /var/cache/samba
    sudo ln -s /tmp/log-samba /var/log/samba

    #reboot
    sudo reboot

  50. Mac
    November 17th, 2016 at 17:22 | #63

    Hi,

    I just followed the steps outlined and everything, even my web server on the pi, is working…except, now I can’t ssh into the PI.

    What’d I get wrong?

    ssh worked before reboot, that’s what I used to make it read only.

    • k3a
      November 17th, 2016 at 18:58 | #64

      Hi. SSH doesn’t require write access by default. It’s difficult to diagnose the problem without inspecting logs but it looks like SSH either isn’t running at all or can’t authenticate you. It may not start because of missing /etc/sshd directory for example. If it is running, check that /etc/passwd and /etc/shadow are accessible by root. If you are using ~/.ssh/authorized_keys, make sure to check .ssh directory and authorized_keys ownership as SSH refuses to authenticate user with insecure permissions to .ssh directory. You can check/fix the SD card on a different computer with GNU/Linux or using a serial (UART) connection to Pi.

  51. Mac
    November 17th, 2016 at 19:45 | #65

    I poked around a bit since there is still a monitor and key board on the PI. There is no ~/.ssh folder. there is no /etc/sshd folder. Root has access to /etc/passwd and /shadow.

    but ssh returns “Connection Refused”.

    I think I saw a failed start OpenBSD ssh message during boot, but logread dosen’t show it.

    I also noted that, for example, “ls -la /” shows /var, /etc., basically /* as writable by root.

  52. Mac
    November 17th, 2016 at 19:49 | #66

    Also, before this ro attempt, the PI showed up on my network with a host name. Now it thinks it’s host name is correct, but the hostname doesn’t not show up under the dhcp lease. Ping finds the IP but not the hostname (as would be expected since it doens’t show up with a hostname on my router.)

  53. Mac
    November 18th, 2016 at 17:07 | #67

    I’m lost.

    I mounted / rw and used raspi-config to enable ssh. No errors, but couldn’t ssh to the pi before or after reboot.

    Since the response is “Connection Refused” and I don’t see anything in the process list with ssh I’m guessing the ssh server isn’t starting.

  54. Mac
    November 23rd, 2016 at 16:48 | #68

    I have re-done this on a fresh image with the same results.

    It appears that several things (dbus, dhcp, random-seed) don’t start with the ro mounts.

    If mounted rw, then dbus starts and ssh works.

    logread complains: unable to create /var/log/lastlog: Read-only file system

    So what am I missing or doing wrong?

  55. Mac
    November 23rd, 2016 at 17:03 | #69

    Sorry for all the follow up posts.

    It appears I’ve figured it out. I’m running jessie and I added the following to fstab:

    # For Debian Jessie
    tmpfs /tmp tmpfs nosuid,nodev 0 0
    tmpfs /var/log tmpfs nosuid,nodev 0 0
    tmpfs /var/tmp tmpfs nosuid,nodev 0 0

    As per instructions here:
    https://hallard.me/raspberry-pi-read-only/

    This seems to have fixed the ssh access.

    It still takes a long time when it runs the dhcpd on all ports, but it does appear to dhcp on the wired port.

  56. k3a
    November 23rd, 2016 at 17:04 | #70

    @Mac
    Have you followed the steps correctly? dbus should not be installed at all, see “4.2 Remove unnecessary services and files”. The article was primarily focused on read-only headless setup (without graphics with SSH/local access only).

    It must be possible to configure it to do whatever you want. In that case, you have to help yourself, though. The idea is:
    1. remove services you don’t need,
    2. from services you do need, make sure they doesn’t make any unnecesary logs or files.
    3. If they need to make files for their functionality (like DHCP mentioned in the article), make symlinks to /tmp or change paths to these files in the service config so that they points to a file in /tmp. Eventually you can create a new tmpfs mountpoint for some service(s) to separate them (/tmp is tmpfs by default so contents “live” in RAM which allows many safe rewrites).

    So in short: what NEEDS writes must be configured to write to /tmp (or redirected using symlinks). Inspect logs to see what needs write (for example by attaching a montor and keyboard, trying to run the service manually and watching logs and/or output or using strace).

    If lastlog is a problem, try make a symlink for it too: ln -sf /tmp/lastlog /var/log/lastlog or if you don’t need lastlog at all, symlink it to /dev/null: ln -sf /dev/null /var/log/lastlog

  57. k3a
    November 23rd, 2016 at 17:06 | #71

    @Mac
    Yep that ‘fixed’ it because the whole /var/log is being mounted as tmpfs. While it works, it’s not a good idea to just do this without inspecting it all. If there are some services making lots of logging, it will fill up your RAM after some time, forcing out of memory killer and killing random services and/or triggering watchdog and rebooting your RPi.

  58. Mac
    November 23rd, 2016 at 18:39 | #72

    I followed the steps very carefully.

    Yes, I agree dbus is removed with the first apt-get. I don’t know what’s up. But, when the system is ro, without tmpfs, it throws errors during boot about starting login service and can’t connect to dbus. (not just once but several times. trying to read as it scrolls by is difficult) And I can’t find any log that mentions the issue. the error also says to use systemctl status systemd-logind.service and dbus.socket but any attempt to use systemctl responds with: Failed to get D-Bus connection: Connection refused

  59. k3a
    November 23rd, 2016 at 18:52 | #73

    @Mac
    Ah, yes. The new distributions are systemd-based which can make it more difficult to setup and debug (I would fear of journalctl databases). :( Additionally it’s possible systemd requires dbus. You should be able to get dbus working by making /var/run/dbus (and/or ~/.dbus) tmpfs or symlinking it. It’s all matter of trial and error. I am sorry the article can’t help you completely but it was written in 2014 for non-systemd Raspian and I don’t have RPi available for testing at the moment so please keep letting me know how it goes.

  60. Mac
    November 23rd, 2016 at 19:15 | #74

    Hmm…well, I added /var/run/dbus tmpfs to fstab (note: only /var/run/dbus, no additional tmpfs mounts) and ssh works now.

    I’m not sure what implications /var/run/dbus on tmpfs might have (i.e. space issues) but, it appears to be working.

  61. k3a
    November 23rd, 2016 at 19:17 | #75

    @Mac
    Check what files are being created in that directory but there is probably just one socket file in which case memory won’t be a problem. :)

  62. Mac
    November 23rd, 2016 at 20:38 | #76

    Yes, that appears to be the case.

    I had to add tmpfs /tmp to get Avahi and random seed to not fail.

    It still takes a 1min 39sec to run dhcp on all interfaces and then fails.

    But it appears to get an IP address…

  63. Mac
    November 26th, 2016 at 18:13 | #77

    Update; got to spend some more time on this:

    dhcp is still failing
    that means dns is not working
    ntp fails (guessing, because dns is not working…)

  64. Mac
    December 5th, 2016 at 14:54 | #78

    I started over again with a fresh image of jessie.

    I followed these steps excluding removal of dbus since it is required by avahi.

    After reboot, creation of randomseed fails and dhcpcd fails.

    I’m guessing this is because /var/log is read only. In fact all the links made (/var/run, /lock etc. are read only. Is this because / is ro and /tmp is mounted under / ? )

    Interesting that the last dhcp lease (which is now in a read only directory/file is used. After the lease times out, the pi can still be accessed by the IP, but not the hostname (expected since the hostname is no longer associated).

    Since /var/log is read only, there are no logs to see to figure out the dhcpcd fail. It could be as simple as the daemon can’t log so it fails…

  65. k3a
    December 5th, 2016 at 21:13 | #79

    @Mac
    Everything on root filesystem is read-only – that’s the point of this guide – no writes to SD card so it won’t wear off. That why we are mounting root filesystem ro = read-only. Then, on top of this root filesystem, you can mount other filesystems on various paths, for example at /tmp there is a filesystem of type tmpfs mounted which is made out of RAM memory and that one is mounted read-write as it is safe to write in RAM memory many times – it’s a different kind of memory than that cheap SD card. You can see all filesystems mounted and their mountpoints using `mount` utility.

    For logging, you can use that busybox-syslogd from Busybox – syslogd captures log messages and stores them in a fixed RAM memory buffer, overwriting the oldest one when allocated memory is full. You can read stored logs using `logread` utility.

    Unfortunately not every app can log into syslog and some apps write directly into log files. Normally, when all is set up, you shouldn’t need any log files. Using syslog should be enough. Of course you could just mount /var/log as tmpfs but that’s bad because if some app would make many logs / is logging too much, it would fill up your RAM and system won’t have enough RAM remaining to work correctly and it may even reboot. You can do that temporarily during experiments though. And you can even limit the maximum size of tmpfs mounts – using the `size` parameter like size=3m for 3 mega or percentage of physical ram, for more read the doc of tmpfs https://www.kernel.org/doc/Documentation/filesystems/tmpfs.txt

    Something like mount -t tmpfs -o size=3m /var/log would mount a 3m tmpfs filesystem at /var/log. To unmount, you would use umount /var/log. To mount during boot, you have to rewrite this command into /etc/fstab like this:

    tmpfs /var/log tmpfs size=3m 0 0

  66. Mac
    December 8th, 2016 at 19:02 | #80

    To clarify: I meant that logread provides no indication as to what happened during boot. Particularly, it has no information about dhcpcd failing, random-seed failing, etc.

    Man pages and web searches on dhcpcd have provided no info (at least for me) as to what it needs to be writable or other requirements (for example, something that might have bee removed that it needs. Such as the case with dbus and avahi). I have attempted to move (ln -s as was done for other directories) /var/log/dhcpcd to /tmp, but dhcpcd still does not start.

    After boot, systemctl status dhcpcd.service reports: “dhcp_bind: write _lease: Read-only file system”. That might be a clue, but, I have not found where a lease is written.

  67. k3a
    December 9th, 2016 at 09:36 | #81

    @Mac
    /var/lib/dhcpcd? Have a look at the dhcpcd config file too. And here is the man https://linux.die.net/man/5/dhcpd.leases

  68. vadbut
    February 10th, 2017 at 04:26 | #82

    Hi, thanks for an awesome tutorial!
    My task requires logging and I’m thinking of creating a separate partition on the SD card specifically for log files, and making the main partition read-only as you described above. I understand that log files might occasionally get corrupted, but is it safe for the rest of the system?
    Thanks

    • k3a
      February 16th, 2017 at 21:09 | #83

      Hi, this is of course possible but I would see two potential problems:

      1. SD cards are not using continuous areas of physical memory (unlike HDDs). Instead, as you write, they recycle least-recently used sections (to prevent wear). I don’t know if one partition can recycle a section previously used by another partition. If so, then partitioning won’t help you at all. But I think that’s highly probable unfortunately because SD card doesn’t know anything about partitions itself. It behaves like one disk and SD controller is doing the wear prevention – you can actually use any partition scheme (MBR/GPT/whatever) like you can on your HDD.
      2. If partitioning still helps then there is another problem – if your log partition fails to mount because of FS corruption, some daemons may not start as they won’t be able to write into a logfile – that can potentially make the system unusable.

      So really the best solution is having SD card read-only. And use some other way of storing logs – in memory or for example sending them over the network into another box using traditional storage.

  69. Michael
    March 22nd, 2017 at 01:51 | #84

    Hi,
    the watchdog module name has changed to “bcm2835_wdt” for newest raspbian. “bcm2708_wdog” does not work anymore.

    btw.: thanks a lot for the tutorial!!!!!

  1. April 8th, 2015 at 08:51 | #1
  2. May 14th, 2015 at 16:50 | #2
  3. February 20th, 2016 at 16:53 | #3
  4. November 2nd, 2016 at 00:10 | #4