Today I decided to protect all my Raspberry Pi SD cards from unproper shutdown and also prevent SD card premature end of life. This means protect them from power outage or just unplugging the power while the Pi is running.
Edit :
April 2016 to works on Jessie and Raspberry PI 3
January 2017 Fix SSH access problem after reboot / moved resolv.conf to tmpfs
Why ?
Most of the time you won’t see or have any incidences when the shutdown is not clean, but depending on what is doing the Pi at this time, you can start to have unpredictable results or data corruption. Also, sometime, applications have really verbose logging and continuously writing log to the SD will for sure bring it to death quicker than expected. SD corruption is not really simple to detect, because sometime it works and sometimes Pi is crashing/blocking and you don’t know why. You have these “hazardous” problems until you decide to change the SD. the worst, is that you were just thinking you done something wrong and searched what you done wrong ? Get time, in doubt, change your SD card first. This is my own experience and as today, I’m on 3 dead SD card.
How ?
Well, the 1st thing I’ve done was to set some folder into temp file system (log file for example) and write the log files to the SD only every hour (a example). It’s working fine but I wanted to go deeper and have more protection, so searched over Internet people doing with read-only filesystem. I found lot of interesting article with different methods and decided to start with this one from k3a, really thanks to him.
Requirement, do it or not ?
Using a read-only file system is mostly used for Pi working 24H/7 a day and with console mode only, so if you’re using X11 or graphic interface, I won’t recommend this method because it won’t work. May be there are some other but it’s not on this article scope. So when you want you’re Pi in read-only mode it’s acting like a server but some server applications are not compatible with read only file system. Mainly database.
So if you’re running a database on your Pi, don’t use read-only filesystem. May be again there are some specific build or database to minimize write to SD, but once again it’s not in today’s scope. I see more and more people using mySQL database on Pi claiming it works fine. Sure it works, until you’ll start to face on SD corruption, and I hope they all have a good backup, because we know all, admins always have backup ;).
May be I’m the only one claiming that except if you change only few data a day, you shouldn’t use a RPi as a 24/7 database engine. For a pretty rock solid database, use a clouded one or one on a NAS like a Synology with redundant hard disk. It will go so much faster and you’ll get a real reliability. More over you will be able to use your Pi to do action and manage this database, just host your database elsewhere, just my 2 cents.
Let’s do it
Okay, I’ve done nothing yet, I’m writing this article while setting my Ambilight hyperion Rpi in read only mode and following the K3A article. You need to connect to your Pi with ssh to run the following commands, I’m connecting as root so you won’t see any sudo command (I know, I’m a bad boy!)
Get to latest current version
|
1
|
apt–get update; apt–get upgrade
|
Reboot in case of bootloader or kernel version change
|
1
|
reboot
|
Wait for reboot and connect back with SSH.
Remove unwanted package and services files
The original article removed cron and fake-hwclock, I decided to let it working assuming my Pi will have the correct date (I never had a problem with date) and also have a date at bootup that is more the last known than 1970 year. fake-hwclock will be written at logout and every day
|
1
|
apt–get remove —purge wolfram–engine triggerhappy anacron logrotate dphys–swapfile xserver–common lightdm
|
|
1
|
insserv –r x11–common; apt–get autoremove —purge
|
Replace log management with busybox one
|
1
|
apt–get install busybox–syslogd; dpkg —purge rsyslog
|
This will put log into circular memory buffer, you will able to see log using
logread
command
Disable swap and filesystem check and set it to read-only.
edit the file
/boot/cmdline.txt
and add the three words
fastboot noswap ro
at the end of the line
mine were (your can be different depending on several options) :
|
1
|
dwc_otg.lpm_enable=0 console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 elevator=deadline rootwait
|
and now looks like :
|
1
|
dwc_otg.lpm_enable=0 console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 elevator=deadline rootwait fastboot noswap ro
|
Move some system files to temp filesystem
|
1
2
3
4
5
6
|
rm –rf /var/lib/dhcp/ /var/run /var/spool /var/lock /etc/resolv.conf
ln –s /tmp /var/lib/dhcp
ln –s /tmp /var/run
ln –s /tmp /var/spool
ln –s /tmp /var/lock
touch /tmp/dhcpcd.resolv.conf; ln –s /tmp/dhcpcd.resolv.conf /etc/resolv.conf
|
On Raspberry PI 3, move some lock file to temp Files System
Thanks to @harlock for the trick
|
1
|
nano /etc/systemd/system/dhcpcd5
|
and be sure to change the line with
|
1
|
PIDFile=/run/dhcpcd.pid
|
to
|
1
|
PIDFile=/var/run/dhcpcd.pid
|
Full file should now looks something like that (on my Jessie Lite on RPI3).
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
|
[Unit]
Description=dhcpcd on all interfaces
Wants=network.target
Before=network.target
[Service]
Type=forking
PIDFile=/var/run/dhcpcd.pid
ExecStart=/sbin/dhcpcd –q –b
ExecStop=/sbin/dhcpcd –x
[Install]
WantedBy=multi–user.target
Alias=dhcpcd5
|
On Debian jessie move random–seed file to writable location
remove existing file
|
1
|
rm /var/lib/systemd/random–seed
|
link the random-seed file to tmpfs location
|
1
|
ln –s /tmp/random–seed /var/lib/systemd/random–seed
|
Since file is on tmpfs it will not be created upon, reboot, but we can do it with a kind of magic of systemd system service, this is so powerfull.
To create file on the tmp area at bootup before starting random-seed service, just edit the file service file to add a pre-command to execute :
|
1
|
nano /lib/systemd/system/systemd–random–seed.service
|
add the line ExecStartPre=/bin/echo “” >/tmp/random–seed under service section, should now looks like this
|
1
2
3
4
5
6
|
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStartPre=/bin/echo “” >/tmp/random–seed
ExecStart=/lib/systemd/systemd–random–seed load
ExecStop=/lib/systemd/systemd–random–seed save
|
Do not use touch instead of echo, it won’t work because checking RO filesystem
Execute following to tell systemd we made changes
|
1
|
systemctl daemon–reload
|
Setup the Internet clock sync
I think this one is not needed because on new Raspbian version, the ntp daemon already do the job, but just in case
|
1
|
apt–get install ntp
|
and be sure to configure your time zone, with raspi-config tool
|
1
|
raspi–config
|
then go to menu “Internationalisation Options” then “Change Timezone” and select your time zone.
Edit the hourly cron script that save clock every hour
/etc/cron.hourly/fake–hwclock
and change it to allow saving clock. It sould become
|
1
2
3
4
5
6
7
8
9
10
|
#!/bin/sh
#
# Simple cron script – save the current clock periodically in case of
# a power failure or other crash
if (command –v fake–hwclock >/dev/null 2>&1) ; then
mount –o remount,rw /
fake–hwclock save
mount –o remount,ro /
fi
|
Edit the the file /etc/ntp.conf to set redirect driftfile to writable zone /var/tmp . My file was like this
|
1
2
3
|
# /etc/ntp.conf, configuration for ntpd; see ntp.conf(5) for help
driftfile /var/lib/ntp/ntp.drift
|
I changed to this, thanks to Gregor for the tip.
|
1
2
3
|
# /etc/ntp.conf, configuration for ntpd; see ntp.conf(5) for help
driftfile /var/tmp/ntp.drift
|
Remove some startup scripts
|
1
|
insserv –r bootlogs; insserv –r console–setup
|
Now time to tell the mounted filesystem that we’re in read-only mode
Add “,ro” flag to both block devices in
/etc/fstab
to tell the system to mount them read-only:
edit the file
/etc/fstab
mine was :
|
1
2
3
|
proc /proc proc defaults 0 0
/dev/mmcblk0p1 /boot vfat defaults 0 2
/dev/mmcblk0p2 / ext4 defaults,noatime 0 1
|
And after modifications it is :
|
1
2
3
4
5
6
7
8
|
proc /proc proc defaults 0 0
/dev/mmcblk0p1 /boot vfat defaults,ro 0 2
/dev/mmcblk0p2 / ext4 defaults,noatime,ro 0 1
# For Debian Jessie
tmpfs /tmp tmpfs nosuid,nodev 0 0
tmpfs /var/log tmpfs nosuid,nodev 0 0
tmpfs /var/tmp tmpfs nosuid,nodev 0 0
|
We done, reboot
That’s it, you should now be able to reboot your Pi
|
1
|
reboot
|
If all went fine, it should boot. Well, on mine it rebooted fine and ambilight was still working, then I ssh’ed onto and issued
|
1
2
3
|
root@ambilight:~# mount | grep /dev/root
/dev/root on / type ext4 (ro,noatime,data=ordered)
root@ambilight:~#
|
you can see root is ro mode
|
1
2
3
|
root@ambilight:~# mount | grep /dev/mmc
/dev/mmcblk0p1 on /boot type vfat (ro,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,errors=remount–ro)
root@ambilight:~#
|
and /boot (fat part of the SD) is also ro mode, great !!!
In case of problem, look at syslog (or logread) and try to find out why. You can try to fix plugging the SD card in a different computer.
Switching from Read-Only mode to Read-Write and vice-versa
Now you’re in read only mode it’s fine and safe, but if you need to install, write or modify files, upgrade, or whatever that need write access, you need to be able to do it, we’ll add this possibility and in visual mode
To set system to Read-Write use
mount –o remount,rw /
and to set it back to Read-Only
mount –o remount,ro /
but I do not remember this tricky syntax and decided to improve things a little. I want to have two simple commands like
ro
for setting mode to read only and
rw
to enable read write mode. I also wanted to know on which mode I am on command prompt.
Add fancy indicating features
Ok for all users just edit the file
/etc/bash.bashrc
and at the end add the following lines
|
1
2
3
4
5
6
7
8
9
10
11
|
# set variable identifying the filesystem you work in (used in the prompt below)
set_bash_prompt(){
fs_mode=$(mount | sed –n –e “s/^\/dev\/.* on \/ .*(\(r[w|o]\).*/\1/p”)
PS1=‘\[\033[01;32m\]\u@\h${fs_mode:+($fs_mode)}\[\033[00m\]:\[\033[01;34m\]\w\[\033[00m\]\$ ‘
}
alias ro=‘sudo mount -o remount,ro / ; sudo mount -o remount,ro /boot’
alias rw=‘sudo mount -o remount,rw / ; sudo mount -o remount,rw /boot’
# setup fancy prompt”
PROMPT_COMMAND=set_bash_prompt
|
Execute this new file and look at the magic, the prompt has also changed and show up file system state
|
1
2
3
4
5
6
7
8
|
root@ambilight:/var/log# . /etc/bash.bashrc
root@ambilight(rw):/var/log# ro
root@ambilight(ro):/var/log# touch test.txt
touch: cannot touch `test.txt‘: Read–only file system
root@ambilight(ro):/var/log# rw
root@ambilight(rw):/var/log# touch test.txt
root@ambilight(rw):/var/log# ro
root@ambilight(ro):/var/log#
|
Use logout to save history and force Read-Only mode
To be sure to avoid setting back to Read-Only at logout, add the following line to the file
/etc/bash.bash_logout
(may be you need to create it)
|
1
2
|
mount –o remount,ro /
mount –o remount,ro /boot
|
If you want to have your bash history file saved and last known good date, also, put these lines instead
|
1
2
3
4
5
|
mount –o remount,rw /
history –a
fake–hwclock save
mount –o remount,ro /
mount –o remount,ro /boot
|
Of course you can also enhance the system like changing colors and/or set file system to read only after a certain amount of time. It’s just the way I’m using it.
PS : depending on your configuration, you may have set prompt elsewhere after /etc/bash.bashrc is executed ( ~/.bashrc for example), that would cause overriding the settings of /etc/bash.bashrc. So if it does not work, test by putting the lines at the end of the user ~/.bashrc profile file ( /root/.bashrc for root)
Bonus : health check using Watchdog
If you follow my blog, you probably know that I’m using micro-controller day by day. I love the watchdog feature they have and reading original K3A post, I saw we can do the same on Raspberry, so I decided to give it a try of course.
It’s for advanced users, be sure knowing what you do if you don’t want your Pi going into reset loop, most users don’t need this.
Set system to Read-Write before executing these commands, you sure remember this ?
|
1
|
rw
|
Enable watchdog module :
|
1
|
modprobe bcm2708_wdog; apt–get install watchdog
|
Edit the file and
/etc/watchdog.conf
add the following lines at the end of the file
|
1
2
3
|
watchdog–device = /dev/watchdog
max–load–15 = 25
watchdog–timeout = 10
|
On raspbian before jessie (old system init.d) set the watchdog to start at boot and start it now:
|
1
|
insserv watchdog; /etc/init.d/watchdog start
|
On Jessie edit the file /lib/systemd/system/watchdog.service and in section [Install] add the following
|
1
2
|
[Install]
WantedBy=multi–user.target
|
Always on Jessie, enable it by
|
1
|
systemctl enable watchdog
|
In addition to the watchdog, you should set up reboot after a kernel panic. This is done by editing
/etc/sysctl.conf
. Add this line:
|
1
|
kernel.panic = 10
|
This will cause to wait 10 seconds after a kernel panic, then automatically safely reboot the box.
Last test and validation
As last operation, set back to Read-Only and reboot the Pi
|
1
2
|
ro
reboot
|
References
Has I already said, I followed the original author (K3A) and just added some features I needed. Now that I have one Pi in this mode, I will update all my Pi and may be I need to add some modifications or enhancement not seen yet. If this the case I will update this post, so stay tuned.
If you want to see more precise explanations, I strongly suggest to read the excellent original K3A article located here
Thank’s to Raspbian France for providing this article image.
