#############################
# Blocked State Cant Reboot #
#############################

Working with systems that are in blocked state (D state in “ps” output) – means they are waiting for IO from slow media (drives etc). There is no way to get a blocked state process to return or kill, you just need to wait it out (which sometimes will probably be forever, or a very long time). So the only solution to end blocked state is a reboot, when the system comes back try to avoid the same commands that got you in a blocked state in the first place (maybe try to figure out why they blocked and try to fix that issue – a common cause of blocked D states with BTFS systems is fragmentation on the volume). However sometimes even a reboot command will get hung up. So you can force a reboot thru the kernel using sysrq-trigger.

First make sure its safe to reboot and it wont hurt data (if it will hurt data, make sure that data is backed up and customer knows the consequences). And just try these simple reboots:

If above fails, here is the fun stuff, first we analyze the system and see why its hung and where its hung:

Analyze whats stuck. Most likely it will btrfs related. Sometimes CPU backtrace output doesnt show anything useful because its the CPU is waiting on IO, so its not going to be processing anything BTRFS related.

Lets say your trying to reboot but your reboot commands are not working?

Try this forced reboot (IGNORE THIS: dont forget to change boot-flash syslinux.cfg file contents of the “default …” line to “default TechSupport” or to “default Normal” depending on what you want – TechSupport to boot to TSM, Normal to boot to Normal mode, which is the default behavior – warning: do not have an extra space at the end of the default line or else it will not work, make sure you have no typos, or you might brick the box – you can confirm that immediately after your Normal or TechSupport is a newline ‘\n’ character by running “od -c syslinux.cfg” and lookg for something like this “d e f a u l t N o r m a l \n \n”)

To do a more conventional safe reboot you can do this:

r*unRaw (take control of keyboard back from X11)
e*tErminate (send SIGTERM to all processes, allowing them to terminate gracefully),
i*kIll (send SIGILL to all processes, forcing them to terminate immediately),
s*Sync (flush data to disk),
u*Unmount (remount all filesystems read-only),
b*reBoot.

On servers that dont use X & thus dont usually dont work with display monitors (devices mainly controlled thru console or ethernet – such as NASes): I dont recommend running r, because we have no keyboard on these console devices, also dont run e and i because that will kill your processes (including the SSH and TELNET process I assume)

Instead just run s,u,b

If all else fails: reboot with power cables and power buttons

More docs:
https://www.kernel.org/doc/Documentation/sysrq.txt
http://www.thegeekstuff.com/2008/12/safe-reboot-of-linux-using-magic-sysrq-key/
https://en.wikipedia.org/wiki/Magic_SysRq_key

Leave a Reply

Your email address will not be published. Required fields are marked *