When working with a Linux system being loaded through PXE / tftpboot / NFS in a device without hard disk it could happen that we need to understand or fix problems at boot time process, part of it is initramfs.

The procedure to unpack an initrd.img is simple:

# create a new directory
root@mainserver:/tftpboot/system# mkdir test
# copy the image into it
root@mainserver:/tftpboot/system# cp initrd_12.04 test/
# change to that directory
root@mainserver:/tftpboot/system# cd test/
# list its content
root@mainserver:/tftpboot/system/test# ls -ltr
total 11136
-rw-r--r-- 1 root root 11383089 jul  5 16:24 initrd_12.04
# extract the image
root@mainserver:/tftpboot/system/test# zcat initrd_12.04 | cpio --extract
47645 blocks
# and list again
root@mainserver:/tftpboot/system/test# ls -ltr
total 11172
-rw-r--r-- 1 root root 11383089 jul  5 16:24 initrd_12.04
-rwxr-xr-x 1 root root     7237 jul  5 16:25 init
drwxr-xr-x 8 root root     4096 jul  5 16:25 scripts
drwxr-xr-x 2 root root     4096 jul  5 16:25 sbin
drwxr-xr-x 2 root root     4096 jul  5 16:25 run
drwxr-xr-x 6 root root     4096 jul  5 16:25 lib
drwxr-xr-x 7 root root     4096 jul  5 16:25 etc
drwxr-xr-x 3 root root     4096 jul  5 16:25 conf
drwxr-xr-x 2 root root     4096 jul  5 16:25 bin

Here we’ll have access to the scripts and configuration of initramfs. I’ve modified the init script and added echo and sleep whenever was necessary  to display the different stages of the execution. Also when starting Linux I’ve added the parameter «debug» to the kernel, so this «case statement» in the init script would be executed:

        debug)
                debug=y
                quiet=n
                exec >/run/initramfs/initramfs.debug 2>&1
                set -x
                ;;

As you can see, errors will be redirected to initramfs.debug but also I wanted to see all the executions with set -x so I’ve commented the exec part.

After changes were made it’s time to pack everything again:

root@mainserver:/tftpboot/system/test# find . 2>/dev/null | cpio --quiet --dereference -o -H newc | gzip -9 > initrd_12.04
cpio: ./etc/ld.so.conf.d/i386-linux-gnu_GL.conf: Cannot stat: No such file or directory
cpio: ./etc/modprobe.d/blacklist-oss.conf: Cannot stat: No such file or directory
cpio: File ./initrd_12.04 grew, 11763712 new bytes not copied

I’ve safely ignored those errors which correspond to broken symlinks. The modified initrd_12.04 was generated and it worked perfectly.

It’s important to notice that the last step of init will be executing:

exec run-init ${rootmnt} ${init} "$@" ${recovery:+--startup-event=recovery} <${rootmnt}/dev/console >${rootmnt}/dev/console 2>&1

In this case run-init which is executed from /usr/lib/klibc/bin/run-init won’t display errors so easily due the «exec», in this case we must divide the debug process in «before run-init» and «after executing run-init». If there’s not a «panic error» usually the startup scripts, certain udev rules and other processes will be executed through SystemV or Upstart.

If the system hangs at «Stopping Userspace bootsplash» and there is no text console to interact, run-init «somehow has finished» and we should look for issues on the startup processes of SystemV or Upstart located at /etc/init.d and /etc/init respectively, disabling heavy services like X, or pulseaudio  is a lucky shot.

In my case I was stuck at «Stopping Userspace bootsplash» with no X server or text console to interact with the device, the system was just «stuck» there. Using other machine and disabling X (in init.d) lead me to a text console and finally taking a look on the services that used  «X»  I found the one causing problems, which was a personalized binary to splash an image at the boot process, after disabling it I had «X» back and everything worked.