What I did over my Summer Vacation

What I did over my Summer
or, Bring Me to Life: in which
I teach an old dog many new tricks
Call me old-fashioned, but I revel in finding vintage systems
and salvaged hardware. One of my favorite YouTube channels
is The 8-bit Guy, who
rescues old Commodores from attics and junk-heaps, and spends
considerable time scrubbing them out, soldering, tinkering, and
releasing them back into service. Then, there is “Play With Junk“, a
European fellow whose disembodied voice accompanies thorough
teardowns of “big iron” enterprise-class systems and
So when Todd offered me the task of
salvaging a decommissioned Xeon rackmount server, I eagerly
accepted his challenge. Together we lugged this beast into my
apartment, where it sat for quite a while. I had so much to do in
my second semester at MCC, and I needed to figure out a way to
position and connect the machine to my other systems.
This is a Dell
PowerEdge 2650
 system, of approximate 2004 vintage. It
features twin 32-bit Xeon CPUs at 2.8GHz with 512K L2 cache. This
is, I believe, the “Prestonia”
Xeon 2.8B
. FSB is 533MT/s (also noted 533MHz, perhaps this
is one in the same.) The bus is PCI/PCI-X with 64-bit
full-length slots
. Installed memory is 12288MB
. Five hot-swap Ultra3 SCSI
hard disk drives
 are installed in front. A venerable 3.5″
1.44MB floppy drive and standard CD-ROM optical are accessed from
the front panel. The I/O ports are all the standard PC-compatible
ports of those bygone days. Notably, there are twin 1000BaseT
Ethernet ports
 and a DRAC3 module for
out-of-band management
, also Ethernet-based.
Twin hot-swap power
supply units, complete with fans, plug into standard 110V power. A
small matrix LCD display scrolls text status messages, and a sturdy
bezel panel can be placed over the front panel, concealing the
unsightly workings behind it, and provides its own multifunction
status lamp for at-a-glance diagnostics. The BIOS revision is A17.
This machine is designed for high
 and fault-tolerance. Its
many fans and its 500W power make it a quite noisy and thirsty
guest in my home office.
I did not open the case itself, although the outside case does
indicate its long period of service, with quite a bit of grime
accumulated. It would benefit from a thorough cleanup and
On June 8, 2018, the Solemnity
of the Sacred Heart of Jesus
, I wrestled this behemoth
into my office and found a way to connect everything necessary.
Soon I decided its hostname would be “corazon”, which in Spanish
means “heart”. With the twin Ethernet interfaces, there is one for
Jesus and one for the Immaculate
Heart of Mary
I plugged in the left-side power supply to a UPS-protected outlet.
I powered on the system, and the front-panel LCD indicated
trouble with fan RPM and one of the two PSUs.
I plugged in the right-side power supply to a UPS-unprotected
outlet, keeping the left-side plugged in. When I powered on again,
the front-panel LCD still complained: “E0412: RPM FAN 2”
This led to several bouts of swap-and-test with both the power
supplies and the HDDs, as there were failures each time. I feel
that I must misunderstand the numbering scheme used by the system
or something about the hot-swap facility, because the failures did
not occur in any predictable or systematic way, such that swapping
could indicate a bad connector, port or device. I made no attempt
to remedy these PSU issues by disassembly, because that is a
high-risk and counterproductive activity for someone who is
untrained and lacks the proper tools.
At first I had assumed that these PS and FAN warnings were
fatal errors and would prevent a system boot, but as it turns out,
the system simply takes its sweet time to run POST-time diagnostics
and probes. Ultimately, I was able to boot an operating system and
begin the installation process.
I perused the BIOS and fiddled with the settings. I enabled
DHCP addressing on the DRAC module.
As I prepared to install the operating system, my next order
of business was a memtest. I had
enabled “OS Install Mode”, which had the effect of limiting RAM to
256MB. That was ostensibly some kind of Windows hack, so I disabled
it, but memtest still threw plenty of errors such as: 0x00effedc80
at 3839.8 MB.
The bezel seems to be missing the link cable for its LED.
For as long as I am performing maintenance on the system, I do not
intend to reinstall the bezel anyway. The connector seems
proprietary and specialized; it may be worth picking one up on
The HDDs gave me their share of trouble. They had arrived
without being seated, which must have been an common-sense test for
me. Once I had them firmly plugged, I found that one wouldn’t
respond; SCSI commands failed and the LED flashed amber. As I
swapped disks around, I could not quite understand whether it was a
disk unit failure, or the controller, etc. Four of the disks
are Seagate 
including the failed one, and the fifth is IBM/Hitachi, which
indicates that perhaps had been a replacement while in service.
Each has a capacity of 68.3GB.
I ran a surface scan on each disk from the Adaptec SCSI controller’s BIOS
screen. No bad block errors were indicated. I enabled RAID 5 and assembled
a new container with the 4 remaining physical disks: a 205GB
container. I triggered a SCRUB, and it ran for several hours.
Ending a second day of labors, I retired to pray and sleep through
the night.
I chose CentOS
 as the preferred Linux distribution here. CentOS is
essentially the free-download version of Red Hat Enterprise Linux.
I could not take advantage of the latest CentOS 7, because hardware
support has already been discontinued. The kernel did indicate
Store Bypass
 (Spectre-NG) vulnerability. The world is
moving to 64-bit systems; developers and vendors find it quite
difficult to support legacy hardware with diminishing returns on
their investment. Likewise, we will find that many 32-bit software
package maintainers will drop support and EOL their applications.
This is a concern from not only a standpoint of bug-fixes, but also
security vulnerabilities. Your security risks grow as you
accumulate more abandoned and unmaintained software in a world
where everyone is connected to the Internet, and new holes are
inevitably discovered in old systems.
CentOS 6.9-minimal was correctly installed,
with LUKS
encrypted filesystem
 and ext4 on top. Unfortunately,
CentOS 6 will not install with XFS. My other wish was
to experiment with ZFS, but
documentation indicated that 32-bit systems were unstable and
unreliable, so I decided to simply play it safe. I undertook an
installation of the groups “X Window System” and “KDE Desktop”,
just temporarily, so that I could have a web browser and terminal
windows while I did more work. The peculiar thing about the OS
workout is that the machine does not act like it has bad RAM, but I
would yet have grave reservations about the errors claimed by
memtest. I partitioned the filesystems into /, /home, and
Since I am quite inexperienced at encrypted filesystems, I
expect to have quite a few learning experiences in my near
future. Pavlos was wise to warn
me about learning to walk before we can run. When I decided to
replace the passphrase with a better one, I was reminded of the
importance of meticulous documentation in such matters. I could not
read my writing and nearly decided all my data was lost when the
new (and old) passphrase wouldn’t work. Then, I discovered that all
three logical volumes had their own passphrases attached, and I had
to run cryptsetup(8) multiple
times in order to first add the new one, and delete the old one: a
list of more than one passphrase can be applied to each.
I connected Ethernet to the management port, which is DRAC
III. There is no way to communicate with the DRAC. Every
modern web browser refuses to connect due to an outdated
RC4 cipher, and other HTTP protocol errors. I
Chromium Version 66.0.3359.181 (Official Build) Built on
Ubuntu , running on Ubuntu 18.04 (64-bit)
Internet Explorer 11.0.9600.19003, update KB4103768
Firefox 60.0.2 (64-bit)
Lynx Version 2.8.9dev.16 (11 Jul 2017) libwww-FM 2.14, SSL-MM
1.4.1, GNUTLS 3.5.17, ncurses 6.1.20180127(wide)

For all intents and purposes, to access the DRAC in a modern
environment, you would need to maintain a legacy OS running an
outdated browser. I came to find out that DRAC3 has been obsolete
since Windows Server 2008. The DRAC is an essential component of
datacenter operations; without this kind of remote management, you
would need to send a human operator onto the machine room floor,
plug in a keyboard and monitor, and power-cycle or diagnose the
system in person. The DRAC allows all of this to happen remotely
from the comfort of your own home, office, or coffeehouse (please
use VPNs and firewalls.)
I continued to install software, while making notes of the
items I would need to scrub before it is taken away from my
personal home network. I used “yum
” for a “kitchen sink” approach to loading software
that may be useful in such an enterprise server environment. I
configured the NFS client to mount from my desktop server, but of
course this Xeon would surely be configured as a fileserver in its
own right.
I snapped some photos of the server while it was up and running. You can see the front panel, indicating a bad HDD with amber LED to the left, and the status LCD readout on the right. The top view includes the CentOS 6.9 installation CD, the Seagate HDD in its hot-swap carrier, the detached front bezel, and one of the two hot-swap PSUs. Finally the rear view, mostly unremarkable.
In summary, I am thankful to be appointed as steward of this
Xeon server which I have christened “corazon”. It has given me
experience with a hardware RAID controller, an up-close encounter
with this and other HA/redundancy features that are now common to
commercial enterprise systems, and I have augmented the education
and experience I gained with my CompTIA A+
I am pleased to do this in service to the AZLoCo
 and the Phoenix Linux User Group  As I
work toward a degree in Linux
Network Administration
 and certifications
in Red Hat Enterprise Linux
, this is exactly the sort of
hands-on experience that will draw attention from prospective
employers and clients. All thanks and praise be to God the Father,
the Source of all blessings and graces, in His only Son, Jesus
Christ, through the working of the Holy Spirit, one God, forever
and ever.
Given by my hand,
Mr. Robert Andrew Earl
Student, Red Hat Academy, Cisco Network Academy
Class of 2020

My February Adventures with Installfest

Robert Andrew Earl CIS126RH 36343 – Mr. Dennis Kibbe Honors Project, final written report February 28, 2018

The world of Free Open Source Software (F/OSS), and that of the Linux family of operating systems in particular, is remarkably distinct from the prevailing culture of proprietary systems and closed-source software. In this paper, I intend to discuss my findings as I explored one of the first steps new users take on their journey into F/OSS: Installing Linux.

Last year as I matriculated at Mesa Community College, I also joined a group of organizations dedicated to Linux evangelism: PLUG (Phoenix Linux User’s Group) and AZLoCo, a team particularly concerned with Canonical Software’s Ubuntu distribution. One of PLUG’s major events is Installfest, which is held on the first and third Saturday of each month at the University of Advancing Technology (UAT). This report is a narrative of my experiences of attending two of these events.

Installfest is a gathering of professionals and amateurs where Linux experts can help neophytes install, diagnose, and repair hardware which is capable of running Linux and those software applications which run atop it. Among the “helper” crew are: Todd, who supervises, Walter, who works for Intel and is a crackerjack troubleshooter, Pavlos, who has helped me a great deal personally, and Don and Tammy, a married couple. UAT graciously hosts us in an upstairs classroom, with room for about 10-12 stations, and provides networking, power, and all the usual facilities.

For my first visit to Installfest on February 3, I first visited Fry’s Electronics across the street, and spent $300 on Raspberry Pi equipment, including a Pi 3 Model B in a Project Kit from Newark, formerly MCM Electronics. The kit included a 16GB MicroSD card with New Out Of Box Software (NOOBS) installed, 2.4A micro-USB power supply, clear case, 6’ HDMI cable, GPIO Breakout Board, breadboard, and assorted electronic components. Also, I purchased a 7″ touchscreen display, wireless mouse, Pi HAT module with enclosure, and extra Patriot MicroSD cards. As of this writing, I have only employed a fraction of these components in my experimentations, but it is interesting, new and different for me.

Once we got down to brass tacks and began the install, it took a mere 11 minutes with gracious assistance from Pavlos. We used the default NOOBS (which is Raspbian with a few extra tools) and it was simply a matter of choosing the correct options from a graphical menu. Post-install, I took about 10 steps to configure additional features and services, including sshd, adding user “rearl” and an attempt to pair with my Bluetooth mouse, which was ultimately unsuccessful because of immature BT support, so I used its USB wireless feature. I named my system “blaise”, after the patron of the day, a bishop and martyr. Logging in as rearl, I noticed that it is necessary to add myself to groups including “video”, for extra permissions such as starting the Xorg server.

For my next adventure, I unpackaged a fresh 32GB MicroSD card and Todd helped me locate and copy a different Raspbian install image to it. I chose stretch-lite-2017-11-29, which is based on Debian “stretch” and excludes the GUI in favor of a command-line-only interface: very lightweight. The major install glitch this time was my copying process, in which I mistakenly used “/dev/sdb1” instead of “/dev/sdb” and had a corrupted image, which could have been much worse if I’d used “/dev/sda” and corrupted Todd’s root filesystem! Once Walter helped me find my error, we recopied the image and it booted, installing with no worries. This install uses 1.1GB on the root filesystem, which is a fairly lightweight install, and could probably be pared down more. There were several important config changes needed once the install finished, and time ran short before I could fully verify full system functionality.

My second visit was on February 17 during Lent. Several days before, I resolved to bring my ancient eMachines M6810 AMD Athlon 64 laptop and, deleting Windows XP, establish a dual-boot double Linux system. I considered Kali Linux, the penetration tester’s favorite; OpenSUSE, which was recommended by Mr. Kibbe, and Puppy Linux, a compact and lightweight distro suitable for liveUSB usage. This laptop has problems, particularly a tendency to overheat and shutdown, an inability to boot from USB devices, flaky DVD drive, not to mention its age and meager resources. However, Todd and the others helped me immensely, and we finally had a working Lubuntu installation when I left that day, and which five days later I ruined in my ham-fisted attempt to install Kali on the spare partition. Highlights of my experience included Todd’s loan of the PLOP boot manager CD, which can chain-boot to the USB device; a controversy over filesystem encryption (Pavlos vehemently forbade it, but Todd seemed to be in favor); and a search for Broadcom 4306 WiFi drivers. Todd informed me that Kali requires Master Mode for effective penetration testing, and since my laptop’s hardware does not support that, I would need a new laptop or at least a USB WiFi interface if I expected to engage in such activities.

Outside of Installfest, I have been evaluating the merits of Puppy Linux, and found it to be very cool! Initial configuration is intuitive, startup features a cute “bark bark!” sound effect, and it seems that the developers took great care to make it as friendly as possible, even to an experienced user. But it is markedly different from normal distributions, due to its special-application nature. Although it was only the second out of many to successfully boot my laptop, attempts to install it to HDD engendered countless woes for me, and after hours of work I ultimately abandoned the laptop for the foreseeable future. Days later, Mr. Kibbe heard my litany of complaints and suggested that I try Debian netinst, which is a very small CD-ROM image that loads packages from the net instead of CD. Following his suggestion, I ran the install yet again and, mirabile dictu, I now have a fully functional Debian stretch installation, complete with encrypted root filesystem.

As I retired for the day, Todd offered to give me a ride home, because he was sending me forth with salvage hardware. I was gifted a Lenovo G555 laptop with a black screen, and a decommissioned Dell PowerEdge Xeon 2650 rackmount server. Since then, I have successfully disassembled the Lenovo, but diagnosing the patient remains elusive. The Dell is being saved for a rainy day.

In conclusion, I shall say that I am very grateful to have connected with the PLUG crew. Every meeting with them is productive and rewarding, as we share knowledge and wisdom about IT systems, and especially the free and open kind. I enjoy challenging myself with new horizons, and so stepping outside of my comfort zone of Ubuntu on desktops was exciting and edifying. I owe a debt of gratitude to everyone who has shouldered my burden in this journey, including Todd, Walter, Pavlos, Don and Tammy, Mr. Kibbe, Brian and Hans. Most importantly, God in Heaven above, without Whom I could accomplish nothing.