“Terrence [Houlahan] and I were working together during what’s probably the most remarkable and extraordinary period in both our careers. While isolated in one of the most challenging parts of Africa, Terrence and I were supporting health projects by building solutions using the latest technology. The knowledge of Terrence has been an incredible asset while building eHealth projects. His drive made sure that each task would not end until everyone was contented. It has been a real pleasure working with Terrence and we can be proud of what we have achieved, all the more when taking into account the circumstances we worked in. Hopefully we will meet again one day, for example at the Four Days Marches in Nijmegen. Wish you all the best!“
– Frank Salet, GIS Team Manager, eHealth Africa (2014)
Security Challenges
Monitoring: Infrastructure
Monitoring: Networking
Network Training
Hardware & Drivers
Operating in Areas Contested by Terrorists
Northern Nigeria was one of three countries- Afghanistan & Pakistan being the other two– where Polio Stubbornly persisted. The same reason Polio persists in such places is the same reason which makes it difficult to eradicate: maintaining sanitary conditions in vast areas which hav e become ungovernable due to sectarian vilence & terrorism is a nightmare. Where the Nigerian Police & Military could maintain order in the larger cities reasonable sanitary conditions were possible. But the North is vast, so much so the government could only concentrate resources on large cities, outside of which were no-man’s lands. Although the work was technically challenging, 40C+ temperatures, dust from the Sahara and the effects of terrorism compounded the difficulties…
eHealth’s director was upfront with me: there’s no security here, nor would he purport to offer guarantees of any. Before Skyping with him, I did some Googling and found a YouTube video- since removed– showing women wailing in a courtyard filled with bodies. These were the Nigerian National Police massacred by Boko Harem at their Kano HQ located about 5 minutes from where I’d be living. When I made my way to the flight, I knew what I was signing-on for.
Shortly after arriving in Kano, Boko Harem murdered 30 people in (2) car bombings not far from where I was. The risk being no longer theoretical nor distant, one of the eHealth staff flew back to Atlanta after that. And it wasn’t long before I learned Boko Harem were sawing people’s heads of- with tree saws.
Although a lot of my work was accomplished from behind a keyboard, some required physical presence in bad neighbourhoods. Although eHealth had another Nigerian Network Engineer, he was terrified of heights. We had variability in networking in our WAN- which spanned the northern half of the country. I realized the AirFibres were slightly misalligned, so this involved climbing up masts and realigning them to within 1db. Well, when you’re the only white person around for a gazillion miles working 100 feet up a mast, it’s pretty difficult to be discrete. And of course falling from the mast is not impossible: step on a part with a brittle weld and you can be in freefall if you don’t have a sufficient grip on the laterals. But there was only 2 network engineers who weren’t afraid of heights, so you muck-in wherever you’re needed.
When planning and executing work, I never asked my Nigerian colleagues to go anywhere I would not go, nor ask them to do anything I myself would not do. I always considered and placed their safety above my own. I was acutely aware that whatever happened to me were we caught, my colleagues being Muslims in the company of a white Infidel that their fate would be much more grim than my own.
Having said all that, the Northern Nigerians are just the most decent, moral and kind folks one could ever hope to meet. Technically all very capable and with balls the size of coconuts; I was indeed privileged to keep their company both as a fellow technologist with these men of great purpose. I was no white saviour though; the success of the Polio Eradication Program was a uniquely Nigerian achievement. I did have a personal interest in solving the problem: my father was so “white privileged” he contracted Polio as a child in an Irish ghetto…
Case 1: Zabbix Monitoring Saved Infrastructure from Total Loss
Abuja had modern infrastructure being the capitol of Nigeria, so all eHealth Nigeria’s servers lived down there where users wrote date and accessed applications across their expansive WAN. Also, there was less risk of loss due to the effects of terrorism; Abuja was largely secure compared to the North. But one thing that is problematic everywhere in Nigeria is reliable power. Even in Abuja the infrastructure was reliant upon generators to keep spinning.
I asked the Abuja Polio EOC (“Emergency Operations Center“) manager what happened if the backup generator failed, to which they responded that it had never happened and was not a possibility worth worrying about. However, a good infrastructure person is by their nature a pessimist and always plans for the worst-case scenario 😉.
Notwithstanding what I was told, I proceeded to configure monitoring on the servers’ IPMI interfaces –in addition to other monitoring– in Zabbix.
A few months later while in Kaduna, a flood of alerts hit my phone: all the servers were HOT.
A call to the security guard at the Abuja EOC revealed BOTH the primary AND backup generator failed concurrently- just 10-15 minutes after all the staff left for the evening; it was a perfect storm.
So the cooling running on mains electricity was DOWN, but now all the servers connected to UPS were UP and puking heat. I instructed the guard to immediately open all the doors & windows to vent the heat and proceeded to perform an orderly shutdown of the DB and other servers.
Had I not configured the monitoring on the servers’ IPMI interfaces, disaster was assured. More than anything, I view that decision to do hardware monitoring on the key assets as one of the most important things I did for eHealth.
Case 2: Using Zabbix to Isolate Faults on a Large Complex Network
The eHealth Network spanned the Northern half of Nigeria; it was a large and complex network connecting Polio Emergency Operations Center’s in remote parts of the country. Extreme weather- heat, wind & storms– could affect the connectivity. By planning the monitoring, I was able to identify reliability & broken connectivity issues quickly to restore network services to hundreds of users scattered across the North. This would not have been possible without prior extensive monitoring experience to configure enterprise monitoring solutions such as Zabbix.
Even if it’s possible- however inefficient it might be– pings & traceroutes will only prove paths/reachability through a network: they will NOT identify reliability issues because there’s no historical data captured. Zabbix accumulated this data which made it easy to identify reliability issues and correlate them to events.
Zabbix has come in leaps and bounds in the past 12 years since it saved my bacon in Africa and is an indispensible tool for operating large complex IT infrastructures.
Network Training
Before sending an EOC Tech out to their assignments, I’d given them practical training in the Kano EOC with my excellent colleague Mukhtar who could design some fiendish broken connectivity scenarios to troubleshoot 😈. I’d go through all the key skills they needed, and we’d give the trainee practice in the Kano EOC- a network with hundreds of users. So a large part of the training was of course networking, working with a 1U local server & printers.
Mukhtar & I would introduce faults into the network- ANYWHERE in the network, not just Kano– and observe how they worked to isolate the cause of the fault. We would increase the complexity of the faults, and introduce multiple faults for the trainee to work through. By the time we were through, they had a high level of self-confidence that they could succeed and we correspondingly had confidence they would succeed.
BTW, the young lady in this picture above was not hired because of gender quotas: she was hired because she was by far the sharpest candidate out of a fairly large field. She earned her appointment; nothing was given to her. Although not a network guru, she had extensive experience as a software developer and so possessed excellent analytical & logical skills. She was EXCLUSIVELY appointed on merit; nothing else. Being a software developer, she had an instinctive skill troubleshooting complex faults which was an important role requirement (Mukhtar is standing to the right of her; he was an excellent trainer & mentor).
And while I was training the EOC trainees with Mukhtar, I was training Mukhtar to be a trainer himself to ultimately carry on after I returned to the UK.
Hardware & Drivers
One of the most serious- and pressing– problems I faced was the Polio Eradication Programme’s was the DB server tipping-over randomly. This was a problem we inherited and it was such a monkey the previous company working on it was fired; no pressure then… 😉
The logs offered no clues, but we remarked that when a new vaccination drive kick-off the problem would manifest.
It was a show-stopper of a problem threatening the Programme, and even the developers in Geneva who configured the database server had no ideas how things were breaking 🤷♂️.
I isolated the fault down to the BIOS and its’ “Green” powersaving settings. When activity dropped, the box would spin-down its’ various interfaces to reduce power consumption. That part of the “Green” power settings worked correctly.
HOWEVER: when activity picked-up again, there were no Kernel hooks to raise the interfaces back up.
I disabled the “Green” powersaving settings in the BIOS and the DB server was now ultimately performant AND reliable.
