About 18 months ago I worked at $VERY_BIG_ISP, whose bazillion legacy servers were interdependant in hard to spot and baffling ways. Changing foousers quota might cause an outage on a server you’d never heard of which scped files to ~foouser so a third server could pick them up from behind a firewall.
To help, they made a very nasty and bureaucratic change management system. It didn’t help (and made you choose the solution with the least red tape rather than the sanest one – see above).
Anyway, it couldn’t answer things like ’What cronjobs around the network rely on the location of a particular CGI script?’ or ’I am in single user mode and want the webserver up. What other services (database, DNS) need to be up?’. Admittedly you could stumble blindly around for the answer to these questions, but 20% of our work was clearing up the shit that had occurred when we guessed wrong (75% was wasted on the change management system, which didn’t leave much time for jobserve).
They needed to describe a tangle of components around the network with nothing much in common ( language / packaging system / OS) except an interdependency.
My basic idea was to ‘draw’ a graph describing how all this ”complexitti spaghetti” was tangled up. The nodes could be services like ’NTP service’, ’20Gb of RAID5’, ’accounts database’, ’IMAP mailbox for user@visp.com’, or whatever. Links would be dependencies,
You be able to show that to be functional, ”user@visp.coms imap mailbox” depended on cyrus, working DNS, LDAP for authentication, and so on.
With this Golden Hammer:
- system documentation would be a graph
- system deployment would be a graph you fed into a tool
- the current system state could be checked against the graph
- changes would be described in terms of updates to the graph
- system initialisation and installation would be a graph
In this happy wonderland, you’d just cross out ‘courier’, write ‘cyrus’ , hand the sketch to your robot minion and go kite flying. (Of course, some poor sod would have to sit down and document the shambles in ‘graphese’ in the first place. Exercise for the reader.)
Despite your obvious skepticism, I actually started implementing this pipedream. It was called ’ratking’ and was really just an excuse to keep my hand in with Ruby. Plus I liked the name.
After a few commits I’d reimplemented rake. Badly.
Each node was essentially just a Proc and a list of dependencies.
I had a nice test suite though.
Then I found that you could’nt serialize Procs, got a new job somewhere less doomed, and gave up on it.
I got back from holiday yesterday to find Jamis Buck has written SwitchTower, which is (at a quick glance) network rake. I’ve just started looking after a maze of twisty webservers, all alike, so his timing couldn’t be better.
Still think my name was better though.