Included in this document :
Structured Systems Administration | The Goals of GHOST | The Philosophy of GHOST | Tools For Results

Structured Systems Administration

In terms of industry sophistication, system administration in distributed computing environments is back where programming was in the 70s; it is dominated by a lot of excellent work being done by people and companies all over the world, but no consistent mechanism exists for developing and deploying tools across the industry.

Just as in the 70s there were super programmers who generated brilliant, but unmaintainable code, so too there are super administrators, whose work becomes useless when they leave an organisation. Just as structured programming methodologies were created to address the problems of staff turnover and programmer memory loss through the introduction of coding and documentation standards, so too we need methodologies for system administration in order to manage our systems in a more consistent, flexible, maintainable manner with less effort.

What is needed is a framework in which to develop, deploy, monitor and maintain system administration tools and practices. This framework must address the issue of standards, documentation, and staff training.

Geoff Halprin's Operational Support Toolkit (GHOST) is a framework and toolkit for structured systems administration.

GHOST represents software engineering in the systems administration context. It is based upon the principles of encapsulation, layers of abstraction, and information hiding. It addresses the mainframe goals of Reliability, Availability, and Serviceability (RAS) in the continually changing Open Systems environment.

By creating a framework which embodies the attributes of scalability, flexibility, uniformity and maintainability, we vastly improve predictability, and hence our measures of RAS.

GHOST is more than just a generalised framework, however. It comprises a number of specialised tools for the improved management of large distributed networks of Unix hosts with tools for: centralised administration, automated administration, and centralised configuration and management of regular system administration duties and house-keeping functions.

GHOST is not "just another control panel product". Off-the-shelf system management products can only capture organisational expertise. So an organisation must already know how to solve the problem before they can use such products to automate that solution. These products cannot ever replace that expertise. GHOST makes no attempt to do so.

GHOST is a framework designed to capture that expertise. Moreover, it allows corporations to harness the efforts of system administrators from around the world by deploying freely available tools from the Internet community in a controlled, secure manner.

Most importantly, it allows your corporation to harness and control the efforts of your staff, and to automate functions in a manner which is self-documenting, maintainable, and directly reduces the huge hidden costs of staff turnover.

The Goals of GHOST

In designing a framework, and in determining appropriate tools to implement as part of GHOST, a simple set of principles and a clearly defined end goal was in mind:

To develop a framework and toolkit which can be deployed at a client site as the controlling step in standardising the system administration practices at that site.

To develop the toolkit such that it will provide the necessary framework to encapsulate all local customisations to standard practices.

To encapsulate advanced, leading-edge learning and practices within the toolkit, such that less experienced system administration staff can maintain the system in a manner consistent with the practices developed and established at the site by the implementation team.

To take advantage of the work being done by system administrators from around the world, and incorporate their tools and learning within the framework and, hence, not have to develop each piece of technology ourselves.

To automate as much of the systems administration workload as reasonable, in order to maximise uptime expectations, and hence reliability and availability.

The Philosophy of GHOST

Embedded throughout GHOST is a philosophy, a simple set of principles. Of course, the nature of system administration dictates that without near total flexibility, any framework will serve to constrain and interfere rather than assist. Douglas Adams best describes our intent with the phrase:

    Rigidly defined areas of doubt and uncertainty
This can be seen in the following underlying design principles:

  1. Treat the network as a single system; not as a large number of loosely coupled, independent hosts.

    If we are to win the war of entropy, we must seek to manage functionality, not hosts. We must think of our network in terms of the services it provides to the user base, not the machines and what processes they are running.

    The GHOST framework provides for this by making it possible to think in terms of:

      Centralised Management
      Centralised Monitoring
      Centralised Configuration

  2. Deploy a uniform framework across all hosts

    In order to make good use of such a framework, we must rely on it being present on every host, and providing a base level of functionality and predictabilty. We must know what assumptions we can make; something that is not possible without this consistency of environment.


      Common framework deployed across all hosts.
      Known points of flexibility and scalability within a well-defined infrastructure.

  3. Use the network to manage the network

    Our tools should not be hard-coded with their behaviour for the week. This is not a maintainable solution. Our system administration tools must be generic, and take their behaviour and configuration information from a central repository. As the duties of a network change, the tools can then follow this without intervention.

Tools For Results

All of these good intentions are wasted if were to not supply a core set of tools which take us towards these goals. GHOST is all about results. It is about taking quantifiable steps towards controlling the entropy of a site.

Thus, GHOST comprises not only the generic framework described above, but also a number of core tools which provide an accelerated path to gaining control of your distributed environment.

Tool      Description
CHANGE      CHANGE is a tool for ensuring the consistent application of revision control and post-change actions and audit trails when modifying system configuration files.
Host Configuration Baseline      The HCB gathers local and remote configuration information into a central baseline and performs a nightly comparison of the current configuration of each host against this baseline, reporting any discrepancies found.

The HCB proactively monitors the systems under its control for change. This (a) provides for increased communication between the support team, and (b) provides a mechanism for the rollback of unauthorised changes, and an accurate history of changes which might lead or have led to system defects.

Central Administration Baseline      The HCB is a PULL system, which gathers information from each system. This captures the uniqueness of each host. (How it differs from when it came out of the box.)

By contrast, the CAB seeks to reduce that uniqueness by providing a central repository of all tools and configuration information such that the support team can centrally administer functionality, rather than individual hosts.

The CAB defines the basic characteristics of each host, and then determines from that database which files and commands to PUSH out to the managed network. When a configuration change is made, all hosts in an affected class can be updated with one simple command. This vastly reduces the overhead of software propogation and synchronisation.

JumpStart - The Next Generation      JSTNG is a framework and control scripts to manage and automate the definition of host characteristics, as required for initial operating system installation, patching and configuration. (Works in conjunction with CAB.)
HOSTDEF      The HOSTDEF Host Characteristics Database is a specialised database for the definition of host and tool characteristics, and is highly optimised for access by system administration tools, such as the other components of GHOST.
Configurator      The CONFIGURATOR is a tool for the merging of templates and HOSTDEF database entries to generate customised system configuration files on demand.
FSIB      The FSIB takes Tripwire and filesystem integrity management to the next level. By recognising the basic nature of filesystem organisation as a set of distributed information repositories, a number of distinct Tripwire repositories are individually maintained, each with its own update characteristics.

For example, an operating system repository contains only those files which form part of the operating system and, therefore, should only change by upgrade or by patch. A separate admin repository contains all those files which control the configuration of the machine, and so may change frequently. The admin repository will typically only take a matter of a minute to update, and to do so does not require re-scanning the entire host and all its filesystems, as with Tripwire in its default configuration.

This makes Tripwire easier to use, and more likely to be accurate and used by the support team as a security tool.

FSWALK      FSWALK is an advanced tool and framework to walk through local filesystems, cleaning up temporary files and gathering statistics for both change notification and trend analysis.

Contact SysAdmin to see how GHOST can be implemented at your site to reduce the overheads of system administration.

Home | Services | Clients | White Papers | About Us | Site Map | Contact Us
©1998 The SysAdmin Group Pty Ltd. All Rights Reserved.