PROWL

Synopsis:

prowl [-s] <bash.script>

Download:

The source for this software can be obtained from here.

The supporting paper can be obtained from here.

Description:

The prowl program was developed by heavily hacking the open source code for bash-4.2. All changes to the original source code are bracketed by #ifdef IJD, making it easy to both identify changes and to compare this code to the original bash-4.2 code.

Powl exploits all of the pre-existing bash functionality to read and parse an arbitrary bash shell script, ignore comments, and to source other referenced bash shell scripts, invoke functions, etc. In particular it assigns values to variables just as bash would, expanding as necessary other variables as part of this exercise.

It differs from bash-4.2 in not invoking either external programs or built in bash commands (with the exception of the built in source command) thus ignoring bash statements such as return and exit, and in following all control paths within the bash script exactly one. It thus executes both the if and the else block within each if statement, and every possible clause specified within a case statement. It executes while and until blocks exactly once irrespective of the result returned by the corresponding test. The intent is to ensure that every logical line of a bash script is at least visited once by prowl.

During the execution of prowl the bash variables are internally annotated with flags that indicate the potential usage made of each variable, and thus the potential relevance of deeming the variable externally configurable with a user specified value. These flags (currently) are:

• USED (Some logical use is made of this variable. IE. at some point the bash script looks up this variables value)
• ASSIGNED (This variable is assigned a value using the assignment command)
• CHANGED (This variable assumes multiple distinct values when examined using prowl)
• EXIT (This variable is used with a return or an exit statement)
• ENV (This variable is initially duplicated from an external run time environment variable)
• SOURCED (This variable is only used in one or more of the sourced shell scripts)
• DERIVED (This variable is assigned a value computed in part from other variables)
• LOOP (This variable is assigned a value within a looping construct)
• FUNC (This variable is assigned a value within a function)
• CONFIG (This variable is assigned using prowler)
• BADEARLY (Variable is used before being assigned a value by prowler)
• BADLATE (Variable is assigned a new value after being set by prowler)

Other flags might easily be added as future requirements dictate.

At end of run those variables marked as USED by prowl are emitted (on standard output) in an XML format to a prototypical configuration file. The above flags are expressed as cumulative XML attributes within the output XML file as follows:

• Not ASSIGNED then tested='y'
  • This variable may never have been involved in an assignment command as consequence of having been using as the argument of a for statement, or may potentially be a configuration value that is tested to see if it exists, but is not assigned any default value when it does not.
• CHANGED then changed='y'
  • Within the execution across all possible paths within prowl this variable assumed multiple distinct values. Typically such variables would not be candidates to be considered a configuration parameter since configuration parameters must assume a constant value. However, a sloppy coding style within scripts is to add alternative values for a configuration parameter and not to comment out earlier now incorrect values for this parameter.
• EXIT then exit='y'
  • This variable was used as the return code of a bash return or exit statement. Such variables would not normally be deemed configuration parameters since their primary function is to assign meaningful names to internal return code values within scripts.
• ENV then env='y'
  • This variable was assigned its initial value from the corresponding named external environment variable. Typically such variables would only be candidates for being treated as configuration parameters if one was already using the run time environment within which a script was executed to determine how a script should be configured at execution time.
• SOURCED then sourced='y'
  • This variable was referenced in scripts sourced by the initial bash script, but not used in this main bash script. If the main script is performing most of the real work, it may be that the variable is not being used, and can be removed from further consideration. However, it is always possible that the core scripting code is itself contained within sourced bash scripts.
• sources='<source1>[, <source2>]*'
  • The list of sourced scripts in which this variable is referenced if any
• DERIVED then derived='y'
  Somewhere within the bash script this variable is assigned a value that is derived (in part) from the current value of other bash variables. Such variables are typically strictly internal to the bash script and so should not be considered to be configuration parameters. However, it is possible that a script becomes more general by permitting such derived values to be assigned initial values determined by a configuration file.
• LOOP then loop='y'
  • Somewhere within the bash script this variable is assigned a value within a for, while or until loop construct.
• CONFIG then config='y'
  • This variable is assigned a value by prowler
• CONFIG and !TESTED then unused='y'
  • This configuration variable is never used
• BADEARLY then BADEARLY='y'
  • This configuration variable is tested before being set by prowler
• BADLATE then BADLATE='y'
  • This configuration variable is assigned a value after being set by prowler

If the -s option is specified then the output is shortened to show only those variables most likely to be deemed configuration variables. Specifically variables that having any of the following flags are not then printed:

• EXIT
• DERIVED
• LOOP
• FUNCTION
• ENV
• !ASSIGNED

It is recommended however that the -s option is generally avoided since it is safer to manually remove potentially invalid configuration variables, than to never see variables that may still potentially be considered configuration variables.

The XML File Format

The XML file format involves 4 layers of hierarchy:

1. The outermost root entity is labelled <prowl> and confirms the XML file is a prowl configuration file.
2. The next entity level identifies the application layer, and may be repeated to support multiple distinct applications within a single XML file if this is deemed desirable. This entity has an optimal attribute version, naming the default configuration for the application if none is provided.
3. The next entry level is repeated with differing tags matching arbitrary version/configuration options associated with the outer named application. Distinct configuration versions of an application will typically describe the same parameters but with differing values.
4. Within each application's version are parameters identified by entity tag name, and parameter values identified by the content enclosed with such tags. Parameters may have attributes as described above. They occur in ascending name order.

PROWLER

Synopsis:

prowler <XML file> <application> [<version>] [<name>=<value>]*

Description

Prowler scans the specified XML file, identifies the section of the configuration file matching the application, and if specified the version of configuration values required within this application. If not specified the default version as specified in the file, or the first if not specified is scanned. Each parameter and associated value is echoed on standard output, in a format suitable for inclusion within bash. When generating the XML file prowl escapes special HTML characters such as '&','<' and '>'. This escaping is reversed by prowler, and quotes within values themselves escaped appropriately.

Assignments specified on the prowler command line for named variables present in the XML configuration file override the XML assignments. To pass script arguments to prowler, either present them to prowler when and where appropriate or use the bash $@ macro to copy all of the arguments presented to the script to prowler.

To assign the correct configuration parameters and their values within an arbitrary bash script simply back quote the above invocation of prowler within bash and precede this expansion of the prowler output into the bash script through the interpretation of the back quotes, with the bash built in command eval, thus causing the variable assignments to be applied within the bash script.

#!/bin/bash
eval `prowler <XML file> <application> <version>`
env

This should retrieve the XML parameters and their values for the named version of the application within the XML file, and assign them within the bash script. The env command will then print them all out.

Because python is more strongly typed than bash, care is now taken to ensure that integer and float values are not quoted in the output from prowler. If all values returned by prowler should be quoted they can subsequently be cast to strings by using the str(...) python cast mechanism.

APPLICATIONS

The XML configuration files offer a number of useful additional capabilities. Since the variables are sorted one can easily:

• Compare distinct configuration files for similarities and differences
• Compare variable usage to documented usage
• Use knowledge about variable usage to improve scripts
• Extend the XML configuration files to allow C programs to also obtain parameterization through the same mechanism as scripts
• Document variables within the XML file in XML comments
• Log the values generated by prowler

Contact information