ESG publisher configuration

The ESG configuration file is an ASCII text file that contains fixed information for all ESG publication applications. The file is used to setup:

• relational database connections
• parameters for communication with node servers such as TDS and LAS
• project-specific information, such as names of models and experiments
  
  

Overriding configuration parameters

Configuration options are defined in a text file, the 'configuration file'. The first of these files found, in the listed order, is used:

1. Value of environment variable ESGINI, if specified.
   
2. $HOME/.esgcet/esg.ini
3. <PYTHON>/lib/python2.5/site-packages/esgcet/config/etc/esg.ini, where <PYTHON> is the Python installation directory. Note: If there is no directory <PYTHON>/lib/python2.5/site-packages/esgcet, then esgcet has been installed with a .egg file. To extract the initialization file from the egg use unzip:

   % unzip <PYTHON_LOCATION>/lib/python2.5/site-packages/esgcet-0.1dev_r7306-py2.5.egg esgcet/config/etc/esg.ini

4. esg.ini in the current working directory.
   

Most ESG publication tools have an option (-i) to specify the configuration file location.

The ESG publication source code contains a file template.ini, that can be used as a starting point for first-time creation of esg.ini.

Configuration file format

The configuration file has a sequence of sections, each with a [section] header and a set of name = value options. Leading and trailing whitespace is stripped from values. A value can contain format strings of the form %(option)s which are substitued with other options  in the same section or a special DEFAULT section. Lines beginning with '#' or ';' are comments.

Each section corresponds to a project or application:

[DEFAULT]
...
[project:ipcc4]
...
[project:cfmip]
...
[initialize]
...

Each section is a list of options of the form

option = value

where value may be:

• a string on the same line. Leading and trailing white space is stripped:
  

  thredds_authentication_realm = THREDDS Data Server

  Multivalued strings may be separated with commas:

  thredds_exclude_variables = a, a_bnds, b, b_bnds, bounds_lat, bounds_lon, height, lat_bnds, lev_bnds, lon_bnds, p0, time_bnds, lat, lon, time, lev, depth, depth_bnds, plev, geo_region

• multiple lines. Use a vertical bar (|) to separate fields on each line:
  

  categories =
    project | enum | true | true | 0
    experiment | enum | true | true | 1
    model | enum | true | true | 2

Format strings

Format strings have the form %(option)s, where option is defined elsewhere in the section. For example:

root_id = pcmdi
parent_id = %(root_id)s.ipcc4

results in parent_id=pcmdi.ipcc4.
Some format strings are predefined:

%(home)s : home directory
%(here)s : current directory
%(pythonbin)s : Python executables directory

Format strings may also be defined dynamically by the project handler - the software module that defines how project-specific metadata is discovered. For example, the IPCC4Handler is used for ipcc4 datasets, and defines the format strings %(project)s and %(model)s among others. The strings are interpreted when the dataset files are scanned. For example, if:

parent_id = %(root_id)s.%(project)s.%(model)s

then for datasets generated by model ukmo_hadcm3 the parent id will be

parent_id = pcmdi.ipcc4.ukmo_hadcm3

DEFAULT section

Options in the default section apply to all other sections. A default option may be overridden in another section. For example, to turn on debug logging during database initialization but only print warnings otherwise:

[DEFAULT]
log_level = WARNING
...
[initialize]
log_level = DEBUG
...

Project sections

Project-specific section headers have the form:

[project:project_name]

For example, the ipcc4 section header is:

[project:ipcc4]

Each project must also be listed in the default project_options:

project_options =
  ipcc4 | IPCC Fourth Assessment Report | 1
...

Each project has a corresponding handler, a Python class that defines the logic of dealing with files from the project.

Categories

Each dataset in a project has as a set of associated categories: string-valued attributes. (In the THREDDS context they are called properties). Some important points about categories:

• Categories are the potential fields used to search for datasets in the gateway.

• They are the fields displayed in the dataset window of esgpublish_gui. The type of widget associated with the category depends on the category_type:
• Enumerated categories are displayed with a pulldown menu. if the field name is xyz, there should also be an option xyz_options that defines the enumerated values as a comma separated list. For example, the 'publisher' options are defined in 'publisher_options'.
  
• 'string' categories are displayed in a single-line text box
• 'text' categories are displayed in a multi-line text box.
• 'fixed' categories are not modifiable.
  
• Categories may be mandatory or optional for a given dataset. Attempting to publish a dataset before all mandatory fields are set raises an exception. Typically the categories used to form the dataset_id should be defined as mandatory.
  
• A category may be a THREDDS property, in which case the data is stored as a <property> element in the THREDDS catalog.

When datasets are published the category values are filled in by the appropriate project handler in one of several ways:

• From command line arguments in the esgpublish script, or user-input values in the esgpublish_gui client.
• By matching directory names to a template. See directory_format.
• By reading metadata from the 'first file' discovered in the dataset.
• From default values, if the category value cannot be determined otherwise. See category_defaults.
  

Some categories are predefined:

• project: defined by project_options, mandatory
• experiment: defined by experiment_options, mandatory
• model: defined by initial_models_table in the [initialize] section, mandatory
• run_name: specific model run that generated the dataset
  
• title: optional string
• creator: optional multiline enumeration, defined by creator_options
• publisher: optional multiline enumeration, defined by publisher_options
• date: optional string publication date in the form yyyy-mm-dd hh:mi:ss
• format: fixed string, file format (e.g., netCDF, CF-1.0)
• source: optional text string
• description: optional text string
  

Maps

When a set of data files is scanned, the associated dataset ID is generated dynamically by matching the directory_format template. This greatly simplifies publication of large numbers of files, and helps ensure that datasets are identifed consistently when published. For example, if:

root_id = pcmdi
directory_format = /ipcc/%(experiment)s/%(submodel)s/%(time_frequency_short)s/%(variable)s/%(model)s/%(run_name)s
dataset_id = %(root_id)s.%(project)s.%(model)s.%(experiment)s.%(run_name)s.%(time_frequency)s

and we are scanning directory /ipcc/20c3m/atm/mo/hfls/cnrm_cm3/run1

then the directory name and directory_format are matched, resulting in:

experiment=20c3m

submodel=atm

time_frequency_short=mo

variable=hfls

model=cnrm_cm3

run_name=run1

Note the dataset_id has the format string %(time_frequency)s which is not part of the directory_format template. What determines the time_frequency? The answer is a map. a table associating a set of string-valued independent variables with a dependent variable:

time_frequency_map = map(time_frequency_short : time_frequency)

  3h    | 3hourly

  da    | daily

  fixed | monthly

  mo    | monthly

  yr    | yearly

Given this mapping the generated dataset_id for this directory will be:

pcmdi.ipcc4.cnrm_cm3.20c3m.run1.monthly

since  time_frequency_short=mo maps to time_frequency=monthly.

To define a map:

1. Add the map name to the map option in the project section.
2. Add an option of the form:
   
   

   map_name = map(independent_variable_1[, independent_variable_2[, ...]] : dependent_variable)

     value_1 [ | value_2 [...]] | dependent_value

     value_1 [ | value_2 [...]] | dependent_value

   Each line has the same number of values as the variables listed in the map() definition.
   
   

Maps are a convenient way to:

• add flexibility to the generation of option values
• make handler-specific associations explicit and configurable.
  

The project options are:

application sections
Some sections apply to specific applications:

• initialize: esginitialize (create and initialize the node database)
  
  

  Option	Description	Valid values	Example
  initial_models_table	Path of the model definition table. This table defines model-specific  metadata for all projects. Each line of the file has the form: project | model_id | URL | description The default table is named esgcet_models_table.txt, and is located in the same directory as the configuration file.		initial_models_table = %(here)s/data/esgcet_models_table.txt
  initial_standard_name_table	Path of the standard name table. This table defines CF standard name metadata, used for optional name validation at scan time. The file is in XML format as defined by the CF standard name table. The default table is named cf-standard-name-table.xml, and is located in the same directory as the configuration file.		initial_standard_name_table = %(here)s/data/cf-standard-name-table.xml

  
  
• hsils: HSI lister for listing HPSS-resident files. Options in this section are enabled if offline_lister=hsi in the [DEFAULT] section.
  
  

  Option	Description	Valid values	Example
  hsi	Absolute path of the hsi executable		hsi = /usr/local/bin/hsi
  offline_lister_executable	Absolute path of the hsils.py script.		offline_lister_executable = %(home)s/work/Esgcet/esgcet/scripts/hsils.py

  
  
• srmls: SRM lister for listing tertiary storage files. Options in this section are enabled if offline_lister=srmls in the [DEFAULT] section.
  
  

  Option	Description	Valid values	Example
  offline_lister_executable	Absolute path of the srmls.py script.		offline_lister_executable = %(home)s/work/Esgcet/esgcet/scripts/srmls.py
  srm_archive	SRM archive		srm_archive = /garchive.nersc.gov
  srm_server	SRM server URL		srm_server = srm://somehost.llnl.gov:6288/srm/v2/server
  srmls	Absolute path of srm-ls		srmls = /usr/local/esg/bin/srm-ls