Command Line Interface tools

This sections describes a series of Command Line Interface (CLI) tools currently available in AQUA. It includes software with a variety of goals, which are mostly made for advanced usage.

Benchmarker

A tool to benchmark the performance of the AQUA analysis tools. The tool is available in the cli/benchmarker folder. It runs a few selected methods for multiple times and report the durations of multiple execution: it has to be run in batch mode with the associated jobscript in order to guarantee robust results. It will be replaced in future by more robust performance machinery.

Grids management

This section describes the tools available to manage the grids used in AQUA, from the download and validation to the synchronization between different HPC platforms.

Grids downloader

The grids used in AQUA are available for download. A script in the cli/grids-downloader/ folder is available

Basic usage:

bash grids-downloader.sh -o <outputdir> all | model

This will download the grids to the <outputdir> used in AQUA. The output directory will be created if it does not exist. For the ClimateDT machines, the target folders are commented in the code for documentation. It is possible to specify if to download all the available grids or to download only a subset of the grids, by specifying the group of grids to download (usually one per model).

Warning

From September 2025, the grids are stored and versioned using DVC (Data Version Control). Grids downloader script still works but it will be deprecated in the near future. Please refer to Access to aqua-dvc data for developers for more information on how to access the data.

Checksum verifications for obs and grids

AQUA built on many grids files to speed up operations for interpolation and area evaluation, especially within the ClimateDT workflow. To provide valuable diagnostics, it also builds on a catalog for observations. These are available on multiple ClimateDT HPC but sometimes the synchronization might not be complete following an update. In order to verify that all the grids files are ported on the used machine the cli/checksum-checker/grids-checker.py and cli/checksum-checker/obs-checker.py scripts are available to verify the checksum of the grid and observation files is the same as it is expected.

To verify that everything is at it should be please run:

./grids-checker.py verify

To generate a new checksum should be please run:

./grids-checker.py generate -o checksum_file.md5

Please notice that not all the grid/obs folders will be checked, but only those defined in the file with GRIDS_FOLDERS and OBS_FOLDERS variables. Option -s can be used as well to scan a single grid/obs folder (e.g. HealPix, or ERA5)

Generation of grid from data

A tool to create CDO-compliant grid files (which are fundamental for proper regridding) specifically for oceanic model in order to ensure the right treatment of masks. Two scripts in the the cli/grid-fromd-data folder are available.

Both hpx-from-source.py and multiIO-from-source.py works starting from specific sources, saving them to disk and processing the final results with CDO to ensure the creation of CDO-compliant grid files that can be later used for areas and remapping computation.

A YAML configuration file must be specified.

Basic usage:

./hpx-from-source.py -c config-hpx-nemo.yaml -l INFO

Grids synchronization

Since the upload of the grids to the SWIFT platform used to store the grids is available only from Levante, a simple script to synchronize the grids from Levante to LUMI and viceversa is available in the cli/grids-downloader/ folder. You will need to be logged to the destination platform to run the script and to have passwordless ssh access to the source platform.

Basic usage:

bash grids-sync.sh [levante_to_lumi | lumi_to_levante]

This will synchronize the grids from Levante to LUMI or viceversa.

Warning

If more grids are added to the Levante platform, the SWIFT database should be updated. Please contact the AQUA team to upload new relevant grids to the SWIFT platform.

Grids uploader

A script to upload the grids to the SWIFT platform is available in the cli/grids-downloader/ folder. You will need to be on levante and to have the access to the SWIFT platform to run the script. With the automatic setup updated folders will be uploaded in the same location on the SWIFT platform and no updates of the links in the grids-downloader.sh script will be needed.

Basic usage:

bash grids-uploader.sh [all | modelname]

Note

The script will check that a valid SWIFT token is available before starting the upload. If the token is not available, the script will ask the user to login to the SWIFT platform to obtain a new token.

ORCA grid generator

A tool to generate ORCA grid files (with bounds) from the mesh_mask.nc. A script in the cli/orca-grids folder is available.

Basic usage:

./orca_bounds_new.py mesh_mask.nc orcefile.nc

HPC container utilities

Includes the script for the usage of the container on LUMI and Levante HPC: please refer to Container

LUMI conda installation

Includes the script for the installation of conda environment on LUMI: please refer to HPC Installation

Weights generator

A tool to compute via script or batch job the generation of interpolation weights which are too heavy to be prepared from notebook or login node. It can be configured to run on all the catalog enties so that it can be used to update existing weights if necessary, or to compute all the weights on a new machine. A script in the cli/generate_weights folder is available.

Basic usage:

./generate_weights.py -c weights_config.yaml

Orography generator

A tool to generate orography files from a source that can be accessed via AQUA. It is located in the cli/orography_from_data folder and it contains all the configurations to generate orography files inside the script file itself.

It has been used to produce the orography files for the Tropical Cyclone diagnostic.

Basic usage:

python orography_generator.py