> 
>
| SUPPORTEDCODES TUTORIAL |   |
Supported Codes and Data Formats
SimulationTools has built-in support for reading the output data from many simulation codes and data formats. In particular, it has support for reading data produced by components of the Einstein Toolkit and also for reading data in the Numerical Relativity Data Format.
It is also designed in a modular way, so that support for new codes can be easily added. Typically, SimulationTools provides a low-level interface for reading the output data from a code, handling file-name and parsing issues, as well as a high-level interface in the domain-language of the user. Where several codes generate equivalent data (for example PunctureTracker and MinTracker), an abstraction has often been built on top (Trackers) so that codes can be written which work with simulations utilising any of the underlying codes without having to support each explicitly.
Since ASCII and HDF5 output from grid functions is also supported, any code which deals with grid function data can also have this data read into SimulationTools.
This tutorial lists the currently supported formats and advises on which simulation parameters should be set to gain the maximum support in SimulationTools.
Structure of a simulation
When SimulationTools searches for data from a particular simulation, it will look for the simulation directory under the directories listed in $SimulationPath. The most basic definition of a “simulation” understood by SimulationTools is therefore just a single directory containing all files pertaining to that simulation.
However, SimulationTools goes beyond this simple concept by adding support for transparently merging chunks from multiple possible restarts, determining filenames based on the type of data requested, etc.
SimFactory
Simulations run with the the Einstein toolkit can make use of SimFactory to manage the running of jobs on clusters, job chaining, etc. SimulationTools understands the directory structure produced by simulations run with SimFactory and can read data from them while transparently joining together data from multiple restarts. It can also read simulation properties such as the number of cores used for a simulation from SimFactory output.
EinsteinToolkit/Cactus simulation data
SimulationTools has support for reading data produced by a large number of codes included in the Einstein Toolkit. Each of these codes has parameters which control the output format. While it is designed to be as flexible as possible, SimulationTools users will benefit from setting certain parameters to values recommended in this section. In addition to the parameters given here, the parameter file for the included example binary black hole simulation includes many more convenient parameter values and can be used as a starting point for other simulations.
Cactus parameters
SimulationTools has support for reading and parsing Cactus parameter files. Parameter values can be read using the parameter name. In order for this parsing to be possible, the parameter file must be contained inside the simulation directory and must have the ‘.par’ file extension.
Additionally, SimulationTools provides a functional interface to some parameters. For example e.g. SimulationFinalTime[“mysim”] reads the value of the Cactus::cctk_final_time parameter.
Grid functions (CarpetIOHDF5, CarpetIOASCII)
SimulationTools can read grid function data output by the Cactus IO mechanism. The optimal format to use is that produced by the CarpetIOHDF5 thorn. 1D, 2D and 3D data can be read, with automatic merging of output from different processes, including support for refinement levels and multiblock data.
For best results, it is recommended that the following parameter values are set:
IO::out_dir = $parfile
IOScalar::one_file_per_group = yes
IOASCII::one_file_per_group = yes
It also currently supports mesh-refined 1D CarpetIOASCII output files, automatically combining data from different processes. However, it should be noted that HDF5 is highly recommended over ASCII for performance and storage-size reasons.
For grid function data to be available, grid function output should be enabled using the IOHDF5::out1D_vars, IOHDF5::out2D_vars and IOHDF5::out_vars parameters.
Waveform extraction (Multipole)
The Multipole thorn produces output for the spherical harmonic modes of arbitrary Cactus grid functions at specified radii.
SimulationTools enables access to this data through its “Waveforms” interface. While SimulationTools supports both HDF5 and ASCII output formats from the Multipole thorn, it is highly recommended to use HDF5 for performance and disk space reasons. This may be controlled by setting the following parameter values:
Multipole::output_hdf5 = yes
Multipole::output_ascii = no
Another option provided by the Multipole thorn is to specify the name to use for the output variable. By default, SimulationTools assumes this variable is named ‘psi4’. To ensure this is the case, it is recommended to set the ‘name’ part of the variables parameter to ‘psi4’. For example, when extracting waveforms using the WeylScal4 thorn one would use
Multipole::variables=”
WeylScal4::Psi4r{sw=-2 cmplx=’WeylScal4::Psi4i’ name=’psi4’}
“
SimulationTools can be made to use another variable name by setting a different value for $MultipolePsi4Variable, e.g.
$MultipolePsi4Variable=”psi0”;
Run information (SystemStatistics, Cactus, Carpet, TimerReport)
SystemStatistics
The SystemStatistics thorn produces output files for simulation statistics such as memory and swap usage, among others. In order for this information to be available, the following parameter values should be set, where <out_every> reflects the frequency in iterations at which output will be available:
IOScalar::outScalar_every = <out_every>
IOScalar::outScalar_vars = “SystemStatistics::process_memory_mb”
TimerReport
Data on Cactus timing statistics can be read from the output of the TimerReport thorn. For best results, it is recommended that the following parameter values be set:
Cactus::cctk_timer_output = full
TimerReport::output_all_timers_together = yes
TimerReport::output_schedule_timers = no
Carpet timers
Timing statistics from the timers included with Carpet can also be read. For best results, it is recommended that the following parameter value is set:
IOASCII::out0D_vars = “Carpet::timing“
Grid structure (Carpet, CarpetRegrid2, CoordBase, Coordinates, Time)
The grid structure of a simulation is obtained by reading the parameters for the Carpet, CarpetRegrid2, CoordBase, Coordinates and Time thorns. For best results, it is recommended to set the following parameter value:
Carpet::grid_coordinates_filename = “carpet-grid.asc”
Initial data (TwoPunctures)
For binary black hole systems, SimulationTools obtains information about initial data (ADM mass, separation, etc.) from the TwoPunctures metadata file, from the TwoPunctures parameters set in the Cactus parameter file, and from the simulation’s standard output. The most convenient of these is the metadata file; to ensure this is created by the simulation, a version of the TwoPunctures thorn more recent than September 25, 2011 should be used.
Trajectories (PunctureTracker)
SimulationTools has support for PunctureTracker output files and makes the data available through the “Tracker” and “Binary” interfaces. In order for this data to be available, the following parameter should be set:
IOASCII::out0D_vars = “PunctureTracker::pt_loc“
Black hole horizons, spins and masses (QuasilocalMeasures,AHFinderDirect)
The black hole horizon data computed by the AHFinderDirect thorn is used throughout SimulationTools where it is required (exposed, for example, through the “Black Hole” interface). In order for this data to be available, AHFinderDirect must be enabled in the simulation.
The black hole spin data computed by the QuasiLocalMeasures thorn is used throughout SimulationTools where it is required (exposed, for example, through the “Black Hole” interface). In order for this data to be available, the following parameter should be set:
IOASCII::out0D_vars = “QuasiLocalMeasures::qlm_scalars“
In addition, for binary black hole simulations, SimulationTools makes assumptions about the indexing of the quasilocal surfaces. For example, it assumes that suface n corresponds to the apparent horizon n+1. For more details and example parameter settings, the the parameter file in the included bbh simulation.
Numerical relativity data format (NRDF)
SimulationTools has support for reading data from simulations in the Numerical Relativity Data Format, a standard for interchanging data between numerical relativity codes, using the same functional interface as data from other simulation formats.