By default, the logfile contains one line with a string corresponding to the `fileheadout' given in methodlist (see below), and one line with a string that is meant to identify the coordinate names, the conserved variables (wnames) and other entries, and then follows a sequence of lines containing numbers: i.e. a single line per requested output time, containing the integer timestep counter it, the time t, the time step to be used in the next time advance dt, the domain averaged value of each conserved variable (nw real numbers, which allows to check perfect conservation across the grid tree when the boundary conditions imply it), the percentage of the domain covered by each allowed grid level (mxnest real numbers between 0.0 and 1.0, with 1.0 indicating 100% coverage: when all mxnest numbers are summed, we get 1.0), the number of grids per allowed grid level (hence, mxnest integers).
For example, the log file could look like this:
it t dt d s1 s2 s3 tau b1 b2 b3 Ds dtr1 lfac xi c1 c2 c3 n1 n2 n3| Xload Xmemory 'Cell_Updates /second/core' 'Active_Blocks/Core' 'Wct Per Code Time [s]' 'TimeToFinish [hrs]' ... 1604490 4.9999E+03 2.9734E-03 3.8971E-07 1.4678E-06 1.2121E-08 1.5896E-06 1.2063E-08 -7.7303E-16 1.4822E-11 6.5156E-11 6.1005E-06 3.8954E-07 1.1271E+00 4.0311E-07 3.5156E-02 1.4844E-01 8.1641E-01 9 152 3344 | 1.05E+00 1.05E+00 4.23E+04 1.99E+01 4.05E+01 1.09E-03 ...
You notice that after the |, the code outputs additional performance metrics: starting with Xload and Xmemory which denote the load- and memory-imbalance. Monitoring this is useful when the grid-activation feature of BHAC is used. 'Cell_Updates /second/core' shows the code performance where we count each stage in the timestepper as cell update. 'Active_Blocks/Core' allows to put this performance in relation to the work for each core. 'Wct Per Code Time [s]' measures the seconds needed to complete one code unit time and finally 'TimeToFinish [hrs]' is the projected time to complete the run to a time of tmax. The latter can be quite inaccurate especially if AMR is involved.
The first part of each *.dat file stores all the conservative variables w for all grids which happen to be present at that time in the tree hierarchy. The Morton-ordered space filling curve through the grid hierarchy, together with the fixed block size, allows us to perform fully parallel I/O read and write operations. Each processor writes all grids stored in its local memory in the order defined by the Morton number of the grid. All processors write simultaneously to a single file when typeparIO=1. Each grid block writes the nw variables out over the full mesh (without the ghost cell layers).
This part is then followed by all extra info needed to reconstruct/restart the simulation, which is written at the end *.dat file by the master CPU. The extra info consists of
(1) the boolean representation of the grid structure; (2) info on grid block size per dimension, actually the mesh size nx^D (3) equation-specific variable values (i.e. the eqpar array) (4) number of active tree leafs nleafs, (5) maximal refinement level present levmax, (6) dimensionality NDIM, (7) number of vector components NDIR, (8) number of variables nw, (9) number of equation-specific variables neqpar+nspecialpar, (10) integer time counter it, (11) global time t.
(1) hdf5_ini: if true (default) you will restart your simulations from a *.hdf5 file. In some cases (mostly for data conversion) it might be useful to restart from an old *.dat snapshot and then dump *.hdf5. In this case set hdf5_ini=.false.
(2) write_xdmf: if true (default) for each *.hdf5 file also a *.xdmf is created, which is necesarry for data visualisation with Visit or Paraview. This happens non-parallel and might take a considerable fraction of the IO-time. If not required (e.g. you only want to checkpoint), just set write_xdmf=.false..
(3) save_GZ: with the HDF5-IO ghostzones are always saved. However, you might not want to see them when visualizing your data. In this case set save_GZ=.false. (default). Note that this will not change the *.hdf5 file, but the *.xdmf file.
(4) saveprim: this switch is deprecated and ignored by the HDF5-IO. Data will always be saved as primitives.
(5) fastIO: if true, you will see a considerable speed-up in output, but also double your memory footprint (data will be aggregated into buffers before dumping). If memory is no problem for your simulation, it is highly recommended to use this switch. If set to false (default) each process dumps its data block by block (but still parallel for all the blocks). In case of simulations, where not all processes control the exact same number of blocks this might lead to some independent data-dumps and thus increase IO-time.
(6) convert_type: In addition to the other convert types you can now also set convert_type='hdf5' in order to convert old *.dat data. This mostly makes sense in combination with hdf5_ini=.false..
(7) autoconvert: this switch allows to additionally output visualization data, e.g. *.vtu files.
(8) typeparIO: HDF5 will always be parallel, so the typeparIO is simply ignored (obviously it's not parallel, if you run only one process, but this is recognized automatically. So no need for typeparIO).
(9) xdmf_type: If 'Node' (default) the data will be displayed smooth, but shifted by half a grid-cell. Setting xdmf_type='Cell' the data will be displayed cell-centered on the correct grid.
Once you have a running version of HDF5 you can proceed to install BHAC as usual. Make sure to set #define HDF5 in the file definitions.h and use an adequate arquitecture. Some arquitectures are provided in the folder arch with the ending *_hdf5.defs. If you create your own arch-file it is highly recommended to set the HDF5 compilation scripts for the compiler flags. They are called h5pfc or h5pfc.openmpi (depending on your HDF5 installation) and are created with the installation of the HDF5 libraries (or downloaded with the binaries). If compilation was successfull, you should try to run the tests/rmhd/shocktube_hdf5/ example. If you get errors, then probably your HDF5 library was not installed properly or you are using the wrong HDF5-compilation script. In this case refer to the support of the HDF5 Group or to weih@th.physik.uni-frankfurt.de.
Note: If you use the non-parallel version of HDF5 (as provided on some clusters) you will use h5fc rather than h5pfc as compiler script. This, however, does not compile with MPI. As BHAC necessarily needs MPI, you can not use non-parallel HDF5 in this way. Changing the corresponding environment variables, so that the script uses an MPI-compiler will not work either, because these scripts were not compiled with MPI to begin with. If you absolutely want to use non-parallel HDF5 with BHAC (if at all possible?), you will have to change the makefiles and circumvent the compiler script h5fc. Hints on how to do this can be found on the HDF5 website.
Note: For visualisation with Visit you might encounter some problems when using AMR. This usually happens when you play with the time-slider and the number of blocks changes from one timeframe to the next. In this case just click on reopen and/or draw. This is a problem specific to the Visit-reader. Something similar might happen with Paraview.