HDF5

.hdf5
Hierarchical Data Format
API/Library to pack large dataset/array into single file (for transfer over network?)
There are serial vs parallel versions.

h5import  	# create hdf5 archive?
h5repack    	# extract single dataset/group from an hdf5 archive ?
h5ls   		# 
h5dump  
h5diff  	ph5diff
h5debug  
h5stat        
h5copy   

h5cc   	h5pcc
h5c++  
h5fc    h5pfc

h5jam  
h5unjam 

h5mkgrp        
h5redeploy  
h5repart  
h52gif  

gif2h5  

h5perf_serial  


NetCDF

.nc
NetCDF - Network Common Data Form.
Often used for geospatial data storing array-oriented scientific data (eg input for CMAQ modeling, oceanography, ArcGIS).
Older CDF format were flat, whereas HDF was Hierarchical. Thus HDF originally provided more grouping functionality to organize data, good for long term maintenance of the data "archive".
NetCDF v4 is the HDF5 format, with some restrictions. (typically considered a simpler API with essentially the same functionality)
CDF5 format is from parallel-netcdf project
There are serial vs parallel versions (even for NetCDF v4?)

One can learn to use nc commands without invoking hdf5 commands directly. And if just using it, the choice "between" .hdf5 vs .nc is form the application on hand, unless one is debating writting a new app.

NetCDF file are self describing, having all info in its header. Array Strucutured in header also provide random seek to desired record.

module load  gccc/6.3.0 hdf5/1.8.20-gcc-p netcdf/4.6.1-gcc-p 

ncdump 			# show structure of file in ascii (from NCAR)

ocprint 
nccopy  
ncgen  
ncgen3  
nc-config  

Related tools

IOAPI

IOAPI is library used by CMAQ to read/write .nc files. (Fortran files are machine specific)
Much of the data in CMAQ are time-series. ie, for the variables/arrays defined, IOAPI automatically apply time stamp or stepping info for them.

JDATE: YYYYDay eg 201935 is Feb 4 of 2019.
JTIME: HHMMSS  (10000 * Hour) + (100 * Minute) + Seconds

Logical filename.  Max 16 chars.  hide NetCDF internal file structure from user/programmer.

IOAPI cmd
From CMAQ manual (v4.6) p50 of pdf
M3XTRACT
extract a subset of variables from a file for a specified time
interval

M3DIFF 
compute statistics for pairs of variables

M3STAT 
compute statistics for variables in a file
build a boundary-condition file for a sub-grid window of
a gridded file

BCWNDW
build a boundary-condition file for a sub-grid window of
a gridded file

M3EDHDR 
edit header attributes/file descriptive parameters

M3TPROC
compute time period aggregates and write them to an output
file

M3TSHIFT 
copy/time shift data from a file

M3WNDW 
window data from a gridded file to a sub-grid

M3FAKE
build a file according to user specifications, filled with
dummy data

VERTOT 
compute vertical-column totals of variables in a file

UTMTOOL 
coordinate conversions and grid-related computations for
Lat/Lon, Lambert, and UTM

CMAQ

Community Modeling ... By EPA.
Ref: CMAQ manual (v4.6)
Modeling components and workflow overview - (see p56 of pdf).
  • CCTM - CMAQ chemistry-trasnport model - main program of CMAQ modeler. Most of the other programs are pre-processor to prepare data for CCTM use.
  • Pre-processors
    Support tools

    BioInformatics File Format

    sam

    
    
    

    ChemInformatics File Format

    PDB

    Protein databank: http://pdb.org or or http://rcsb.org/pdb/

    Sample protein:
    1tii		A 900+ residues protein, not huge, but a sizable molecule for testing 3D rendering (especially in space filling model)
    
    11AS, 117E	Some pdb file around size of Haemoglobin
    
    1A00            Hemoglobin
    1z1g            Topoisomerases, a large protein, "Molecule of the Month" for Dec 2006.
    
    

    .smi

    Smiles - http://www.daylight.com/dayhtml/doc/theory/theory.smiles.html
    SMILES (Simplified Molecular Input Line Entry System) is a line notation (a typographical method using printable characters) for entering and representing molecules and reactions. Some examples are:o
    CC  			ethane  	
    [OH3+]  		hydronium ion
    c1ccccc1  		benzene  	
    N[C@H](C)C(=O)O  	D-alanine
    COc1cc(cc(OC)c1OC)C(=O)N\N=C(/C)\c2ccc(cc2)S(=O)(=O)N[C@@H](C)C(=O)O	some carboxylic acid
                      
    
    OpenEye Omega take .smi input smiles and produces .oeb.gz files.
    
    

    .oeb

    .oeb receptor file containing active site for docking.
    Probably OpenEye propietary.  Fred can open these files (and maybe ROCS?)
    



    
    Engineering is the art of making compromises.
    Science is the reverse engineering of the compromises made by nature.
    Medicine is the hacking of the scientific knowledge base.   - A comp sci student :-)
    
    






    [Doc URL: http://tin6150.github.io/psg/sci-file.html]
    (cc) Tin Ho. See main page for copyright info.


    hoti1
    bofh1