Sys Admin Pocket Survival Guide - SCI
Home	Sci-File	Sci-App	mpi	lsf

HDF5

.hdf5

Hierarchical Data Format
API/Library to pack large dataset/array into single file (for transfer over network?)
There are serial vs parallel versions.


h5import  	# create hdf5 archive?
h5repack    	# extract single dataset/group from an hdf5 archive ?
h5ls   		# 
h5dump  
h5diff  	ph5diff
h5debug  
h5stat        
h5copy   

h5cc   	h5pcc
h5c++  
h5fc    h5pfc

h5jam  
h5unjam 

h5mkgrp        
h5redeploy  
h5repart  
h52gif  

gif2h5  

h5perf_serial

NetCDF

.nc

NetCDF - Network Common Data Form.
Often used for geospatial data storing array-oriented scientific data (eg input for CMAQ modeling, oceanography, ArcGIS).
Older CDF format were flat, whereas HDF was Hierarchical. Thus HDF originally provided more grouping functionality to organize data, good for long term maintenance of the data "archive".
NetCDF v4 is the HDF5 format, with some restrictions. (typically considered a simpler API with essentially the same functionality)
CDF5 format is from parallel-netcdf project
There are serial vs parallel versions (even for NetCDF v4?)

One can learn to use nc commands without invoking hdf5 commands directly. And if just using it, the choice "between" .hdf5 vs .nc is form the application on hand, unless one is debating writting a new app.

NetCDF file are self describing, having all info in its header. Array Strucutured in header also provide random seek to desired record.


module load  gccc/6.3.0 hdf5/1.8.20-gcc-p netcdf/4.6.1-gcc-p 

ncdump 			# show structure of file in ascii (from NCAR)

ocprint 
nccopy  
ncgen  
ncgen3  
nc-config

Related tools

ncBrowse - view java graphics, 3D viz
ncview - view multi dimention data with changing color map, etc. X-based?
NCL - NCAR Command Language - viz netCDF files (and other formats)

IOAPI

IOAPI is library used by CMAQ to read/write .nc files. (Fortran files are machine specific)
Much of the data in CMAQ are time-series. ie, for the variables/arrays defined, IOAPI automatically apply time stamp or stepping info for them.


JDATE: YYYYDay eg 201935 is Feb 4 of 2019.
JTIME: HHMMSS  (10000 * Hour) + (100 * Minute) + Seconds

Logical filename.  Max 16 chars.  hide NetCDF internal file structure from user/programmer.

IOAPI cmd

From CMAQ manual (v4.6) p50 of pdf

M3XTRACT
extract a subset of variables from a file for a specified time
interval

M3DIFF 
compute statistics for pairs of variables

M3STAT 
compute statistics for variables in a file
build a boundary-condition file for a sub-grid window of
a gridded file

BCWNDW
build a boundary-condition file for a sub-grid window of
a gridded file

M3EDHDR 
edit header attributes/file descriptive parameters

M3TPROC
compute time period aggregates and write them to an output
file

M3TSHIFT 
copy/time shift data from a file

M3WNDW 
window data from a gridded file to a sub-grid

M3FAKE
build a file according to user specifications, filled with
dummy data

VERTOT 
compute vertical-column totals of variables in a file

UTMTOOL 
coordinate conversions and grid-related computations for
Lat/Lon, Lambert, and UTM

CMAQ

Community Modeling ... By EPA.
Ref: CMAQ manual (v4.6)
Modeling components and workflow overview - (see p56 of pdf).

CCTM - CMAQ chemistry-trasnport model - main program of CMAQ modeler. Most of the other programs are pre-processor to prepare data for CCTM use.

Pre-processors

MM5 or WRF - Meteorology model (input?)
SMOKE - Emission model (not part of CMAQ)
MCIP. Meteorological models, produce GRIDDESC (one sub-type of IOAPI format) suitable for CCTM.
ICON - prepare initial condition (eg from ascii), produce netcdf file as input to CCTM. The input are specific to the modeling grid and chem parameterization.
Data source could be ascii or previous CCTM output.
BCON - Boundary Condition. produce .nc file as input for CCTM.
Data source could be ascii or previous CCTM output.
JRPOC - output is a lookup table of phtolysis rate (in clear sky condition), which is needed by CCTM to do its modeling.

Support tools

PARIO - Govern communication of Parrallel run of CCTM.
STENEX - diagnostic tool for CMAQ

BioInformatics File Format

sam

sam/bam/cram :
BCFtools: BCF2/VCF/gVCF
VCF: Variant call format. Mutation centric view

ChemInformatics File Format

PDB

Protein databank: http://pdb.org or or http://rcsb.org/pdb/

Sample protein:
1tii		A 900+ residues protein, not huge, but a sizable molecule for testing 3D rendering (especially in space filling model)

11AS, 117E	Some pdb file around size of Haemoglobin

1A00            Hemoglobin
1z1g            Topoisomerases, a large protein, "Molecule of the Month" for Dec 2006.

.smi

Smiles - http://www.daylight.com/dayhtml/doc/theory/theory.smiles.html

SMILES (Simplified Molecular Input Line Entry System) is a line notation (a typographical method using printable characters) for entering and representing molecules and reactions. Some examples are:o
CC  			ethane  	
[OH3+]  		hydronium ion
c1ccccc1  		benzene  	
N[C@H](C)C(=O)O  	D-alanine
COc1cc(cc(OC)c1OC)C(=O)N\N=C(/C)\c2ccc(cc2)S(=O)(=O)N[C@@H](C)C(=O)O	some carboxylic acid
                  

OpenEye Omega take .smi input smiles and produces .oeb.gz files.

.oeb

.oeb receptor file containing active site for docking.
Probably OpenEye propietary.  Fred can open these files (and maybe ROCS?)


Engineering is the art of making compromises.
Science is the reverse engineering of the compromises made by nature.
Medicine is the hacking of the scientific knowledge base.   - A comp sci student :-)

[Doc URL: http://tin6150.github.io/psg/sci-file.html]
(cc) Tin Ho. See main page for copyright info.

hoti1
bofh1