This Linux guide is my own notes on Linux, for stuff that I did not like to documents that I have found on the internet. It is RedHat centric.
CentOS is supposed to be a "rebadged/un-commercialized" version of RHEL, keeping binary-level compatibilities at major release level, thus the discussion should apply equally to CentOS and Scientifi Linux.
Ubuntu is conquering the world now, so there are some Debian things here and there. What is Linux anyway? :)



RHEL vs CentOS vs Scientific Linux

Linux Admin 101

Linux init mode are not "cumulative". So, if default init is 5, script in run level 3 would not be run. If adding a Start script, may want to add them in both in rc3.d and rc5.d, unless it is really only wished to be available at one of the run level and not the other. Kill script should be placed in rc6.d, which shutdown/reboot scan.

RHEL3 - binary compatible with FC2 (but FC2 is EOL)
RHEL4 - binary compatible with FC3 (and maybe FC4)

Single User Mode RHEL 6

To boot into single user mode for maintenance,
enter "linux single" or "linux emergency" at the LILO boot: prompt.

If using GRUB, then, at the menu for choosing kernel, do:
 hit 'e'
 then arrow down to the kernel line
 hit 'e' again
 add 'single' to the end of the line
 hit enter
 hit 'b' to boot
 lather, rinse, repeat

Single User Mode RHEL 7
linux16 /vmlinux...  ro               
linux16 /vmlinux...  rw init=/sysboot/bin/bash 

linux /vmlinux...    

remove rhgb quiet
rhgh is redhat splash
quiet remove most verbose msg

GRUB_CMDLINE_Linux=... (remove rhgb quiet)

Single User Mode RHEL 7 with pxe boot with chrooted env

ln -sf /lib/systemd/system/ /etc/systemd/system/

Old days Core dump

Linux kernel panic does not core dump to a file.
it will print output of the kernel core dump result to the stdout device:
a vga screen or a serial console.  
To setup serial console, do:
console=ttyS0,115200 console=tty0
but this would mean console boot/shutdown/panic message will no longer
be send to the VGA, but to serial port.  VGA may get a spawned
login prompt after OS is up.


yum install kexec-tools
chkconfig kdump on 
/etc/kdump.conf ::
core_collector makedumpfile -c --message-level 1 -d 31
# -d specify the dump filter (stuff to exclude)
# 1 	Zero pages
# 2 	Cache pages
# 4 	Cache private
# 8 	User pages
# 16 	Free pages 
# -c to enable compression
sshkey /etc/acs/kdump/id_rsa-sa_kdump
path /data/kdump/
blacklist pvscsi
blacklist vmmemctl
/boot/grub/grub.conf : add crashkernel clause
	        kernel /boot/vmlinuz-2.6.18-371.11.1.el5 ro root=LABEL=/ crashkernel=768M@32M

test.  this WILL CRASH the machine:
echo 1 > /proc/sys/kernel/sysrq
echo c > /proc/sysrq-trigger
To look at vmcore file:
yum install crash
debuginfo-install kernel	# to install necessary "decoder" to analyze the crash file

crash  /var/crash/.../vmcore   /usr/lib/debug/lib/modules/.../vmlinux    # need matching kernel file
  bt       	# for backtrace
  ps		# display processes
  files		# lsof
somewhat helpful shell function to see if there are recent core dumps
lsdump () {
    for entry in $(ls /kdump|grep ^[1-9]); do
        ip=$(echo $entry|cut -d- -f1)
        hostname=$(dig +short -x $ip)
        date=$(echo $entry|cut -d- -f2-)
        echo "$date    $hostname ($ip)"
    done | sort

console in serial port

--kargs="nopti console=tty0 console=ttyS1,115200n8"

Linux "deviation" from Solaris

Some of the most notable things that Linux does rather differently than Solaris or traditional Unix.

RHEL/CentOS/Scientific Linux vs Ubuntu

Ubuntu is Debian-based. Some of the more often run-into diff for sys admin:

RHEL Ubuntu
nic /etc/sysconfig/network-scripts/ifcfg-NIC /etc/network/interfaces
Firewall systemctl enable/start/status iptables
systemctl enable/start/status firewalld
ufw enable/start/status

Most of this doc is CentOS centric.

64-bit Platforms

RHEL 3.0 and 4.0 maintains simultaneous relese/update levels for 64-bit platforms as their popular 32-bit x86 OS. Just need to get the "special" distro ISO to install on the 64-bit hardware and it will be good to go. Not all the AS/ES/WS flavor maybe available to all platform.
RHEL 4.0   	

rhel-ia64-as-4		Red Hat Enterprise Linux AS (v. 4 for 64-bit Intel Itanium)
rhel-x86_64-as-4	Red Hat Enterprise Linux AS (v. 4 for 64-bit AMD64/Intel EM64T)
rhel-ppc-as-4		Red Hat Enterprise Linux AS (v. 4 for 64-bit IBM POWER)
rhel-s390-as-4		Red Hat Enterprise Linux AS (v. 4 for 31-bit IBM S/390)
rhel-s390x-as-4		Red Hat Enterprise Linux AS (v. 4 for 64-bit IBM zSeries)
rhel-i386-as-4		Red Hat Enterprise Linux AS (v. 4 for 32-bit x86)   [typical intel pentium 4/80x86 release]
rhel-i386-ws-4		Red Hat Enterprise Linux WS (v. 4 for 32-bit x86)

RHEL 3.0 	

rhel-ia64-as-3		Red Hat Enterprise Linux AS (v. 3 for Itanium)
rhel-x86_64-as-3 	Red Hat Enterprise Linux AS (v. 3 for AMD64/Intel EM64T)   
rhel-ppc-as-3		Red Hat Enterprise Linux AS (v. 3 for iSeries and pSeries)
Commands for 64-bit info:
ld -V		# shows supported emulation for a given machine	
		# eg: elf_x86_64 elf_i386 i386linux

64-bit and rpm

AMD64 RedHat Linux utilize lots of rpm that has ARCH set to x86_64 (instead of the traditional i386). rpm -q by default won't tell you. Use something like

rpm -q --qf '%{NAME} %{VERSION} %{RELEASE} (%{ARCH})\n' Package_Name

to see which one you have installed. In 64-bit machines, most of the packages are x86_64. But quite often you will have a matching package of the same name, but is i386 ARCH. Some apps don't understand 64bit libs, so these older libs are sometime needed. There are also some compat libs. My friend Vic says:

* Compatibility Arch Support
Crapload of i386 binaries and libraries that run on x86_64 RedHat systems

* Compatibility Arch Development Support
You'll need some packages in Compatibility Arch Development Support to build
i386 RPMs or compile 32 bit binaries on an x86_64 system.

I find it annoying to install the Compatibility Arch Support group, though.

For instance, let's say you have an x86_64 arch server. You want some i386
compatibility for some random library.  Let's say zlib.i386 for whatever
reason.  Maybe some 3rd party application needs to run as a 32 bit binary and
requires zlib's /usr/lib/ instead of zlib.x86_64's

You can just install Compatibility Arch Support and not worry about it, or go
back an install zlib.i386 specifically.  If you install the whole
Compatibility Arch Support group, you are going to end up with all the
freakin' i386 packages designed to run on x86_64 systems.  Which means, you'll
get a lot of crap you don't want to be living on a light weight server.  For
instance, kdebase.i386, gnome-blah.i386 and such.

One thing that is very useful to have is firefox.i386 or mozilla.i386 on an
x86_64 system. 
Most plugins don't play well with 64-bit browsers.

So, in short -- you can remove what you don't need :)


The Linux community embraces the source code, and compatibility between different release is to recompile the code. Binary compatibility is not maintained between different (minor) releases of the glibc. So, be careful not to update the libraries, kernels, and gcc compiler unless you are ready to recompile everything in the system! (And if you have 3rd party libraries dependencies, then really think twice before changing the system library level).
If using up2date for RHEL, it would be safe as only compatible updates are listed in the channel. I would assume the same is true for yum.

System Release			kernel		glibc		gcc			compat-glibc
----------------------------	-------------	-------------	-------------		-------------
SuSE SLUS  9 (patch  3)	2.3.3-98.94	3.3.3-43.54		
SuSE SLUS 10 (patch 10)	2.4-31.5	4.1.2_20070115-0.11	

RHEL 3				2.4.21		   		3.2
RHEL 4.5 (WS)			2.6.9-55	2.3.4-2.36	3.4.6-8			2.3.2-95.30
RHEL 5				2.6.18		2.5		4.1			

RHEL 5.9                        2.6.18-348      2.5-107         4.1.2-54
Fedora Core 3                   2.6.9
Fedora Core 6			2.6.19-1	2.5-10.fc6	4.1.1-51.fc6		
Fedora 8                        2.6.23

RHEL 8.1			4.18.0-147.3.1  2.28-72		8.3.1-4.5

RHEL 2.1, 3, 4.  Increasing support cost: WS,                    WS HPC, 		ES (2 socket max),      AS
             5.                           Desktop + workstation, HPC Compute Nodes, 	Server, 		Advance Platform

For RH, each category has separation of basic, std, premium. HPC has a 4 socket version. See comparison chat.

Initial Setup

Things to keep in mind when installing Linux.

For RHEL 3 and 4, once the OS is installed, additional international language support cannot be added (easily), RH recommends a reinstall. So, if you might need to support any given natural language, install it when you do the system install!
Starting with Fedora Core 4, additional language can be added from system-configure-packages.

CentOS Network Install CD
Use Web as source of rpm.  
Server:  			(no http:// prefix)
Directory: yum/pub/centos/5.1/os/x86_64/  	(leading slash should not matter)
Directory: /centos/5.2/os/x85_64
But unfortunately don't support proxy, so likely have to setup something locally.
The dir containing images sub dir is the right one, it need to have the rpm available rather than just .iso
Essentially, same as internal network install, but the web server is over the WAN, and not using a pre-defined kickstart.cfg file :)
Other mirror can be used, but some of them only offer ISO files.


A step-by-step command listing for setting up kickstart, (hopefully easier to read than Red Hat kickstart instructions, once it is working, tweaking it is much easier to comprehend). ...
Here is a rough outline:

  1. iso loopback mount DVD or copy all content of CD/DVD to a dir
  2. cretae a dir on web server to host the ks.cfg file, make file accessible as http://apache/ks/ks.cfg
  3. ks.cfg uses http or nfs install, indicating path where rpm can be retrieved. eg:
    nfs --server= --dir=/mnt/loopback/rhel-5.1-server-x86_64-dvd
    url --url
  4. client, use cd 1, at boot prompt, enter:
    linux ks=
running commands after kickstart rpm are installed, everything is masked as ran from /:
echo "manual log to execution ran on post section of kickstart "  > ${LOGFILE}
pwd                                                              >> ${LOGFILE}

# Add yum repos (rpm from original dvd for easy install via yum)
echo "[rhel5]
name=RHEL5 Kickstart Server 
gpgcheck=0" > /etc/yum.repos.d/rhel5.repo

# install additional packages 
yum -y install j2re AdobeReader_enu

# Create symlink for java browser plugin
ln -s /usr/java/j2re1.4.2_07/plugin/i386/ns610-gcc32/ /usr/lib/mozilla/plugins/

# Local services
chkconfig ntpd on

echo "nfserver:/export/home    /nfshome    nfs    rw,soft,intr,tcp,rsize=32768,wsize=32768,vers=3 0 0
" >> /etc/fstab

mkdir /nfshome/
touch /nfshome/MOUNT+PENDING
mount /nfshome

ln -s /bin/csh  /usr/bin/csh
ln -s /bin/tcsh /usr/bin/tcsh

echo "sn     ALL=(ALL) ALL" >> /etc/sudoers

mv /etc/yp.conf /etc/yp.conf.orig
wget http://apache/conf/yp.conf -O /etc/yp.conf

# Modify SSHD allowed protocols to use only ssh v2:
# sed -i.bak -c "s/#Protocol 2,1/Protocol 2/" /etc/ssh/sshd_config

Updating OS

RHEL 4.x  - Use up2date
RHEL 5.x  - Use yum
	    rhn_register # automatically prompt for root password when run as user, X-based.
SuSE      - Use rug

Version numbers with dots matter for kernel, glibc, etc. Only version number after dash are bug fixes and don't change behaviour. See compatibility section for more details.

Quick Ref

Hardware related command

see also
  • performance section
  • bios.html
    lspci			# list pci info
    lsmod			# display all kernel loaded modules.  
    modprobe -l		# display all kernel loaded modules.  now lsmod.
    hwbrowser		# GUI hardware browser, in /usr/bin  (RH9, RHEL 4)
    /dev/mic*		# many integrated core, for Phi cpu when setup as co-processor, accelerator (NOT the Dell C6320p which has Phi as main cpu)
    udev			# some hw persisten naming thing, eg for hd that may change name due to san multipath, etc.

    Random Tidbits

    Changing Timezone
    /etc/cron.daily/logrotate	# cronjob that runs daily
    /etc/logrotate.d/syslog		# where /var/log/messages is stated to get rotated, compression, etc.
    /var/lib/logrotate/logrotate.status	# when a given log was last rotated.  new entry may have to wait eg 1 week before it get rotated
    /usr/sbin/logrotate -v /etc/logrotate.conf	# manually invoke logrotate with verbose output, stating why file are rotated or not.


    enabling XDMCP will allow program like X Manager to use browser and login via virtual screen, 
    fully X Manager, Remote Display Manager style, rather than having to login 
    using ssh and start VNC first.  However, VNC is nicer in that 
    the session stays on the server, XDMCP, if client is a laptop and disconnect, 
    session will be reset.
    Note that botyh XDMCP and VNC are insecure by default!
    vi /etc/X11/fs/config
    # don't listen to TCP ports by default for security reasons
    ###no-listen = tcp
    ### commented out line above to enable XDMCP
    service xfs restart
    vi /etc/X11/xdm/xdm-config
    ! SECURITY: do not listen for XDMCP or Chooser requests
    ! Comment out this line if you want to manage X terminals with xdm
    !!!DisplayManager.requestPort:  0
    !!!Line above commented out to enable XDMCP
    vi /etc/X11/xdm/Xaccess
    # *                                     #any host can get a login window
    *       # have start by itself will allow all host to get login window
    vi /etc/X11/gdm/gdm.conf
    ### XDMCP is enabled using the above clause
    ### this file need to be updated when GNOME is default windows manager/Display Manager
    vi /etc/kde/kdm/kderc
    ### XDMCP is enabled using the above clause
    ### this file need to be updated when KDE is default windows manager/Display Manager
    chmod 444 /etc/X11/xdm/Xservers                 # probably correct already
    chmod 755 /etc/X11/xdm/Xsetup_0
    older config need to update /etc/X11/XF86Config, but circa 2003
    most distro use Xorg,
    check /etc/X11/xorg.conf
    and ensure FontPath uses Xserver:
            FontPath     "unix/:7100"
    restart X:
    - killall gdm-binary 	# if running GNOME by default
    - ctrl+alt+bacckspace	# if in front of maching/keyboard
    - init 3; init 5	# This may work
    - reboot		# :)


    gnome-terminal, configure to source .login/.cshrc when launched:
    Edit menu, Current Profile... Titles and Command tab, 
    check "Run command as a login shell".
    This should be saved in the user's config file somewhere under the user's home dir.  
    # to view user's setting.
    gconftool-2 --get /apps/gnome-terminal/profiles/Default/login_shell	
    # command line to set to source login shell, per user.
    gconftool-2 --type boolean --set /apps/gnome-terminal/profiles/Default/login_shell true			
    # read system wide setting
    gconftool-2 --direct --config-source xml:read:/etc/gconf/gconf.xml.defaults --get /apps/gnome-terminal/profiles/Default/login_shell		
    # set global settings as root, to source login when term opens.
    gconftool-2 --direct --config-source xml:readwrite:/etc/gconf/gconf.xml.defaults --type boolean --set /apps/gnome-terminal/profiles/Default/login_shell true		

    authentication configuration

    authconfig is the tool to use to configure authentication services in RHEL7/CentOS7. Manually changing nsswtich.conf isn't enough. There are pam.d, selinux, firewall updates.
    authconfig-tui is being deprecated, so for interactive config would need to rely on authconfig-gtk or system-config-authentication.
    See also ipa-client-install and realmd

    sssd is the daemon for System Security Services. It provides a unified interface to manage cache daamon, etc. nsswitch.conf refer this as sss.
    /etc/sysconfig/authconfig	# config file (multiple server use space as delimiter)
    authconfig --test		# show/test current auth service
    authconfig ... --update		# for any changes, always end with --update!
    authconfig --enablenis --nisdomain=research --nisserver=nis1,nis2 --disableldap --update	
    	# change authentication method of system to use nis, not ldap
    	# this thing may fail silently... 
    	# or other config in file not expressed in the cli doesn't get changed 
    	# and affect final behavior :(
    	# run test a/o check the resulting config file to be sure.


    See the LDAP doc for more details.

    authconfig 		# TUI, select to use ldap, etc.
    			# upon completion, ldap should work for user login.
    			# multiple server can be listed using comma separation
    			# or space separated in /etc/sysconfig/authconfig
    /etc/sysconfig/autofs	# update BASEDN to define where autofs will look for maps.  eg
    			# BASEDN="ou=us,dc=unixville,dc=com"
    service autofs restart	# restart autofs to effect the changes
    /usr/lib/autofs/autofs-ldap-auto-master		# list all known automount maps from ldap


    ypcat (group, passwd)
    ypstart		# linux
    ypinit 		# client, specify machines to bind to
    		# server, specify whether master or slave
    		# Often messed up client, "up2date yp" to ensure running latest code with bug fixes.
    ypwhich		# tell what NIS server machine is bound to.
    yppasswd (obsolete, for combatibility only)
    ypcat -k auto.master	(-k to display the key name too!)
    ypcat -k	(the direct map, use /-)
    /etc/init.d/autofs	# most standard automount from auto.master, etc.  
    			# /net need to be enabled by uncommenting entry in /etc/auto.master .
    			# Some version of autofs (eg RH 4.0) have nasty bug that stale NFS mount
    			# may result in autofs daemon wanting to erase all files on server,
    			# at the very least, seen it cause lot of errors in syslog.
    /etc/init.d/amd		# an alternate for automount for /net/HOST/... paths
    			# install as up2date am-utils, and enable always start via ntsysv
    			# amd maybe a bit older than autofs and performance may not be as good.

    Network Connectivity

    mii-tool -vv		# determine NIC speed, duplex.
    ethtool -i eth0 	# list ethernet driver info
    Spawn login session on serial port:
    echo "7:2345:respawn:/sbin/agetty -h 9600 ttyS0" >> /etc/inittab
    echo "8:2345:respawn:/sbin/agetty -h 9600 ttyS1" >> /etc/inittab
    above does NOT redirect VGA to console, to do that, use
    (nothing will show on VGA output at boot):
    	[check zambeel notes, TBA]
    adding route:
    route add default gw
    route delete -net netmask eth3
    route add -net netmask gw eth2
    (but going out from first interface)
    route add -net netmask gw
    metric 1 dev eth2
    route add -net netmask dev eth3
    dhclient	= get dhcp address from server for current client.
    dhcpcd		= slackware, start the dhcp client daemon, then interfaces 
    		  will get dhcp address.
    ifconfig hme0 dhcp		see purple book.  (Solaris?)
    Link Aggregation
    Here is a script to setup bonding:
    # script to change machine with single eth0 to use bond0 (eth0+eth1)
    # completely automatic, generic for all hosts.
    # no testing of conditions, assume only 1 nic with 1 ip.
    # For RHEL 5.3 with bonding modules for kernel already pre-installed.
    # copy script to local drive so that it doesn't depend on network avail !!
    echo "alias bond0 bonding"                              >> $ModConfFile
    echo "options bond0 mode=balance-rr miimon=100"         >> $ModConfFile
    echo "#"                                                                                                >> $ModConfFile
    echo "# see"     >> $ModConfFile
    echo "# Section 1 and 6 Switch config for more info of different modes."                                >> $ModConfFile
    echo "# balance-rr(0), balance-xor, boradcast: works with Cisco EtherChannel "                             >> $ModConfFile
    echo "# balance-xor is safer but less optimal than balance-rr"                                          >> $ModConfFile
    echo "# 802.3ad is the IEEE standard, aka lacp mode"                                                    >> $ModConfFile
    echo "# active-backup(1), balance-tlb and balance-alb modes do not require any specific configuration of the switch" >> $ModConfFile
    echo "#"                                                                                                >> $ModConfFile
    echo "# sed -i 's/^options bond0 mode=balance-alb/options bond0 mode=balance-rr/' $ModConfFile "        >> $ModConfFile
    # create a bond  interface
    echo '
    ' > /etc/sysconfig/network-scripts/ifcfg-bond0
    cat /etc/sysconfig/network-scripts/ifcfg-eth0 | grep ^IPADDR=10.140    >> /etc/sysconfig/network-scripts/ifcfg-bond0
    # update eth1 config
    echo '
    ' >> /etc/sysconfig/network-scripts/ifcfg-eth1
    sed -i 's/ONBOOT=no/ONBOOT=yes/'        /etc/sysconfig/network-scripts/ifcfg-eth1
    # update eth0 config
    sed -i 's/^IPADDR=/#IPADDR=/'                           /etc/sysconfig/network-scripts/ifcfg-eth0
    sed -i 's/^NETMASK=/#NETMASK=/'                         /etc/sysconfig/network-scripts/ifcfg-eth0
    sed -i 's/^BOOTPROTO=static/BOOTPROTO=none/'            /etc/sysconfig/network-scripts/ifcfg-eth0
    echo '
    ' >> /etc/sysconfig/network-scripts/ifcfg-eth0
    # next two lines need to run in sequence, network connectivity lost after modprobe
    /sbin/modprobe bonding ; /etc/init.d/network reload
    /etc/init.d/network status
    /sbin/ifconfig bond0
    /sbin/ifconfig eth0
    /sbin/ifconfig eth1
    cat /proc/net/bonding/bond0


    iptables (RHEL)

    For more details, see firewall.html and net.html#firewall
    /etc/sysconfig/iptables 	# firewall rule config file
    system-config-firewall		# GUI tool to set iptables firewall rules
    firewall-config			# use this instead of above in RHEL7 (since it use shorewalls)

    SAN Connectivity

    Basic SAN Commands

    Utility/Function Description
    fdisk Command used to create and manipulate partition tables.
    fsck Command used to check and repair a Linux filesystem.
    mkfs Command used to create a Linux filesystem on a device partition.
    insmod Utility used to dynamically load a single module into a running kernel.
    rmmod Utility used to unload loadable modules from the running kernel 
           if they are not in use and if other modules are not dependent upon those being removed.
    modprobe Utility used to load or remove a set of modules that can be either a single module or a stack of dependent modules.
    lsmod Utility used to list the currently loaded modules.
    lspci Utility used to display information about all of the PCI buses in the system and all of the devices connected to those buses.
    scsiinfo Utility to query information from a scsi device.
    Ref: Emulex/EMC Linux Guide.

    HBA - Emulex

    HBAnyware is an optional add-on utility, but it is not a driver req for FC functionality.
    Emulex HBA Driver is included in RHEL kernel since 4.1.  To verify:
    lsmod | grep lpfc
    modprobe -l | grep lpfc
    modinfo lpfc | fgrep version
    	# if installing HBAnyware + lputl, must find exact version matching lpfc driver !!
    CNA (convergent net+hba) driver on the other hand is not in kernel of std os distro.
    hbanyware		# Java GUI tool for emulex HBA, in /usr/sbin/hbanyware
    hbacmd			# cli
    sudo /usr/sbin/hbanyware/hbacmd listhbas	# show hbas on the host
    /usr/sbin/lpfc/lputil	# util to check config and coutner for Emulex HBA
    /usr/sbin/lpfc/lun_scan all	# check for new LUN, delete dead ones
    /usr/sbin/lpfc/lun_scan -r all	# supposed to check for size change of LUN, don't work in RHEL 5.2

    Restarting HBA driver
    Ref: p58 Unix user manual (by EMC)
    Run the following commands to stop the Emulex application daemons:
    cd /usr/sbin/hbanyware
    Run the following command to remove the Emulex Application
    Helper Module (lpfcdfc driver):
    /etc/init.d/elxlpfc stop
    Finally, unload the Emulex driver:
    modprobe -r lpfc
    To reload the Emulex driver, the Application Helper Module (lpfcdfc
    driver), and the Application Kit use the following steps.
    Load the Emulex driver:
    modprobe lpfc lpfc_log_verbose=0x10
    Reload the Emulex Application Helper Module (lpfcdfc driver):
    /etc/init.d/elxlpfc start
    Start the Emulex Application daemons:
    cd /usr/sbin/hbanyware

    HBA - Qlogic

    Driver typically install some command tools in /opt/QLogic_Corporation
    scli	- san surfer cli
    qaucli	- convered network cli
    these can be run in a menu prompted interface.  see HBA info, WWN, BIOS version, etc.
    Can flash HBA using this command in linux and providing a .BIN file.
    getting driver to work in upgraded kernel was quite painful.
    Maybe should have avoided building the initrd image, which clobber with subsequent 
    kernel compilation...?
    The DMKS driver didn't work.  
    the Converged Network Driver in RPM worked out okay.  Had to hand edit modprobe.conf that 
    added module but didn't work in new kernel... then things worked.
    BIOS is not explicitly listed, but part of some package like converged console cli package.
    (and there there is a zip file that contain a .BIN (and DOS utility to install flash, but can be done in linux)).

    parallel scsi

    new LUN from SAN to a physical host, hba card can scan for new FC disk:
          /usr/sbin/lpfc/lun_scan all
    ESX direct raw lun mapping provides good old fashion scsi disk, not FC.  Command to use for scanning scsi bus is:
    	  echo "- - -" > /sys/class/scsi_host/host0/scan			# scan for new device, change host0 to appropriate controller
          echo 1 > /sys/block/sda/device/rescan					    # rescan for changed size, change sda to appropriate drive
    the hostX dir has to do with scsi id, but not necessarily assigned in same numbring.
    default 0:x device are in host0, but when i added scsi 2:10, 
    it was placed in host1.  I had skipped 1:x, presumably later on when this is added, 
    this would be assigned the next hostX dir.
    but /dev/sdc and /dev/sdd are not assigned as per scsi id number, but sequentially
    so sdc was 2:10, and sdd was 1:2   :(
    the scsi target id number can be found by looking at
    	ls -ld /sys/block/sd*/device
    Overall, in linux, scsi device in /dev/sdb, sdc, etc are not necessary static.  
    changes to scsi bus and boot order may make them scan at different order and 
    thus assigned different device path.  
    fs labeling and fstab mounting using such label circumvent around the problem.
    but no such labeling in raw scsi devices.
    emc powerpath provides a uniq /dev/emcpower* device path for the LUN.
    in its absence, can use udev naming rules.
    scsi_id : retrieve and generate a unique SCSI identifier
    	  the id presented should be consistent when presented to multiple
    	  host in a cluster environment.
    udev    : dynamic dev management  - seems to allow for mapping scsi_id 
    	  to specific path user desire, and make such path persistent 
    	  to the same physical disk independent of machine path, 
    	  since it uses id.
    linux machine path eg: /sys/block/sde 
    scsi id for the device can be retrieved by:
    	scsi_id -g -u -s /block/sde
    KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -s %p", RESULT=="3600601607ba02900b63b5cf7c4b3e011", SYMLINK+="data10_d1_p%n"
    To test the rule w/o reboot, use the command "udevtest  /block/sde"
    Oracle ASM can use /dev/emcpowerX1 for data storage.  (need to have a partition in it).
    or use the symbolic name created by udev, like /dev/data10_d1_p1
    (udev create sym link at boot like /dev/data10_d1_p --> sde, /dev/data10_d1_p1 --> sde1)
    maps emcpowerX to /dev/raw/rawX so that scsi id reservation for voting can work.
    if no power path, then use scsi_id 
    ACTION=="add", KERNEL=="emcpoweri", RUN+="/bin/raw /dev/raw/raw1 %N"
    ACTION=="add", KERNEL=="sd*[!0-9]", PROGRAM=="/sbin/scsi_id -g -u -s %p", RESULT=="3600601607ba0290056a27f08c6b3e011", RUN+="/bin/raw /dev/raw/raw1 %N"
    when the above rule runs, it will create "character special devices" path in 
    Oracle RAC uses /dev/raw/raw1 thru 5 to run its voting and quorum selection to power the cluster membership algorithm.
    No partitions are needed in such disk.  only scsi reservation commands are issued to it.


    Info moved to infiniband.html


    cat /sys/block/sda/queue/rotational
    0 = ssd
    1 = hdd


    cat /proc/mdstat 
    mdadm --detail /dev/mdNNN
    mdadm --detail --scan 			# could redirect output to like /etc/mdadm.conf to save config
    mdadm --create --verbose /dev/md127 --level=linear --raid-devices=3  /dev/sda1 /dev/sdb1 /dev/sdc1 
    have to assemble the device each time when running:
    mdadm --assemble /dev/md127 /dev/sda1 /dev/sdb1 /dev/sdc1 
    mdadm --assemble --scan
    mdadm --stop /dev/md127
    Ref: Linux RAID


    storcli is the replacement for megacli, should be easier to use and script.
    eg use for LSI Logic / Symbios Logic MegaRAID SAS-3 3108

    Virtual Disk should be the center of gravity for work to present a "LUN" from the raid controller to the host OS.
    /c0 		# is controller 0
    /c0/v0		# is controller 0, virtual disk 0, eg RAID 1 for OS
    /c0/v1		# is controller 0, virtual disk 1, eg RAID 6 for data
    /c0/eall	# eall = all enclosure.  e252, e8
    /c0/eall/sall	# sall = all slot.  
    /c0/e252/s1	# is controller 0, enclosure 252 (really just channel number), slot 1 (really hd on the channel).  ie hd 252:1 EID:Slot
    /c0/e8/s7	# is controller 0, hd at EID:Slt of 8:7 (eg RAID 6 volume for data)
    storcli64 /c0/eall/sall | grep Model\ Num	# get all hard drives model on controller 0
    storcli64 /c0 add vd raid6 drives=8:0-3		# create a LUN with disk 8:0 thru 8:3 (SID:Slt)
    storcli64 /c0/v1 start migrate type=raid6 option=add drives=8:4-7	# add disks 8:4 thru 8:7 to existing /c0/v1 VD (via migrate process)
    									# raid type remains the same
    storcli64 /c0/v1 start migrate type=raid5	# change from prev raid6 to raid5.  no disk addition needed.
    storcli64 /c0/vall show migrate			# show add progress (eg take 3 days to add 4 disks, 18 days to restripe from raid6 to raid5)
    storcli64 /c0 show				# constroller info, new disks show as UGood.
    storcli64 /c0/v1 show all			# display VD/LUN info
    storcli64 /c0/vall show init			# init is not always req.  will be done automatically in big for VD with ~5+ disks
    						# but before creating LVM/FS, may want to force it to start on small VD, just to be sure
    storcli64 /c0/vall show cc			# consistency check, seems to req that init have been done.  recommend 1+/month
    storcli64 /c0/v1 start cc			# cc (or init) can run in bg or fg, pause, resumed, etc.
    storcli64 /c0/v1 pause cc
    storcli64 /c0/v1 resume cc
    storcli64 /c0/v1 stop cc
    storcli64 /c0/v2 set name=dev_sdc		# set name of VD, best to put linux path on it for easy mapping.
    						# hopefully os don't remap it after reboot...
    # lsblk
    # lsscsi: 								# storcli64: 
    [0:0:8:0]    enclosu SMC      SC846P           0c1f  -        
    [0:2:0:0]    disk    AVAGO    MR9361-8i        4.68  /dev/sda 		# /c0/v0
    [0:2:1:0]    disk    AVAGO    MR9361-8i        4.68  /dev/sdb 		# /c0/v1
    [0:2:2:0]    disk    AVAGO    MR9361-8i        4.68  /dev/sdc 		# /c0/v2
     c:  vd 
        scsi id/path map to raid controller path ::
    ie, first number in lsscsi is likely the LSI/Avago controller number c0
        third number in lsscsi is likely the LSI/Avago controller Virtual Disk number eg VD1
    lsscsi -d	# device major and minor number
    lsscsi -w -s	# wwn, size
    lsscsi -i 	# -i = udev derived scsi_id, probably picked by zfs
    lsscsi -w -i -d	# 
                         -w                                   -d        -i                                 -s
    [0:0:11:0]   disk    0x5000c5009443358f          /dev/sda [8:0]     35000c5009443358f                  10.0TB
    [0:2:0:0]    disk    AVAGO    MR9361-8i    4.68  /dev/sdy [65:128]  3600605b00cd891a021fd7ddc04f32b71   1.02TB
    Bluemoon settings
    /opt/storcli64 /call show bootdrive	# see which VD is used to boot the machine (OS drive)
    /opt/storcli64 /c0 show migraterate      # check migration rebuild %, default is 30
    /opt/storcli64 /c0 set  migraterate=90   # upping to 90% on idle drive doesn't improve progress speed.
    /opt/storcli64 /c0/e8/sall show rebuild
    /opt/storcli64 /c0/v1 set wrcache=wt    # set write cache ## p221 of pdf has cmd for other settings
    ### p190 has controller properties.
    /opt/storcli64 /c0 show alarm 		# whether controller alarm is enabled (probably whether alarm is usable, not whether it is beeping) [only for 9750 and 9690a)
    storcli64 /call show event		# display controller events, presumably fail drive, error status, etc would be listed here
    p210 drive locate command
    /opt/storcli64 /c0/e8/s1 start locate
    /opt/storcli64 /c0/e8/s1 stop  locate
    storcli64 /c0/cv show all   		# check battery backup
    storcli64 /call/dall show all 		# get disk detail
    ## example commands and output in


    megaRAID controller is often used in white box low cost "home made" storage arrays.
    megacli is the main command to manipulate such raid.
    there is a gui called raid manager.
    /usr/local/MegaRAID\ Storage\ Manager/./    # GUI
    # ?? not sure what these params are exactly:
    # -a    = array controller number.  
    # -a0   = array controller 0
    # -aAll = all array controllers (if there were multiple)
    # -l    = ??
    # -L    = Logical Disk?
    MegaCli64 -LDInfo -lall -aall	# array info and state
    MegaCli64 -LDInfo -lall -aall | egrep --color "Adapter|^Virtual Drive|Name|^RAID Level|^Size|State|Number Of Drives"
    MegaCli64 -AdpAllInfo -aALL	# HW info, whether BBU (Battery Backup Unit) is present, error counters 
    MegaCli64 -AdpAllInfo -aALL	| egrep --color "^Adapter|Product Name|Serial No|FW Version|Mfg. Date"  # controller hw info
    MegaCli64 -AdpAllInfo -aALL     | egrep --color "^Adapter|Virtual Drives|Degraded|  Offline"		# virtual  drive state
    MegaCli64 -AdpAllInfo -aALL	| egrep --color "^Adapter|Physical Devices|Disks|Critical|  Failed"	# physical drive stte
    MegaCli64 -LDInfo -Lall -aALL | egrep  --color -i "virtual|state|adapter"
    MegaCli64 -PDInfo -PhysDrv [1:2] -aALL | more
    MegaCli64 -Pdgetmissing -a2		# see if there are missing drive in adapter 2
    MegaCli64 -PdList -aall | grep --color -i firmware	# get firmware info
    MegaCli64 -LdPdInfo -aAll # logicak disk info
    ## how to find out which linux /dev/sdX maps to which VirtualDisk (LUN) on the RAID controller ??
    MegaCli64 -ShowSummary -aAll
    MegaCli64 -EncInfo -aAll						# enclosure info
    MegaCli64 -AdpEventLog -GetEventLogInfo -aAll	# event log
    MegaCli64 -AdpEventLog -GetEvents
    MegaCli64 -AdpBbuCmd -aAll 						# Backup Battery Unit (BBU) state
    MegaCli64 -AdpBbuCmd -GetBbuStatus -a0
    MegaCli64 -AdpBbuCmd -GetBbuDesignInfo -a1
    useful megacli cmd blog by Frederik :
    # eg of converting from RAID6 to RAID5 to get more space! 
    /usr/local/sbin/MegaCli64 -LDRecon -Start -r5 -L0 -a0
    echo "- - -" > /sys/class/scsi_host/host0/scan			# scan for new device, change host0 to appropriate controller
    echo 1 > /sys/block/sda/device/rescan				# rescan for changed size, change sda to appropriate drive
    MegaCli64 -CfgLdAdd -r0'[32:3]' -a0     # create new config for Logical Disk with raid0 (with disk 32:3 ie one plain disk)
    										# -a0 = on array controller #0
    MegaCli64 -CfgLdAdd -r5'[252:5,252:6,252:7]' -a0		# create new raid5 LUN
    MegaCli64 -LDRecon -Start -r5 -Add -PhysDrv[32:3] -L0 -a0	# extend existing LUN with 1 new Physical Drive
    MegaCli64 -LDRecon ShowProg L0 -a0				# view reconstruction progress (when replacing bad drive?, or extending VD?)
    MegaCli -LDGetProp -DskCache -LAll -aALL			# check physical HD cache, recommend enable iff have UPS
    MegaCli64 -PDList -aALL  					# find out about physical HD info
    MegaCli64 -PDList -aALL | egrep --color "Adapter|Enclosure|^Coerced Size|Device Id"
    MegaCli64 -PDList -aALL | egrep --color "Adapter|Enclosure|^Coerced Size|WWN|Inquiry Data|Device Id"
    # SAS Addresses are subsequent to WWN, eg:
    #            WWN:  5000C500631B3D3C
    # SAS Address(0): 0x5000c500631b3d3d
    # SAS Address(1): 0x5000c500631b3d3e
    #                                  ^
    MegaRAID level explanation" server fault:
    /opt/MegaRAID/MegaCli/MegaCli64 -LDInfo -Lall -aALL | grep RAID
    ['Primary-0, Secondary-0, RAID Level Qualifier-0'] = RAID-0
    ['Primary-1, Secondary-0, RAID Level Qualifier-0'] = RAID-1
    ['Primary-5, Secondary-0, RAID Level Qualifier-3'] = RAID-5
    ['Primary-6, Secondary-0, RAID Level Qualifier-3'] = RAID-6
    ['Primary-1, Secondary-3, RAID Level Qualifier-0'] = RAID-10
    Primary-1   = "Mirrored array"
    Secondary-3 = "Spanned 0x03 A combination of stripping and concatenations involving Basic VDs of different sizes."
    RAID Level Qualifier-0 = "Mirroring across two extents"
    Additional info at
  • megaraid.html
  • MegaCLI cheetsheet
  • #hp

    HD Partitions

    From DOS days, disks are typically partitioned. Linux typically have up to 4 primary partions, and extended partitions if necessary. Technically, ext2/3 can be created on the whole disk without partition, ie, one can do
    mkfs -t ext3 /dev/sda
    instead of defining /dev/sda1, /dev/sda2, etc.
    But this is not recommended, another sys admin may think the disk is not in use. Partition also help add flags to indicate what fs is in a given partition.
    fdisk is the traditional command for manipulating partitions. It is reasonably user friendly, but can't handle large drive (2+ TB? or 8+ TB?). For really large drive, gnu parted will need to be used. parted can manipulate some fs, should be able to resize FAT fs/partitions.

    sudo parted /dev/sdb print		# show partition table
    sudo parted /dev/sdb mklabel gpt	# initialize disk as gparted, erase all existing partition!  then add label
    sudo parted /dev/sdb mklabel msdos	# initialize partition using msdos mbr
    sudo parted /dev/sdb mkpart primary ext3 1 2048		# shoudl create a 2G partition
    sudo parted /dev/sdb mkpart primary amufs 1 100%	# create a primary part
    	# instead of 100%, can use:
    	# cyl eg 123456000cyl
    	# mb     123456000    (ie, default is mb, finding disk size is tricky)
    	##   amufs cuz parted don't have lvm and don't want ext2/3
    sudo parted /dev/sdb set 1 LVM		# optional
    	## the LVM flag isn't likely relevant, but will be printed in output
    	## to help remind sys admin it is LVM and not say ext3 fs.
    parted /dev/sdz mkpart primary linux-swap 32GiB 40GiB # type id 82 # use GiB instead of MB so that it align correctly
    parted /dev/sdz mkpart primary ext2 40GiB 52GiB       # type id 83
    parted /dev/sdz mkpart primary ext2 52GiB 98%         # 98% of remaining space.  leave a bit at the end for possible conversion to gpt
    parted /dev/sdz align-check optimal 1
    parted /dev/sdz align-check optimal 2	# need to check alignment of each partition one at a time
    # repartitioning disk carefully could leave data seems untouched :)
    # need partition number and end size.  
    # if omitted will prompt, can use size in GB or % in addition to sector.
    # -1 is last block on disk.  
    sudo help parted # display help.  
    # eg resize partition 1, use 100% of avail space (eg after VM hd grew):
    sudo parted /dev/sda resizepart 1 100%  
    sudo parted /dev/sdb rm 1		# remove partition 1
    Don't use fdisk on big drive where parted was needed to create partition table.
    fdisk will screw things up.
    fdisk -l		# see list of drives
    fdisk /dev/sda		# modify a specific drive
    			# m for help


    File System:	FS: one ext3 fs on top of a given LV.
    Logical:	LV: many virtual/logical partitioins on top of a single VG.
    Aggregation:	VG: volume group, encapsulate many PVs. 
    Physical:	PV: place extends "markings" on a partition or whole disk 
    partition	sda1 or sdd
    kernel          -
    device		/dev/sda
    In VM environment, best to add a separate virtual disk, 
    and add the new disk to LVM using pvcreate+vgextend.
    Can use VM's feature to extend the existing LUN, 
    but would still need to use fdisk to create a new partition and then
    add this partition to LVM using pvcreate+vgextend.  
    (RHEL 5.4 need reboot to see expanded LUN, LVM partition can
    be an extended partition, but after fdisk creation of volume, reboot needed
    before pvcreate can mark the partition for use with vgextend.
    Newer Linux kernel can scan for new space without reboot, 
    echo "- - -" > /sys/class/scsi_host/host0/scan		# scan for new device, change host0 to appropriate controller
    echo 1       > /sys/block/sda/device/rescan		# rescan for changed size, change sda to appropriate drive
    pvs -o +pe_start		# show offset, new lvm default to 1m and usually good enough to get correct alignment
    lvs --segment
    lvdisplay -vm 			# show segemts and write policy (linear vs stripped)
    iostat 5 
    iostat -x -k 1
  • Percona alignment theory
  • SysAdmin Cast linear vs stripped LVM (but not sure if it is necessary or modern lvm takes care of it automatically?)
    PV on whole disk or partition?
    I used to like partition, just so that fdisk -l give more useful info.
    Now I am all for LVM using the whole disk without partition markings.
    This is because in a RAID controller environment (and lesser extend SAN), 
    the VirtualDisk (LUN) could grow (cuz added disks to the Raid Group).
    pvresize can then handle expanding the PV (it adds Extends), but:
    - if no partition, then pvresize will just expand to the new end.  not too painful.
    - if PV on partition, fdisk will see more space.  but partition is static.  
      Probably just add a new partition, then pvcreate on the new partition.
      (not sure if there are any way to enlarge existing partition via fdisk, maybe via gparted)
      Overall, having partition for LVM that wants to grow add an layer of complication.
    After pvexnted (or pvcreate), use the regular vgextend, lvresize, xfs_grow, etc to get more space.
    has a thread where pvresize on partitioned raid disk was done on LVM2 back in 2009 
    for more info, search on ServerFault for "lvm pvresize"...
    Separately, whole disk without partition does not create possible alignment problem.
    Modern fdisk would create partition that is correctly alignment with disk sector/block size (not so with older fdisk).
    Not using partition avoid this problem.
    ZFS does create partitions, and ZFS require a database of disks actual geometry (instead of "lie" it reports to windows-minded stack).  Lack of such correct definition could/would result in misalignment and reduces performance.
    LVM ccommand
    system-config-lvm			# GUI for LVM.  Pretty neat.
    lvm					# interactive shell for LVM, all cmd avail as stand alone.
    pvdisplay				# display info such as PV size, PE size, avail PE.
    pvcreate /dev/sda1			# create pv on partition, def RHEL 4 setup  
    pvcreate /dev/sdd			# whole disk allocated to LVM, can't hold swap or /boot, but typically ok for data volume
    pvresize /dev/sdd			# change block device , while LVM is live
    vgdisplay				# display vol grp info such as lvm/lvm2.
               -v				# list all LV on the vol group, which PV it uses.
    vgcreate VolGroup00 /dev/sda2		# create a new VG, this is the def in RHEL 4 setup.
    vgextend  VolGroup00 /dev/sda5
    vgreduce				# reduce number of disks in vol grp
    vgremove  VolGroup00			# remove the whole volume group
    lvscan						# brief list of LV and their size.
    lvdisplay					# show info of all LV: path, LV size, Status, which VG it resides in.
    lvdisplay /dev/VolGroup00/LogVol01		# display specific info of a single LV (instead of all LV)
    lvcreate -n LogVol00 --size 500M VolGroup00	# create a new LV called LogVol00 of size 100 MB
    lvextend -L+100M /dev/VolGroup00/LV01		# extend existing LV (hosting swap of fs) by given size
    						# -L20G would set exact size of 20G
    lvremove /dev/mapper/VolGroup00/LV01		# remove unused LV
    mkfs -j /dev/VolGroup00/LogVol02	# create fs on the LV (-j will use jounal, ie ext3)
    					# NOTE: old kernel may give error.  2.6.16-1.2069_FC4 works.
    mkswap /dev/VolGroup00/LogVol02		# create swap partition on the LV 
    					# Doesn't seems to work, actually confuses kernel to the point that
    					# all subsequent mkfs command will also be refused.  BUG??!!
    e2fsadm					# extend or reduce fs and LVM1 at the same time 
    					# but not avail with LVM2 (eg FC4, RHEL 4)
    ext2online /dev/VolGroup00/LogVol00	# extend ext3 FS online (for LVM 1 or 2)  (RH4)
    ext2online /				# can use mount point instead of dev path
    ext2resize				# change ext2 FS after it has been UNMOUNTED.  ?? does not exist??
    resize2fs /dev/VolGroup00/LogVol00 	# RH 5, extend the FS to max space that the LVM has allocated.
    					# op done online in RH5, no umount needed.
    resize2fs /dev/VolGroup00/lv101 1000000	# reduce ext2/ext3 FS to size to exactly 1,000,000 bytes. 
    xfs_growfs /mount/point 		# expand xfs, to max space of eg LV
    xfs_growfs /mount/point -D size
    lvreduce -L1G /dev/VolGroup00/LogVol00	# reduce the LV the FS is on 
    					# (Don't make it smaller than FS size!)
    					# After these procedure, there maybe some left over space,
    					# if using ext3, can use ext2online to reclaim the space.
    					# ext2 somehow can't be re-expanded, use ext3 !!
    					# use tunefs -j to convert to ext3 :)
    /etc/fstab				# typically list which LV dev used by which FS
    ??	 				# no cmd to display what LV is being used by which FS :(
    Importing a new hard drive that has LVM:
    vgexport VolGrpName			# export a volume so that it can be imported (work on already "migrated" hd)
    vgimport VolGrpName			# import for use
    lvscan					# newly imported volume group will be listed as inactive
    vgchange -ay VolGrpName			# activate volume group
    mount -t ext3 /dev/to/fs /mnt/myfs	# mount the file system on the newly imported volume
    Removing LVM stuff
    sudo umount /data
    sudo lvremove /dev/mapper/VolGroup02-LV_data
    sudo vgremove VolGroup02
    sudo pvremove /dev/sdb1

    Software Mirror Boot and/or OS volume

    There are several ways to setup software mirror for the OS volume (eg /) and/or the /boot volume.
    For simple server just needing mirror across two disks, best to let installer do the mirror during initial setup. Then the fact that it is not using RAID controller is largely transparent in daily sys admin task. Maybe RAID controller is warranted only when doing storage server with many disks, but then ZFS maybe a better choice there. RAID is also useful if want to keep CPU free. Also see discussion in hardware RAID versus mdadm versus LVM-striping
  • sw raid tends to be more flexible and easier for most sys admin to manage.
  • hw raid offload load and let cpu do other work.
  • hybrid hw + sw may give convinient large fs, but then painful in recovery situations. should avoid.

    Installer setup mdadm mirror for /boot FS and LVM volume for OS FS
    This is the easiest method, use when setting the machine from scratch using the RedHat installer (anaconda).
    See CentOS 7 Installation with LVM RAID 1 – Mirroring by tuxfixer, with screenshots of interactive GUI install.
    or How to create LVM on Software RAID (mirroring) during installation of Red Hat Enterprise Linux using kickstart file?(RedHat, login req)

    Overall, installer wiil:
    1. select two disks for mirror during disk selection step of install process.
    2. /boot is setup as RAID device (mdadm)
    3. swap is a LVM volume group of RAID 1 - create new "centos" VG -- LV is centos-swap
    4. / is a LVM volume group of RAID 1 - re-utilize "centos" VG -- LV is centos-root

    OS update don't need special steps. /boot is mirrored constantly by mdadm.
    Check on grub.conf though.
    So other than initial setup, normal operation is oblivious whether a HW RAID controller doing the mirror or SW mdadm. Disk recovery would be different, and some pre-emptive health check using smartctl, lmsensors and disk scrubbing.

    Verify mirror setup after machine is build:
    cat /proc/mdstat 	# should see two mdNNN device
    blkid | grep boot 	# see 2 /dev/sd* disk and 1 /dev/mdNNN mdadm dev
    pvdisplay		# see single PV on the /dev/mdNN  mdadm dev, so LVM don't need to know about mirroring.
    vgdisplay		# see single VG called "centos"
    lvdisplay  		# see typical LV setup for a RHEL machine.  again, these don't know they are mirrored by mdadm
    mdadm mirror whole disk - manual config after OS had been setup
    See How do I convert my root disk to RAID1 after installation of Red Hat Enterprise Linux 7? (redhat, login req)
    The target result is essentially same as above done by the anaconda installer. However, since exinsting partition/volume/fs exist, some data movement dance is required, thus taking many tedious steps to do. Characterisitcs
    1. This method use mdadm and mirror the whole disk (actually, multiple partitions on each disk that are raid devices)
    2. quite a number of steps, to be done after initial OS is setup
    3. mdadm mirror for /boot using /dev/md* device
    4. Then run LVM on top of the /dev/md* mirrored disk. so LVM doesn't know that it is mirrored. /boot and whatever volume that is placed inside the LVM get the underlaying mirroring done transparently.
    5. still need to update grub manually
    mdadm --create /dev/md0 --level=1 --raid-disks=2 missing /dev/sdb1 --metadata=1.0
    pvmove /dev/sda2 /dev/md1   # move VG from regular disk to md device (mirrored)
    dracut -f --mdadmconf

    LVM mirror only OS volume(s)
    See How do I convert my LVM root disk to LVM mirror after installation of Red Hat Enterprise Linux 7? (redhat, login required)
    The mdadm methods above are likely better. But this does not need to use mdadm if doesn't want to deal with that. But leave /boot non mirrored and need manual sync each time there is an update.
    Also, if doing mirror after OS has been setup, LVM-only approach req less acrobatic on the existing data thus a bit easier to carry out.

    1. It uses mirror at the LV level (lvconvert).
    2. Creates mirror (legacy) or raid1 (rhel6,7) segment types to hold /, swap, etc.
    3. /boot is left to be rsync manually.
    4. grub need to be setup for each boot disk/partition
    5. Long term require more manual maintenance
    6. Maybe easier to boot up in disaster scenario, assuming homework was done to mirror /boot and had grub up to date.
    7. overall, probably not the best solution.
    1. linear volume (single lvm with NO RAID?)
    2. segment type (aka LV type) --type
    3. mirrored volume (used up to RHEL 6, considered legacy in 2018)
    4. raid1 (rhel7 default)
    vgextend system_vg /dev/vdb2     # add normal disk device into LVM purview
    lvconvert -m1 --type raid1 /dev/system_vg/root /dev/sdb     # create mirror at LV level, using specific disk that LVM has control of.
    lvconvert -m1 datavg/testlv
    # converts from linear to mirrored volume 
    lvconvert -m1 datavg/testlv /dev/sdc
    # remove mirror, /dev/sdc is the device that is DETACHED
    lvs -a -o+devices
    dmsetup table
    work be for RHEL7
    convert single disk EFI boot to sw raid AFTER OS install
    If using EFI, may have to check into this: < A HREF="">How to convert single disk EFI boot to software RAID after installation (redhat "solution in progress").

    File System Related Command

    See also: fs for loopback mount, samba, etc.
    mkswap  /dev/hdd3			# def RHEL 2.1 swap dev
    mkswap  /dev/VolGroup00/LogVol01	# def RHEL 4   swap dev
    swapon  -s 				# list swap dev (eg, which LV is swap partion hosted on)
    swapon  -a 				# turn on  all swap partitions/files
    swapoff -a 				# turn off all swap partitions/files
    mkfs -t ext3 /dev/hdd2	# create a new fs on ide drive secondary slave.
    			# probably only create as ext2;  then:
    tune2fs -j /dev/hda3 	# add journaling to it via tune2fs  (manual for ext2 in 2.1)
    			# (probably automatic/default in ext3/as 3.0)
    e2label /dev/hda3 [Partition-Label]
    		view/change label of a partition
    		used in /etc/fstab LABEL=/label-name for mounting
    		so that mapping does not have to be maintained in fstab
    findfs LABEL=/boot	# determine which disk host a given FS with the assigned label.
    raven:/usr/local/bin/wshaper  traffic shaping and throttle specific connection bandwidth usage.
    du -kxS * 	= display space usage Summary for only (x) one file sys (ie local)
    				linux only
    du -kl		= local disk in solaris?
    df -kl 		= report fs space usage of local fs, solaris and linux

    Undelete and data recovery tools

    There is no undelete in XFS or Ext or most unix FS.
    However, if an inode is unlinked but neither it nor its associated data blocks get immediately re-used and overwritten, there is some small chance to recover the file from the disk.
    Umount the file system as soon as realize needing to do data recovery. This help preserve as many inodes and data blocks before they get overwritten by new files.
    If hardware is failing, dd the disk/partition image to a new device and recover from that.
    If accidentally deleted lot of files from rm -r, proceed carefully, keeping the FS mounted as RO is okay by most tool. Restore to new location so as not to overwrite any block.
    ext4magic is the best undelete tool for ext I have used so far (had more success undeleting files from ext4 fs than extundelete).
    apt install ext4magic # avail in ubuntu
    ext4magic /dev/sda3 -L # list recoverable files
    ext4magic /dev/sda3 -r
    ext4magic /dev/sda3 -m # works when FS still mounted RO, find more files than shown by -L
    recovered data in ./RECOVERDIR
    Was not successful in defining syntax to undelete only one specific file
    ext4 is journaled 
    extundelete will scan the journal, inode, and retrieve files.
    But FS should be unmounted ASAP so that data are preserved as much as possible.
    eg run:
    mount /dev/sdb1 /RESTORE	# eg new external hd to store restored files
    cd /RESTORE
    extundelete /dev/sda3 --restore-all 	
    would retrieve all files it can undelete, and save in current dir, 
    under a dir called RESTORED_FILES
    it will preserve original directory structure and filename.
    EPEL have this package for CentOS7:
    if compiling from source, ensure these packages are installed:
    photorec is a signature based recovery tool that work largely independent of FS (thus work for FAT, NTFS, ext2, etc, but not RaiserFS because it store small amount of data in the b-tree itself for better performance).
    It does more than AV datas. Can recover zip, txt, mathlab mdl, CAD, pdf, ps, etc. It handles some 300 file families
    Unfortunately it does *NOT* recover file name or directory structure. it will determine extension from the file magic/signature. Scripts exist that help sort files by type into directories, recover filename from EXIF, MP3 tags, etc.
    qPhotoRec is a GUI.
    Static link binary of TUI for both PhotoRec and TestDisk avail from single tar.bz2 at cgsecurity download.
    TestDisk use a TUI to recover recover parition table and undelete file.
    It also undelete files, presumably scanning FS table to recover deleted file, can provide name and directory structure.
    For Ext, may work similarly to extundelete, ext4magic.
    Handles other FS type like FAT, NTFS, HFS+.
    Note that EXT is listed under "MS DATA" partition type.
    It recommends using PhotoRec if desired file is not undeleted.
    Commercial tools
  • SysDev lab sale two product lines. I wonder if they are really just different in the UI skin and menus. Both use a GUI, and recover XFS, EXT2/3/4, and other.
    They were the only tool I found (July 2018) that preserve filename and directory structure for XFS.
    The base version likely suffice for recovery from accidental rm on files. RAID version could be used if destroyed raid volume? and the Pro version has (maybe) gimmicky viewer tools or on-the-fly edit of FS? Trial version works, but restrict recovery size to 700K per file.
    There is a IntelliRAW feature, that just scan for say picture or video, file will have no name and could be truncated. Useful only if it portion recovery of content is useful. For structured data file this won't be all that useful.
    On retrospect, maybe similar to PhotoRec, but need to define rules, so not played with.

  • R-Linux Free undelete tool for ext2/3/4. /usr/bin/rlinux. GUI. Has ability to create an image of the partition needing data recovery. For my ext4 thumbdrive test this tool worked horribly. scanned and found old FAT files and recovered none of ext4 files. Didn't produce any filename or directory structure.
  • R-studio Data Recovery. Same GUI as r-linux, with support for additional FS types: Ext2/3/4, HFS, UFS, NTFS, exFAT, but NO XFS!!. Demo version limit to 256KB per file.
    Fedora rpm worked in centos7. run: /usr/bin/rstudio
    Probably less useful tools
    Ref: XFS FAQ
    stack exchange


    system boot up doesn't do any of the check.
    only do mount...
    only run the below if have suspect of fs consistency problem.
    mount and umount the system cleanly before running the check, 
    so that xfs log is in proper state
    xfs is a parallel fs.
    parallelism is governed by agcount (aggregate group count).
    each ag is a parallel thread.
    But don't set agcount per cpu thread, but on number of spinning disk.
    agcount of 4 to 16 are common.  really large array could have larger number.
    don't have agcount in thousands, it will have detrimental impact on performance.
    # use xfs_info to see agcount.  if FS grew from GB to TB, it may have too many ag for efficient work.
    # when too many ag (eg agcount=5295839, agsize=6336 blks), kworker spends many minutes scanning thru them before actual file operation can take place.
    # recreate the fs  with eg agcount=125, agsize=268435328 blks and things will be much faster
    # xfs_repair is equally faster and more "responsive" with a more normal agcount (hundres to thousands).
    # agcount * agsize = FS size.  largest agsize is 1 TiB, thus agcount on reallly large fs would be in hundreds.
    # xfs tuning per XFS fag is said to be largely not necessary, as default works well most of the time.
    # agcount = thousands means write may have to consider all such AG to find optimal place to write.
    # when not catched, this result in lots of disk head seek.  
    # see
    # ideally, agcount ~ number of disk strips (cpu core should not matter, as the slow component is disk)
    # thus, agcount of 4 to 16 are common number.
    # small agsize also means large files will be divided into many ag... thus creating fragmentation
    # (and lots of search for ag to use)
    # xfs was orig written by SGI for supercomputer, which operate on large files
    # so lots of small file access may not be xfs' strength.
    xfs_info				# see fs info, like agcount, agsize, sunit, swidth
    xfs_repair			    	# fs must be unmounted, or resulting fs maybe inconsistent (per RHEL7.4 man page)
    xfs_repair -n /dev/datavg/datavol	# dry run method, but (skip?) extended attributes
    xfs_check				# deprecated and not avail in rhel7
    both of these commands use xfs_db to do its work.
    xfsmisc/ -f			# stats, not avail for rhel7
    Use the -p option to xfs_bmap to view unwritten extents.
    # xfs_io -f -c 'resvsp 0 10m' -c 'bmap -vp' /tmp/foo
    these commands can be used to make copy of the xfs metadta (which is also journaled) and then can then use xfs_repair -L on such backup image to see effect and decide whether to proceed with the repair.
    serverfault Ref:
    XFS vs EXT3


    ZFS is a combination of LVM and file system into single fused storage function.
    At the same time, it also bring many enterprise storage system features to the open source world. Originally developed by Sun for Solaris, Oracle can't backtrack it. CDDL stopping many distribution from including it in their .ISO. Cannonical still ship it with Ubuntu and provide support for it. Has yet to be sued by Oracle :) Centos can get a zfs.repo and get it. May still need some kernel modif (DMKS? no need to manually compile kernel?).
    ZFS basic cmd
    zfs list		# 
    zfs mount		# display currently mounted zfs fs.  may as well use zfs list
    zfs share -a		# share all zfs, done automatically when zfs "boot"
    zpool list
    zpool status		# list all disks that make up a pool and their status.  how "raid group" is defined.
    ZFS pool creation
    Many example from
    zpool create POOLNAME drive		# single drive, so raid0 :)
    zpool create storage  /dev/sdc
    zpool create POOLNAME drives		# multiple drives, raid0 (no redundancy)
    zpool create storage  da0 da1 da2
    zpool create POOLNAME raidz drives... 	# raidz (raid5) with 3 drives (loose 1 drive for redundancy)
    zpool create storage  raidz da0 da1 da2
    zpool create POOLNAME raidz2 drives...	# raidz2  (raid6, loose 2 drives for redundancy)
    zpool create storage  raidz2 da0 da1 da2 da3 da4 da5 da6 da7 da8 da9 da10 da11 da12 da13 da14 da15 da16 da17 da18 da19 da20 da21 da22 da23
    zpool create POOLNAME raidz3 drives...	# raidz3  (raid7, loose 3 drives for redundancy, min 5 drives)
    zpool create storage  raidz3 da0 da1 da2 da3 da4
    # 3 mirror-pair, concatenated together.  use 6 drives, loose space of 3
    zpool create POOLNAME VVVVVV   ...    VVVVVV  ...     VVVVVV ...
    zpool create storage  mirror da0 da1  mirror da2 da3  mirror da4 da5
            NAME        STATE     READ WRITE CKSUM
            storage     ONLINE       0     0     0
              mirror-0  ONLINE       0     0     0
                da0     ONLINE       0     0     0
                da1     ONLINE       0     0     0
              mirror-1  ONLINE       0     0     0
                da2     ONLINE       0     0     0
                da3     ONLINE       0     0     0
              mirror-2  ONLINE       0     0     0
                da4     ONLINE       0     0     0
                da5     ONLINE       0     0     0
    # 2 raid6 stripes concatenated together.  each strip loose 2 drives to redundancy, total 4 drives overhead
    # 4TB drives, 12 of them.  each stripe has 16TB usable, totalling ~30 TB.
    zpool create POOLNAME VVVVV ...		             VVVVV ... 
    zpool create storage  raidz2 da0 da1 da2 da3 da4 da5 raidz2 da6 da7 da8 da9 da10 da11
            NAME        STATE     READ WRITE CKSUM
            storage     ONLINE       0     0     0
              raidz2-0  ONLINE       0     0     0
                da0     ONLINE       0     0     0
                da1     ONLINE       0     0     0
                da2     ONLINE       0     0     0
                da3     ONLINE       0     0     0
                da4     ONLINE       0     0     0
                da5     ONLINE       0     0     0
              raidz2-1  ONLINE       0     0     0
                da6     ONLINE       0     0     0
                da7     ONLINE       0     0     0
                da8     ONLINE       0     0     0
                da9     ONLINE       0     0     0
                da10    ONLINE       0     0     0
                da11    ONLINE       0     0     0
    # once a pool is created, a FS is automatically created with the same name??
    zpool add -f geek1 raidz /dev/sde /dev/sdf 	# add new drives to existing zpool caked "geek1"
    zpool get mountpoint
    zpool set mountoint ...
    zpool offline POOLANEM drive			# offline a drive
    zpool replace POOLNAME ...			# replace drive (when there are unused disks from a different slot)
    zfs get quota
    zfs set quota=...
    zfs snapshot ...				# take snapshot.  avail in .zfs/snapshot of top of FS
    zfs sharenfs ...				# build-in nfs export if desired to use it
    ZFS Alignment, Performance
    Automatic alignment may not work, cuz HD lies about their geometry. Linux may need -o ashift. see ZFS Perf Tuning
    Top-level vdevs contain an internal property called ashift, which stands for alignment shift. It is set at vdev creation and it is immutable. It can be read using the zdb command.
    zfs iostat -v
    zpool status
    ZFS fs, dataset
    zfs create POOLNAME/volname1		# create a file system called "volume1" under the zpool POOLNAME  ??
    zfs create 
    There is a concept of dataset (like qtree in netapp) below the file system, and itseems like it can be nested. see Unix Area for example.
    ZFS other
    /usr/sbin/smcwebserver start # start a admin web portal, at “https://system-name:6789/zfs”
    ZFS Ref

    Package/Software management

    rpm is a basic command to manage software packages, available in just about every linux distro. up2date is a Red Hat Network moneyware to automatically download updates and install them. YUM is the freeware version, used by Fedora (from Yellow Dog distro) and actually better than up2date. But for those paying money to RH for support, keep to their tool maybe advisible.
    rpm -ivh [path/file.rpm]     # install rpm, vh = verbose hash for progress monitor
        --nodeps                 # install pkg even if fail dependencies check
        --noorder                # 
    for File in `ls -1`; do rpm -ivh --nodeps  $File; done
                                 # install all rpm in a dir eg cdrom
                                 # it seems that already installed packages will be overwritten by default
                                 # there is no "force" options needed to overwrite package to redo bad install.
    rpm -qa                      # query for all installed rpm packages in the machine
    rpm -qa arch=i386	     # same as above, specifying 32-bit binaries
    rpm -qa arch=x86_64	     # same as above, specifying 64-bit binaries for AMD 64 / Intel EM64T
    rpm -qa | grep glibc         # query for glibc version
    rpm -qpl [rpmfile]           # list where are the files installed to.
    rpm -ql  [pck_name]          # query installed package to see where files were send to
                                 #  (partial package name will work). 
    rpm -qi                      # query info of a specified package.
    rpm -qf /path/file           # query see which package installed the given file 
    rpm -e [pck_name]            # erase pcka_name as listed in rpm -qa from computer.  stop if it break deps
    rpm -e [pck_name] --nodeps   # forceful erase, even if it breaks dependencies.
    -Uvh [pkgname]               # Upgrade (erasing old packages, which install don't do)
                                 #  h= print hash sign as it goes, 
                                 #  v = nicely formated output
    rpm -Fvh *.rpm               # Freshen/update all rpm if an older version is installed
                                 # eg.  Refresh from AS 4.0 Update 1 to Update 2 via -F of all the rpm from cd.
    -V [pkgname]                 # Verify integrity of installed package.
    rpm -qa --qf '%{NAME} \t\t %{VERSION} \t %{RELEASE} \t %{ARCH}\n' 	
    	# display whether a given package is 32-bit i386 or x86_64.
    echo "%_query_all_fmt     %%{name}-%%{version}-%%{release}.%%{arch}" >> ~/.rpmmacros 
    	# to set the rpm query arch stuff permanently in the rpm query macro
    	# fedora does something like this by default.

    RedHat's up2date

    up2date				# GUI/CLI program to download patches, need RH registration.
    up2date ypbind			# ypdate ypbind client package to latest version
    #up2date nis			# update the nis package to latest version (??)
    up2date libstdc++-3.2.3-42	# update a specific rpm
    up2date -l			# list available downloads
    up2date -d			# download patches (as per config)
    up2date -i 			# install downloaded patches
    up2date --config		# run configuration (GUI or TUI), register machine, set exclusion, etc

    YUM (yellow dog, fedora, centos, redhat)

    yum			# yellow dog update modified, better rpm updated than up2date
    			# run cmd by itself will do all update automatically.
    			# not installed by default in RHEL 4, only in FC.
    yum update		# can be added to cronjob for auto update
    yum list		# list avail pkg and version number
    yum list  autofs	# list installed and avail pkg/ver for the pkg autofs
    yum info [pkgname]	# only display info of avail packages, opt followed by pkg name.
    yum install yumex	# donwload and install the Yum Extender (GUI) rpm
    yum search wireshark	# search for package... give more info than list | grep...
    yum --enablerepo=epel --enablerepo=dag ...	# tmp enable named repos for this run (they have to be in /etc/yum.repos.d but disabled)
    repoquery --whatprovides '*bin/yes'   # coreutils
    yum       --whatprovides '*bin/yes'   # coreutils
    yum --setopt=tsflags=noscripts -y install boinc-client  # pass --noscripts to rpm when installing the boinc-client*rpm
    # it is like setting yum.conf tsflags option that make rpm install package without triggering execution of install script"
    # yum package bundles/collections (meta package?)
    yum grouplist
    yum groupinfo "Development Tools" 		# cmake, cvs, and many other stuff is in the optional part :(
    yum --setopt=group_package_types=optional groupinstall "@Development tools"
    Creating YUM Repository
    For RHEL 5.2 and older, 
    dvd dir has some media id in it.
    can't have yum and kickstart use same dir list :(
    It creates a series of xml files under the dir repodata/  (where all the RPMs are)
    Therefore, one need to copy all RPM from CD/DVD (RPMS or Server dir) to NFS or HTTP server location.
    cd /path/to/rpm-repository ; createrepo .
    createrepo is an rpm in RHEL5 not installed by default.
    (make sure .discinfo and .treeinfo are copied from root of DVD before running createrepo)
    For RHEL 5.3, such restrictions doesn't apply anymore, and 
    yum repo can point to the same dir tree that kickstart use,
    both can be a loopback mount to a DVD iso.
    Client side config: 
    add file to /etc/yum.repos.d/
    eg call it rhel5.repo
    name=RHEL 5.1 Server

    apt (debian, centos, redhat) recommends use of apt and rpmsource for getting packages automatically,
    including dependencies.  However, it doesn't seems to like yum, and recommends command for its 
    removal (apt-get --fix-broken install).
    Sample package that is good to use are Apache mod_perl modules RH doesn't package but DAG does.  
    eg perl-Apache-DBI (Apache::DBI mod_perl),  perl-Compress-Zlib 
    install apt...rpm and rpmforge...rpm
    apt-get update				# update repository db
    apt-get -d install perl-Compress-Zlib	# install a specific package
    					# -d = download only, stored in /var/cache/apt/archives
    apt-get -y --quiet install DEBname   # answer yes.  DEBIAN_FRONTEND=noninteractive dont always stop the questioning :/
    aptitude search foo			# kinda equiv to yum list | grep foo
    aptitude install foo			# equiv to yum install foo
    aptitude show genome-music		# brief info about pkg
    					# info stored in /var/lib/dpkg/available
    cat/var/lib/dpkg/info/genome-music.list # list files installed by a given package
    dpkg -l					# list installed packages (.dpkg) = rpm -qa
    dpkg -i file.deb		# rpm -ivh
    apt-get install ./file.deb	# probably better, solve dependencies
    dpkg  --search file			# find which package has file
    rpm    -qif    file
    note that sometime the dependencies of a package is to be satisfied by OS distro rpm, 
    in such case, apt-get would fail saying broken package.  Just install the necessary rpm (eg perl-Digest-HMAC) 
    from CD and re-run apt-get.

    SuSE Enterprise (SLUS/SLED)

    rug se [PKG-NAME]	# find rpm package from pre-configured repository.

    Open SuSE

    Open SuSE is the name for the former commercial SuSE that Novell ditched after it publised Enterprise SUSE.
    yast			# TUI far-style

    System Services

    chkconfig                       # Easiest way to manipulate rc services, 
                                    # bettter than ntsysv or mangling xinetd file manually!
    				# for up to rhel6
    chkconfig --list xinetd         # see if xinetd service is enabled
    chkconfig --list rsh            # see if specific xinetd service is running, eg rsh
    chkconfig --list telnet         # see if specific xinetd service is running, eg telnet
    chkconfig --list autofs         # see if specific xinetd service is running, eg autofs automount
    chkconfig --list amd            # see if specific xinetd service is running, eg amd automount
    chkconfig --add ypbind		# register ypbind into service db
    chkconfig --level 345 ypbind on	# enable ypbind on run level 3,4,5
    chkconfig --level 345 rsh    on # enable rsh on run level 3,4,5, starting the service now also.
    chkconfig --level 35  httpd  on # enable httpd in init 3 and 5
    chkconfig vsftpd on             # enable  ftp  as xinetd controlled service (level 2,3,4,5)
    chkconfig talk   off            # disable talk as xinetd controlled service (all level)
    service iptables off            	# temporary stop firewall service (till reboot)
    service iptables status         	# check running status 
    chkconfig --level 345 iptables stop 	# disable firewall service from starting on run level 3,4,5
    					# RHEL 2.1 use ipchains.
    for SVC in `ls -1 /etc/xinetd.d`; do chkconfig --list $SVC; done
                                    # see which xinetd service is on or off.
    for SVC in `ls -1 /etc/init.d`; do chkconfig --list $SVC; done
                                    # see which init service run when
    				# but some file are not actually service
    serviceconf			# GUI, see what service run at what run level.
    redhat-config-services 		# same as above
    ntsysv 				# TUI of serviceconf (vsftp/telnet/rlogin, etc)
    				# Need to have ntsysv*rpm
    RHEL7, some newer ubuntu?
    systemctl status firewalld


    sudo service SuSEfirewall2_setup status		# firewall service  
    sudo /sbin/SuSEfirewall2 status			# alt script to check status  
    chkconfig --level 345 SuSEfirewall2_setup  off	# turn off firewall


    see also hw cmd
    numactl -H
    numactl ... cmd  # run command with specific memory config info (eg, restrict malloc to Phi MCDRAM?) 
    lstopo-no-graphics aka hwloc-ls
    turbostat 	# cpu current speed in Bzy_MHz column
    cpupower frequency-info
    cpupower frequency-set -d 1000000
    cat /proc/cpuinfo | grep MHz		# current core speed in ubuntu, max core speed for rhel7.2 (7.4 may work better)
    sudo cat /sys/devices/system/cpu/cpu*/cpufreq/cpuinfo_cur_freq  # ubuntu, rhel 7.4+ 
    sudo cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_cur_freq	# ubuntu, fedora?
    sudo cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor  # rhel, but not present if no governor avail
    RHEL7 may need CPUfreq driver to be installed
    or cpupower from kernel-tools
    can have this run via singularity
    possible kargs for governors:
    intel_pstate=disable # this maybe bad, no lower MHz in idle, no turbo?   was tried in specific cascadelake checking on IB osu_latency
    intel_pstate=force # don't use acpi-cpufreq even if preferred on a given system, careful: thermal implication.  ok on skylake?
    no_hwp, hwp_only, support_acpi_ppc, per_cpu_perf_limits
    pstate avail for most Intel SandyBridge and later CPU.
    pstate with hwp (hardware managed pstate) is recommended and default policy.  cpu decide on pstate, os can give it hints.  ?hwp avail to skylake and later?
    sudo dmidecode -t processor | grep Speed	# but "Current Speed" is from ACPI and maybe stale
    						# Max Speed 
    						# don't seems good on ubuntu laptop
    						# show max speed for SL7 (ie match /proc/cpuinfo MHz info) 
    /etc/security/limits.conf ::
    *             -   memlock        unlimited
    *          soft   memlock        unlimited
    *          hard   memlock        unlimited
    *      hard memlock unlimited
    oracle hard nproc 16384
    oracle soft stack 10240
    oracle soft core unlimited
    oracle soft nproc 16384
    oracle soft nofile 131072
    oracle hard nofile 131072
    *      hard core 0
    oracle hard core unlimited
    *      soft memlock unlimited
    oracle hard stack 10240
    /etc/sysctl.conf ::
    net.ipv4.ip_forward = 0
    net.ipv4.conf.default.rp_filter = 1
    net.ipv4.conf.default.accept_source_route = 0
    kernel.sysrq = 0
    kernel.core_uses_pid = 1
    net.ipv4.tcp_syncookies = 1
    kernel.msgmnb = 65536
    kernel.msgmax = 65536
    kernel.shmmax = 68719476736
    kernel.shmall = 4294967296
    kernel.sem = 250 32000 100 128
    net.core.rmem_default = 4194304
    net.core.rmem_max = 16777216
    net.core.wmem_default = 262144
    net.core.wmem_max = 16777216
    net.ipv4.tcp_wmem = 4096 262144 16777216
    net.ipv4.tcp_rmem = 4096 4194304 16777216
    fs.file-max = 6815744
    net.ipv4.ip_local_port_range = 9000 65500
    fs.aio-max-nr = 1048576
    vm.swappiness = 0
    cat /proc/cmdline	# list of kargs passed to kernel at boot
    			# also reported in the first few lines of dmesg
    kargs are to be space separated.
    the comma used in some args are specific to that one arg, eg "console=ttyS1,115200n8"  are options for console, not kargs proper.
    "iommu=pt net.ifnames=0 biosdevname=0 acpi_irq_nobalance console=tty0 console=ttyS1,115200n8"
    iommu=pt is used by AMD for nic 
    net.ifnames=0 biosdevname=0 	# tell rhel 7 to use eth1, eth2 etc nic name rather than enp196s0f1 


    $ sensors
    acpitz-virtual-0						[1]
    Adapter: Virtual device						[2]
    temp1:        +42.0°C  (crit = +104.0°C)			[3]
    thinkpad-isa-0000						[4]
    Adapter: ISA adapter						[5]
    fan1:        3773 RPM						[6]
    coretemp-isa-0000						[7]
    Adapter: ISA adapter						[8]
    Physical id 0:  +42.0°C  (high = +87.0°C, crit = +105.0°C)	[ 9l  9i  9m  9c]
    Core 0:         +42.0°C  (high = +87.0°C, crit = +105.0°C)	[10l 10i 10m 10c]
    Core 1:         +33.0°C  (high = +87.0°C, crit = +105.0°C)	[11l 11i 11m 11c]
    # ubuntu cpu temperature (readable by user)
     9l 	/sys/bus/platform/devices/coretemp.0/hwmon/hwmon2/temp1_label
     9i	/sys/bus/platform/devices/coretemp.0/hwmon/hwmon2/temp1_input   #  42000
     9m	/sys/bus/platform/devices/coretemp.0/hwmon/hwmon2/temp1_max	#  87000
     9c	/sys/bus/platform/devices/coretemp.0/hwmon/hwmon2/temp1_crit	# 105000
       	/sys/bus/platform/devices/coretemp.0/hwmon/hwmon2/temp1_crit_alarm #   0 (times reached alarm level?)
    10l 	/sys/bus/platform/devices/coretemp.0/hwmon/hwmon2/temp2_label	# Core 0
    10i	/sys/bus/platform/devices/coretemp.0/hwmon/hwmon2/temp2_input	#  42000
    11i	/sys/bus/platform/devices/coretemp.0/hwmon/hwmon2/temp3_input	#  31000
    # sl7 cpu temperature 
    grep [0-9] /sys/bus/platform/devices/coretemp.*/temp*_input



    To disable nouveau driver from loading (so that commercial nvidia driver can be installed) (newer linux use this driver before even X, thus blacklist in modprobe don't cut it) ref:
    append to boot argument to kernel: rd.driver.blacklist=nouveau 	# don't remember if this really worked, maybe not
    append to boot argument to kernel: nouveau.modeset=0 		# this worked on exalearn 2018.11
    edit /etc/default/grub :: 
    GRUB_CMDLINE_LINUX="[ . . . ] rhgb quiet rd.driver.blacklist=nouveau"  		# **ml4hep1 2020.1215**
    append to boot argument to kernel: nouveau.modeset=0 				# this worked on exalearn 2018.11
    (remove rhgb quiet for verbose text boot)
    grub2-mkconfig -o /boot/grub2/grub.cfg      # BIOS mode, UEFI need a diff config
    check with lsmod | grep nouveau
    cat /proc/cmdline
    sudo update-grub
    sudo update-grub2
    Ubuntu/Mint seems to use quiet splash for GUI boot. Mint added nomdmonddf nomdmonism
    nouveau.blacklist=1 in grub.conf didn't work in Mint.
    Driver Manager was the right tool to change from nouveau to nvidia driver in Mint.

    Kernel Compilation

    compiling linux kernel  
    (similar procedure for old 2.4, new 2.4, and 2.6)
    download kernel source (srpm, rpm -ivh ...)
    Get the version applicaple to the base OS (eg RH 8.0) and kernel similar to the one installed 
    (latest maybe okay, but then other support may not be available).
    For ES2.1/RH7.2, 
    building custom kernel just req getting RPMS packages for:
    kernel-headers-2.4.9-e.12	(disk1)
    kernel-source-2.4.9-e.12 	(disk2)
    SRPMS are not needed!  Though there are dependencies such as gcc, glibc-dev, 
    kernel dev, etc, which are RPMS, not Source RPMS!
    Files from SRPMS/ are not really needed unless really do heavy dev, 
    and they do not show up in rpm -qa!
    Note that RH does not provide support for customized kernel.
    cd /usr/src/linux-2.4* [ use /usr/src/kernels/2.6.11-* for FC4 ]

    make menuconfig
    this lauches a text menu program to createa config file.
    * indicates compiled into kernel
    M indicates loadable modules.

    make xconfig
    # GUI, an alnternate to the TUI menuconfig.
    # RHEL 2.1 (RH 7.2) GUI is a big grid
    # 2.6 kernel is a tree based GUI with lot of explanations; auto compile when done.

    make dep; make bzImage; make modules
    make modules_install
    #make install		# may not be avail.
    cp vmlinux... ...
    For old machines runing LiLo (RHEL 2.1 and older):
    cp /etc/lilo.conf.anaconda /etc/lilo.conf
    vi /etc/lilo.conf
    /sbin/lilo -v -C  /etc/lilo.conf
    to install a new boot loader w/ new config into MBR.
    For newer machines running Grub (RHEL 3.0/RH 8.0/Fedora):
    /etc/grub.conf (link to /boot/grub/grub.conf).
    edit of such file doesn't require lilo install, but then need to reactivate grub...
    From README
     - make config : standard kernel compiling config.
     - Alternate configuration commands are:
            "make menuconfig"  Text based color menus, radiolists & dialogs.
            "make xconfig"     X windows based configuration tool.
            "make oldconfig"   Default all questions based on the contents of
                               your existing ./.config file.
    Then do:
    make bzImage	# create compressed kernel image
    		# files are placed in .../linux/arch/i386/boot/bzImage
    make bzdisk	# optional, make boot floppy disk.
    make modules 	# needed if anything is in loadable module
    make modules_install	# suggested to do backup, but not sure where

    Upgrading Kernel

    Upgrading RH kernel, just need to get the later RPMS, package format kernel-VER.rpm, and install that.  
    No other packages are needed, and no recompiling needed.   It insall a stock RH tested general kernel.
    rpm -Uvh new-kernel-ref.rpm
    Install stuff like /boot/vmlinux-... vmlinuz-... initrd-... 
    plus lot of files with version number on filename into /lib, etc
    The rpm has to be from RH.  Changes to /boot/grub/grub.conf may be needed.
    Easiest is just to use up2date and let it update kernel also, 
    then just reboot, it will do all the necesary update and changes.
    OS upgrade typically update grub.conf so that old kernel can be booted.
    Especially useful when certain drivers are tied to the kernel and don't work after kernel upgrade.
    eg. HBA driver from QLogic, PowerPath.
    DKMS would be really helpful as system is patched to new kernel.

    Kernel Tuning

    For general performance tuning, see the tool page
    ipcs -l 	# display interprocess communication parameters
    sysctl 		main tool to viewing kernel parameters
    	-a	display parameter and its setting.
    	-w	write new value for a given param
    	-n 	supress printing of key, just print value.
    	-p 	re-read /etc/sysctl.conf and make changes effective immediately
    $ sysctl kernel.threads-max
    kernel.threads-max = 20479
    $ sysctl -n kernel.shmmax
    $ sysctl -a
    kernel.tainted = 0
    kernel.core_name_format = core
    kernel.core_uses_pid = 0
    kernel.child-runs-first = 1
    kernel.panic = 0
    kernel.domainname =
    kernel.hostname = lancer
    kernel.version = #1 SMP Tue Feb 11 02:24:10 EST 2003
    kernel.osrelease = 2.4.9-e.12smp
    kernel.ostype = Linux
    fs.aio-max-pinned = 163836
    fs.aio-max-size = 131072
    Alternatively, the kernel params are listed in /proc/sys
    and they can be viewed or changed like all the other /proc parameters.
    $ cat /proc/sys/kernel/sem 	# semaphore :: semmsl semmns semopm semmni
    250     32000   32     128
    $ cat /proc/sys/kernel/shmmax
    $ cat /proc/sys/kernel/threads-max
    Changing kernel parameters:
    Edit /etc/sysctl.conf for permanent changes that persist thru reboot.  
    Run sysctl -p to read this file and make changes effective immediately
    eg, for ora10g R2 install, add the following to /etc/sysctl.conf
    to modify equiv params in /proc/sys/kernel/shmmax and sem:
    kernel.sem=250 32000 100 128
    Or, for one time change:
    echo VALUE > /proc/sys/kernel/file
    echo "250     32000 100 128" > /proc/sys/kernel/sem
    smartctl	# tool to read hd SMART data.  gnome-disks is a GUI front end of this.
    		# ubuntu# apt-get install smartmontools

    Kernel Tuning Links describe the major /proc entries in Linux, what they do and stuff.

    Oracle 10g install on linux params are described in: A thread where it seems linux kernel only support 1024 threads. Changes would be in the config file and recompile kernel. Info on 2.4 Older info, 2.2 kernel. Info about source file to change to alter kernel param.

    Kernel Modules

    modprobe -c		# list all modules
    insmod			# simple, no dependencies check, not too user friendly
    rpm -q --scripts kernel		# find out what kernel script runs and what modules are loaded.
    mkinitrd -v -f initrd-2.6.18-53.1.13.el5 2.6.18-53.1.13.el5	# create new initrd image


    Dynamic Kernel Module System
    Source code living outside kernel source, get rebuild automatically when new kernel is installed. Used by eg Dell PERC RAID controller, which add the mpt2sas driver (but not always newer than what comes with the OS distribution).
    rpm -ivh dkms-...rpm 		# install the dkms package/feature
    rpm -ivh driver...dkms.rpm	# these would actually be source codes 
    dkms status			# see if dkms is installed/configured (/etc/dkms)
    modinfo mpt2sas			# could be from native os or dkms

    SELinux - Security Enhanced Linux

    useradd -Z user_u 	# when adding new user, also give it a default context. 
    sudo semanage login -l 	# list defined context for all users 
    			# (but not for user not added with useradd?)
    sudo semanage login -a -s user_u tin	
    			# hopefully will rid "Unable to get valid context for [USER]" in Mint when Gnome starts.
    semanage port -l | grep 514	# see if syslogd can use that port (@@ vs @ for tcp vs udp in rsyslog.conf)
    ps -eZ | grep syslog		# see process selinux context
    id -Z	# context of current user
    	# but not getting the info from $( semanage login -l ) ??!!
    	# user need to be in sysadm_t to be able to run sudo, su
    by default, unconfined_u user run in the unconfined_t domain
    seinfo			# summary info of context
    seinfo -t		# list all context types
    semanage fcontext -l 	# list all file path and context applied to them
    ls -Z file		# show context of a specific file
    cp --preserve=context foo bar	# copy file, preserving its context
    sesearch ...
    chcon --type var_t /var/www/html/index.html
    findcon ...
    restorecon ...
    matchpathcon  path
    sesearch ...
    sudo semanage fcontext -a -t git_user_content_t  "/export/tin/context_managed_dir(/.*)?"
    (?) add file entry to context db , but context is not applied to file, not visible in ls -Z 
    /etc/selinux/targeted/contexts/files/file_contexts.local	local context db
    getsebool -a
    getsebool    use_nfs_home_dirs=1	# get specific setting
    setsebool -P use_nfs_home_dirs=1	# allow sshd to read authorizedkeys in NFS mounted home dir 
    Ansible module:
    - seboolean:
        name: use_nfs_home_dirs
        state: yes
        persistent: yes
    Controlling file contexts yourself

    Allow binary to open tcp port
    eg: change ssh to use port 2222 in rhel7 denials noted in /varlog/audit/audit.log semanage port -a -t ssh_portt -p tcp 2222
    SELinux for RHEL 7
    sestatus # check status
    Can only be disabled by editing config file and reboot.
    ping in rhel7 does not rely on setuid big, but on selinux context to gain elevated priv to do the ping.
    Seems like RHEL7 has SELinux enable by default, and it is very usable, don't tend to run into problem/restrictions with most generic sys admin config as desktop machine.

    SELinux for Ubuntu 14
    Ubuntu by and large don't talk about SELinux and is disabled by default. Older distro don't even have it installed by default. It is installed on Ubuntu 14.04/Mint 17.2.
    Documentation is refered to the Debian page at:
    apt-get install selinux-basics selinux-policy-default auditd 
    apt-get install setools	# may have lots of dependencies in older debians
    download script from
    cp -p ~/Downlaods/_load_selinux_policy   /usr/share/initramfs-tools/scripts/init-bottom/
    update-initramfs -u
    ls -la /.autorelabel
    Old info for SELinux from RHEL 4 days
    Starting from Red Hat Enterprise Linux 4.0 ships with an implementation of Security Enhanced Linux. They use "targeted policy" which only limits functionality to a pre-defined set of deamon processes. The defined set will then have to obey the Mandatory Access Control (MAC) and become more secure even when exploited. The rest of the process are monitored but are freely allowed to change state, and thus they will operate more like the traditional Discretionary Access Control (DAC).

    Overall, enabling SELinux in RHEL 4 is largely safe and don't cause too much headache. A few known places are known to break, such as chroot, dhcp. The old RHEL4 settings is largely what is the "discretionary", ie monitoring/logging SELinux mode.

    Some places it may cause lots of logging. Specific daemons (eg SNMPD) can be excempt from the SELinux targeted policy via:
    A1.  Modify the /etc/selinux/targeted/booleans file.  
    A2.  Change snmpd_disable_trans=0 to snmpd_disable_trans=1 
    A3.  reboot your server
    B1.  echo "1 1" > /selinux/booleans/snmpd_disable_trans
    B2.  echo "1"   > /selinux/commit_pending_bools
    B3.  Check that the value in /etc/selinux/targeted/booleans is changed to
    Starting with RHEL6 (or 5?), there is an enforcing mode. This will add an extra layer of restrictions. eg, httpd that got hacked, but if running in SELinux environment, the kernel will still block access to things beyond its context (eg access to user's home dir).

    sudo chcon -v  --type=httpd_sys_content_t /var/www/html/index.html	# change SELinux context for a file, 
    									# this eg allow apache httpd to be allowed access to the file 
    									# (SELinux restrictions will show up in error_log).
    sudo chcon -vR --type=httpd_sys_content_t /var/www/html			# change SELinux context for a dir, Recursively.
    sestatus					# check status and mode of SELinux, such as enforcing or monitoring
    ls -Z /var/www/html/index.html			# -Z works for many commands, show SELinux context
    ps -efZ


    Ubuntu can use x-swap PPA driver for nvidia, much easier than downloading drivers directly from nVidia and trying to install it for ubuntu. ref:
    Zorin used ppa to install nvidia-390 for Quadro 4000 card.
    Centos 7
    blacklist nouveau kernel driver...
    systemctl  enable --force	# not necessary?
    systemctl  set-default    # no GUI, good for server, esp those with GPU cards!
    systemctl  start
    systemctl  set-default
    systemctl  get-default 	# see what's the default target
    ligthadm stop
    cp /etc/X11/xorg.conf /etc/X11/xorg.conf.bak
    system-configure-display	# rhel6
    Installing nvidia if nouveau had been installed could be rather painful. see devtalk.nvidia on steps to rid old nvidia.ko

    ANSI escapes

    Ansi is the most common escape sequence to draw colors on screen.
    more colorfile.txt will parse them correctly.
    less colorfile.txt does not (yet) display file with color.  to rid them:
    cat colorfile.txt | strip-ansi | less
    npm install --global strip-ansi-cli
    sed can likely remove them, but not as easy as above.
    vt100 dont support color, but still support ncurses/readline to draw character on screen  (?)
    docker build, 
    Rscript install
    apt-get install
    are tools that tend to draw on screen and make for very annoying logs :/

    Vendor Specific Tools/Packages

    HP (Intel Hardware)

    HP Smart array rpm eg: hpacucli-7.60-18
    part of HP/Compaq ProLiant Support Pack (PSP)
    More info:
    => ctrl all show 			# list all controller
    => ctrl all show status			# show controller info
    => ctrl slot=0 array all show 		# list all array in a given controller
    => ctrl slot=0 array all show status	# array status, an array is a mirror set.
    => ctrl slot=0 array A   show		# give info and config of a specific array
    To find RAID config, simply:
    cat /proc/driver/cciss/cciss0
    HP System Health Application and Isight Management Agent - hpasm
    For Proliant servers eg DL 585
    rpm -ivh hpasm-7.6.0-111.rhel3.i386.rpm
    hpasm activate			# activate and configure SNMP strings, etc
    /etc/init.d/hpasm  reconfigure 	# change settings
    it create a bunch of cma* process
    For command usage, see


    Collect log for support using
    techsupreport   collect
    jobqueue view
    techsupreport   export -f
    jobqueue delete --all

    VMware ESX


    upstart replaces /etc/inittab in rhel6, ubuntu 14, allow specifying user, etc.
    but abandoned in rhel7, ubuntu 15 in favor of systemd

    For pure upstart process (and not just link to init/rc script), To enable/disable a service, one have to vi the upstart config file, eg /etc/init/smbd.conf and comment out the "start on..." line
    service smbd start
    service smbd stop
    initctl connects/controls init daemon using D-Bus (Ubuntu 14)
    initctl --system list			# similar to chkconfig --list, but with status
    initctl --system list | grep ufw	# (ubuntu) uncomplicated firewall
    initctl list-env

    Ref: upstart doc from Ubuntu

    /etc/init/couchdb.conf ::
    # couchdb - a RESTful document oriented database
    description "Start the system-wide CouchDB instance"
    author "tin ho"
    # upstart will start couchdb automatically on boot based on this config file
    # for manual control, run these as root:
    # initctl start couchdb
    # initctl stop couchdb
    # initctl status couchdb
    #start on filesystem and static-network-up
    #stop on deconfiguring-networking
    #start on runlevel [345]
    start on stopped rc RUNLEVEL=[345]
    stop  on runlevel [!345]
    env COUCHDB=/usr/local/bin/couchdb
    env COUCHDB_USER=couchdb
    env COUCHDB_STDOUT_FILE=/var/run/couchdb/stdout.log
    env COUCHDB_STDERR_FILE=/var/run/couchdb/stderr.log
    pre-start script
        mkdir -p /var/run/couchdb
        chown couchdb:couchdb /var/run/couchdb
        logger -i -s -t upstart_couchy -p 'initctl couch reached pre-start script section'
    end script
            # remember, for su, username is at the end!
            #exec su -c "logger -i -s -t upstart_couchy -p 'this is a test msg from initctl couchy'" couchdb
            logger -i -s -t upstart_couchy -p 'initctl couch starting couchdb'
            exec su -c "$COUCHDB -a /usr/local/etc/couchdb/default.ini -a /usr/local/etc/couchdb/local.ini"  $COUCHDB_USER
    end script
    post-stop script
        logger -i -s -t upstart_couchy -p 'initctl couch reached post-stop script section'
    end script

    RHEL7 new stuff

    systemctl list-unit-files			# replaces chkconfig --list
    systemctl list-dependencies 			# show service dependency tree
    systemctl list-dependencies 	# show service dependency tree for a specific boot target
    systemctl show bruker_lmgr.service		# get details of the bruker lmgrd service
    						# init script fragments consolidated into /etc/systemd/system
    systemctl daemon-reload bruker_lmgr		# reload service after changes to init script fragment
    /etc/systemd/system				# dir where startup files are stored
    /etc/systemd/system/	# expect sym link to /usr/lib/systemd/system
    /etc/systemd/system/                    	# can drop a file here and allow systemctl enable SVC_NAME ## preferred
    /etc/systemd/system/	# can drop a file here (uniq name that does not conflict with other service in /usr/lib/systemd/system) and it will be run on boot
    systemd-analyze blame						# time line of service starting, bottom = first service, top = last service
    systemctl enable  vncserver0			# add sym link to the target dir to start service on "runlevel" change
    systemctl start   vncserver0
    systemctl status  vncserver0			# look at output/error message from the start up process
    systemctl disable vncserver0			# remove sym link, but seems like service still listed...
    # /etc/systemd/system/vncserver0.service  # ensure executable
    # systemctl daemon-reload
    # systemctl enable vncserver0			# add sym link to to start the service
    Description=Remote desktop service (VNC)
    #Type=forking   # for process that deamonize itself and return to the command prompt within ~120sec
    ExecStart=/usr/sbin/runuser -l sn -c "/usr/bin/x0vncserver -PasswordFile=/home/sn/.vnc/passwd -AlwaysShared=1 -display=:0"

    Network Manager

    CentOS7 install and use Network Manager if it is adds a Desktop or Developer sw group.
    Server incidentally would still use the traditional network stack.

    If not changing location and don't need DHCP, really better off disabling network manager!
    ypbind would NOT bind if NetworkManager is running (even when ifcfg-* has NM_CONTROLLED=no)!! NetworkManager is really evil in the server world! But if so much as include some developer or gui package in SL7, NM will be installed and enabled!!

    Mostly Disabling Network Manager in centos7
    vi /etc/NetworkManager/NetworkManager.conf 
    vi /etc/sysconfig/network-scripts/ifcfg-WhatEver
    nmcli device 			# should show the interface no longer managed by Network Manager
    #systemctl enable network ??	# not enabled on system that use Network Manager exclusively?
    systemctl start  network 
    systemctl enable network       # redirect to something else...
    								# machine with Network Manager does not have this enabled (centos 7 default!)
    Completelyt Disabling Network Manager in centos7
    systemctl disable NetworkManager.service;
    systemctl stop NetworkManager.service;
    systemctl enable network 
    ##systemctl start  network 
    /usr/sbin/service   network start 
    /usr/sbin/chkconfig network on			# still present in rhel7 !
    Network Devices
    nic has new naming convention (in rhel7 and ubuntu 15) 
    eno1, eno2, ...	on-board ethernet nic.  index provided by firmware.
    ens1, ens2, ... PCIe hotplug nic
    enp2s0       	eth nic on PCI bus address 02:0:0 , ie bus #2, slot #0, function 0.
    wlp0s3f1u4	wifi. p0s3 is USB controller's PCI address.  f1u4 is USB addressing.
    enx112233445566	mac-based naming (but would need to change it when NIC is replaced).
    #nm* are NetworkManager commands (which tends to be GUI in laptop)
    nmtui					# Text UI
    nmtui-connect				# select wifi, or activate LAN port
    nmcli h					# cmd help
    nmcli n					# whether nm is enabled
    nmcli nm ?				# list interfaces managed by network-manager
    nmcli d					# show devices
    nmcli c					# show connections
    nmcli connection up ifname wlp3s0	# actiate a specific connection	
    nm-connection-editor			# nm GUI app, not the dock panel thingy
    /etc/NetworkManager/system-connections  # location for Debian NM interfaces info (?)
    systemctl restart NetworkManager	# restart the network stack, rhel7
    systemctl restart network		# restart the network stack, even if running NM
    service network-manager restart		# ubuntu 14
    iproute2 (ip cmd)	# replaces net-tools (ifconfig, route, netstat, arp)
    			# for the subcommands of ip, can use a single or first two letters, so long as it is non ambiguous.
    ip [opt] [sub-cmd]
    ip addr			# show ip address, aka ip a, replaces ifconfig.  will show NO-CARRIER.  UP just indicate driver is loaded?
    ip ad  show wlp3s0	# show info for a specif ineterface only
    ip route		# see routing table, replaces netstat -r
    ip route ...		# replaces route...
    ip link			# cumulative stat (subset of ip addr output)
    ip -s link		# -s provides stats info for all links
    ip neigh		# mac to ip mapping, replaces arp -a
    ip -4 neigh
    ip -4 addr		# show ip address, all nics, IPv4 only
    ip -6 addr show ens1
    ifstat			# LAN stats, replaces netstat -i
    ss -a			# replaces netstat -a, same output format.
    ss -au 			# udp only, 
    ss -at4			# tcp only, IPv4 only
       -r 			# resolve ip to hostname 
       -l			# show listening service only
    ethtool eno1		# set duplex, etc of an interface
    DNS settings under evil watch of Network Manager
    /etc/NetworkManager/NetworkManager.conf		# NM config
    	dns=none				# tell it not to mangle /etc/resolv.conf
    NM_CONTROLLED=yes				# set to no and the ifcfg will be on traditional "networking" but not NetworkManager will not change it IP, do dhcp with it, etc.
  • Debian Admin Handbook (also as .epub in google book)

    authentication in rhel7

    authconfig is the tool to use to configure authentication services in RHEL7/CentOS7. Manually changing nsswtich.conf isn't enough. There are pam.d, selinux, firewall updates.
    See also ipa-client-install and realmd

    firewall in rhel7

    firewalld is default firewall, does NOT write to /etc/sysconfig/iptables
    iptables		# rhel 6 default, boot up executes /etc/sysconfig/iptables ...
    firewall-config		# GUI tool.  DONT use system-config-firewall
    			# shorewall, not default, but supported as os package.

    RHEL8 New STuff

    default, no /usr/bin/python , which may break things like ansible (unless otherwise configured to explicityly use python3)
    alternatives --set python /usr/bin/python3

    Linux Desktop Progies

    sublime		text editor (not IDE, so light) good for python programming .  avail in Linux, Win, Mac.
    mtv  		mpegtv ($10 shareware req reg) for viewing mpeg video.
    mvtp		cli of mtv, free.  
    		-z = 2x2 zoom.  
    		-G+10+10 = place window in specified +x+y geometry 
    mpeg_play mpeg video player.  in old a.out (ZMAGIC) binary though.
    		avail for solaris, etc.
    mpegplay	mpeg video player found in Knoopix distro.
    xine		free video player (vcd, dvd, avi, but no mpeg?).
    totem		alternate front end UI for xine.
    mpg123		cli for playing mp3 audio
    mpg321		drop in replacement for mpg123
    xmms		winamp like mp3 player (GUI for mpg123?)
    kate		KDE editor w/ many extras, like CLI window, mgnt tabs, somewhat like Visual Studio editor
    lynx		text based web browser
    elinks		a much better text based browser
    		set proxy server http"//address:port thru env var http_proxy
    BitchX		a text based irc client 
    irssi		a newer text based irc client, some cmd same as BitchX
    iEvince/GPdf 	GNOME pdf viewer
    Okular/KPDF	KDE pdf viewer
    evince		Document viewer, PDF, PS, XPS, TIFF
    xpdf		X Window pdf viewer
    PDFedit		GPL pdf internal structure editor
    CUPS-PDF	PDF generator extension to CUPS printing system, generate pdf file
    		in /var/spool/cups-pdf, but somehow doesn't work in x86_64 
    		http://localhost:631/	- cups web interface
    ssh-agent | xclip		# send output to xclip , require working X 
    xclip -o				# paste output from X clipboard , just echo
    xclip -o | bash			# pipe to bash to execute 
    clip 					# part of geomview  ?? 
    readreg works with screen clipboard
    pbcopy for OS X
    cygwin, windows has /dev/clipboard
    fsview			# A GUI to see directory/file space usage, kinda interesting.
    redhat-logiviewer	# GUI tool to parse many diff logs
    gqview			# acdsee like image viewer
    eog			# eyes of genome, graphics file viewer
    eom			# eyes of mate
    eeyes			# electric eyes
    gimp			# "photoshop"
    Oothers to ponder:
    sudo snap install irfanview # via wine
    gthumb			# thumbview, crop, resize (probably not resample)
    ethereal is replaced by wireshark
    but that only install text-based snooper called /usr/sbin/tshark
    older ethereal comes with a gtk+, gnome, kde package.
    But those are somehow no longer shipped (by up2date, rpmfind).
    xxd -r 			# reverse hex to ascii conversion, from vim-common rpm
    			# xxd /etc/passwd | xxd -r 
    od			# octal dump
    hexdump			# hex dump
    xterm, use rxvt color scheme, grey on black. MiscFixed font, 8pt.
    kterm ??
    iptraf 	- ip trafic monitor
    ss	- socket usage (related to netstat)
    pmap	- process memory usgae
    mpstat	- multiprocessor usage
    KSysGuard - kde system guard - real time reporting and graphing 
    	    similar one on Gnome
    gnome-system-monitor	# windows-style performance meter
    ntop 	- network top (need kernel tapping?)
    etherApe - bubble graphics of where traffic is going, but some bubbls become too big.
    vnstat	- network traffic monitor
    htop	- better top, 
    mtr	- ping and traceroute fused has good overview.
    diff tools
    vimdiff    file1.txt file2.txt          # character-based side-by-side diff, handle up to 4 files.
    					# always default to side by side diff.
                                            # ^w J = vertical split   (toggle)
                                            # ^w H = horizontal split 
    vimdiff -o A.txt B.txt C.txt will open three horizontal splits for each file.
    vimdiff -O A.txt B.txt C.txt will open three vertical splits for each file
    - fldiff # X, side by side, opt to ignore space, best diff so far ## avail in  wsl, zorin, not in centos7
    - meld # lots of deps on KDE or Qt?  but new version is very good.  fairly close to github web diff.  only side by side diff. 
    - icdiff # text color sdiff by char, very nice (but no ignore space) ## not in zorin, ubuntu?
    - dwdiff # doesnt seems to be in centos7
    - colordiff # -yw for side and ignore space, but not too good
    - kdiff3,  GUI, up to 3 files diff? can't handle large diff.  depends/install kde-runtime 
    - kdiff3-qt # lots of qt stuff needed
    - mgdiff # clone of xdiff, old school X, no ignore space option
    - xxdiff # directory diff and merge tool
    - wdiff  # gnu word diff
    - docdiff
    See also apple#serial
    minicom		# telix like program in Linux (a bit thicker than Solaris tip)
    minicom -o -m	# -o = no modem init
    		# -m = use meta key (instead of ctrl-a + key, can use ALT+key
     		#      or ESC, key.  eg ALT-O for options)
    		#      useful when running minicom inside "screen", 
    		#      which also use ^A for escape, thus req ^a, a, key combo.
    minicom -o -m  -D /dev/tty.usbserial -b 9600 --8bit 
    one anoying thing seems that minicom param change in ALT-O is not fully
    effective right away, especially on change of serial port.  Need to save it
    eg as default .dlf, then restart minicom for it to become effective!
    screen /dev/tty.usbserial 9600   # worked in mint 19
    	# ^a k 		: kill the connection (terminate screen and its use of /dev/tty...)
    	# ^a ^b    	: send break (for sun's go prompt)   (don't need ^B (upper case B))
    	# ^a b    	: send break (for sun's go prompt)   (after ^a, it is ^b OR just b, not a triplet of key seq to send the break)
    serial speed to try:
    Solaris, use 9600, 8n1, no flow control (hardware of software).
    Acopia seems to have required Flow Control DTS/DTR.
    NetApp now use 115200 ?
    dmesg | egrep --color -i serial\|tty # should determine serial port, eg ttyS0 ttyUSB0
    /dev/cu* are said to be obsolete circal kernel 2.x days.
    kernel 3.x and 4.x has build in driver for most of the usb-serial adapter?
    mint and wall-e has driver for the start-tech (blue) and belkin.  
    if usb dev removed, /dev/ttyUSB0 will be removed.

    iso2usb - making bootable usb thumbdrive from .iso

    CentOS 8, 7, 6.5 and newer
    Need dd, their iso format has a portion that windows cannot copy correctly.
    dd if=centos.iso of=/dev/sdz
    must write to the whole usb, not a partition in it!  ie, can't use /dev/sdz1
    so will need to dedicate a whole usb key to it.
    Resulting image looks like (partition is read only, and extra space on usb is wasted):
        Disk /dev/sdc: 15.1 GiB, 16170196480 bytes, 31582415 sectors
        /dev/sdc1  *        0 15335423 15335424  7.3G  0 Empty
        /dev/sdc2       23772    44151    20380   10M ef EFI (FAT-12/16/32)
        /dev/sdc1 on /media/tin/RHEL-8-1-0-BaseOS-x86_64 type iso9660 (ro,nosuid,nodev,relatime,nojoliet,check=s,map=n,blocksize=2048,uid=43413,gid=100,dmode=500,fmode=400,uhelper=udisks2)
    Fedora Media Writer for windows work, easy, but overwrite whole USB and create read-only FS (or is it ext2?).
      It likely use dd underneath.
    * UNetbootin ? works for ubuntu?  
    * livecd-iso-to-disk said can preserve files on usb 
    * etcher for ubuntu... work for fedora.iso ? ... 
        - appImage (multiplatform), GUI.
        - on Ubuntu 18.04, wiped the whole USB and created an iso9660 RO FS
          resulting usb has single partition, of type 17: Hidden HPFS/NTFS
          /dev/sdd1  *        0 3710975 3710976  1.8G 17 Hidden HPFS/NTFS
          may as well just use dd?  it does "label" the disk, eg: /media/tin/SL-75-x86_64-LiveDVDgnome
    * live-iso
    SL7 could use dd, but also have other options listed in
    * live usb creator gui -
      yum --enablerepo=sl-addons install liveusb-creator
    livecd-tools (cli)
    for SL7 can be found here:
    livecd-iso-to-disk SL-64-i386-2013-04-17-LiveCD.iso /dev/sdb1
    To reserve space for data persistence use option --overlay-size-mb: 
    livecd-iso-to-disk --overlay-size-mb 1024 SL-64-i386-2013-04-17-LiveCD.iso /dev/sdb1
    windoze tools
    * LiLi, was old favorite, but stopped working for CentOS 7 or 8 iso.  maybe okay for ubuntu/debian.
      lili writes to existing FAT32 partition, so could use the usb for regular storage as well.
    * UNetbootin (win/linux/mac).  Write to existing FAT32, make it bootable.
      Seems a bit slow?
      Cant work for Centos 7 iso.
    * PS.  DVD sizes are: 4.7, 8.5, 9.4, 17.08 GB.  So, maybe create a partition of 18GB for hosting such boot media.


    Stress test program, to help diag nasty hardware problems:


    # cmd.linux.ref
    # some content formerly adapted from cmd.admin.ref.
    Red Hat 4.0   (and not available in 3.0)
    /usr/sbin/system-config-kickstart	kickstart configurator.
    system-config-netboot			pxe, tftp boot setup
    system-config-language			GUI language config tooll for the machine
    many of these system-config-* had been named redhat-config-* in RH 3.0
    Red Hat 9.0    2003/07
    Also for RH 3.0 (and presumably available in RH 4.0)
    /etc/X11/XF86Config	# config file read by X server.
    redhat-config-xfree86	# config tool to create XF86Config, try system-config-display in ES 4.0 and later.
    redhat-config-*		# lot of red hat gui config tool.
    nautilus		# file explorer
    gqview			# acdsee like image viewer
    Red Hat ES 2.1  (7.2 base)
    /usr/sbin/setup		# rh version of aix smitty, not all that.
    gnorpm				# GUI rpm manager, not in 3.0 or 4.0 (/usr/bin)
    edit DS line to read
    and sv-smtp will become the smarthost that will be used to realy all the outbound mail.
    /etc/rc[35].d/		# newer RH system, note that links need exist in 3 and 5
    runlevel		# what init level machine currently in
    who -r			# RH 9 and AS 3.0 (cmd exist in Solaris)
    /etc/xinetd.conf, /etc/xinetd.d/
    /etc/printcap  or /etc/printer.conf
    /etc/release		# see which os cd was used eg sol 8 01/00 for Jan 2000 build. 
    /etc/resolv.conf	# dns setup
    /etc/nsswitch.conf	# order of search for files/dns/nis
    printtool  (red hat print config)
    e-conf		enlightment configuration
    linux-conf	most essential config of linux in here
    control-panel	old red hat panel, configure network, printer, etc

    Keyboard shortcuts


    ctrl + shf + PrtScr ==> clipboard (draw rectangle)
           shf + PrtScr ==> File in Pictures folder
    ctrl + alt + PrtScr ==> clipboard (current window)
           alt + PrtScr ==> File in Pictures folder.

    Know Your Penguins - From @Linux Tweeter feed

    [Doc URL:]
    (cc) Tin Ho. See main page for copyright info.
    Last updated: 2020-12-18