Pile of notes

Speed-up OSX

noreply@blogger.com (Eugene Dedits) — Sun, 08 Sep 2013 19:32:00 +0000

Animations in OSX are ridiculously slow! Let's make some of them a little faster:

Show dock faster:
defaults write com.apple.dock autohide-delay -float 0.0; killall Dock

Make MissionControl faster (apple still calls it Expose; ehw..., real expose was so much better):
defaults write com.apple.dock expose-animation-duration -float 0.2; killall Dock

To restore defaults (why would I?) do:
defaults delete com.apple.dock expose-animation-duration; killall Dock

Unfortunately that's it. Super annoying animation for spaces switching stays super annoying for now...

gnu screen

noreply@blogger.com (Eugene Dedits) — Sun, 08 Sep 2013 06:33:00 +0000

Amazing piece of software. How could I live without it?
Here is a list of things I do not want to look for all over the internet:

C-a c -- create new session
C-a " -- get list of open sessions
C-a A -- set title of current session
C-a n -- go to next session
C-a p -- go to previous one

Beauty of it:
C-a d -- detach session. This let's me to start screen, run something inside of it (like long build procedure), detach from it and log out. After that log back in and attach to that screen again!
screen -r -- re-attach it!

Now let's create some windows!
C-a S -- split horizontaly
C-a | -- split verticaly
C-a <tab> -- go to next window
C-a : layout save somename -- this allows to save the layout after detach/reattach.

Vertical split feature requires screen-4.1.X and older. I got it from
https://github.com/skoneka/screen-session/tree/master/screen-4.1.0

Needed to comment out the following lines in the src to be able to build:
process.c://# include <sys/stropts.h>
pty.c://# include <sys/stropts.h>
screen.c://# include <sys/stropts.h>
tty.c://#include <sys/stropts.h> /* for I_POP */

openmpi on OSX

noreply@blogger.com (Eugene Dedits) — Sat, 31 Aug 2013 05:44:00 +0000

Provided you have gcc and gfortran installed on your machine, this is relatively easy procedure. GCC: go to http://hpc.sourceforge.net/ and grab latest binaries for compilers

sudo tar xvf gcc-blabla.tar -C /
sudo tar xvf gfortran-blabla.tar -C /

Make sure that /usr/local/bin is ahead of /usr/bin or /bin. Get the latest openmpi, untar and prepare for installation.

./configure --prefix=/usr/local/openmpi/1.6.5 CC=gcc CXX=gcc F77=gfortran FC=gfortran --disable-vt
make 
sudo make install

Thats it!

Finally:

export PATH=$PATH:/usr/local/openmpi/1.6.5/bin
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/openmpi/1.6.5/lib
export MANPATH=$MANPATH:/usr/local/openmpi/1.6.5/share/man

Normal 'clear' in emacs eshell

noreply@blogger.com (Eugene Dedits) — Sun, 14 Jul 2013 20:51:00 +0000

Everybody is soo used to ctrl-l in terminal. How to achieve the same behavior in emacs 'eshell'? There goes:

(defun eshell-clear ()
  "Clears the shell buffer ala Unix's clear or DOS' cls"
  (interactive)
  ;; the shell prompts are read-only, so clear that for the duration
  (let ((inhibit-read-only t))
 ;; simply delete the region
 (delete-region (point-min) (point-max)))
        (eshell-send-input) )
(add-hook 'eshell-mode-hook
           '(lambda () (define-key eshell-mode-map "\C-l" 'eshell-clear)))

SSH, sudo and X

noreply@blogger.com (Eugene Dedits) — Wed, 29 May 2013 02:10:00 +0000

ssh to the remote host:

ssh -X username@host

run

xauth list

will get something like

host.name.edu/unix:11  MIT-MAGIC-COOKIE-1  ae2be4ded3dedcd0483b6fa3540a248e

Copy this and "sudo -i". Once root do

xauth add host.name.edu/unix:11  MIT-MAGIC-COOKIE-1  ae2be4ded3dedcd0483b6fa3540a248e

Run "xterm" or whatever is needed. Bingo!

noreply@blogger.com (Eugene Dedits) — Wed, 22 May 2013 17:30:00 +0000

I want to see which nodes are available for the gpu queues.

for i in `seq 0 47`; do qmgr -c "p n gpute-$i"; done > YYY1

After this I do

grep available.accel ./YYY1

and see which gpute-s are marked as connected to accelerator and which are not.

CUDA Toolkit/SDK/Driver installation notes

noreply@blogger.com (Eugene Dedits) — Mon, 15 Apr 2013 20:56:00 +0000

* Toolkit/SDK
Nvidia used to ship SDK and toolkit separately. Starting with
version 5.0 they are distributing these two in the form of single
bundle. Following their earlier conventions, we used to install
SDK in a separate directory (/share/apps/cudasdk). This is done
for 5.0 as well. However we may want to reconsider this practice
and start installing both toolkit and SDK into the same directory,
following Nvidia approach.

We install SDK after toolkit and driver.

* Driver
1. Nvidia driver is only installed on the gpute-x nodes. It needs to be
installed with "keep all files" option. One would normally do it on
/opt/cuda directory.
2. Once installation is done driver directory needs
to be copied to the head node under /share/apps/cuda/5.0/driver
(make sure to keep old driver directory under some distinct name).
3. Inside this directory symlinks libcuda.so and libcuda.so.1 are to
be created (same inside 32)
4. After installation of the driver (on the compute node) run
'nvidia-smi' to see if everything went OK.
5. IMPORTANT: every time nvidia-smi is run as root, permissions on
nvidia devices are set to 600. This needs to be fixed with:
chmod 666 /dev/nvidia*
6. Once all of the above done, driver installation directory on the
compute node may be deleted.
7. We've created an image with 310.44 driver as a part of it. If newer
driver is installed, this image needs to be updated.

* Detailed installation on compute node
Make sure gpus are powered on!
1. Check current driver (nvidia-smi, lspci, ls /dev nv*)and see if
device is present.
2. we run installation form /opt/cuda/build
3. Create text file 'params' with lines
------------
Accept
Y
Y
------------

4. Run installer

./NVIDIA-Linux-x86_64-310.44.run --ui=none --target /opt/cuda/installed --keep < params > output 2>&1 &
Output will be store in 'output'
5. nvidia-smi for check
6. check for the permissions on the devices /dev/nvidia*
permissions need to be 666
chmod 666 /dev/nvidia0
chmod 666 /dev/nvidia1
chmod 666 /dev/nvidiactl
or simply
chmod 666 /dev/nvidia*
7. On the head node
cd /share/apps/cuda/5.0/driver
scp -r gpute-0:/opt/cuda/installed/* ./
we need these files (libs, etc... ) so that users can compile...
8. Finally copy over all the logfiles form the compute node
to the same location on the head node:
scp -r gpute-0:/opt/cuda/build/output ./
scp -r gpute-0:/var/log/nvidia*.log ./
9. Now we can run some tests.

Installation and configuration of PBS

noreply@blogger.com (Eugene Dedits) — Mon, 15 Apr 2013 20:10:00 +0000

Installation

1) Server and scheduler
Installation of PBS server and scheduler is done on the head node
under /share/apps/pbs/versionID
PBS installation script creates 'default' symlink automatically.
Inside this directory:
- /raw -- raw installation package (zipeed or tarred original)
- /build -- build takes place here
- csi_config_ANDY -- all the config file for this system

To install:
./INSTALL
During installation you will be asked for license information.
After installation /etc/pbs.conf has to be modified
I saved current (04-15-2013) version of /etc/pbs.conf in
/share/apps/pbs/11.3.0.121723/build/pbs.conf_04152013
Once all of this is done:
/etc/init.d/pbs start

2) MOM
Installed on the compute nodes in /opt/pbs/version
Use 'execution only' mode when asked
Use input file mom_install.in:

     > cat mom_install.in
     /opt/pbs/11.3.0.121723
     /var/spool/PBS
     2
     y
     service0
     y
     y
     n

Configuration
All the relevant configs are stored in csi_configs_ANDY directory.
1) NODES: node_config
Creates all the 'nodes' for PBS to use. Assign certain nodes to
specific queue. Typical example:

     # Create and define vnode gpute-47
     # <----------------------------->
     create node gpute-47 MOM=gpute-47
     #<--- vnode general attribute settings --->
     set node gpute-47 max_running = 8
     set node gpute-47 resv_enable = true
     set node gpute-47 no_multinode_jobs = false
     #set node gpute-47 queue = qdev2_gpu
     #<--- vnode group,user attribute settings --->
     #set node gpute-47 max_user_run = 8
     #set node gpute-47 max_group_run = 8
     #<--- vnode predefined resource settings --->
     set node gpute-47 resources_available.arch = linux
     # The ncpus resource is picked up from /proc/cpuinfo,
     # but is set explicitly here to the physical number
     # of cores per node to spoof hyperthreading. --rbw
     set node gpute-47 resources_available.ncpus = 8
     #<--- vnode local custom resource settings --->
     set node gpute-47 resources_available.ngpus = 2
     set node gpute-47 resources_available.switch = qdr
     set node gpute-47 resources_available.accel  = fermi
     set node gpute-47 resources_available.qlist  = "qint1_gpu,qdev2_gpu"

This definition creates virtual resource "gpute-47" and makes it
queue-able from queue qint1_gpu and qdev2_gpu. All these
instructions need to be fed into qmgr.

Sometimes one need to manualy re-assign nodes for various
purposes. For example when we lost Fermi cards on few compute
nodes on the gpu side of Andy, we decided to temorarily re-assign
this nodes to the ddr side of the machine. File
nodes_config.bad_gpus has definitions of 'bad gpute nodes' that
perform such re-assignment. Other fine-tunnigs are done in:
- nodes_config.gau -- nodes for gaussian jobs
- nodes_config.pck -- nodes for packed

2) QUEUES: queue_config
Creates and sets paramaters of all the queues (both execution and
routing) By setting resources minimuns and maximums we can
control placement of the job into this or the other queue. Queue
can be disabled with
set queue qname enabled = False
set queue qname started = False

When setting up routing queue in section:

     set queue production route_destinations  = qserial
     set queue production route_destinations += qmax
     set queue production route_destinations += qlong64
     set queue production route_destinations += qshort64
     set queue production route_destinations += qlong16
     set queue production route_destinations += qshort16

order matters! PBS will place the job into first execution queue
that matches requested resources. We don't want to impose qshort
time limits on the job that does not request specific amount of
time. Therefore qlongXX is in front of qshortXX

Matlab queues definitions are located in directory
'matlab_queues'.

3) SERVER: server_config
Defines some jeneral things like server defaults (what will be
allocated to a the user if nothing else is specified), server
limits, licensing information, priorities formula. Is to be
pushed to qmgr.

4) SCHEDULER: sched_config
This file contains information for the scheduler. Things like
starving of the job, backfilling, resources that scheduler needs
to be aware of, mom resources, etc...

Is to be placed under /var/spool/PBS/sched_priv

5) RESOURCES: resourcedef
Contains definition of additional resources that we may be
interested in managing. For example 'accel' can be equal to
'fermi' or 'tesla' on a cluster with both types of GPUs. Based on
what user requires PBS can send him either to the compute node
with tesla or to the compute node with fermi.

Another example is gscratch. This resource is created to manage
allocation of scratch space fr gaussian users.

File 'resourcedef' is to be placed under /var/spool/PBS/server_priv

6) MOM: mom_config
Setting for mom. List of users not to be killed upon login to the
compute node, max lod, external dynamic resources (lscratch for
gaussian users). lscratch runs script
/share/apps/pbs/default/bin/gscratch to get amount of scratch
space available for gaussian job.

Resides on compute nodes under /var/spool/PBS/mom_priv

PBS Analytics

1. Lisence server for PBS and PBS analytics is installed on
Neptune under /opt/altair/flex/security
2. To generate new license one needs to
- obtain hostid with './bin/lmutil lmhostid'
- login to PBS support account.
- download license file (altair_lic.dat)
- place it under /opt/altair/flex/security
- start daemon:
/etc/init.d/altairmgrd start/stop
3. License manager must be ran by user=group=nobody
4. Data collectors need to be installed on those machines that run
PBS server. We install them under /opt/gridworks/10.4.2
Logs, binaries, configs are under
/opt/gridworks/10.4.2/datacollector/etl/modules/pbsdata
5. Datacollectors a ran by daemon /etc/init.d/pbsdata
6. Portal is running on HPCAdmin. Start/stop with
/etc/init.d/pbsportal start/stop

Salk's storage manager

noreply@blogger.com (Eugene Dedits) — Wed, 10 Apr 2013 17:13:00 +0000

Salk's storage manager allows to ... manage storage array.

To get to it:
1) login to salkadmin as root with X tunneled
2) cd /opt/SMgr/client
3) ./SMclient

WRF installation

noreply@blogger.com (Eugene Dedits) — Wed, 10 Apr 2013 16:13:00 +0000

Wrf/WRF-chem installation procedures on Andy and Cray machine.

* On Andy needed to install a whole bunch of packages first:
- szip (reinstalled 2.1 with new intel)
- zlib (installed 1.2.7 with gcc. original version there was also built with gcc)
- curl (installed 7.29.0 with new intel)
- udunits (installed 2.1.24)
- png (installed 1.6.0)
- jasper (rebuilt 1.900.1 with new intel)
- g2clib (installed 1.2.3)
- jpeg (reinstalled 6b with new intel)
- hdf5 (installed 1.8.10 with new intel)
- hdf4 (installed 4.2.9) - pnetcdf (installed 1.3.1)
- netcdf - NCAR (finally gave up on this one and grabbed latest binary distribution ./share/apps/6.1.2_bin)

Unpack all of the WRF and WPS tarballs. Directory structure should look like:

ll /share/apps/wrf/3.4.1
total 12
drwxr-xr-x  7 root root 4096 Aug 16  2012 WPS
drwxr-xr-x 16 root root 4096 Aug 16  2012 WRFV3
drwx------  2 root root 4096 Mar 19 11:48 raw

** Building WRF
1) export all the env variables that are needed:

   
   export NETCDF=/share/apps/netcdf/4.2.1.1s_c
   export PNETCDF=/share/apps/pnetcdf/1.3.1
   export HDF5=/share/apps/hdf5/1.8.10s
   export PHDF5=/share/apps/hdf5/1.8.10p
   export JASPER=/share/apps/jasper/1.900.1
   export JASPERLIB=/share/apps/jasper/1.900.1/lib
   export JASPERINC=/share/apps/jasper/1.900.1/include
   export WRFIO_NCD_LARGE_FILE_SUPPORT=1
   export NCARG_ROOT=/share/apps/ncar/6.1.2_bin

2) run ./configure script
3) configure.wrf script will be created. It needs to be modified. Therefore I save it as configure.wrf_orig. Modified version is named configure.wrf_csi
4) Compile "em_real" case with "./compile em_real > COMP.out 2>&1 &"

** Building WPS

1) variables needed to be exported (see (1) for building WRF)
2) run ./configure cp configure.wps configure.wps_orig
3) Modify original configure.wps to include all the external lib. Saved as configure.wps_csi
4) compile with "./compile > COMP.out 2>&1 &"
5) create directory /share/apps/wrf/3.4.1/WPS_DATA with geod data and JAN00 data (latter is used in the wiki for a test case)
6) Set all the permissions to allow users to access binaries and relevant data:

   chmod 755 /share/apps/wrf/3.4.1
   chmod 755 /share/apps/wrf/3.4.1/WPS
   chmod 755 /share/apps/wrf/3.4.1/WPS
   chmod 755 /share/apps/wrf/3.4.1/WPS_DATA
   chmod 755 -R /share/apps/wrf/3.4.1/WPS_DATA
   chmod 755 /share/apps/wrf/3.4.1/WRFV3/main/*exe
   chmod 755 /share/apps/wrf/3.4.1/WPS/metgrid.exe
   chmod 755 /share/apps/wrf/3.4.1/WPS/ungrib.exe
   chmod 755 /share/apps/wrf/3.4.1/WPS/geogrid.exe
   chmod 755 -R /share/apps/wrf/3.4.1/WPS/geogrid

** Building WRF_CHEM
Procedure is explained in the /share/apps/wrfchem/3.4.1/WRFV3/README.csi
Basically it is:

1) export all the env variables that are needed:

      
      export NETCDF=/share/apps/netcdf/4.2.1.1s_c
      export PNETCDF=/share/apps/pnetcdf/1.3.1
      export HDF5=/share/apps/hdf5/1.8.10s
      export PHDF5=/share/apps/hdf5/1.8.10p
      export JASPER=/share/apps/jasper/1.900.1
      export JASPERLIB=/share/apps/jasper/1.900.1/lib
      export JASPERINC=/share/apps/jasper/1.900.1/include
      export WRFIO_NCD_LARGE_FILE_SUPPORT=1
      export NCARG_ROOT=/share/apps/ncar/6.1.2_bin
      # For WRF Chem
      export EM_CORE=1
      export WRF_CHEM=1

2) run ./configure
3) modify configure.wrf (original version is kept under name configure.wrf_orig)
4) modify the 'makefile' in external/io_netcdf (compare to original version makefile_orig)
5) run ./compile

** Building WRF_CHEM WPS part

This is pretty much the same.
1) export all vars (see above)
2) adjust configure.wps file (see original for differences)
3) run ./compile

* On Cray: needed to install:
a. pnetcdf (installed 1.3.1)
b. jasper c. png (installed 1.6.0; needed for WPS)
d. ncar (got binary there)
netcdf, HDF5, PHDF5 -- will use Cray's versions.

** Installing WRF
1) Load all the required stuff

   module load netcdf
   module load PrgEnv-cray
   module load xtpe-mc8
   module load libfast
   export NETCDF=/opt/cray/netcdf/4.2.1.1/cray/81
   export PNETCDF=/share/apps/pnetcdf/1.3.1
   export HDF5=/opt/cray/hdf5/1.8.9/cray/81
   export PHDF5=/opt/cray/hdf5-parallel/1.8.9/cray/81
   export JASPERINC=/share/apps/jasper/1.900.1/include
   export JASPERLIB=/share/apps/jasper/1.900.1/lib
   export JASPER=/share/apps/jasper/1.900.1

2) The rest of installation is the same as the one described below:
1) Modify configure.wrf (uncomment -lfast_mv)
# module load libfast
LIB_LOCAL = -lfast_mv
2) compile the 'em_real' binary with: nohup ./compile em_real > COMPILE.out 2>&1 &
This worked OK (though procedure is very slow). Moving to WPS installation.

** Installing WPS
1) Export all the needed stuff

   export NETCDF=/opt/cray/netcdf/4.1.3/cray/73
   export PNETCDF=/share/apps/pnetcdf/1.2.0
   export HDF5=/opt/cray/hdf5/1.8.7/cray/73
   export PHDF5=/opt/cray/hdf5-parallel/1.8.7/cray/73
   export JASPERINC=/share/apps/jasper/1.900.1/include
   export JASPERLIB=/share/apps/jasper/1.900.1/lib
   export JASPER=/share/apps/jasper/1.900.1
   export PNGINC=/share/apps/png/1.6.0/include
   export PNGLIB=/share/apps/png/1.6.0/lib
   export PNG=/share/apps/png/1.6.0
   export WRFIO_NCD_LARGE_FILE_SUPPORT=1
   export NCARG_ROOT=/share/apps/ncar/6.1.2_bin
   module load netcdf
   module load PrgEnv-cray
   module load xtpe-mc8
   module load libfast

2) make modifications to configure.wps and ./util/src/Makefile (compare to their _orig counterparts)
3) ./compile

** Installing WRFCHEM WRFV3 on Cray
1) untar WRF_CHEM into this directory
2) First I had to set the environment to include all the needed stuff:

   module load netcdf
   module load PrgEnv-cray
   module load xtpe-mc8
   module load libfast

   export NETCDF=/opt/cray/netcdf/4.2.1.1/cray/81
   export PNETCDF=/share/apps/pnetcdf/1.3.1
   export HDF5=/opt/cray/hdf5/1.8.9/cray/81
   export PHDF5=/opt/cray/hdf5-parallel/1.8.9/cray/81
   export JASPERINC=/share/apps/jasper/1.900.1/include
   export JASPERLIB=/share/apps/jasper/1.900.1/lib
   export JASPER=/share/apps/jasper/1.900.1
   # For WRF Chem
   export EM_CORE=1
   export WRF_CHEM=1

3) Second, run ./configure. See the output file for details.
4) I copied the newly generated configure.wrf over to configure.wrf_orig
5) Make changes to configure.wrf to include and link all that is needed. 'diff configure.wrf configure.wrf_orig' to see what is different
6) Finally ./compile em_real > COMP.out 2>&1 &" will build all. See COMP.out for details.

** Installing WRFCHEM WPS on Cray
1) export all the required vars:

   module load netcdf
   module load PrgEnv-cray
   module load xtpe-mc8
   module load libfast
   export NETCDF=/opt/cray/netcdf/4.2.1.1/cray/81
   export PNETCDF=/share/apps/pnetcdf/1.3.1
   export HDF5=/opt/cray/hdf5/1.8.9/cray/81
   export PHDF5=/opt/cray/hdf5-parallel/1.8.9/cray/81
   export JASPERINC=/share/apps/jasper/1.900.1/include
   export JASPERLIB=/share/apps/jasper/1.900.1/lib
   export JASPER=/share/apps/jasper/1.900.1
   export PNGINC=/share/apps/png/1.6.0/include
   export PNGLIB=/share/apps/png/1.6.0/lib
   export PNG=/share/apps/png/1.6.0
   # For WRF Chem
   export EM_CORE=1
   export WRF_CHEM=1

2) run ./configure see ouput in the CONFIG.out
3) the above step generated configure.wps which I have copied to configure.wps_orig
4) make adjustments to configure.wps in order to include all the needed stuff. compare to configure.wps_orig to see what's changed.
5) Finally "./compile > COMP.out 2>&1 &" will build all. See COMP.out for details.

Commenting/Uncommenting block of code in vi

noreply@blogger.com (Eugene Dedits) — Wed, 10 Apr 2013 15:24:00 +0000

go to the first line of the block to be commented

Ctrl-V

go to the last line of the block

insert comment symbol (like "#" or whatever) end exit

ESC