Playing with Chef

Recently I got a bit overwhelmed with all the servers I maintain and have to update and so decided to ease my life and start using an automation system. Don't really care that much about my real servers. But having quite a lot of dedicated virtual machines, I guess it was time to simplify this process.

Bootstrapping a system

knife bootstrap systemname-x username -P password --sudo

Installing a cookbook from the central repository

knife cookbook site install name-of-cookbook

Uploading a cookbook to the server

knife cookbook upload name-of-cookbook

Adding a recipe to a system

 knife node run_list add HOSTNAME 'recipe[recipe]'

Updating all systems

knife ssh "*:*" "sudo chef-client" -x username -P password

Updating/Adding a data bag

knife data bag from file BAG_NAME ITEM.json

Changing the environment for nodes

knife exec -E 'nodes.transform("chef_environment:dev") { |n| n.chef_environment("production") }'

Adding a role to all nodes in a certain environment

knife exec -E 'nodes.transform("chef_environment:Fiehnlab") {|n| puts n.run_list << "role[user-management]"; n.save }'

Overriding the runlist for a single run

chef-client -o  recipe["rocks-cluster-6.1::computenode"]

Just a small overview for me to remember some commands.

Basic Rocks linux stuff.

Some basic Rocks Linux Cluster stuff... just some stuff I need to remember and always forget:

list all host interfaces:


rocks list host interface

changing the mac address of a network card: 


rocks set host interface mac HOSTNAME iface=eth1 mac=00:00:00:00:00:02



please be a aware that you also have to remove the mac address definition in the following file:


/etc/sysconfig/networking/devices/ifcfg-eth*


you should be able to just uncomment the mac address line, without any ill effects and this will simplify this process next time.


and possible in the dhcpd.conf file


vim /etc/dhcpd.conf


in case you modify the eth0 interface. To ensure that the dhcpd configuration still works as supposed to.





installing vmware server on ubuntu 11.10

so I'm trying to install a ubuntu base vmware image hosting platform right now keep running into issues.

done on a system with the following kernel config:

Linux sun 3.0.0-15-server #26-Ubuntu SMP Fri Jan 20 19:07:39 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

first you need to link some files to make the script start generating the modules...

$ sudo ln -s /lib/modules/your_kernel_version/build/include/generated/utsrelease.h /lib/modules/your_kernel_version/build/include/linux/utsrelease.h
$ sudo ln -s /lib/modules/your_kernel_version/build/include/generated/autoconf.h /lib/modules/your_kernel_version/build/include/linux/autoconf.h

which now makes the installation fail with some error of missing smp.h header files

/tmp/vmware-root/modules/vmmon-only/linux/driver.c:40: fatal error: linux/smp_lock.h: No such file or directory
compilation terminated.

next step was to downgrade the kernel to the last version where I found enough info about it to make it compile with vmware.

Linking the files again

ln -s /lib/modules/2.6.38-10-server/build/include/generated/utsrelease.h /lib/modules/2.6.38-10-server/build/include/linux/utsrelease.h
ln -s /lib/modules/2.6.38-10-server/build/include/generated/autoconf.h /lib/modules/2.6.38-10-server/build/include/linux/autoconf.h

and now it fails with the following error

/tmp/vmware-config0/vmmon-only/linux/driver.c:1423:4: error: too many arguments to function ‘smp_call_function’
include/linux/smp.h:73:5: note: declared here
/tmp/vmware-config0/vmmon-only/linux/driver.c: In function ‘LinuxDriver_Ioctl’:
/tmp/vmware-config0/vmmon-only/linux/driver.c:1987:18: error: ‘struct task_struct’ has no member named ‘euid’
/tmp/vmware-config0/vmmon-only/linux/driver.c:1987:35: error: ‘struct task_struct’ has no member named ‘uid’
/tmp/vmware-config0/vmmon-only/linux/driver.c:1988:11: error: ‘struct task_struct’ has no member named ‘fsuid’
/tmp/vmware-config0/vmmon-only/linux/driver.c:1988:29: error: ‘struct task_struct’ has no member named ‘uid’
/tmp/vmware-config0/vmmon-only/linux/driver.c:1989:18: error: ‘struct task_struct’ has no member named ‘egid’
/tmp/vmware-config0/vmmon-only/linux/driver.c:1989:35: error: ‘struct task_struct’ has no member named ‘gid’
/tmp/vmware-config0/vmmon-only/linux/driver.c:1990:11: error: ‘struct task_struct’ has no member named ‘fsgid’
/tmp/vmware-config0/vmmon-only/linux/driver.c:1990:29: error: ‘struct task_struct’ has no member named ‘gid’
/tmp/vmware-config0/vmmon-only/linux/driver.c:2007:7: error: too many arguments to function ‘smp_call_function’
include/linux/smp.h:73:5: note: declared here
make[2]: *** [/tmp/vmware-config0/vmmon-only/linux/driver.o] Error 1
make[1]: *** [_module_/tmp/vmware-config0/vmmon-only] Error 2
make[1]: Leaving directory `/usr/src/linux-headers-2.6.38-10-server'

getting frustrated I went back to a bit more car research. Still at the point of 335i vs Cayman vs Boxster vs Z4. It all depends on the european delivery of if I can find a nice 1-2 year old cpo'ed car in my area.

calmed down a lot and so the next step was to rename my vmware server archive to the right extension:

mv VMware-server-2.0.2-203138.x86_64.gz VMware-server-2.0.2-203138.x86_64.tar.gz

and following the instructions here

and it worked

Starting VMware services:
Virtual machine monitor done
Virtual machine communication interface done
VM communication interface socket family: done
Virtual ethernet done
Bridged networking on /dev/vmnet0 done
Host-only networking on /dev/vmnet1 (background) done
DHCP server on /dev/vmnet1 done
Host-only networking on /dev/vmnet8 (background) done
DHCP server on /dev/vmnet8 done
NAT service on /dev/vmnet8 done
VMware Server Authentication Daemon (background) done
Shared Memory Available done
Starting VMware management services:
VMware Server Host Agent (background) done
VMware Virtual Infrastructure Web Access
Starting VMware autostart virtual machines:
Virtual machines done

The configuration of VMware Server 2.0.2 build-203138 for Linux for this
running kernel completed successfully.

Housekeeping...
Thank you for using the script!
Patch provided by:
Ramon de Carvalho Valle
http://risesecurity.org
updated by sirusdv (http://ubuntuforums.org/member.php?u=1289848)
Script author:
Radu Cotescu
http://radu.cotescu.com

now I just need to convince grub to use the 'old' kernel by default...

rocks linux - virtual hosts with apache

currently I have a cluster setup in the office with about 5 nodes and a bit over 50 cpus, so this morning I decided to rebuild some of the nodes, since I needed to make some changes to the cluster.

20 minutes into this procedure, I keep getting odd error messages, like file 'update.img' not found and so.

So while backtracking the latest changes I did to the server. I realized that I setup 20 virtual hosts in the apache configuration, which ended up screwing with the kickstart configuration. It turns out that the order of the virtual host seems to quite important and that the kickstart configuration always needs to be first, and than you have to define the other virtual hosts.

Example of a virtual host configuration, which allows the kickstart configuration to work,


vim /etc/httpd/conf.d/rocks.conf


actual file:


<IfModule mod_mime.c>
AddHandler cgi-script .cgi
</IfModule>

UseCanonicalName Off


DirectoryIndex index.cgi

<Directory "/var/www/html">
Options FollowSymLinks Indexes ExecCGI
AllowOverride None
Order allow,deny
Allow from all
</Directory>

<Directory "/var/www/html/proc">
Options FollowSymLinks Indexes ExecCGI
AllowOverride None
Order deny,allow
Allow from 10.1.0.0/255.255.0.0
Allow from 127.0.0.1
Deny from all
</Directory>

<Directory "/var/www/html/pxelinux">
Options FollowSymLinks Indexes ExecCGI
AllowOverride None
Order deny,allow
Allow from 10.1.0.0/255.255.0.0
Allow from 127.0.0.1
Deny from all
</Directory>

<VirtualHost *:80>
ServerName kickstart.host.com
DocumentRoot "/var/www/html"
</VirtualHost>

<VirtualHost *:80>
ServerName virtual.host.com
DocumentRoot "/var/www/cts/html"
</VirtualHost>


rocks 5.4 cluster - installing subversion

sometimes the most trivial things turn out to be rather interessting.

For example my latest drama with rocks was that I just needed subversion, to convert one of my BinBase images to a development image for BinBase.

Little did I know, that in the rocks 5.4 version, the installer fails to provide you with the base repository. Quality control anyone?

Anyway, to get your yum to work, as exspected. Just open your


/etc/yum.conf


file and add the following section



[base]
name=CentOS-$releasever - Base
mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=os



at the end of the file and than you can install subversion like always using:



yum -y install subversion



openCl - not recognizing GPU, during remote logins

in the last couple of days I had some issues with my Ubuntu/Ati GPU based calculation units. They refused to execute any calculations, whenever I tried to use SSH to run them.

So a bit of googleling later I found the following informations in a hidden pdf:



a. Add the following lines at the end of /etc/gdm/Init/Default,
before the exit 0, to modify the security settings, allowing remote
sessions to access the X server and ensuring that remote sessions have
access to the necessary device files when communicating with the GPU:
xhost +
chmod uog+rw /dev/ati/card*
b. If you normally use bash, add the following line to the end of
/etc/bashrc file to ensure remote sessions know which X server to
access.
case $DISPLAY in ’’) export DISPLAY=:0;; *) ;; esac
NOTE: ’ ’ are two single quotes, not a single double-quot



Taken from:

http://developer.amd.com/sdks/AMDAPPSDK/assets/App_Note-Running_AMD_APP_Apps_Remotely.pdf

Now everything works fine and all is dandy again.

registering jboss to start by default and comforms to chkconfig

currently I had yet another binbase install and it turned out that my standard jboss script is not chkconfig compatible.

So it was time todo a rewrite


#! /bin/bash
#
# network Bring up/down jboss
#
# chkconfig: 345 20 80
# description: Starts and stops the jboss server
#
# /etc/rc.d/init.d/jboss
# See how we were called.

#define where jboss is - this is the directory containing directories log, bin, conf etc
JBOSS_HOME=/share/apps/jboss

#define the user under which jboss will run, or use 'RUNASIS' to run as the current user
JBOSS_USER=${JBOSS_USER:-"root"}

#make sure java is in your path
JAVAPTH=/usr/java/latest/bin/java

#configuration to use, usually one of 'minimal', 'default', 'all'
JBOSS_CONF=${JBOSS_CONF:-"all"}
JBOSS_HOST=0.0.0.0

JBOSS_CONSOLE=$JBOSS_HOME/console.log
#if JBOSS_HOST specified, use -b to bind jboss services to that address
JBOSS_BIND_ADDR=${JBOSS_HOST:+"-b $JBOSS_HOST"}

#define the classpath for the shutdown class
JBOSSCP=${JBOSSCP:-"$JBOSS_HOME/bin/shutdown.jar:$JBOSS_HOME/client/jnet.jar"}

#define the script to use to start jboss
JBOSSSH=${JBOSSSH:-"$JBOSS_HOME/bin/run.sh -c $JBOSS_CONF $JBOSS_BIND_ADDR"}

if [ "$JBOSS_USER" = "RUNASIS" ]; then
SUBIT=""
else
SUBIT="su - $JBOSS_USER -c "
fi

if [ -n "$JBOSS_CONSOLE" -a ! -d "$JBOSS_CONSOLE" ]; then
# ensure the file exists
touch $JBOSS_CONSOLE
if [ ! -z "$SUBIT" ]; then
chown $JBOSS_USER $JBOSS_CONSOLE
fi
fi



finding all external reference names in pubchem sdf files

currently I'm working on a simple way to find all external reference names in the pubchem sdf files. Which is rather trivial, but time consuming:



cat *.sdf | grep PUBCHEM_EXT_DATASOURCE_NAME -A 1 | grep -v PUBCHEM_EXT_DATASOURCE_NAME | grep -v '\-\-' | sort | uniq



once this finished it should give us a list of all possible data sources from pubchem as a unique list of names.


...
Ambinter
Burnham Center for Chemical Genomics
Calbiochem
CC_PMLSC
ChEBI
ChemSpider
DiscoveryGate
Emory University Molecular Libraries Screening Center
InFarmatik
KUMGM
LipidMAPS
MICAD
MLSMR
MMDB
MTDP
Nature Chemical Biology
NCGC
NIAID
NMMLSC
NMRShiftDB
ORST SMALL MOLECULE SCREENING CENTER
PCMD
ProbeDB
Prous Science Drugs of the Future
R&D Chemicals
Sigma-Aldrich
Specs
SRMLSC
Structural Genomics Consortium
The Scripps Research Institute Molecular Screening Center
Thomson Pharma
UM-BBD
UPCMLD
...


renaming all files in a directory

some of the simple things in live




for x in *.php5; do n=${x/.php5/.php}; mv $x $n; done



taken from here

http://www.debianadmin.com/rename-multiple-files-to-another-extension-in-linux.html

scala/groovy on rocks linux

well since there is not scala/groovy roll for rocks we need to install it the traditional way.

• go into the directory /shared/apps on the frontend
• if apps doesn't exist create it
• copy your scala/groovy tgz there
• gunzip and untar it
• edit your extend-compute.xml as shown here
• add a new file modification section like this



<file name="/etc/profile" mode="append">

GROOVY_HOME=/share/apps/groovy
SCALA_HOME=/share/apps/scala

export GROOVY_HOME
export SCALA_HOME

PATH=$GROOVY_HOME/bin:$PATH
PATH=$SCALA_HOME/bin:$PATH

export PATH

</file>



• rebuild your dist as shown here
• reinstall you nodes as shown here

rocks linux cluster - mounting an nfs share on all nodes

after setting up the latest cluster I tried to provide to all nodes a couple of nfs shares, since user demanded this.

Well in rocks linux it's rather simple, once you understand the concept behind.

So a step to step tutorial.

• go to the profile directory
cd /export/rocks/install/site-profiles/5.3/nodes/
• make a copy of the skeleton file
cp skeleton.xml extend-compute.xml
• edit file to tell it that we need to create a directory and add a line to the fstab. The right place for this is in the post section
  
  
mkdir -p /mnt/share

<file name="/etc/fstab" mode="append">
server:/mount /mnt/share nfs defaults 0 0
</file>

  
• change back to the main install dir
cd /export/rocks/install
• rebuild rocks distibution
rocks create distro
• rebuild nodes
ssh compute-0-0 '/boot/kickstart/cluster-kickstart'

congratulations if you did everything right your node should now boot up and have a directory mounted.

maven2 - choose dependcies based on platform

today I run into an old friend. I was compiling my stuff happily under oxs and building eclipse plugins with maven2 and the psteclipse plugin, which I by now more or less rewrote and adapted to my project to perform as good as possible. Anyway, basically I encountered the issue that eclipse ships with different dependencies under osx/win32 and linux.

So promptly it fails to compile under linux...

solution for this problem?

maven2 profiles, your best friend for cross platform development. They actually allow you define dependencies based on the current platform.

Example for out case would be:


<profiles>
<profile>
<id>dev-windows</id>

<dependencies>
</dependencies>

<activation>
<activeByDefault>true</activeByDefault>
<os>
<family>windows</family>
</os>
</activation>
</profile>
<profile>
<id>dev-mac</id>
<dependencies>
<dependency>
<groupId>psteclipse</groupId>
<artifactId>org.eclipse.swt.cocoa.macosx</artifactId>
<version>3.5.1</version>
<scope>provided</scope>
</dependency>

</dependencies>

<activation>
<activeByDefault>false</activeByDefault>
<os>
<family>mac</family>
</os>
</activation>
</profile>
<profile>
<id>dev-linux</id>

<dependencies>
<dependency>
<groupId>psteclipse</groupId>
<artifactId>org.eclipse.swt.gtk.linux.x86</artifactId>
<version>3.5.1</version>
<scope>provided</scope>
</dependency>
</dependencies>

<activation>
<activeByDefault>false</activeByDefault>
<os>
<name>linux</name>
</os>
</activation>
</profile>
</profiles>


And sets a different library for osx and linux. Sweet!

Method "ApplyConfiguration" with signature "" on interface "org.gnome.SettingsDaemon.XRANDR" doesn't exist

today I wanted todo nothing else than the change the primaery output of my laptop to the 40" display attached over DVI to watch movies on it.

Sadly since ubuntu 8.something I kept getting the following error message


Method "ApplyConfiguration" with signature "" on interface "org.gnome.SettingsDaemon.XRANDR" doesn't exist


So I upgraded to the newest ubuntu version. Didn't make it much better. So I could had given up here and start shouting all kinds off stuff about linux or trying to make it work.

So here is the solution:

the gui is basically broken and you need to execute the commands to adjust everything your self. Using 'xandr'

for example to turn off a monitor:


xandr --output LVDS --off


or to change the resolution


xandr --output DVI-0 --mode 1360x768 --pos 0x0


2 minutes later I had my configuration up and running.