Wednesday, October 10, 2012

submitting all files in a directory to qsub

recently we needed a small script to submit all files in a directory to a script, executed by qsub.

#!/bin/bash if [ $# -lt 2 ]; then echo Missing arguments... echo "Use: process.sh 'input dir' 'output dir'" exit 1 fi if [ -d $1 ]; then for file in `ls $1` do qsub -cwd -p -512 run.sh $1$file $2 done else echo "Missing or incorrect input directory..." exit 1 fi

and the actual run.sh script is just a small java program, which takes our two parameters.

java -Xmx1024m -jar $HOME/data/jars/DataExtractor-0.1.jar $1 $2

Wednesday, October 3, 2012

maven-assembly-plugin - stackoverflow

currently I'm developing several helper tools for the alchemy project and promtply run into the issue of a stackoverflow exception with maven. Basically I try to assemble several archive, including all the required jars and generate the corresponding MANIFEST.MF files for a couple of main classes. apparently with maven3, you have to set the following flag: export MAVEN_OPTS=-Xss2m to avoid having a stackoverflow exception. Annoying... solution found here