adding program shortcuts to the context menu of windows

ever since I got the surface pro 3, loving it, I ended up doing a lot of programming under windows and rediscovering this platform.

So one of my first things I had todo was to add/optimize some context menu entries to simplify the progress of getting console2 to open at specific directories. Basically trying to replace the cmd program as much as possible.

regedit.exe

got to:

HKEY_CLASSES_ROOT\Folder\shell

edit 'Command Prompt here' 

set as new command

"C:\terminal\Console.exe"  -d  "%L"

simple oneliner to create an index over all the massbank indexes

I'm quite often playing around with massbank these days and always like to have a simple index of all there files in a single directory instead of dealing with all there sub directories.

find ./ -name '*.txt' -exec bash -c "f=\$(basename {} ); ln -fs {} ./all/\$f" \;

basically takes care of all these issues for you

Using grails for restful services

Well if you use grails for restful services, it saves a lot of time to ensure that you add the following mapping to your controller.

class SubmitterController extends RestfulController{

//response format avoid the error that views are not found
    static responseFormats = ['json']

    static allowedMethods = [list:"GET", save: "POST", update: "POST", delete: "DELETE", show: "GET"]

    public SubmitterController(){
        super(Submitter)
    }


}

once you did this, everything works as it should and it's rather obvious. But sometimes we forget. Nearly as stupid as font-awesome not working fonts, if you use...

< i class="fa-minus-square"></i>

you will only see a stupid utf square. Instead you need to use...

< i class="fa fa-minus-square"></i>

adding the head node of rocks as compute node

happily stolen from:

https://wiki.rocksclusters.org/wiki/index.php/Sun_GridEngine

cause I keep forgetting it...

 Add Frontend as a SGE Execution Host in Rocks

To setup the frontend node to also be a SGE execution host which queued jobs can be run on (like the compute nodes), do the following:

[edit]

Quick Setup

# cd /opt/gridengine
# ./install_execd    (accept all of the default answers)
# qconf -mq all.q    (if needed, adjust the number of slots for [frontend.local=4] and other parameters)
# /etc/init.d/sgemaster.frontend stop
# /etc/init.d/sgemaster.frontend start
# /etc/init.d/sgeexecd.frontend stop
# /etc/init.d/sgeexecd.frontend start

[edit]

Detailed Setup

1. As root, make sure $SGE_ROOT, etc. are setup correctly on the frontend:

# env | grep SGE

It should return back something like:

SGE_CELL=default
SGE_ARCH=lx26-amd64
SGE_EXECD_PORT=537
SGE_QMASTER_PORT=536
SGE_ROOT=/opt/gridengine

If not, source the file /etc/profile.d/sge-binaries.[c]sh or check if the SGE Roll is properly installed and enabled:

# rocks list roll
NAME          VERSION ARCH   ENABLED
sge:          5.2     x86_64 yes

2. Run the install_execd script to setup the frontend as a SGE execution host:

# cd $SGE_ROOT
# ./install_execd 

Accept all of the default answers as suggested by the script.

• NOTE: For the following examples below, the text should be substituted with the actual "short hostname" of your frontend (as reported by the command hostname -s).

For example, if running the command hostname on your frontend returns back the "FQDN long hostname" of:

# hostname
mycluster.mydomain.org

then hostname -s should return back just:

# hostname -s
mycluster

3. Verify that the number of job slots for the frontend is equal to the number of physical processors/cores on your frontend that you wish to make available for queued jobs by checking the value of the slots parameter of the queue configuration for all.q:

# qconf -sq all.q | grep slots
slots                 1,[compute-0-0.local=4],[.local=4]

The [.local=4] means that SGE can run up to 4 jobs on the frontend. Be aware that since the frontend is normally used for other tasks besides running compute jobs, it is recommended that not all the installed physical processors/cores on the frontend be available to be scheduled by SGE to avoid overloading the frontend.
For example, on a 4-core frontend, to configure SGE to use only up to 3 of the 4 cores, you can modify the slots for .local from 4 to 3 by typing:

# qconf -mattr queue slots '[.local=3]' all.q

If there are additional queues besides the default all.q one, repeat the above for each queue.
Read "man queue_conf" for a list of resource limit parameters such as s_cpu, h_cpu, s_vmem, and h_vmem that can be adjusted to prevent jobs from overloading the frontend.

• NOTE: For Rocks 5.2 or older, the frontend may have been default configured during installation with only 1 job slot ([.local=1]) in the default all.q queue, which will only allow up to 1 queued job to run on the frontend. To check the value of the slots parameter of the queue configuration for all.q, type:

# qconf -sq all.q | grep slots
slots                 1,[compute-0-0.local=4],[.local=1] 

If needed, modify the slots for .local from 1 to 4 (or up to the maximum number of physical processors/cores on your frontend that you wish to use) by typing:

# qconf -mattr queue slots '[.local=4]' all.q

• NOTE: For Rocks 5.3 or older, create the file /opt/gridengine/default/common/host_aliases to contain both the .local hostname and the FQDN long hostname of your frontend:

# vi $SGE_ROOT/default/common/host_aliases

.local .mydomain.org

• NOTE: For Rocks 5.3 or older, edit the file /opt/gridengine/default/common/act_qmaster to contain the .local hostname of your frontend:

# vi $SGE_ROOT/default/common/act_qmaster

.local

• NOTE: For Rocks 5.3 or older, edit the file /etc/init.d/sgemaster.:

# vi /etc/init.d/sgemaster.

and comment out the line:

/bin/hostname --fqdn > $SGE_ROOT/default/common/act_qmaster

by inserting a # character at the beginning, so it becomes:

#/bin/hostname --fqdn > $SGE_ROOT/default/common/act_qmaster

in order to prevent the file /opt/gridengine/default/common/act_qmaster from getting overwritten with incorrect data every time sgemaster. is run during bootup.

4. Restart both qmaster and execd for SGE on the frontend:

# /etc/init.d/sgemaster. stop
# /etc/init.d/sgemaster. start
# /etc/init.d/sgeexecd. stop
# /etc/init.d/sgeexecd. start

And everything will start working. :)

postgres killing queries

once in a while our database server get's terrible overloaded, cause people are running thousands of long running queries against it.

1. How to find out who is generating queries:

select datname, client_addr from pg_stat_activity;


2. Killing queries from a specific IP:

select pg_terminate_backend(procpid) from pg_stat_activity where client_addr='IP';

This can be done as postgres user.

playing with logfiles...

After setting up our zabbix system at work to monitor most of the servers, I'm still trying to analyze our database server and logs a bit more. Since the system seems to be under quite a high load recently.

The easiest way todo this, was like always a little awk scripts to analyze some statistics from our log files and point me in the right directions

cat pg_log/postgresql-2013-08-*.log |  grep duration | awk -F ':' '{ print $7, $0 }' | grep execute | awk  '{ total += $1; count++; print $0 } END { print "average query speed: ", total/count, " count of queries: ",  count }'

X-Forwarding in qlogin on rocks linux > 5

to enable X forwarding from your nodes to your original remote session:

In rocks 4.3,I get the output
# qconf -sconf | grep qlogin
qlogin_command               /opt/gridengine/bin/rocks-qlogin.sh
qlogin_daemon                /usr/sbin/sshd -i

But in rocks 5.3,I get
# qconf -sconf | grep qlogin
qlogin_command               builtin
qlogin_daemon                builtin

So I changed it in rocks 5.3
# qconf -mconf global
qlogin_command               /opt/gridengine/bin/rocks-qlogin.sh
qlogin_daemon                /usr/sbin/sshd -i

And modify the script:

/opt/gridengine/bin/rocks-qlogin.sh

to include this part

/usr/bin/ssh -Y -p $PORT $HOST

The Y basically allows none secured X authentification.

 

Playing with Chef

Recently I got a bit overwhelmed with all the servers I maintain and have to update and so decided to ease my life and start using an automation system. Don't really care that much about my real servers. But having quite a lot of dedicated virtual machines, I guess it was time to simplify this process.

Bootstrapping a system

knife bootstrap systemname-x username -P password --sudo

Installing a cookbook from the central repository

knife cookbook site install name-of-cookbook

Uploading a cookbook to the server

knife cookbook upload name-of-cookbook

Adding a recipe to a system

 knife node run_list add HOSTNAME 'recipe[recipe]'

Updating all systems

knife ssh "*:*" "sudo chef-client" -x username -P password

Updating/Adding a data bag

knife data bag from file BAG_NAME ITEM.json

Changing the environment for nodes

knife exec -E 'nodes.transform("chef_environment:dev") { |n| n.chef_environment("production") }'

Adding a role to all nodes in a certain environment

knife exec -E 'nodes.transform("chef_environment:Fiehnlab") {|n| puts n.run_list << "role[user-management]"; n.save }'

Overriding the runlist for a single run

chef-client -o  recipe["rocks-cluster-6.1::computenode"]

Just a small overview for me to remember some commands.

cdk and grails - why don't get they along. This time using maven for dependency management...

...

Recently I posted about grails2 and the CDK using the BuildConfig.groovy approach.

Now it's only fair that I show the maven approach as alternative. It's aint pretty...

But it works and allows you to build grails 2.1 project with the CDK. Now i just need to integrate scala into the mix...

<?xml version="1.0" encoding="utf-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>edu.ucdavis.fiehnlab.cts</groupId>
    <artifactId>cts</artifactId>
    <packaging>grails-app</packaging>
    <version>2.0-beta-2</version>
    <name>cts</name>
    <description>2.0-beta-2</description>

    <parent>
        <artifactId>web</artifactId>
        <groupId>edu.ucdavis.fiehnlab.cts.web</groupId>
        <version>1.0-SNAPSHOT</version>
    </parent>

    <dependencies>
        <dependency>
            <groupId>org.grails</groupId>
            <artifactId>grails-dependencies</artifactId>
            <version>${grails.version}</version>
            <type>pom</type>
        </dependency>

        <dependency>
            <groupId>org.grails</groupId>
            <artifactId>grails-test</artifactId>
            <version>${grails.version}</version>
            <type>pom</type>
        </dependency>


        <dependency>
            <groupId>org.grails</groupId>
            <artifactId>grails-plugin-testing</artifactId>
            <version>${grails.version}</version>
            <scope>test</scope>
        </dependency>


        <dependency>
            <groupId>net.sf.jni-inchi</groupId>
            <artifactId>jni-inchi</artifactId>
            <version>${jni.inchi.version}</version>
        </dependency>

        <dependency>

            <groupId>org.openscience.cdk</groupId>
            <artifactId>cdk-inchi</artifactId>
            <version>${cdk.version}</version>

            <exclusions>
                <exclusion>
                    <groupId>xerces</groupId>
                    <artifactId>xmlParserAPIs</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>xerces</groupId>
                    <artifactId>xercesImpl</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>xalan</groupId>
                    <artifactId>xalan</artifactId>
                </exclusion>

                <exclusion>
                    <groupId>java3d</groupId>
                    <artifactId>vecmath</artifactId>
                </exclusion>

                <exclusion>
                    <groupId>xmlpull</groupId>
                    <artifactId>xmlpull</artifactId>
                </exclusion>

                <exclusion>
                    <groupId>cml</groupId>
                    <artifactId>cmlxom</artifactId>
                </exclusion>

                <exclusion>
                    <groupId>xom</groupId>
                    <artifactId>xom</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>jdom</groupId>
                    <artifactId>jdom</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>xpp3</groupId>
                    <artifactId>xpp3</artifactId>
                </exclusion>

                <exclusion>
                    <groupId>jama</groupId>
                    <artifactId>jama</artifactId>
                </exclusion>

            </exclusions>
        </dependency>

        <dependency>

            <groupId>org.openscience.cdk</groupId>
            <artifactId>cdk-fingerprint</artifactId>
            <version>${cdk.version}</version>

            <exclusions>
                <exclusion>
                    <groupId>xerces</groupId>
                    <artifactId>xmlParserAPIs</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>xerces</groupId>
                    <artifactId>xercesImpl</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>xalan</groupId>
                    <artifactId>xalan</artifactId>
                </exclusion>

                <exclusion>
                    <groupId>java3d</groupId>
                    <artifactId>vecmath</artifactId>
                </exclusion>

                <exclusion>
                    <groupId>xmlpull</groupId>
                    <artifactId>xmlpull</artifactId>
                </exclusion>

                <exclusion>
                    <groupId>cml</groupId>
                    <artifactId>cmlxom</artifactId>
                </exclusion>

                <exclusion>
                    <groupId>xom</groupId>
                    <artifactId>xom</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>jdom</groupId>
                    <artifactId>jdom</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>xpp3</groupId>
                    <artifactId>xpp3</artifactId>
                </exclusion>

                <exclusion>
                    <groupId>jama</groupId>
                    <artifactId>jama</artifactId>
                </exclusion>

            </exclusions>
        </dependency>
        <dependency>

            <groupId>org.openscience.cdk</groupId>
            <artifactId>cdk-renderawt</artifactId>
            <version>${cdk.version}</version>

            <exclusions>
                <exclusion>
                    <groupId>xerces</groupId>
                    <artifactId>xmlParserAPIs</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>xerces</groupId>
                    <artifactId>xercesImpl</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>xalan</groupId>
                    <artifactId>xalan</artifactId>
                </exclusion>

                <exclusion>
                    <groupId>java3d</groupId>
                    <artifactId>vecmath</artifactId>
                </exclusion>

                <exclusion>
                    <groupId>xmlpull</groupId>
                    <artifactId>xmlpull</artifactId>
                </exclusion>

                <exclusion>
                    <groupId>cml</groupId>
                    <artifactId>cmlxom</artifactId>
                </exclusion>

                <exclusion>
                    <groupId>xom</groupId>
                    <artifactId>xom</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>jdom</groupId>
                    <artifactId>jdom</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>xpp3</groupId>
                    <artifactId>xpp3</artifactId>
                </exclusion>

                <exclusion>
                    <groupId>jama</groupId>
                    <artifactId>jama</artifactId>
                </exclusion>

            </exclusions>
        </dependency>


        <dependency>

            <groupId>org.openscience.cdk</groupId>
            <artifactId>cdk-render</artifactId>
            <version>${cdk.version}</version>

            <scope>runtime</scope>


            <exclusions>
                <exclusion>
                    <groupId>xerces</groupId>
                    <artifactId>xmlParserAPIs</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>xerces</groupId>
                    <artifactId>xercesImpl</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>xalan</groupId>
                    <artifactId>xalan</artifactId>
                </exclusion>

                <exclusion>
                    <groupId>java3d</groupId>
                    <artifactId>vecmath</artifactId>
                </exclusion>

                <exclusion>
                    <groupId>xmlpull</groupId>
                    <artifactId>xmlpull</artifactId>
                </exclusion>

                <exclusion>
                    <groupId>cml</groupId>
                    <artifactId>cmlxom</artifactId>
                </exclusion>

                <exclusion>
                    <groupId>xom</groupId>
                    <artifactId>xom</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>jdom</groupId>
                    <artifactId>jdom</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>xpp3</groupId>
                    <artifactId>xpp3</artifactId>
                </exclusion>

                <exclusion>
                    <groupId>jama</groupId>
                    <artifactId>jama</artifactId>
                </exclusion>

            </exclusions>
        </dependency>


        <dependency>
            <groupId>org.codehaus.groovy.modules.http-builder</groupId>
            <artifactId>http-builder</artifactId>
            <version>0.5.2</version>
            <scope>runtime</scope>

            <exclusions>
                <exclusion>
                    <groupId>xerces</groupId>
                    <artifactId>xmlParserAPIs</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>xerces</groupId>
                    <artifactId>xercesImpl</artifactId>
                </exclusion>

                <exclusion>
                    <groupId>xalan</groupId>
                    <artifactId>xalan</artifactId>
                </exclusion>
            </exclusions>
        </dependency>


        <dependency>
            <groupId>postgresql</groupId>
            <artifactId>postgresql</artifactId>
            <version>8.2-504.jdbc3</version>
        </dependency>


        <dependency>
            <groupId>org.grails.plugins</groupId>
            <artifactId>cache</artifactId>
            <version>1.0.1</version>
            <type>zip</type>
        </dependency>


        <dependency>
            <groupId>org.grails.plugins</groupId>
            <artifactId>jquery-ui</artifactId>
            <version>1.8.24</version>
            <type>zip</type>
        </dependency>


        <dependency>
            <groupId>org.grails.plugins</groupId>
            <artifactId>jquery</artifactId>
            <version>1.8.3</version>
            <type>zip</type>
        </dependency>


        <dependency>
            <groupId>org.grails.plugins</groupId>
            <artifactId>export</artifactId>
            <version>1.5</version>
            <type>zip</type>
        </dependency>


        <dependency>
            <groupId>org.grails.plugins</groupId>
            <artifactId>resources</artifactId>
            <version>1.1.6</version>
            <type>zip</type>
        </dependency>


        <dependency>
            <groupId>org.grails.plugins</groupId>
            <artifactId>jquery-ui-extensions</artifactId>
            <version>0.5.8</version>
            <type>zip</type>
        </dependency>

        <dependency>
            <groupId>org.grails.plugins</groupId>
            <artifactId>hibernate</artifactId>
            <version>${grails.version}</version>
            <type>zip</type>

        </dependency>

        <dependency>
            <groupId>org.grails.plugins</groupId>
            <artifactId>spock</artifactId>
            <version>0.6</version>
            <type>zip</type>
            <scope>test</scope>
        </dependency>

        <dependency>
            <groupId>org.spockframework</groupId>
            <artifactId>spock-grails-support</artifactId>
            <version>0.6-groovy-1.8</version>
            <scope>test</scope>
        </dependency>


        <dependency>
            <groupId>org.grails.plugins</groupId>
            <artifactId>functional-test</artifactId>
            <version>1.2.7</version>
            <type>zip</type>
            <scope>test</scope>
        </dependency>

        <dependency>
            <groupId>org.grails.plugins</groupId>
            <artifactId>webxml</artifactId>
            <version>1.4.1</version>
            <type>zip</type>
        </dependency>

        <dependency>
            <groupId>org.grails.plugins</groupId>
            <artifactId>tomcat</artifactId>
            <version>2.1.0</version>
            <type>zip</type>
            <scope>runime</scope>
        </dependency>


    </dependencies>

    <build>
        <pluginManagement/>

        <plugins>
            <!-- Disables the Maven surefire plugin for Grails applications, as we have our own test runner -->
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <configuration>
                    <skip>true</skip>
                </configuration>
                <executions>
                    <execution>
                        <id>surefire-it</id>
                        <phase>integration-test</phase>
                        <goals>
                            <goal>test</goal>
                        </goals>
                        <configuration>
                            <skip>false</skip>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-clean-plugin</artifactId>
                <version>2.4.1</version>
                <configuration>
                    <filesets>
                        <fileset>
                            <directory>plugins</directory>
                            <includes>
                                <include>**/*</include>
                            </includes>
                            <followSymlinks>false</followSymlinks>
                        </fileset>
                    </filesets>
                </configuration>
            </plugin>

            <plugin>
                <groupId>org.grails</groupId>
                <artifactId>grails-maven-plugin</artifactId>
                <version>${grails.version}</version>
                <configuration>
                    <!-- Whether for Fork a JVM to run Grails commands -->
                    <fork>true</fork>
                </configuration>
                <extensions>true</extensions>
            </plugin>
        </plugins>
    </build>

    <repositories>
        <repository>
            <id>grails</id>
            <name>grails</name>
            <url>http://repo.grails.org/grails/core</url>
        </repository>
        <repository>
            <id>grails-plugins</id>
            <name>grails-plugins</name>
            <url>http://repo.grails.org/grails/plugins</url>
        </repository>

        <repository>
            <id>ambit-public</id>
            <name>ambit-public</name>
            <url>http://ambit.uni-plovdiv.bg:8083/nexus/content/repositories/public/</url>
        </repository>


        <repository>
            <id>ambit-thirdparty</id>
            <name>ambit-thirdparty</name>
            <url>http://ambit.uni-plovdiv.bg:8083/nexus/content/repositories/thirdparty/</url>
        </repository>


        <repository>
            <id>codehaus</id>
            <name>codehaus</name>
            <url>http://snapshots.repository.codehaus.org</url>
        </repository>


    </repositories>
</project>

scala yield vs map which is nicer?

currently I'm playing a bit more with scala in an attempt to leverage the power of my new pair of radeon 7970 for GPU calculations and I'm a touch unsure, which syntax I prefer and is easier to read.

example a:


scala> 1 to 10 map ( _ * 2)
res4: scala.collection.immutable.IndexedSeq[Int] = Vector(2, 4, 6, 8, 10, 12, 14, 16, 18, 20)


versus example b:

scala> for (i <- 10="" 1="" 2="" i="" nbsp="" to="" yield="">
res5: scala.collection.immutable.IndexedSeq[Int] = Vector(2, 4, 6, 8, 10, 12, 14, 16, 18, 20)


They both produce the same result, but I feel the b is easier to understand from a none scala perspective. But a. Is shorter and somewhat clearer

Now if we change this to:


scala> val result  = 1 to 10 map ( _ * 2)
result: scala.collection.immutable.IndexedSeq[Int] = Vector(2, 4, 6, 8, 10, 12, 14, 16, 18, 20)

and

scala> val result = for (i <- 10="" 1="" 2="" i="" nbsp="" to="" yield="">
result: scala.collection.immutable.IndexedSeq[Int] = Vector(2, 4, 6, 8, 10, 12, 14, 16, 18, 20)

now the first version seems to make a lot more sense and the second looks unwieldy complicated.

cdk and grails, why don't they get along...

so apperently the cdk depends on some xml libaries, which causes us all kind of error messages like this in grails:

| Loading Grails 2.1.0
| Configuring classpath
| Error Error executing script Help: loader constraint violation: when resolving overridden method "org.apache.tools.ant.helper.ProjectHelper2$RootHandler.setDocumentLocator(Lorg/xml/sax/Locator;)V" the class loader (instance of org/codehaus/groovy/grails/cli/support/GrailsRootLoader) of the current class, org/apache/tools/ant/helper/ProjectHelper2$RootHandler, and its superclass loader (instance of ), have different Class objects for the type org/xml/sax/Locator used in the signature (Use --stacktrace to see the full trace)

IDEA hook: Grails not found!
| Error java.lang.NullPointerException
| Error  at org.jetbrains.groovy.grails.rt.Agent$2.run(Agent.java:135)
| Error  at java.lang.Thread.run(Thread.java:680)

which are rather annoying since all the mailing list say, to just run the dependency report to see what causes this problem. Great idea exact that grails crashes before it can generate the report...

So after a while of searching and goggling and shouting random words at my imac, I found an ugly solution which works rather well for now.
Basically make your BuildConfig.groovy file look like this and it should fix your current issues...

    dependencies {
        
        runtime 'postgresql:postgresql:8.2-504.jdbc3'

        compile('xmlpull:xmlpull:1.1.3.1')

        compile('org.openscience.cdk:cdk-fingerprint:1.4.16') {
            transitive = false
        }
        compile('org.openscience.cdk:cdk-inchi:1.4.16') {
            transitive = false
        }
        compile('org.openscience.cdk:cdk-standard:1.4.16')
        compile('org.openscience.cdk:cdk-interfaces:1.4.16')
        compile('org.openscience.cdk:cdk-annotation:1.4.16')
        compile('org.openscience.cdk:cdk-io:1.4.16')
        compile('org.openscience.cdk:cdk-isomorphism:1.4.16')
        compile('org.openscience.cdk:cdk-render:1.4.16')
        compile('org.openscience.cdk:cdk-renderbasic:1.4.16')
        compile('org.openscience.cdk:cdk-renderawt:1.4.16')
        compile('org.openscience.cdk:cdk-smarts:1.4.16')  {
            transitive = false
        }
        compile('org.openscience.cdk:cdk-extra:1.4.16')  {
            transitive = false
        }
        compile('org.openscience.cdk:cdk-dict:1.4.16')  {
            transitive = false
        }
        compile('jama:jama:1.0.2')  {
            transitive = false
        }
        compile('org.openscience.cdk:cdk-formula:1.4.16')
        compile('org.openscience.cdk:cdk-smsd:1.4.16')
        compile('xpp3:xpp3:1.1.4c')
        compile('java3d:vecmath:1.3.1')
        compile('net.sf.jni-inchi:jni-inchi:0.7')
    }

Maybe this help someone, who has the same issues with old XML-Apis and Xerces and so

Lift - getting access to a database

Since I recently started to play more and more with lift. I discovered that it's not always as easy as you think it is and once you find a solution it was easy indeed.

For example I want to connect to my database server, but have no desire to actually use any form of OR mapping.

So after googleing a bit and woundering how todo this. It's all rather simple.

In your Boot.scala file

def boot {
    // where to search snippet
    LiftRules.addToPackages("code")

    // Build SiteMap
    LiftRules.setSiteMap(sitemap)

    //rest based stuff

    /**
     * connection url for lift
     */
    val vendor = new StandardDBVendor("org.postgresql.Driver","jdbc:postgresql://****/****",Full("username"),Full("password")) with ProtoDBVendor{
      override def maxPoolSize = 100
      override def doNotExpandBeyond = 200
    }
    
    DB.defineConnectionManager(DefaultConnectionIdentifier, vendor)

  }

This basically defines what to connect to and a connection pool with a max of 100 connections.

To actually access this connection and todo something with it, well this is remarkable easy.

          DB.performQuery("select * from data")

scala + lift + json

So since I really really really dislike soap and Java, I decided to play a bit more with scala and rest based web services. Basically parse some JSON strings with scala.

Now after browsing countless articles and remembering the JSON structure, I finally found the 'lift-json' project, which simplifies my live as busy developer a bit.

How to parse JSON with scala?

package edu.ucdavis.fiehnlab.alchemy.core.communication.services.reader

import edu.ucdavis.fiehnlab.alchemy.core.process.types.LibrarySpectra

import net.liftweb.json.parse

import net.liftweb.json.DefaultFormats

/**

 * connects to the alchemy webservice and fetches all registered co

 */

class WebserviceLibraryReader {

  /**

   * internal converter class to simplify things

   */

  case class JsonCompound(name: String, inchikey: String, retentiontime: Double, theoretical: Double, massspectra: String)

  case class CompoundWapper(compound: Array[JsonCompound])

  implicit val formats = DefaultFormats

  /**

   * downloads the compounds for the given URL and returns it as a set of library spectra

   */

  def fetchCompounds(url: String): Set[LibrarySpectra] = {

    val compounds: CompoundWapper = parse("""

        

        {"compound":[{"name":"test","inchikey":"ADVPTQAUNPRNPO-UHFFFAOYSA-N","retentiontime":11111,"theoretical":11,"massspectra":"111:1 112:2 113:3"},{"name":"test","inchikey":"ADVPTQAUNPRNPO-UHFFFAOYSA-N","retentiontime":11111,"theoretical":11,"massspectra":"111:1 112:2 113:3"}]}

        

        """).extract[CompoundWapper]

    compounds.compound.foreach { x: JsonCompound =>

      println(x)

    }

    

    

    null

  }

}

object WebserviceLibraryReader {

  def main(ars: Array[String]) = {

    println(new WebserviceLibraryReader().fetchCompounds("http://localhost:8080/alchemy-admin/services/queryAllCompoundsForMethod/test"))

  }

}

The complete tutorial can be found here

using ssd's in a raid0 over 4 drives.

Recently I played a bit more with ssd's, since frankly they are getting affordable enough to use as scratch drives for our new software product.

Basically I was wondering, if a raid0 over 4 ssd's is fast enough for us, or if I should use a PCI-Express card.



root@****:/mnt/luna/scratch/gert# dd if=/dev/zero of=/mnt/scratch/bs.img bs=8048 count=81920
81920+0 records in
81920+0 records out
659292160 bytes (659 MB) copied, 0.529044 s, 1.2 GB/s
root@****:/mnt/luna/scratch/gert# dd of=/dev/null if=/mnt/scratch/bs.img bs=8048 count=81920
81920+0 records in
81920+0 records out
659292160 bytes (659 MB) copied, 0.129838 s, 5.1 GB/s


Sure the sample size is not perfect, but for a first test I'm rather surprised by this result and looking forward, if the speed keeps up like this.

grails 1.3.8 and java 1.6

since we still have a lot of grails 1.3.8 application, we are forced to use them and apparently sometimes all the test fail with the following exception:


java.lang.IllegalAccessError: tried to access class org.apache.xml.serializer.XSLOutputAttributes from class org.apache.xalan.transformer.TransformerImpl at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(TransformerImpl.java:2387) at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(TransformerImpl.java:2255) at org.apache.xalan.lib.Redirect.write(Redirect.java:212) at org.apache.xalan.extensions.ExtensionHandlerJavaClass.processElement(ExtensionHandlerJavaClass.java:495) at org.apache.xalan.templates.ElemExtensionCall.execute(ElemExtensionCall.java:230) at org.apache.xalan.templates.ElemApplyTemplates.transformSelectedNodes(ElemApplyTemplates.java:395) at org.apache.xalan.templates.ElemApplyTemplates.execute(ElemApplyTemplates.java:177) at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(TransformerImpl.java:2336) at org.apache.xalan.transformer.TransformerImpl.applyTemplateToNode(TransformerImpl.java:2202) at org.apache.xalan.transformer.TransformerImpl.transformNode(TransformerImpl.java:1276) at org.apache.xalan.transformer.TransformerImpl.transform(TransformerImpl.java:673) at org.apache.xalan.transformer.TransformerImpl.transform(TransformerImpl.java:1192) at org.apache.xalan.transformer.TransformerImpl.transform(TransformerImpl.java:1170) at org.apache.tools.ant.taskdefs.optional.TraXLiaison.transform(TraXLiaison.java:187) at org.apache.tools.ant.taskdefs.XSLTProcess.process(XSLTProcess.java:709) at org.apache.tools.ant.taskdefs.XSLTProcess.execute(XSLTProcess.java:333) at org.apache.tools.ant.taskdefs.optional.junit.AggregateTransformer.transform(AggregateTransformer.java:264) at org.apache.tools.ant.taskdefs.optional.junit.XMLResultAggregator.execute(XMLResultAggregator.java:158) at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:288) at org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106) at _GrailsEvents_groovy$_run_closure5.doCall(_GrailsEvents_groovy:58) at _GrailsEvents_groovy$_run_closure5.call(_GrailsEvents_groovy) at _GrailsTest_groovy$_run_closure1.doCall(_GrailsTest_groovy:199) at TestApp$_run_closure1.doCall(TestApp:82) at gant.Gant$_dispatch_closure5.doCall(Gant.groovy:381) at gant.Gant$_dispatch_closure7.doCall(Gant.groovy:415) at gant.Gant$_dispatch_closure7.doCall(Gant.groovy) at gant.Gant.withBuildListeners(Gant.groovy:427) at gant.Gant.this$2$withBuildListeners(Gant.groovy) at gant.Gant$this$2$withBuildListeners.callCurrent(Unknown Source) at gant.Gant.dispatch(Gant.groovy:415) at gant.Gant.this$2$dispatch(Gant.groovy) at gant.Gant.invokeMethod(Gant.groovy) at gant.Gant.executeTargets(Gant.groovy:590)
fix in the BuildConfig.groovy file:


inherits("global") { // uncomment to disable ehcache // excludes 'ehcache' excludes 'serializer' }

git deleting remote branch

deleting a remote branch is rather clunky, but easy:

git push origin :branch

please ensure that you have the colon (:) infront of your branch name to ensure that the branch will deleted.

git memorizing branches

First, you must create your branch locally

git checkout -b your_branch

After that, you can work locally in your branch, when you are ready to share the branch, push it. The next command push the branch to the remote repository origin and tracks it

git push -u origin your_branch

Teammates can reach your branch, by doing:

git fetch
git checkout origin/your_branch

You can continue working in the branch and pushing whenever you want without passing arguments to git push (argumentless git push will push the master to remote master, your_branch local to remote your_branch, etc...)

git push

Teammates can push to your branch by doing commits and then push explicitly

... work ...
git commit
... work ...
git commit
git push origin HEAD:refs/heads/your_branch

Or tracking the branch to avoid the arguments to git push

git checkout --track -b your_branch origin/your_branch
... work ...
git commit
... work ...
git commit
git push

Found at stackoverflow

submitting all files in a directory to qsub

recently we needed a small script to submit all files in a directory to a script, executed by qsub.

#!/bin/bash if [ $# -lt 2 ]; then echo Missing arguments... echo "Use: process.sh 'input dir' 'output dir'" exit 1 fi if [ -d $1 ]; then for file in `ls $1` do qsub -cwd -p -512 run.sh $1$file $2 done else echo "Missing or incorrect input directory..." exit 1 fi

and the actual run.sh script is just a small java program, which takes our two parameters.

java -Xmx1024m -jar $HOME/data/jars/DataExtractor-0.1.jar $1 $2

maven-assembly-plugin - stackoverflow

currently I'm developing several helper tools for the alchemy project and promtply run into the issue of a stackoverflow exception with maven. Basically I try to assemble several archive, including all the required jars and generate the corresponding MANIFEST.MF files for a couple of main classes. apparently with maven3, you have to set the following flag: export MAVEN_OPTS=-Xss2m to avoid having a stackoverflow exception. Annoying... solution found here

Maven - generate executable jar file

Recently I started to work a bit more with scala again and also finally upgraded to my new nemesis, maven3. More torture than maven2, but I could not live without it.

So what I wanted todo is rather simple, build an archive containing all my classes, a generated manifest and make it executeable.

10 minutes later we found the solution:



<plugin> <artifactId>maven-assembly-plugin</artifactId> <executions> <execution> <id>convert-to-asci</id> <phase>package</phase> <goals> <goal>single</goal> </goals> <configuration> <archive> <manifest> <mainClass>edu.ucdavis.fiehnlab.alchemy.core.process.util.ConvertMZXmlToAscii</mainClass> <packageName>edu.ucdavis.fiehnlab.alchemy.core.process.util</packageName> <addClasspath>true</addClasspath> </manifest> </archive> <descriptors> <descriptor>src/main/descriptor/jar.xml</descriptor> </descriptors> <finalName>ConvertMzXmlToASCII-${project.version}</finalName> <appendAssemblyId>false</appendAssemblyId> </configuration> </execution> <execution> <id>alchemy-converter-zip</id> <phase>package</phase> <goals> <goal>single</goal> </goals> <configuration> <descriptors> <descriptor>src/main/descriptor/zip.xml</descriptor> </descriptors> <attach>true</attach> <finalName>alchemy-converter-${project.version}</finalName> </configuration> </execution> </executions> </plugin> The important part is to set the appendAssembyId to false, otherwise you get all kinds of annoying exceptions....