Coding and More

grails plugins and feedback

2012-01-26T11:47:00.000-08:00

once in a while I get emails like this, which makes me really happy to put effort into open source projects.

Hi Gert,

I'm fairly new to Grails and Java and I've been using your Grails jprogress plugin the past few days and it works great - thanks.

I wanted to update the message text as a process went along but didn't see that featured defined in the service. I assumed that you were using the jProgressBar java class and that I could define it's setString method but as I look around I don't think that's the case. Is there a way to do that with your plugin?

Thanks,

Greg

obviously it also means I need todo some more stuff to keep people happy, hence even less spare time for other stuff :(

CKEditor plugin + grails submitTorRemote == fail

2011-12-14T15:33:00.000-08:00

so people in my office talked to me now a couple of times that they like to have a formating option for text in the comment field.

So my first thought was to use the mighty ckeditor and there is a grail plugin already around. SO we should be done with this in 5 minutes...

Wrong it took a bit longer, since my form kept refusing to save my content and always reported the original content instead.

After quite a bit of googleling and the mandatory

I'm pissed about grails beer, this time it was a nice Aventinius

I found a solution to make it work.

Basically if you want to use CKEditor with JQuery you need to update it fields before (the plugin should really handle this! Do you hear me? I mean I fixed to many plugins by now...)




<g:form name="createNewComment" id="createNewComment" controller="comment">
   

        <ckeditor:editor height="300px" width="80%" name="text  ">

            test
        </ckeditor:editor>

        <g:hiddenField name="id" value="${id}"/>
        <g:hiddenField name="type" value="${type}"/>
        <g:hiddenField name="destination" value="${destination}"/>


    <g:javascript>
        function CKupdate() {
            for (instance in CKEDITOR.instances)
                CKEDITOR.instances[instance].updateElement();
            return true;
        }
    </g:javascript>

            <g:submitToRemote before="CKupdate();" class="save" action="saveAjax" update="${destination}" value="save comment"/>
  

</g:form>

And this is all, now before we press the button, we always call the method to update the editor and everybody is happy...

jQuery + grails and my missing buttons...

2011-12-09T14:19:00.000-08:00

currently I'm switching miniX over to use jQueryUi style buttons, since this makes life so much easier to maintain them.

Except during my event processing my buttons keep disappearing, which is rater annoying, to say at least.

Couple of hours later I finally found a crude but working solution. I just create a simple event handler, which re renders all by buttons...


$(document).ajaxComplete(function() {
    $("input:submit, a.button, input:button, div.qq-upload-button").button()
});

$(document).ready(function() {
    $("input:submit, a.button, input:button, div.qq-upload-button").button()
});

scala and instanceOf -

2011-11-30T16:49:00.000-08:00

I'm for some reason a fan of instanceOf in java and just love to be able to check at runtime of what kind a class is and do some action depending on this outcome.

But sometimes it's a bit tedios. For example



Object a = new String("tada");
Object result = null; 

if(a instanceOf Number){
  result = "its a number";
}
else if(a instanceOf String){
  result = "its a string";
}


System.out.println(result);

ok it's a silly example, now the pretty scala approach using match



 var a:Any = "tada..."

      val result = a match {
        case n:Number => "it's a number"
        case s:String => "it's a string"
      }
      println(result)

oh so pretty and easy to read.

casting and scala

2011-11-28T16:40:00.000-08:00

since a couple of month's I work more and more with scala and really start to like this language. But some of the concepts are really confusing and make me wounder if this really is necceasary.

A prime example would be the whole casting of variables.

In java you simple do:


String value = (String)object;

now in scala


val value:String = object.asInstanceOf[String]

now I admit it looks very clean, but sure took a while to find this one little details.

Otherwise it's always better to use matchers.

rocks linux - virtual hosts with apache

2011-11-04T09:35:00.000-07:00

currently I have a cluster setup in the office with about 5 nodes and a bit over 50 cpus, so this morning I decided to rebuild some of the nodes, since I needed to make some changes to the cluster.

20 minutes into this procedure, I keep getting odd error messages, like file 'update.img' not found and so.

So while backtracking the latest changes I did to the server. I realized that I setup 20 virtual hosts in the apache configuration, which ended up screwing with the kickstart configuration. It turns out that the order of the virtual host seems to quite important and that the kickstart configuration always needs to be first, and than you have to define the other virtual hosts.

Example of a virtual host configuration, which allows the kickstart configuration to work,


vim /etc/httpd/conf.d/rocks.conf

actual file:


<IfModule mod_mime.c>
AddHandler cgi-script .cgi
</IfModule>

UseCanonicalName Off


DirectoryIndex index.cgi

<Directory "/var/www/html">
Options FollowSymLinks Indexes ExecCGI
AllowOverride None
Order allow,deny
Allow from all
</Directory>

<Directory "/var/www/html/proc">
Options FollowSymLinks Indexes ExecCGI
AllowOverride None
Order deny,allow
Allow from 10.1.0.0/255.255.0.0
Allow from 127.0.0.1
Deny from all
</Directory>

<Directory "/var/www/html/pxelinux">
Options FollowSymLinks Indexes ExecCGI
AllowOverride None
Order deny,allow
Allow from 10.1.0.0/255.255.0.0
Allow from 127.0.0.1
Deny from all
</Directory>

<VirtualHost *:80>
ServerName kickstart.host.com
DocumentRoot "/var/www/html"
</VirtualHost>

<VirtualHost *:80>
ServerName virtual.host.com
DocumentRoot "/var/www/cts/html"
</VirtualHost>

grails quartz plugin - don't repeat jobs

2011-11-02T14:42:00.000-07:00

well it might be obvious, but sometimes you just want to execute a simple job once during the startup and than let them be.

For an example I needed this to synchronize a database and run statistics over it. So the simples trigger I could come up with, was


    static triggers = {
        simple name: 'synchronize', startDelay: 60000, repeatCount: 0
    }

An alternative way could had been todo schedule the job during the startup phase


SynchronizeExperimentJob.triggerNow([experimentId: id, database: database])

this line should be put into your bootstrapping file.

Personally I feel the trigger is more elegant.

kinesis avantage pro - day one

2011-11-01T11:40:00.000-07:00

so the keyboard got finally delivered and now Im in the progress of learning typing again and to say i'm inefficient is quite and understatement. I mean the layout is so weird and my hands feel way to big for this keyboard and i'm constantly hunting for the letters.

most frustrating things so far:

- only one space button instead of two
- i'm using muscles in my hand I never used before, I'm actually getting sore from typing after just a minute or two
- the enter button is in the most awkward spot ever, why cant the keyboard be a bit wider, so that there can be a dedicated enter button to be hit with my right pinky and a dedicated backspace button for the left pinky
- the right side of the keyboard if way more confusing than the left side
- the buttons are just a touch to large
- no dedicated number block, only a funky food pedal to switch some buttons around

rocks 5.4 cluster - installing subversion

2011-10-27T20:35:00.000-07:00

sometimes the most trivial things turn out to be rather interessting.

For example my latest drama with rocks was that I just needed subversion, to convert one of my BinBase images to a development image for BinBase.

Little did I know, that in the rocks 5.4 version, the installer fails to provide you with the base repository. Quality control anyone?

Anyway, to get your yum to work, as exspected. Just open your


/etc/yum.conf

file and add the following section



[base]
name=CentOS-$releasever - Base
mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=os

at the end of the file and than you can install subversion like always using:



yum -y install subversion

google is weird...

2011-10-26T20:20:00.000-07:00

ok I love the autocomplete function in google, but sometimes it just results in to odd of search recommendations...

example:

The fun part is that robin and I were actually researching for smurf costumes (it's bring your smurf to school day on monday...), but that the first suggestion on 'do sm...'

do smurfs have tails

I really did not see that comming

grouping dates in postgres using date_trunc

2011-10-21T14:39:00.000-07:00

well recently I started to write some statistics functions for my miniX application and so I was woundering, how can you easily execute queries like:

give me the count of object for the last 24h grouped by hour

which should be rather simple, if you know about the functions 'date_trunc' in postgres.

So 5 minutes later some sql appeared...

       
"select max(object_count) as object,date_trunc('hour',created_at) as date from bbstatistic_snapshot where object_name = ? and created_at between ? and ? group by date_trunc('hour',created_at) order by date_trunc('hour',created_at)"

which works rather well and get's the job done, but the speed is a touch slow.

Results should look like:

object | date
--------+---------------------
1742 | 2011-10-13 15:00:00
1742 | 2011-10-13 16:00:00
1742 | 2011-10-13 17:00:00
1742 | 2011-10-13 18:00:00
2102 | 2011-10-13 19:00:00
2102 | 2011-10-13 20:00:00
2057 | 2011-10-13 21:00:00
1899 | 2011-10-13 22:00:00
1803 | 2011-10-13 23:00:00
1742 | 2011-10-14 00:00:00
1742 | 2011-10-14 01:00:00
1742 | 2011-10-14 02:00:00
1742 | 2011-10-14 03:00:00
1742 | 2011-10-14 04:00:00
1742 | 2011-10-14 05:00:00
1742 | 2011-10-14 06:00:00

reinstalling a compute node in rocks

2011-10-19T11:54:00.001-07:00

this is actually a no brainer, but since I barely ever have todo it, here is the procedure:

log onto compute node
execute
```
/boot/kickstart/cluster-kickstart-pxe
```
wait for the install to finish

that's it

cron - useful triggers

2011-10-11T12:53:00.000-07:00

just a nice list of commonly used cron triggers I keep forgetting.

Cron Expression to fire the trigger every five minutes:
0 0/5 * * * ?
(e.g. 3:00:00, 3:05:00, 3:10:00

Cron Expression to fire the trigger every hour:
0 0 * * * ?

Cron Expression to fire the trigger every two Hours:
0 0 0/2 * * ?

Cron expression to fire the trigger every Four Hours:
0 0 0/4 * * ?

Cron Expression to fire the trigger every 10 Minutes 20 Seconds:
20 0/10 * * * ?
(e.g. 10:00:20, 10:10:20, 10:20:20)

Every half an hour between 1 AM to 7 AM on 20th of Every Month
0 0/30 1-7 5, 20 * ?
(e.g. 2010-07-20 1:00:00, 2010-07-20 1:30:00m 2010-07-20 2:00:00)

Cron Expression to fire the trigger on the 25th Minute from 1 AM to 5 PM on every Monday and Wednesday
0 25 1-5 ? * MON,WED

Cron Expression to fire the trigger at Noon every day
0 0 12 * * ?

Cron Expression to fire the trigger at Noon every day only in 2010
0 0 12 * * 2010

Cron Expression to fire the trigger Every 10 minutes between 9 AM to 5 PM (until 5:50) Every Day
0 0/10 9, 17 * * ?

Cron Expression to fire the trigger at 2:30 PM on the last Saturday of every month.
0 30 14 ? * 7L

Cron Expression to fire the trigger at 7:10 AM every Monday, Tuesday, Wednesday, Thursday and Friday
0 10 7 ? * MON-FRI

Cron Expression to fire the trigger at 2:10 PM and at 2:40 PM every Sunday in the month of July.
0 10,40 14 ? 7 SUN

source: cron-expression-to-fire-the-trigger-every-hour

groovy - a quick way to copy a list of files to the local directory

2011-08-10T12:54:00.000-07:00

don't we love these tasks? We get a list of files to look at and they are in a directory with thousands of other files and now we need to copy them one, by one, by one and there is no clear patter, which could ease the pain of doing so.

But like always there is a simple groovy way todo this in a quicker fashion.


def values = [
1333020,
1332872,
1332428,
1332280,
1331984,
1331836,
1331688,
1331540
]

values.each{def value ->
new File("${value}.xml") << new File("/Users/****/Documents/metadata/${value}.xml").text
}

this does simplify your life once in a while to copy a quick list of text files to a different folder.

remote ssh forwarding

2011-08-09T15:12:00.000-07:00

for some reason I always forget the syntax to forward a port on my server to my local system to work remotely from home on some of my webapps.

Now to not forget it again, I shall write it down on my blog, so I can just look at it...


 ssh -L port:remoteServer:port remoteServer

now this let's me access the 'port' on my 'remoteServer' on my local system and work on the webapp. Now I just have to rewrite all the absolute paths to relative paths, so that I can find my css stylesheets too...

grails + jquery + checkboxes

2011-08-08T16:17:00.000-07:00

sometimes forms based on checkboxes can be rather annoying. An example would be that this is the return of a standard html checkbox,

html defined checkboxes:


<g:checkBox name="compare" id="1"/>
<g:checkBox name="compare" id="2"/>
<g:checkBox name="compare" id="3"/>

result of the send parameters to the server:



    def compareBins = {
        println params
    }

out:

compare:[on, on, on]

now this is rather useless, we rather would see the id of the checkbox in the result. So let's call jQuery to the rescue and see how it can save out day!



    jQuery(document).ready(function() {
        $('input:checkbox').each(
        function() {
            $(this).click(function() {

                if (this.checked) {
                    var value = $(this).attr('id');
                    $(this).val(value);
                }
            });
        }
        );
    });

and promptly we have the following result if all 3 boxes are selected


compare:[1,2,3]

or if 1 and 3 are selected


compare:[1,3]

so we can actually tell, which elements where checked instead of getting just 'on'

Arduino + Ethernet Shield + Temperature 2

2011-08-04T15:57:00.000-07:00

After monitoring the thermometer for a couple of hours, it turned out the temperature sensor is a bit off. About 2.5 degrees.
At least compared to me being cold and a good old HG thermometer.

Now this meant we need some improvement in our little thermometer or better, let's find a way to calibrate it.

So looking around in the tool box I quickly discovered a potentiometer and some more cables and thought, we can wire this to another pin and scale it to +/- 5 degree celsius and apply this correction.

This is how it roughly looks now:

And the new code to apply the correction settings is here


#include <SPI.h>
#include <Ethernet.h>


byte mac[] = { 0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xED };
byte ip[] = { 192,168,1, 2 };

Server server(80);

float coreVoltage = 5.0;
float correctionScale = 10; //in degree celsius +/- half scale

void setup()
{
// start the Ethernet connection and the server:
Ethernet.begin(mac, ip);
server.begin();
}

void loop()
{
// listen for incoming clients
Client client = server.available();
if (client) {
// an http request ends with a blank line
boolean currentLineIsBlank = true;
while (client.connected()) {
if (client.available()) {
char c = client.read();
// if you've gotten to the end of the line (received a newline
// character) and the line is blank, the http request has ended,
// so you can send a reply
if (c == '\n' && currentLineIsBlank) {
// send a standard http response header
client.println("HTTP/1.1 200 OK");
client.println("Content-Type: application/json");
client.println();

// output the value of each analog input pin


//read twice to improve precission
float voltageSensor = analogRead(0);
voltageSensor = analogRead(0) * coreVoltage;

//get temperature of the potentio meter
float voltagePotentioMeter = analogRead(1);
voltagePotentioMeter = analogRead(1)*coreVoltage/1000;

float debugVotlage = voltagePotentioMeter;

//scale it to +/- 1/2 coreVoltage
voltagePotentioMeter = voltagePotentioMeter - (coreVoltage/2);

//calculate the correction value in celsius, based on our scale
float correctionValue = voltagePotentioMeter * correctionScale / coreVoltage ;

voltageSensor /= 1024.0; 

//temperature calculation
float temperatureC = (voltageSensor - 0.5) * 100 + correctionValue;  //converting from 10 mv per degree wit 500 mV offset
float temperatureF = (temperatureC * 9.0 / 5.0) + 32.0;


client.print("sensor {");

client.print("temperature_voltage:");
client.print(voltageSensor);
client.print(",");


client.print("temperature_celsius:");


client.print(temperatureC);
client.print(",");

client.print("temperature_fahrenheit:");

client.print(temperatureF);
client.print(",");

client.print("temperature_correction:");

client.print(correctionValue);
client.print(",");

client.print("voltage_potentiometer:");
client.print(debugVotlage);
client.print("}");

break;
}
if (c == '\n') {
// you're starting a new line
currentLineIsBlank = true;
} 
else if (c != '\r') {
// you've gotten a character on the current line
currentLineIsBlank = false;
}
}
}
// give the web browser time to receive the data
delay(1);
// close the connection:
client.stop();
}
}

Also since we want to keep track of the correction values, we updated the expected json string a bit to reflect the changes in our implementation.


sensor {temperature_voltage:0.75,temperature_celsius:21.98,temperature_fahrenheit:71.56,temperature_correction:-3.22,voltage_potentiometer:0.89}

I have to say, I'm really start to enjoy playing with this little thing. The next step should be have the transmission of several sensors done wirelessly over XBees and to send them to one central location...

Ardunio + Ethernet shield

2011-08-04T13:03:00.000-07:00

After I received the ethernet shield, my first task was it to figure out how to monitor the temperature in my office over the network and possible in a way, which allows me to store this result in a database.

So this got me thinking, JSON is the way togo.

easy to create
easy to read
possible to be understood without an interpreter
very lightweight
everybody uses it

So let's see how the code would look for the arduino


#include <spi.h>
#include <ethernet.h>

byte mac[] = { 0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xED };
byte ip[] = { 192,168,1, 2 };

Server server(80);

void setup()
{
// start the Ethernet connection and the server:
Ethernet.begin(mac, ip);
server.begin();
}

void loop()
{
// listen for incoming clients
Client client = server.available();
if (client) {
// an http request ends with a blank line
boolean currentLineIsBlank = true;
while (client.connected()) {
if (client.available()) {
char c = client.read();
// if you've gotten to the end of the line (received a newline
// character) and the line is blank, the http request has ended,
// so you can send a reply
if (c == '\n' && currentLineIsBlank) {
// send a standard http response header
client.println("HTTP/1.1 200 OK");
client.println("Content-Type: application/json");
client.println();

// output the value of each analog input pin


//read twice to improve precission
float voltage = analogRead(0) * 5.0;
voltage = analogRead(0) * 5.0;

voltage /= 1024.0; 

float temperatureC = (voltage - 0.5) * 100 ;  //converting from 10 mv per degree wit 500 mV offset
float temperatureF = (temperatureC * 9.0 / 5.0) + 32.0;

client.print("sensor {");

client.print("temperature_voltage:");
client.print(voltage);
client.print(",");


client.print("temperature_celsius:");


client.print(temperatureC);
client.print(",");

client.print("temperature_fahrenheit:");

client.print(temperatureF);
client.print("}");


break;
}
if (c == '\n') {
// you're starting a new line
currentLineIsBlank = true;
} 
else if (c != '\r') {
// you've gotten a character on the current line
currentLineIsBlank = false;
}
}
}
// give the web browser time to receive the data
delay(1);
// close the connection:
client.stop();
}
}

now this was rather simple and results in the following json string on port 80 of the arduino address


sensor {temperature_voltage:0.74,temperature_celsius:24.22,temperature_fahrenheit:75.59}

Now back to my actual project, trying to monitor the PH value and ammonia value with an arduino and adjust the ph, if needed...

MedPlan

2011-08-02T15:43:00.000-07:00

Over the last couple of days I was thinking about creating a small grails to show the speed of development with grails. The goal was to create an application in less than 24h, which is able to track your medications and at the same time has some important features.

The original idea came from spending the last 5 days with my wife's sick sister and see how difficult and confusing it was to track all her different medications. Specially since there were 3 people involved doing this and nobody really know, which medications did she already receive.

What does the user want:

easily enter a new prescription
see which medications have been taken
show when to take a medication
show the medication's in form of a calendar

What did I want as a developer:

easy to use
multi user access
ajax driven/jquery
calendar interface
auto complete
multi platform support (ipad/windows/linux/osx)
able to run without a tomcat installation
don't look like a standard grails scaffolding application
use as few mouse clicks as possible

how does it look after 16h of work?

weekly overview

prescription overview

adding a prescription

What is missing?

I think at this point in time the application does what it's supposed todo, but some nice features would be:

iPad support as a native application
internationalization
share your prescription plan with another user
send out an email, 30 minutes before a medication has to be used
soap/rest based access

Where can I get it?

You can download the application here

grails - export plugin, at dynamically more attributes

2011-07-25T16:57:00.000-07:00

What is this about?

currently I'm writing on a little tech study application, called times. Which can be accessed here

This application is a very simplistic timetracker, but allows me to play with some api's I normally don't use or to simply test some grails functions.

Right now I had a simple idea.

I want to export all my data, using the grails export plugin, without cluttering my database model with dozens of attributes, which will never be queried. I also did not want to create pojo's, just for the export. Instead I decided to play a bit with the EMC MetaClass again and see if, we can do this on the fly.

Example:



def exports = []

def fields = ["name","organization","project","beginDate","endDate","timeSpend"]
def labels = ["name":"Name","organization":"Organization","project":"Project","beginDate":"Start","endDate":"Finished","timeSpend":"Durations (s)"]
Task.list().each {Task t ->
if (t.endDate != null) {
t.metaClass.timeSpend = (t.endDate.time - t.beginDate.time)/1000

exports.add(t)
}
}
exportService.export(params.format, response.outputStream, exports,fields,labels, [:], [:])

this registers the attribute 'timeSpend' to the lifespan of the Task instance 't'. Which is only be required during the export.

The important line is



t.metaClass.timeSpend = (t.endDate.time - t.beginDate.time)/1000

Short, it can simplify your life a bit and reduces the need of introducing a pojo, just to export data or to permanently add this object to your database. Specially since it's a calculated value and can be regenerated on the fly.

openCl - not recognizing GPU, during remote logins

2011-07-18T11:44:00.000-07:00

in the last couple of days I had some issues with my Ubuntu/Ati GPU based calculation units. They refused to execute any calculations, whenever I tried to use SSH to run them.

So a bit of googleling later I found the following informations in a hidden pdf:



a. Add the following lines at the end of /etc/gdm/Init/Default, 
before the exit 0, to modify the security settings, allowing remote 
sessions to access the X server and ensuring that remote sessions have 
access to the necessary device files when communicating with the GPU:
xhost +
chmod uog+rw /dev/ati/card*
b. If you normally use bash, add the following line to the end of 
/etc/bashrc file to ensure remote sessions know which X server to 
access.
case $DISPLAY in ’’) export DISPLAY=:0;; *) ;; esac
NOTE: ’ ’ are two single quotes, not a single double-quot

Taken from:

http://developer.amd.com/sdks/AMDAPPSDK/assets/App_Note-Running_AMD_APP_Apps_Remotely.pdf

Now everything works fine and all is dandy again.

Groovy HTTP Builder- fetching JSON data over post and supply parameters

2011-07-10T21:20:00.000-07:00

Right now I'm playing around with the MtGox api to issue online transactions to buy and sell bitcoins. Yes I'm still obsessed with these and don't think this is going to change anytime soon, since It's starting to actually generate money.

But the big pain right now is that you have togo to so many different websites todo a simple transaction and I really want to have this all combined in one single tool.

So the api exspects from you to be queried over post arguments and the first thing which came to my mind was to use the fantastic HttpBuilder in groovy.

so let's define a simple method to simplify life a bit for us


private executeQuery(Map parameter, String path) {

def http = new HTTPBuilder("https://mtgox.com/")

def result = ""
http.post(body: parameter, path: path, requestContentType: URLENC) { resp, json ->

if (resp.statusLine.statusCode == 200) {
result = json
}
else {
result = false
}

}

return result

and a second method to actually call this


def getCurrentBalance() {

if (MtGoxAccessHandler.isConfigured()) {
def values = executeQuery([

name: MtGoxAccessHandler.getUserName(), 
pass: MtGoxAccessHandler.getPassword()], 
"/code/getFunds.php")

return [usd: values.usds, coins: values.btcs]
}
else {
return false
}
}

The class MtGoxAccessHandler is just a little helper, which stores the username and password and allows me to simplify the code a bit and easy testing.

This post is also related to the google code project DeepBitView

JSON and SSL in groovy, how to ignore certificates

2011-07-04T14:27:00.000-07:00

In the last couple of days I became more and more interested in bitmining and the first thing I noticed was the utter lack of integrated tools and hence statistics I would like to associated with my miners.

This resulted in me creating a tiny grails application to synchronzie my deepbit statistics with the current market data from mtgox to calculate at which point it becomes pointless to mine these thing. I basically hope that this pays for the 700$ for my 2x6950 radeons and associated cooling the 100+F heat we have here in California.

Now originally you could assume that this is rather straightforward. After all you got json data, which just need to be parsed.


Map jsonArray = JSON.parse(new InputStreamReader(new URL("https://mtgox.com/code/data/ticker.php").openStream()))

println jsonArray

but sadly the java ssl security manager does not agree here with us and tosses a fit..



javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at com.sun.net.ssl.internal.ssl.Alerts.getSSLException(Alerts.java:174)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1649)

Now java is kinda smart, if want's to force you install a security certificate from the server you access and authorize yourself. But sometimes you don't want this. In this case it's overkill, since we are having a readonly transaction.

So what has to be done?

you need to create your own security manager implementation. Which basically is a 3 step procedure

implement an interface
class TrustManager implements X509TrustManager { public java.security.cert.X509Certificate[] getAcceptedIssuers() { return null; } public void checkClientTrusted( java.security.cert.X509Certificate[] certs, String authType) { } public void checkServerTrusted( java.security.cert.X509Certificate[] certs, String authType) { } }
register the interface and update the context configuration
TrustManager[] trustAllCerts = new TrustManager[1] trustAllCerts[0] = new TrustManager() try { SSLContext sc = SSLContext.getInstance("SSL"); sc.init(null, trustAllCerts, new java.security.SecureRandom()); HttpsURLConnection.setDefaultSSLSocketFactory(sc.getSocketFactory()); } catch (Exception e) { }
and continue like always with your json script
Map jsonArray = JSON.parse(new InputStreamReader(new URL("https://mtgox.com/code/data/ticker.php").openStream()))

and the result should be now the current ticker for the USD to BitCoin exchange rate.


{"ticker":{"high":15.4989,"low":13.31415,"avg":14.726503862,"vol":42862,"last":13.31415,"buy":13.331,"sell":13.35999}}

groovy - parsing reading builder content from an external file

2011-06-22T15:47:00.000-07:00

we all know that groovy builders are great and fun and can make your life so much easier.

For example if you want to create a simple xml file in your groovy script you can just do the following


def writer = new StringWriter()
def xml = new MarkupBuilder(writer)
xml.records() {
  car(name:'HSV Maloo', make:'Holden', year:2006) {
    country('Australia')
    record(type:'speed', 'Production Pickup Truck with speed of 271kph')
  }
  car(name:'P50', make:'Peel', year:1962) {
    country('Isle of Man')
    record(type:'size', 'Smallest Street-Legal Car at 99cm wide and 59 kg in weight')
  }
  car(name:'Royale', make:'Bugatti', year:1931) {
    country('France')
    record(type:'price', 'Most Valuable Car at $15 million')
  }
}

looks easy and is easy. But now you come to the point where you think. Mhm it would be nice, if I could just read the builder code from an external file and generate the xml on the fly using this. For example you have to generate an xml configuration file and rather simplify this process using groovy and this has to be done more than once...

After a couple of hours googleing the best solution I came up with, was to provide your own binding to the builder, which simplified things quite a bit.



class XmlBinding extends Binding{
 def builder
 
 
 Object getVariable(String name) {
  return { Object... args ->  builder.invokeMethod(name,args) }
 }
}

let's assume we like to read the following builder code, defined in a file named 'test.groovy'



   config{
    bin{
     allow(minimumClassSize:1)
    }
   }

which all you need to do, now you could for example have a main method somewhere containing this:


  def groovyScriptContent = new File("test.groovy").text
  def writer = new StringWriter()
  def xml = new MarkupBuilder(writer)
  def binding = new XmlBinding()
  binding.builder = xml
  
  def dslScript = new GroovyShell().parse(groovyScriptContent)
  dslScript.binding =  binding
  dslScript.run()
  
  println writer.toString()

and as a result it generates the following xml document


<config>
  <bin>
    <allow minimumClassSize='1' />
  </bin>
</config>

vmware fusion - changing nat address range on the host

2011-03-17T10:52:00.000-07:00

recently I was installing a lot of vmware virtual machines on 3 different macs to test some binbase issues and it turns out the vmware fusion assigns a random iprange on every box.

Rather annoying if you virtual machine needs a fixed IP, thank you rocks guys for this...

Anyway to change the vmnet8 interface configuration you need to modify the following files:



sudo vim /Library/Application\ Support/VMware\ Fusion/vmnet8/nat.conf
sudo vim /Library/Application\ Support/VMware\ Fusion/vnetworking
sudo vim /Library/Application\ Support/VMware\ Fusion/vmnet8/dhcpd.conf

and update the default range with you wished range.

afterwards execute:


./boot.sh --restart

in the directory:


 /Library/Application\ Support/VMware\ Fusion/

and you should be good togo!

More about this can be found at http://blog.mclaughlinsoftware.com/2010/03/01/vmware-fusion-nat/

locking an application to a space in OSX Snowleopard

2011-02-10T12:02:00.001-08:00

since i have several monitors connected to my max I like to have some apps always in one window on the same screen and not have to keep searching for it.

Most importantly I always want to keep my browser and terminal on the right screen and on the left screen I have several different IDE's.

Like...

eclipse in space 1
intellij in space 2
etc

solution?

Go to your System Preferences and enable Spaces.
Now, in the box in the middle of the Spaces settings that says Application Assignments, add an application using the + button.
Just navigate to the chosen program in your Applications folder.
Now the trick here is in the Space column to choose Every Space from the picklist.
rinse and repeat for all applications

registering jboss to start by default and comforms to chkconfig

2010-12-21T12:37:00.000-08:00

currently I had yet another binbase install and it turned out that my standard jboss script is not chkconfig compatible.

So it was time todo a rewrite


#! /bin/bash
#
# network Bring up/down jboss
#
# chkconfig: 345 20 80
# description: Starts and stops the jboss server
#
# /etc/rc.d/init.d/jboss
# See how we were called.

#define where jboss is - this is the directory containing directories log, bin, conf etc
JBOSS_HOME=/share/apps/jboss

#define the user under which jboss will run, or use 'RUNASIS' to run as the current user
JBOSS_USER=${JBOSS_USER:-"root"}

#make sure java is in your path
JAVAPTH=/usr/java/latest/bin/java

#configuration to use, usually one of 'minimal', 'default', 'all'
JBOSS_CONF=${JBOSS_CONF:-"all"}
JBOSS_HOST=0.0.0.0

JBOSS_CONSOLE=$JBOSS_HOME/console.log
#if JBOSS_HOST specified, use -b to bind jboss services to that address
JBOSS_BIND_ADDR=${JBOSS_HOST:+"-b $JBOSS_HOST"}

#define the classpath for the shutdown class
JBOSSCP=${JBOSSCP:-"$JBOSS_HOME/bin/shutdown.jar:$JBOSS_HOME/client/jnet.jar"}

#define the script to use to start jboss
JBOSSSH=${JBOSSSH:-"$JBOSS_HOME/bin/run.sh -c $JBOSS_CONF $JBOSS_BIND_ADDR"}

if [ "$JBOSS_USER" = "RUNASIS" ]; then
  SUBIT=""
else
  SUBIT="su - $JBOSS_USER -c "
fi

if [ -n "$JBOSS_CONSOLE" -a ! -d "$JBOSS_CONSOLE" ]; then
  # ensure the file exists
  touch $JBOSS_CONSOLE
  if [ ! -z "$SUBIT" ]; then
    chown $JBOSS_USER $JBOSS_CONSOLE
  fi
fi

grails - providing your own plugins configuration

2010-12-18T01:13:00.000-08:00

sometimes you want to share some services in a plugin, which needs to be configured at runtime.

The easiest way is to provide a simple file in the grails conf diretory of your plugin and just read it and access it at runtime.

How to?

our little helper:


package binbase.core

import grails.util.GrailsUtil
import org.codehaus.groovy.grails.commons.GrailsApplication

/**
 * Created by IntelliJ IDEA.
 * User: wohlgemuth
 * Date: 12/18/10
 * Time: 1:00 AM
 * To change this template use File | Settings | File Templates.
 */
class BinBaseConfigReader {

  private static ConfigObject config = initialize()

  /**
   * initializes the object
   * @return
   */
  static ConfigObject initialize() {
    return new ConfigSlurper().parse(new GroovyClassLoader(BinBaseConfigReader.class.getClassLoader()).loadClass('BinBaseConfig'))
  }

  /**
   * returns the server
   * @return
   */
  static String getServer() {
    if (GrailsUtil.getEnvironment().equals(GrailsApplication.ENV_TEST)) {
      return config.binbase.test.server.toString()
    }
    else if (GrailsUtil.getEnvironment().equals(GrailsApplication.ENV_DEVELOPMENT)) {
      return config.binbase.development.server.toString()
    }
    else if (GrailsUtil.getEnvironment().equals(GrailsApplication.ENV_PRODUCTION)) {
      return config.binbase.production.server.toString()

    }
    else {
      throw new RuntimeException("unexspected enviroment found")
    }
  }
}

our configuration


/**
 * this file contains the binbase configuration for this plugin.
 *
 */
binbase {
  production {
    key = "dasdasd"
    server = "10.1.1.1"
  }
  test {
    key = "dasdasd"
    server = "10.2.2.2"
  }
  development {
    key = "dasdasd"
    server = "127.0.0.1"
  }
}

and to access it in your, say spring configuration?


  jndiBinBaseTemplate(org.springframework.jndi.JndiTemplate) {

    environment = [
            "java.naming.factory.initial": "org.jnp.interfaces.NamingContextFactory",
            "java.naming.factory.url.pkgs": "org.jboss.naming:org.jnp.interfaces",
            "java.naming.provider.url": "${BinBaseConfigReader.getServer()}:1099".toString()
    ]
  }

easy, isn't it?

grails - testing service, which accesses ejb's

2010-12-18T00:06:00.000-08:00

this night I just wanted to write a quick grails plugin to access my BinBase database system.

Since I'm a smart developer I start with a simple spring dsl file and write a test.



// Place your Spring DSL code here
beans = {

  jndiBinBaseTemplate(org.springframework.jndi.JndiTemplate) {

    environment = [
            "java.naming.factory.initial": "org.jnp.interfaces.NamingContextFactory",
            "java.naming.factory.url.pkgs": "org.jboss.naming:org.jnp.interfaces",
            "java.naming.provider.url": "localhost:1099"
    ]
  }

  /**
   * connection to the cluster configuration
   */
  clusterConfigService(SimpleRemoteStatelessSessionProxyFactoryBean) { bean ->
    businessInterface = "edu.ucdavis.genomics.metabolomics.binbase.cluster.ejb.delegate.ClusterConfigService"
    jndiName = "clusterservice/ClusterConfigServiceBean/remote"
    jndiTemplate = ref(jndiBinBaseTemplate)
  }

}

and the test code is simple enough



class BinBaseSchedulingServiceTests extends GroovyTestCase {


  ClusterConfigService clusterConfigService

  protected void setUp() {
    super.setUp()
  }

  protected void tearDown() {
    super.tearDown()
  }

  public void testTest(){
    assertTrue(clusterConfigService.username != null)
  }
}

the variable 'ClusterConfigService' is an stateless session bean in a jboss container and I know it works...

except grails is complaining:



org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'clusterConfigService': Invocation of init method failed; nested exception is javax.naming.NameNotFoundException: Name [clusterservice/ClusterConfigServiceBean/remote] not bound; 0 bindings: []
 at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1412)
 at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:519)
 at org.codehaus.groovy.grails.commons.spring.ReloadAwareAutowireCapableBeanFactory.doCreateBean(ReloadAwareAutowireCapableBeanFactory.java:135)
 at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)
 at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:291)
 at 
....lots and lots of more...
Caused by: javax.naming.NameNotFoundException: Name [clusterservice/ClusterConfigServiceBean/remote] not bound; 0 bindings: []
 at org.springframework.mock.jndi.SimpleNamingContext.lookup(SimpleNamingContext.java:132)
 at javax.naming.InitialContext.lookup(InitialContext.java:392)
 at org.springframework.jndi.JndiTemplate$1.doInContext(JndiTemplate.java:154)
 at org.springframework.jndi.JndiTemplate.execute(JndiTemplate.java:87)
 at org.springframework.jndi.JndiTemplate.lookup(JndiTemplate.java:152)
 at org.springframework.jndi.JndiTemplate.lookup(JndiTemplate.java:178)
 at org.springframework.jndi.JndiLocatorSupport.lookup(JndiLocatorSupport.java:95)
 at org.springframework.jndi.JndiObjectLocator.lookup(JndiObjectLocator.java:105)
 at org.springframework.ejb.access.AbstractRemoteSlsbInvokerInterceptor.lookup(AbstractRemoteSlsbInvokerInterceptor.java:100)
 at org.springframework.ejb.access.AbstractSlsbInvokerInterceptor.refreshHome(AbstractSlsbInvokerInterceptor.java:122)
 at 
...lots and lots of more...
[23:59:35,938] [ERROR] [main] [StackTrace] [Sanitizing stacktrace:]
javax.naming.NameNotFoundException: Name [clusterservice/ClusterConfigServiceBean/remote] not bound; 0 bindings: []
 at org.springframework.mock.jndi.SimpleNamingContext.lookup(SimpleNamingContext.java:132)
 at javax.naming.InitialContext.lookup(InitialContext.java:392)
 at org.springframework.jndi.JndiTemplate$1.doInContext(JndiTemplate.java:154)
 at org.springframework.jndi.JndiTemplate.execute(JndiTemplate.java:87)
 at org.springframework.jndi.JndiTemplate.lookup(JndiTemplate.java:152)

mhm doesn't make much sense. Reason for the failure is that grails decides to mock the access, which is normally ok. But in our case really really not desired.

After all we want to access the application server context!

But it keeps jumping to the 'org.springframework.mock.jndi.SimpleNamingContext' mocking context.

So the only solution is this slightly dirty hack...


/***
 * we do not need mocking in this plugin...
 */
if (org.springframework.mock.jndi.SimpleNamingContextBuilder.currentContextBuilder) {
  org.springframework.mock.jndi.SimpleNamingContextBuilder.currentContextBuilder.deactivate()
}

and promptly it passes.

Since we only want to change this for test cases, we add yet another conditon and our resources.groovy file ends up looking like


import org.springframework.ejb.access.SimpleRemoteStatelessSessionProxyFactoryBean
import grails.util.GrailsUtil
import org.codehaus.groovy.grails.commons.GrailsApplication

/** *
 * we do not need mocking in this plugin...
 */
if (GrailsUtil.getEnvironment().equals(GrailsApplication.ENV_TEST)) {
  println "warning disabling mocking context!!! to make remote ejb's work"
  if (org.springframework.mock.jndi.SimpleNamingContextBuilder.currentContextBuilder) {
    org.springframework.mock.jndi.SimpleNamingContextBuilder.currentContextBuilder.deactivate()
  }
}

// Place your Spring DSL code here
beans = {

  jndiBinBaseTemplate(org.springframework.jndi.JndiTemplate) {

    environment = [
            "java.naming.factory.initial": "org.jnp.interfaces.NamingContextFactory",
            "java.naming.factory.url.pkgs": "org.jboss.naming:org.jnp.interfaces",
            "java.naming.provider.url": "eros.fiehnlab.ucdavis.edu:1099"
    ]
  }

  /**
   * connection to the cluster configuration
   */
  clusterConfigService(SimpleRemoteStatelessSessionProxyFactoryBean) { bean ->
    businessInterface = "edu.ucdavis.genomics.metabolomics.binbase.cluster.ejb.delegate.ClusterConfigService"
    jndiName = "clusterservice/ClusterConfigServiceBean/remote"
    jndiTemplate = ref(jndiBinBaseTemplate)
  }

}

Grails 1.3.5 webtests - custom steps to reset the database

2010-12-16T09:59:00.000-08:00

once in a while I have the trouble that I want to reset the complete database during webtests or to make a snapshot of the current database while running the test.

So we use the great dbunit plugin to dump the database in one step:


import javax.sql.DataSource
import grails.plugin.remotecontrol.RemoteControl
import org.dbunit.database.DatabaseConnection
import org.dbunit.dataset.IDataSet
import org.dbunit.database.IDatabaseConnection
import org.dbunit.dataset.xml.FlatXmlDataSet
/**
* Created by IntelliJ IDEA.
* User: wohlgemuth
* Date: Nov 24, 2010
* Time: 3:46:47 PM
* To change this template use File | Settings | File Templates.
*/
class DumpDatabaseStep extends com.canoo.webtest.steps.Step {

String fileName = "result.xml"
/**
* resets the database
*/
void doExecute() {

RemoteControl remote = new RemoteControl()
try {

def outputFile = fileName
remote {
File file = new File("target/test-reports/data")
file.mkdirs()

DataSource dataSource = ctx.dataSource

IDatabaseConnection connection = new DatabaseConnection(dataSource.connection)

IDataSet fullDataSet = connection.createDataSet()

File out = new File(file, outputFile)

int counter = 1;

boolean run = out.exists()
while (run) {
out = new File(file, "${counter}-${outputFile}")

run = out.exists()
if (counter == 100) {
println "killed after 100 runs.."
run = false
}
counter++
}

FlatXmlDataSet.write(fullDataSet, new FileOutputStream(out))

null
}
}
catch (Exception e) {
e.printStackTrace()

throw e
}

}
}

and also to reset the complete database in another step


import grails.plugin.remotecontrol.RemoteControl
import javax.sql.DataSource
import groovy.sql.Sql

/**
* simple step which resets the complete database
*/
class ResetDatabaseStep extends com.canoo.webtest.steps.Step {

/**
* resets the database
*/
void doExecute() {

try {
DbUnitOperator.create()


RemoteControl remote = new RemoteControl()

remote.exec {
try {
DataSource dataSource = ctx.dataSource
Sql sql = Sql.newInstance(dataSource)

sql.execute("DROP SEQUENCE MINIX_ID")
sql.execute("CREATE SEQUENCE MINIX_ID START WITH  1000 INCREMENT BY 1")
}
catch (Exception e) {
e.printStackTrace()
throw e
}
}


}
catch (Exception e) {
println "error: ${e.getMessage()}"
e.printStackTrace()
throw new RuntimeException(e)
}

}
}

Grails 1.3.5 webtests and JQuery trouble

2010-12-16T09:52:00.000-08:00

recently I migrated my prototype code over to JQuery and promptly all my webtest's failed with a lot of error messages and exceptions.

Like:


"TypeError: Cannot find function createComment in  
object [object]. (FunctionalTestCase.groovy#372)"

So during some googeling I found these reports:

old htmlunit version

replacing version

Which led to me reporting:

my approach

and the final solution was to put this into your 'BuildConfig.groovy'


plugin("webtest") {
test('net.sourceforge.htmlunit:htmlunit:2.8-SNAPSHOT') {
excludes 'xalan'
excludes 'xml-apis'
}
test('com.canoo.webtest:webtest:3.1-SNAPSHOT') {
excludes 'xalan'
excludes 'xml-apis'
}
test('xalan:xalan:2.7.0') {
excludes 'xml-apis'
}
}

grails many to many

2010-11-23T15:25:00.000-08:00

currently grails many to many relation ships are giving me a lot of grief with exceptions like



org.hibernate.StaleObjectStateException:
Row was updated or deleted by another transaction (or unsaved-value mapping was 
incorrect): [ClassName#id]

or constraint excpetions.

Which all turned out to be related that the optimistic locking fails from time to time and needs to be disabled.
With version = false in the domain mapping.

References:

disable optimistic locking
delete many to many relations

BinBase - generate histograms

2010-09-30T00:43:00.000-07:00

sql statements to generate histograms, to learn more about your data

similarity


select count("range"),"range" from 
(
    select
        case 
            when match > 500 and match <= 550 then '501 - 550'             when match > 550 and match <= 600 then '550 - 600'             when match > 600 and match <= 650 then '600 - 650'             when match > 650 and match <= 700 then '650 - 700'             when match > 700 and match <= 750 then '700 - 750'            when match > 750 and match <= 800 then '750 - 800'            when match > 800 and match <= 850 then '800 - 850'            when match > 850 and match <= 900 then '850 - 900'            when match > 900 and match <= 950 then '900 - 950'            when match > 950 and match <= 1000 then '950 - 1000'         end             AS "range" from spectra a, bin b where a.bin_id = b.bin_id and b.export = 'FALSE')a group by "range"

purity


select count("range"),"range" from 
(
    select
        case 
            when a.purity > 0 and a.purity <= 0.1 then '0 - 0.1'             when a.purity > 0.1 and a.purity <= 0.5 then '0.1 - 0.5'             when a.purity > 0.5 and a.purity <= 0.9 then '0.5 - 0.9'             when a.purity > 0.9 and a.purity <= 1.3 then '0.9 - 1.3'             when a.purity > 1.3 and a.purity <= 1.7 then '1.3 - 1.7'            when a.purity > 1.7 and a.purity <= 2.1 then '1.7 - 2.1'            when a.purity > 2.1 and a.purity <= 2.5 then '2.1 - 2.5'            when a.purity > 2.5  then '> 2.5'
         end 
            AS "range" from spectra a, bin b where a.bin_id = b.bin_id and b.export = 'TRUE'
)
a group by "range"

signal noise

BinBase - reset database

2010-09-28T12:46:00.001-07:00

simple sql script to reset the binbase database


delete from bin where bin_id not in (select bin_id from standard)
GO
delete from spectra where bin_id not in (select bin_id from standard)
GO
delete from correction_data
GO
delete from samples where sample_id not in (select sample_id from bin)
GO
delete from rawdata
GO
delete from result_link
GO
delete from result
GO
delete from runtime
GO
delete from quantification
GO

BinBase - simulate the sifter algorithm directly on the database

2010-09-27T20:15:00.001-07:00

currently I'm generating statistics for the BinBase system and so needed to simulate the annotation algorithm in sql.

It was rather simple....



-- 0.05 settings
select 
count(*) as "count", 
lower('0 < 0.05') as purity,    lower('> 500') as signalNoise,
lower('> 800') as similarity

from spectra where purity > 0 and purity < 0.05 and signal_noise > 500 and match > 800

UNION

select 
count(*) as "count", 
lower('0 < 0.05') as purity,    lower('> 50 && < 500') as signalNoise,    lower('> 700') as similarity

from spectra where purity > 0 and purity < 0.05 and signal_noise > 50 and signal_noise <= 500 and match > 700


UNION

select 
count(*) as "count", 
lower('0 < 0.05') as purity,    lower('> 5 && < 50') as signalNoise,    lower('> 600') as similarity

from spectra where purity > 0 and purity < 0.05 and signal_noise > 5 and signal_noise <= 50 and match > 600

-- 0.3 settings

UNION

select 
count(*) as "count", 
lower('0 < 0.3') as purity,    lower('> 500') as signalNoise,
lower('> 700') as similarity

from spectra where purity > 0 and purity < 0.3 and signal_noise > 500 and match > 700

UNION

select 
count(*) as "count", 
lower('0 < 0.3') as purity,    lower('> 50 && < 500') as signalNoise,    lower('> 600') as similarity

from spectra where purity > 0 and purity < 0.3 and signal_noise > 50 and signal_noise <= 500 and match > 600

UNION

select 
count(*) as "count", 
lower('0 < 0.3') as purity,    lower('> 5 && < 50') as signalNoise,    lower('> 500') as similarity

from spectra where purity > 0 and purity < 0.03 and signal_noise > 5 and signal_noise <= 50 and match > 500

-- all ither settings

UNION

select 
count(*) as "count", 
lower('> 0.3') as purity,
lower('> 500') as signalNoise,
lower('> 600') as similarity

from spectra where purity > 0.3 and signal_noise > 500 and match > 600

UNION

select 
count(*) as "count", 
lower('> 0.3') as purity,
lower('> 50 && < 500') as signalNoise,    lower('> 500') as similarity

from spectra where purity > 0.3 and signal_noise > 50 and signal_noise <= 500 and match > 500

UNION

select 
count(*) as "count", 
lower('> 0.3') as purity,
lower('> 5 && < 50') as signalNoise,    lower('> 450') as similarity

from spectra where purity > 0.03 and signal_noise > 5 and signal_noise <= 50 and match > 450

-- total --

union

select 
count(*) as "count", 
lower('no filter') as purity,
lower('no filter') as signalNoise,
lower('no filter') as similarity

from spectra

leco 2.22 settings


-- 0.1 settings
select 
count(*) as "count", 
lower('0 < 0.1') as purity,    lower('> 250') as signalNoise,
lower('> 800') as similarity

from spectra where purity > 0 and purity < 0.1 and signal_noise > 250 and match > 800

UNION

select 
count(*) as "count", 
lower('0 < 0.1') as purity,    lower('> 25 && < 250') as signalNoise,    lower('> 700') as similarity

from spectra where purity > 0 and purity < 0.1 and signal_noise > 25 and signal_noise <= 250 and match > 700


UNION

select 
count(*) as "count", 
lower('0 < 0.1') as purity,    lower('>35 && < 25') as signalNoise,    lower('> 600') as similarity

from spectra where purity > 0 and purity < 0.1 and signal_noise > 3 and signal_noise <= 25 and match > 600

-- 1.5 settings

UNION

select 
count(*) as "count", 
lower('0 < 1.5') as purity,    lower('> 250') as signalNoise,
lower('> 700') as similarity

from spectra where purity > 0 and purity < 1.5 and signal_noise > 250 and match > 700

UNION

select 
count(*) as "count", 
lower('0 < 1.5') as purity,    lower('> 25 && < 250') as signalNoise,    lower('> 600') as similarity

from spectra where purity > 0 and purity < 1.5 and signal_noise > 25 and signal_noise <= 250 and match > 600

UNION

select 
count(*) as "count", 
lower('0 < 1.5') as purity,    lower('> 3 && < 25') as signalNoise,    lower('> 500') as similarity

from spectra where purity > 0 and purity < 1.5 and signal_noise > 3 and signal_noise <= 25 and match > 500

-- all other settings

UNION

select 
count(*) as "count", 
lower('> 1.5') as purity,
lower('> 250') as signalNoise,
lower('> 600') as similarity

from spectra where purity > 1.5 and signal_noise > 250 and match > 600

UNION

select 
count(*) as "count", 
lower('> 1.5') as purity,
lower('> 3 && < 250') as signalNoise,    lower('> 500') as similarity

from spectra where purity > 1.5 and signal_noise > 3 and signal_noise <= 250 and match > 500

UNION

-- total --

select 
count(*) as "count", 
lower('no filter') as purity,
lower('no filter') as signalNoise,
lower('no filter') as similarity

from spectra

and as query to filter polysiloxanes out of the result for leco 2.22




-- 0.1 settings
select 
count(*) as "count", 
lower('0 < 0.1') as purity, lower('> 250') as signalNoise,
lower('> 800') as similarity

from spectra a, bin b  where a.purity > 0 and a.purity < 0.1 and a.signal_noise > 250 and match > 800 and a.bin_id = b.bin_id and b.export = 'TRUE'

UNION

select 
count(*) as "count", 
lower('0 < 0.1') as purity, lower('> 25 && < 250') as signalNoise, lower('> 700') as similarity

from spectra a, bin b  where a.purity > 0 and a.purity < 0.1 and a.signal_noise > 25 and a.signal_noise <= 250 and match > 700 and a.bin_id = b.bin_id and b.export = 'TRUE'


UNION

select 
count(*) as "count", 
lower('0 < 0.1') as purity, lower('>35 && < 25') as signalNoise, lower('> 600') as similarity

from spectra a, bin b  where a.purity > 0 and a.purity < 0.1 and a.signal_noise > 3 and a.signal_noise <= 25 and match > 600 and a.bin_id = b.bin_id and b.export = 'TRUE'

-- 1.5 settings

UNION

select 
count(*) as "count", 
lower('0 < 1.5') as purity, lower('> 250') as signalNoise,
lower('> 700') as similarity

from spectra a, bin b  where a.purity > 0 and a.purity < 1.5 and a.signal_noise > 250 and match > 700 and a.bin_id = b.bin_id and b.export = 'TRUE'

UNION

select 
count(*) as "count", 
lower('0 < 1.5') as purity, lower('> 25 && < 250') as signalNoise, lower('> 600') as similarity

from spectra a, bin b  where a.purity > 0 and a.purity < 1.5 and a.signal_noise > 25 and a.signal_noise <= 250 and match > 600 and a.bin_id = b.bin_id and b.export = 'TRUE'

UNION

select 
count(*) as "count", 
lower('0 < 1.5') as purity, lower('> 3 && < 25') as signalNoise, lower('> 500') as similarity

from spectra a, bin b  where a.purity > 0 and a.purity < 1.5 and a.signal_noise > 3 and a.signal_noise <= 25 and match > 500 and a.bin_id = b.bin_id and b.export = 'TRUE'

-- all other settings

UNION

select 
count(*) as "count", 
lower('> 1.5') as purity,
lower('> 250') as signalNoise,
lower('> 600') as similarity

from spectra a, bin b  where a.purity > 1.5 and a.signal_noise > 250 and match > 600 and a.bin_id = b.bin_id and b.export = 'TRUE'

UNION

select 
count(*) as "count", 
lower('> 1.5') as purity,
lower('> 3 && < 250') as signalNoise, lower('> 500') as similarity

from spectra a, bin b  where a.purity > 1.5 and a.signal_noise > 3 and a.signal_noise <= 250 and match > 500 and a.bin_id = b.bin_id and b.export = 'TRUE'

UNION

-- total --

select 
count(*) as "count", 
lower('no filter') as purity,
lower('no filter') as signalNoise,
lower('no filter') as similarity

from spectra a, bin b where a.bin_id = b.bin_id and b.export = 'TRUE'

BinBase queries

2010-09-22T11:14:00.000-07:00

short overview of binbase queries to generate statistics



count of spectra

select count(*) from spectra a, samples b where  a.sample_id = b.sample_id and visible = 'TRUE'

count of spectra with no annotations

select count(*) from spectra a, samples b where  a.sample_id = b.sample_id and visible = 'TRUE' and bin_id is null

count of samples

select count(*) from samples where visibile = 'TRUE'

known species

select distinct value from sample_info a, meta_key b where a.key_id = b.key_id and b.key = 'Species'

count of species

select count(distinct value) from sample_info a, meta_key b where a.key_id = b.key_id and b.key = 'Species'

sample count for species

select count(a.sample_id),a.value from sample_info a, meta_key b,samples c where a.key_id = b.key_id and b.key = 'Species' and a.sample_id = c.sample_id and c.visible = 'TRUE' group by a.value

bin generation by species

select a.value,count(d.spectra_id) from sample_info a, meta_key b,samples c,bin d where a.key_id = b.key_id and b.key = 'Species' and a.sample_id = c.sample_id and d.sample_id = c.sample_id group by a.value

ri correction successful

select count(*) from samples where visible = 'TRUE' and correction_failed = 'FALSE'

ri correction successful

select count(*) from samples where visible = 'TRUE' and correction_failed = 'FALSE'

named bins

select count(bin_id) from bin where name = TO_CHAR(bin_id,'FM9999999999')

distribution of similarities


select count("range"),"range" from (
select
    case
        when match >= 0 and match <= 100 then '0 - 100'         when match > 100 and match <= 200 then '101 - 200'        when match > 200 and match <= 300 then '201 - 300'        when match > 300 and match <= 400 then '301 - 400'        when match > 400 and match <= 500 then '401 - 500'        when match > 500 and match <= 600 then '501 - 600'        when match > 600 and match <= 700 then '601 - 700'        when match > 700 and match <= 800 then '701 - 800'        when match > 800 and match <= 900 then '801 - 900'        when match > 900 and match <= 1000 then '901 - 1000'    end AS "range" from spectra ) agroup by "range"

grails - provide a dynamically generated file as download

2010-09-08T12:28:00.000-07:00

today I had the rather simple request

generate a file on the fly and provide it as download.

Well the easiest solution was to just create a closure in a controller and set the content type in there.



def download = {

File file = new File(params.file)
response.setHeader "Content-disposition", "attachment; filename=${file.name}.txt"
response.contentType = 'text-plain'
response.outputStream << file.text     response.outputStream.flush()    }

dynamic fields with groovy

2010-08-25T17:11:00.000-07:00

one of the features I love is to just define new fields in objects whenever I want.

like:



def test = ["a","b","c,"]

Object a = new Object()

test.each{

a.metaClass."${it}" = it
}

generates te following fields in the object a

a.a = "a"
a.b = "b"
a.c = "c"

works incredible well with the grails export plugin :)

Bash - One line based For Loop

2010-07-23T14:14:00.001-07:00

today I needed to start 40 jobs in a row and the easiest way I could think about was to use a one line bash script todo this for me


for i in `seq 40`; do ./node.sh; done

tada...

finding all external reference names in pubchem sdf files

2010-07-13T19:39:00.000-07:00

currently I'm working on a simple way to find all external reference names in the pubchem sdf files. Which is rather trivial, but time consuming:



cat *.sdf | grep PUBCHEM_EXT_DATASOURCE_NAME -A 1 | grep -v PUBCHEM_EXT_DATASOURCE_NAME | grep -v '\-\-' | sort | uniq

once this finished it should give us a list of all possible data sources from pubchem as a unique list of names.


...
Ambinter
Burnham Center for Chemical Genomics
Calbiochem
CC_PMLSC
ChEBI
ChemSpider
DiscoveryGate
Emory University Molecular Libraries Screening Center
InFarmatik
KUMGM
LipidMAPS
MICAD
MLSMR
MMDB
MTDP
Nature Chemical Biology
NCGC
NIAID
NMMLSC
NMRShiftDB
ORST SMALL MOLECULE SCREENING CENTER
PCMD
ProbeDB
Prous Science Drugs of the Future
R&D Chemicals
Sigma-Aldrich
Specs
SRMLSC
Structural Genomics Consortium
The Scripps Research Institute Molecular Screening Center
Thomson Pharma
UM-BBD
UPCMLD
...

ubuntu - stucked in graphical mode without an editor

2010-07-05T15:45:00.001-07:00

well for some reason I thought it was smart to install on a freshly installed ubuntu linux server the gdm package in a VMWare console.

20 Seconds later it turned out that I had

- no terminal application
- could not switch to a virtual terminal, vmware kept ignoring the commands
- had ssh disabled

short I was stuck. Till I remembered to write a simple script to just install xterm and execute it. But wait I need to enter parameters and passwords after the prompt.

Quick forward

#! /bin/bash
echo 'password' | sudo -S apt-get --yes install xterm > log.txt

as script in a simple text file, which was set to be executable gave me all I need to install my xterm and be happy again.

making a jmx depend on an ear

2010-06-22T06:09:00.000-07:00

this morning I discovered that my JBoss application server sometimes deploys my jmx beans before a depending ear. The fix is like always easy just define in your jboss-service.xml



<server>
        <mbean
                code="edu.ucdavis.genomics.metabolomics.binbase.bci.server.jmx.LifeTimeEvictionPolicyConfigJMX"
                name="binbase.cache:service=LifeTimeEvictionPolicy">
 <depends>jboss.j2ee:ear=clusterservice.ear</depends>

</mbean>

Getting the hostname for the local machine

2010-06-17T00:37:00.000-07:00

there are always the simple things which I'm getting stuck with, sometimes simple things like just getting the hostname for a local ip address in java.

solutions:



InetAddress addr = InetAddress.getLocalHost();
String hostName = addr.getCanonicalHostName();

2010-06-16T15:26:00.001-07:00

sure shortly after the iPad with the unlimited dataplan got released, they change the plan to

2GB A month

what a joke...

Data per month Price per month
250MB/month $14.99
2GB/month $25.00

now I'm glad that I cancelled my pre order for one. What do 2GB give me these times? Nothing...

renaming all files in a directory

2010-05-13T12:04:00.001-07:00

some of the simple things in live



for x in *.php5; do n=${x/.php5/.php}; mv $x $n; done

taken from here

http://www.debianadmin.com/rename-multiple-files-to-another-extension-in-linux.html

moving indexes in postgres to a new tablespace

2010-05-07T10:56:00.000-07:00

Recently I upgraded my server and added a SSD raid system of 3 128GB disks and for obvious reasons want to have my indexes on it. So the first step was to create a new tablespace


create tablespace 'name' owner 'owner' location 'directory'

and than creating a quick script to move all the indexes


select 'ALTER INDEX '||indexname||' SET TABLESPACE compoundindex ;' from pg_catalog.pg_indexes where schemaname = 'public' order by tablename;

which than just needs to be executed

using perl to replace all occurences of a word with another in large files

2010-05-04T23:17:00.001-07:00

rather simple, we want to replace all occurences of the word 'volatile' with 'volatile2' in a large text file and so just execute


perl -i -p -e 's{volatile}{volatile2}g' /mnt/storage/data/export

I can't believe I'm using perl again...

flushing BinBase

2010-05-04T14:16:00.000-07:00

once in a while you think you want to reprocess all data in a binbase database and for this reason you want to flush all the content except the retention index standards. This should never be required, but if you decided to regenerate all the bins with different settings, well you gotta do what you gotta do.

here is a quick sql script todo this for you


delete from comments

go

delete from sample_info

go

delete from meta_key

go

delete from metainformation

go 

delete from runtime

go

delete from bin_compare

go

delete from virtual_bin

go

delete from bin a where a.bin_id not in (select bin_id from standard)

go

delete from samples a where a.sample_id not in (select sample_id from bin)

go

delete from reference_class

go

delete from reference

go

delete from result_link

go

delete from bin_ratio

go

delete from bin_references

go

delete from classification

go

delete from  configuration where configuration_id not in (select configuration_id from samples)

go

delete from spectra where sample_id not in (select sample_id from samples)

go

delete from qualitycontrol

go

delete from result

go

delete from standard_hist

go

delete from structure

go

delete from substance_classes

go

delete from synonyme

go

delete from user_validation

go

delete from substance_classes

go

griffon

2010-03-24T16:41:00.000-07:00

The last couple of days I spend with developing a quick application in griffon and have to say this is a rather impressive framework.. Specially since it's a young framework with a version of 0.3

what I like?

using the grails approach
very simple to use
saves a lot of time in the initial setup of the project
clear separation into controller/model/views

what I'm missing?

better support in IDE's
no SWT support
no gorm (it will come sooner or later)
documentation, with the current documentation i can't use it for production code

i just love groovy for tasks like this...

2010-03-11T22:22:00.000-08:00

sometimes you have these annoying little tasks on your hands and I always forget the bash syntax and it’s been a while since I seriously played with python.

So I thought, hey let’s try groovysh.

task?

read wordlist
convert to lowercase
remove parts of words

solution

new File("en_US.dic").text.split("\n").each{ new File("result.txt").append "${it.split("/")[0].toLowerCase()}\n" }

ok after a bit more coding I actually wrote a groovy script

read file
read file with words we do not want in word list
convert all values to lowercase
remove duplicated values
save to output file

really nothing fancy, but get’s the job done and took like 3 minutes to write and won’t be used again.

Set<String> cache = new HashSet<String>()
Set<String> result = new HashSet<String>()

new File(“whitelist.txt”).text.split(“\n”).each{ it.split("\t").eachWithIndex{

        String s, int index ->

                if(index > 0){
                        cache.add(s.toLowerCase())
                }
        }
}

new File(“blacklist.txt”).text.split(“\n”).each{
        if(cache.contains(it.toLowerCase()) == false){
                result.add(it.toLowerCase())
        }
}

File out = new File("wordlist.txt")

result.each{
        out.append it
        out.append "\n"
}

calculating exact molare masses using the cdk 1.3.1

2010-03-10T16:51:00.000-08:00

I needed a simple way to calculate the exact mass for a couple of millions compounds and so decided to give it another try with the CDK.

After googleing a bit I found something what put me in the right direction and worked with the current version of the cdk, so it was rather simple
 
class ConvertInchi {

/**
* converts an inchi code to a molare mass
* @param inchi
* @return
*/
public static double convertInchiToMolareMass(String inchi) {

IMolecularFormula moleculeFormula = MolecularFormulaManipulator.getMolecularFormula(convertInchiToMolecularFormula(inchi),DefaultChemObjectBuilder.getInstance())

return MolecularFormulaManipulator.getTotalExactMass(moleculeFormula)

}

/**
*
* converts an inchi to a molecular formula
* @param
inchi
* @return
*/
public static String convertInchiToMolecularFormula(String inchi) {
return inchi.split("/")[1]

}
}

Postgres - partial indexes and sort indexes

2010-03-05T15:37:00.000-08:00

this morning it was a sad moment. Normally people ask me all the time about SQL optimization or SQL tuning or when to create an index and so on.

This morning i had another DOOOH moment.

Basically while trying to optimize BinBase even further I noticed that some queries take an absurd amount of time to finish. Like 500 seconds to query the bin table.

So I started to investigate and discovered that some of my queries are using seq scans over huge tables. For no apparent reason.

It turned out that the sort column was not indexed...


CREATE INDEX bin_retention_index ON bin (retention_index ASC NULLS LAST);

which was greatly improved after executing this statement.

Now there was still a huge wait time for a certain query


explain select * from BIN where bin_id not in ( SELECT bin_id from SPECTRA where sample_id = 1630733 and bin_id is not null )ORDER BY retention_index ASC

and it turned out that indexes were never used for the 'is not null' part. After some research and a lot of head scratching it turned out that postgres supports partial indexes for exactly this case.


create index spectra_bin_id_partial_is_not_null on spectra (bin_id) where bin_id is not null

Now afterward we some improvement but the actually slow data access is caused by something else. We always need the complete table - a few compounds and this operation takes some time.

Time to optimize the internal database lookup...

rocks linux cluster - adding a new parallel environment

2010-03-04T14:30:00.000-08:00

by default rocks ships with a couple of environments, which execute stuff on different nodes. But sometimes you just want to have a node all to your self and take over all it's slots.

Todo this you can just create a new environment and which gives you a defined number of cpus for a specified job.

create a file which describes the paralell environment like this


pe_name            threaded
slots              999
user_lists         NONE
xuser_lists        NONE
start_proc_args    /bin/true
stop_proc_args     /bin/true
allocation_rule    $pe_slots
control_slaves     FALSE
job_is_first_task  TRUE
urgency_slots      min
accounting_summary FALSE


qconf -Ap file.txt

add it to the list of available envionments


qconf -mq all.q
pe_list               make mpich mpi orte threaded

test it with qlogin


qlogin -pe threaded 4

SetupX - tuning and the missing indexes

2010-03-03T14:38:00.000-08:00

after analyzing a couple of hundred sql statements in setupx I noticed that there is no real use of indexes for some reason. Why there are no indexes escapes my mind, but since we prefer higher query speed I suggest the creation of the following indexes (which are far from perfect)

create index pubdata_relatedExperimentID_index on pubdata(relatedExperimentID)
create index NCBIClassifierID_ncbiID_index on NCBIClassifierID(ncbiID)
create index formobject_question_index on formobject(question)
create index formobject_discriminator_index on formobject(discriminator)
create index formobject_value_index on formobject(value(100))
create index query_querytype_index on query(querytype)
create index query_userid_index on query(userID)
create index datafile_source_index on datafile (source)
create index cache_experiment_id_index on cache (experimentID)
create index user_password_index on user (passwd)
create index user_username_index on user (username)
create index BlockedIP_blockedIp_index on BlockedIP(blockedIP)

this should improve the performance nicely and is currently being applied to our production system.

SetupX - rawdata access

2010-03-03T10:37:00.000-08:00

ever had the desire to access the rawdata in SetupX, because the gui doesn't give you easy or access at all to it?

First you need to understand what's happening in the background. The used approache for setupx looks like a mapping from a tree to a table structure. Which is a valid approach and keeps it very flexible in therory, but is not a real pratical solution. Specially since it's a real pain to write queries for.

Get all species:

select distinct value from formobject where question = 'species' and value != ''

Get all organs:

select distinct lower(value) from formobject where question in ( 'organ','organ name','Organs','Organ specification' ) and value != ''

we need to use lower, cause people have a 'strange' way of spelling things

work in progress!

jboss 4.2.1GA and Java6 and jboss based webservice

2010-03-02T15:42:00.000-08:00

thanks to this report I was able to fix this issue. Well it's not a fix, it's a workaround...

Ariela Hui - 13/Aug/08 04:23 PM
I was able to solve this problem. This is what environment I have:
winXP
JDK 1.6.0
JBoss 4.2.1

In the [jboss_home]/lib/endorsed add:
jaxb-api.jar
jboss-jaxrpc.jar
jboss-jaxws.jar
jboss-saaj.jar

copy them from [jboss_home]/server/default/lib

I've also added to endored jboss-logging-spi.jar
This I copied from jboss-5.0.0 client folder.

scala/groovy on rocks linux

2010-03-01T10:43:00.000-08:00

well since there is not scala/groovy roll for rocks we need to install it the traditional way.

go into the directory /shared/apps on the frontend
if apps doesn't exist create it
copy your scala/groovy tgz there
gunzip and untar it
edit your extend-compute.xml as shown here
add a new file modification section like this



<file name="/etc/profile" mode="append">

GROOVY_HOME=/share/apps/groovy
SCALA_HOME=/share/apps/scala

export GROOVY_HOME
export SCALA_HOME

PATH=$GROOVY_HOME/bin:$PATH
PATH=$SCALA_HOME/bin:$PATH

export PATH

</file>

rebuild your dist as shown here
reinstall you nodes as shown here

backup! we got REG Expressions!

2010-02-26T21:20:00.000-08:00

recently I got an interesting task handed

please extract all words out of a text...

ok simple enough, we know regular expression and we know word boundaries.

So we just do


\b\w+\b

and this simple expression applied to the following sentence

Glucose (Glc), a monosaccharide (or simple sugar) also...

gives us the following list

Glucose
Glc
a
monosaccharide
or simple
sugar
also

Sadly the sentence didn't stop at this and continued to include the following tricky statement...

...including as much as possible chemical names

Ok time to read on the reg expressions in groovy and java
Now let's discover a regular expression which helps us with this.

Ok let's try this again with a different sentence

...as glucose, only one of which (D-glucose) is biologically...

our first expression would miss

D-glucose

and return for this

D
glucose

so we need to modify it a bit to include the first seperation. So it becomes

\b(\w\-)*\w+\b

and the day is saved till we try a new sentence and try to discover compounds like

1,3-Diaminopropane
N-(3S-hydroxydecanoyl)-L-serine
3,9-divinyl-2,4,8,10-tetraoxaspiro[5.5]undecane
2-(allyloxy)-1,3,5-trimethylbenzene
3-hydroxy-2-butanone
3,3'-Oxybis(1-propene)
1,1,1,2,2,3,3,4,4-nonafluoro-4-(1,1,2,2,3,3,4,4,4-nonafluorobutoxy)butane
2-(Formamido)-N1-(5-phospho-D-ribosyl)acetamidine
1,6,9,13-tetraoxadispiro[4.2.4.2]tetradecane
3-(N, N-Diethylamino)-1-propylamine
D-Glucose
(R)-3-Hydroxybutyric acid

Or to make it more realistic, find all the words in this completely pointless and scientific wrong text

bunch of rumble to find 1,3-Diaminopropane in D-Glucose and 1,1,1,2,2,3,3,4,4-nonafluoro-4-(1,1,2,2,3,3,4,4,4-nonafluorobutoxy)butane.
It's also nice to have 3,3'-Oxybis(1-propene) or (R)-3-Hydroxybutyric acid. Last bot not least I'm a huge fan or 3,9-divinyl-2,4,8,10-tetraoxaspiro[5.5]undecane.
Also it's a great feeling if we can find (glucose) in brakets without finding statement like (help i'm surrounded by brackets).

So do you see a pattern here?

everything in () or [] or {} can be part of a chemical so we use ((\[.*\])|($.*$)) for this part
everything separated by a ',' and followed by another character ending with a dash can be a chemical, so we use (\w+(,\w+)*\-) for this part
it ends all with a word \w+ or a ) (masked as \) )

so this expression would work for all these


 (\([\w\+]+(,\w+)*\)-)?\b[(\w+(,\w+[\'])*\-)*((\[.*\])|(\(.*\))|(\{.*\}))*\w+]+(\b)( (acid)|(anhydride)|(\sbenzoate)|(\sketone)|(\sether)|(\sester)|(\scyanide))?

except

3-(N, N-Diethylamino)-1-propylamine
glucose instead we get 'glucose)'

no solution for the 1 or 2 yet. Still trying to figure it out.

now the nicest thing is the groovy closure + match example to get all the content in a text.


def match = (text =~ pattern)

congrats now all you words are in the match variable! Text is just a string containing our text.

rocks linux cluster - mounting an nfs share on all nodes

2010-02-24T23:09:00.000-08:00

after setting up the latest cluster I tried to provide to all nodes a couple of nfs shares, since user demanded this.

Well in rocks linux it's rather simple, once you understand the concept behind.

So a step to step tutorial.

go to the profile directory

cd /export/rocks/install/site-profiles/5.3/nodes/

make a copy of the skeleton file

cp skeleton.xml extend-compute.xml

edit file to tell it that we need to create a directory and add a line to the fstab. The right place for this is in the post section

mkdir -p /mnt/share <file name="/etc/fstab" mode="append"> server:/mount /mnt/share nfs defaults 0 0 </file>
change back to the main install dir

cd /export/rocks/install

rebuild rocks distibution

rocks create distro

rebuild nodes

ssh compute-0-0 '/boot/kickstart/cluster-kickstart'

congratulations if you did everything right your node should now boot up and have a directory mounted.

playing around with threads

2010-02-24T18:37:00.000-08:00

currently I got back to my hobby and play a bit with multithreading to tune an algorithm.

So the first step was to write a smpl class to test if the threading api works on my system and what is better than to calculate primes?


import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.TimeUnit;

public class Test {

 public static void main(String args[]) throws InterruptedException {

  ExecutorService service = Executors.newFixedThreadPool(Runtime.getRuntime().availableProcessors());
  for (int i = 0; i < 500000000; i++) {   final int postion = i;   service.execute(new Runnable() {    @Override    public void run() {     try {      int i = postion;      int i1 = (int) Math.ceil(Math.sqrt(i));      boolean isPrimeNumber = false;      while (i1 > 1) {

       if ((i != i1) && (i % i1 == 0)) {
        isPrimeNumber = false;
        break;
       }
       else if (!isPrimeNumber) {
        isPrimeNumber = true;
       }

       --i1;
      }

      if (isPrimeNumber) {
       System.out.println(Thread.currentThread().getName() + " - prime " + i);
      }
      
     }
     catch (Exception e) {
      e.printStackTrace();
     }
    }
   });
  }
  service.shutdown();
  service.awaitTermination(4000, TimeUnit.DAYS);
 }
}

and the output is nicely:

pool-1-thread-2 - prime 469237
pool-1-thread-2 - prime 469241
pool-1-thread-2 - prime 469253
pool-1-thread-4 - prime 466553
pool-1-thread-4 - prime 469267
pool-1-thread-4 - prime 469279
pool-1-thread-2 - prime 469283
pool-1-thread-3 - prime 467869
pool-1-thread-3 - prime 469303
pool-1-thread-2 - prime 469321

while all 4 CPU's are at 100% use.

translation, java executor api seems to work quite well.

Now time to tune the binbase algorithm...

postgres and insert performance with grails

2010-02-20T23:28:00.000-08:00

recently I spend a lot of time tuning a postgres database and an algorithm (based on grails) to try to insert 80M chemical compounds into a database with reference checks and assurance for there uniqueness.

The main issue is that the inserts get slower and slower over time and end up taking to long. To improve the performance we started with the following approach:

more memory
database indexes (obvious)
flush the hibernate session all 100 inserts
flush the grails cache all 100 inserts
tune several postgres parameters
re index the database all 10k inserts
enable vacuum on a 10k row basis
enable analyze on a 10k row basis
set auto vacuum to check all 10 minutes

But after all this work we are still stuck with a problem, the inserts get gradually slower over time, which can be related to the index building time.

Graph over 1M inserts: TODO, it's still in the calculation...

Postgres and utf8

2010-02-09T11:28:00.000-08:00

Recently I discovered that my postgres datbase is not running with UTF-8 support which causes all kinds of headdache,

so what do you have todo to change this?

backup your database
set Lang to: LANG = en_US.UTF-8
execute initdb
recreate the database and specify the encoding: createdb --encoding=unicode
import your data

you should now have a working utf-8 database

muss das sein?

2010-02-03T21:06:00.001-08:00

[INFO] [compiler:compile]
[INFO] Compiling 399 source files to /Users/wohlgemuth/Documents/workspace-private/oscar3-chem/branches/cdk-1.3.1/target/classes
[INFO] ------------------------------------------------------------------------
[ERROR] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Compilation failure
Failure executing javac, but could not parse the error:
An exception has occurred in the compiler (1.5.0_20). Please file a bug at the Java Developer Connection (http://java.sun.com/webapps/bugreport) after checking the Bug Parade for duplicates. Include your program and the following diagnostic in your report. Thank you.
com.sun.tools.javac.code.Symbol$CompletionFailure: file org/openscience/cdk/annotations/TestClass.class not found

Failure executing javac, but could not parse the error:
An exception has occurred in the compiler (1.5.0_20). Please file a bug at the Java Developer Connection (http://java.sun.com/webapps/bugreport) after checking the Bug Parade for duplicates. Include your program and the following diagnostic in your report. Thank you.
com.sun.tools.javac.code.Symbol$CompletionFailure: file org/openscience/cdk/annotations/TestClass.class not found

I mean common, do I really have to deal with compiler bugs now...

correction

nvm forgot the cdk annotations library....

cdk-maven-mojos

2010-02-03T13:34:00.000-08:00

recently I'm doing an awful lot with the CDK library and since I always 'like' to work with maven I thought it's time to write a couple of mojos to help me with the CDK work.

The first one of the list is a mojo which deploys the cdk library to my local maven repository and can be found under google code.

There are most likely more mojo's to come as I work more and more with the cdk.

regular expression for common chemical identifiers

2010-02-02T01:54:00.000-08:00

This is basically a small collection for regular expressions which I use from time to time to distinguish chemical identifiers. Please feel free to add more to the list to make it grow and more complete.

The first line is the name, the second is the valid groovy/java version. All are validate with at least thousand examples

std inchi

InChI=1S/([^/]+)(?:/[^/]+)*\\S

std inchiKey

[A-Z]{14}-[A-Z]{10}-[A-Z,0-9]

CAS

\\d{1,7}-\\d\\d-\\d

KEGG

C\d{5}

LipidMaps

LMFA[0-9]{8}

HMDB

HMDB[0-9]*

how to handle 400 Billion rows in postgress

2010-01-24T00:16:00.000-08:00

well currently I try to optimize an denoising and replacement algorithm. This includes calculating ion traces over thousands files which have 20k peaks or 1 Million ions each.
To make this more challenging we don't do this once, we do it during every BinBase export. Since this is the base algorithm for the Zero replacement.

Short we do the same calculation over and over again, like to calculate the minimum intensity for 100 files between 15 and 20 seconds for the ion 87.

Short this shouts use a SQL database.

Now how much data do we have?

40k files
each file has around 20k spectra
each spectra has up to 500 ion

which translates into

400 000 000 000 rows

Performance Issues and how to make Postgres handle this amount of data

The first attempt was to just store all these files into a single table using hibernate. Which caused an OutOfMemory exception, surprise surprise.

So the next attempt was to use a stateless session which worked quite well and up to 1 Billion rows we had a query speed of under 20ms. Quite nice for no tuning of the database. But once we hit a bit over 1 Billions rows the query speed got worse and worse. Shor this was not an acceptable solution.

So we tried to use partions with postgres and define one table for each ion, inheriting from a master table.


Table "public.iontrace"
Column   |       Type       | Modifiers 
------------+------------------+-----------
id         | bigint           | not null
intensity  | double precision | 
ion        | integer          | 
spectra_id | bigint           | 
Indexes:
"iontrace_pkey" PRIMARY KEY, btree (id)
Foreign-key constraints:
"fk4f93923dc3b62c9a" FOREIGN KEY (spectra_id) REFERENCES spectra(id)
Rules:
rule_index_iontrace_100 AS
ON INSERT TO iontrace
WHERE new.ion = 100 DO INSTEAD  INSERT INTO iontrace_100 (id, intensity, ion, spectra_id) 
VALUES (new.id, new.intensity, new.ion, new.spectra_id)
rule_index_iontrace_101 AS
ON INSERT TO iontrace
WHERE new.ion = 101 DO INSTEAD  INSERT INTO iontrace_101 (id, intensity, ion, spectra_id) 
VALUES (new.id, new.intensity, new.ion, new.spectra_id)
rule_index_iontrace_102 AS
ON INSERT TO iontrace
WHERE new.ion = 102 DO INSTEAD  INSERT INTO iontrace_102 (id, intensity, ion, spectra_id)

which didn't work with hibernate. It kept complaining about


org.hibernate.StaleStateException: Batch update returned unexpected row count from update [0]; actual row count: 0; expected: 1

so we had to write our own batcher as was described here

a bit later we rewrote the example to actually work, since the field batch size is set to private.


/**
* hibernate partion batcher
*/
public class HibernatePartionBatcher extends BatchingBatcher {

public HibernatePartionBatcher(org.hibernate.jdbc.ConnectionManager connectionManager, org.hibernate.Interceptor interceptor) {
super(connectionManager, interceptor)
}


@Override
protected void doExecuteBatch(PreparedStatement ps) throws SQLException, HibernateException {

//use reflections to access the private field of the super class ugly but neccessaery
Field field = this.getClass().getSuperclass().getDeclaredField("batchSize")
field.setAccessible(true)

int value = field.getInt(this)

if (value != 0) {
try {
ps.executeBatch()
}
catch (RuntimeException re) {
throw re;
}
finally {
field.setInt(this, 0)
}
}
}
}

/**
* hibernate factory to intitialize the factory
*/
public class HibernatePartitionBatcherFactory extends BatchingBatcherFactory {

public HibernatePartitionBatcherFactory() {
}

public org.hibernate.jdbc.Batcher createBatcher(org.hibernate.jdbc.ConnectionManager connectionManager, org.hibernate.Interceptor interceptor) {
return new HibernatePartionBatcher(connectionManager, interceptor);
}
}

and to register this in our groovy script


def hibProps = [
"hibernate.jdbc.factory_class": HibernatePartitionBatcherFactory.class.getName()
]

A first test with just a single sample reveals that we are executing a sequential scan over all partions.


explain SELECT * from iontrace where ion = 105;
QUERY PLAN                                                  
-------------------------------------------------------------------------------------------------------------
Result  (cost=0.00..1163.25 rows=3374 width=28)
->  Append  (cost=0.00..1163.25 rows=3374 width=28)
->  Seq Scan on iontrace  (cost=0.00..10.46 rows=7 width=28)
Filter: (ion = 105)
->  Index Scan using index_iontrace_20 on iontrace_20 iontrace  (cost=0.00..1.57 rows=7 width=28)
Index Cond: (ion = 105)
->  Index Scan using index_iontrace_21 on iontrace_21 iontrace  (cost=0.00..1.57 rows=7 width=28)
Index Cond: (ion = 105)
->  Index Scan using index_iontrace_22 on iontrace_22 iontrace  (cost=0.00..1.57 rows=7 width=28)
Index Cond: (ion = 105)
->  Index Scan using index_iontrace_23 on iontrace_23 iontrace  (cost=0.00..1.57 rows=7 width=28)
Index Cond: (ion = 105)
->  Index Scan using index_iontrace_24 on iontrace_24 iontrace  (cost=0.00..1.57 rows=7 width=28)
Index Cond: (ion = 105)
->  Index Scan using index_iontrace_25 on iontrace_25 iontrace  (cost=0.00..1.57 rows=7 width=28)
Index Cond: (ion = 105)

to avoid this you have to change a parameter in postgresql.


SET constraint_exclusion = on;

and the new result shows that we only work on the correct partition now


explain SELECT * from iontrace where ion = 105;
QUERY PLAN                                                  
-------------------------------------------------------------------------------------------------------------
Result  (cost=0.00..13.03 rows=14 width=28)
->  Append  (cost=0.00..13.03 rows=14 width=28)
->  Seq Scan on iontrace  (cost=0.00..10.46 rows=7 width=28)
Filter: (ion = 105)
->  Index Scan using index_iontrace_105 on iontrace_105 iontrace  (cost=0.00..2.57 rows=7 width=28)
Index Cond: (ion = 105)

it still executes a sequential scan, but now it only does it on the correct partion with the stored data.

Now to reduce the likelihood for an index scan over the database we will define indexes on all the table (the script at the end of this post does this for you)

the new query plan is


explain SELECT * from iontrace where ion = 105;
QUERY PLAN                                                  
-------------------------------------------------------------------------------------------------------------
Result  (cost=0.00..4.15 rows=14 width=28)
->  Append  (cost=0.00..4.15 rows=14 width=28)
->  Index Scan using index_iontrace_ion on iontrace  (cost=0.00..1.57 rows=7 width=28)
Index Cond: (ion = 105)
->  Index Scan using index_iontrace_105 on iontrace_105 iontrace  (cost=0.00..2.57 rows=7 width=28)
Index Cond: (ion = 105)

The result is slightly different since we generate by now more data in the database. Sorry for that. But you can see that the query plan is much more efficient with the index.

Important is that you need an index for ion on every partition and on the main 'iontrace' table, if you don't have an index on the main table you will keep getting sequential scans.

The next snatch we encountered was the speed of the actual data insert. Since the check constraints seem to be rather slow and expensive.

First this is a hibernate insert on the main datatable:


LOG:  duration: 8.961 ms  statement: EXECUTE   [PREPARE:  insert into IonTrace (intensity, ion, spectra_id, id) values ($1, $2, $3, $4)]
LOG:  duration: 8.951 ms  statement: EXECUTE   [PREPARE:  insert into IonTrace (intensity, ion, spectra_id, id) values ($1, $2, $3, $4)]
LOG:  duration: 8.966 ms  statement: EXECUTE   [PREPARE:  insert into IonTrace (intensity, ion, spectra_id, id) values ($1, $2, $3, $4)]

Second this is plain old sql inserted into the main iontrace table:


LOG:  duration: 6.608 ms  statement: EXECUTE   [PREPARE:  insert into iontrace(id, intensity, ion, spectra_id) values (nextval('SEQ_TRACE'),1418.0,237,700013)]
LOG:  duration: 6.610 ms  statement: EXECUTE   [PREPARE:  insert into iontrace(id, intensity, ion, spectra_id) values (nextval('SEQ_TRACE'),1373.0,238,700013)]
LOG:  duration: 6.605 ms  statement: EXECUTE   [PREPARE:  insert into iontrace(id, intensity, ion, spectra_id) values (nextval('SEQ_TRACE'),1383.0,239,700013)]

Third this is plain old sql inserted in each partition:


LOG:  duration: 0.055 ms  statement: EXECUTE   [PREPARE:  insert into iontrace_475(id, intensity, ion, spectra_id) values (nextval('SEQ_TRACE'),27.0,475,680203)]
LOG:  duration: 0.058 ms  statement: EXECUTE   [PREPARE:  insert into iontrace_489(id, intensity, ion, spectra_id) values (nextval('SEQ_TRACE'),13.0,489,680203)]
LOG:  duration: 0.051 ms  statement: EXECUTE   [PREPARE:  insert into iontrace_496(id, intensity, ion, spectra_id) values (nextval('SEQ_TRACE'),13.0,496,680203)]
LOG:  duration: 0.138 ms  statement: EXECUTE   [PREPARE:  insert into Spectra (file_id, retentionTime, id) values ($1, $2, $3)]

From this we can see that it makes the most sense to use plain sql to insert the data into the related partitions and to avoid the checks during the insert phase.

For this we used the following approach


//build the ion trace
Statement statement = session.connection().createStatement()

for (int mass = 0; mass < mz.length; mass++) {           if (mass >= beginMass && mass <= endMass) {             if (intensity[mass] > 0) {
IonTrace trace = new IonTrace(ion: mz[mass], intensity: intensity[mass])
trace.spectra = spectra


statement.addBatch("insert into iontrace(id, intensity, ion, spectra_id) values (nextval('SEQ_TRACE'),${intensity[mass]},${(int)(mz[mass])},${spectra.id})")
}
}
}

statement.executeBatch()

which worked quite well. As a conclusion I removed the complete hibernate layer from this application, since we lost most of it's benefits by now.

To further optimize the insert speed we replace the Statements with PreparedStatements, which improves the speed of the insert by close to 50%.


LOG:  duration: 0.035 ms  statement: EXECUTE   [PREPARE:  insert into iontrace_127(id, intensity, ion, spectra_id) values (nextval('seq_trace'),$1,$2,$3)]
LOG:  duration: 0.035 ms  statement: EXECUTE   [PREPARE:  insert into iontrace_127(id, intensity, ion, spectra_id) values (nextval('seq_trace'),$1,$2,$3)]
LOG:  duration: 0.035 ms  statement: EXECUTE   [PREPARE:  insert into iontrace_127(id, intensity, ion, spectra_id) values (nextval('seq_trace'),$1,$2,$3)]

Storage

as you can imagine this amount of rows can take up quite some space. An initial estimate say it's roughly 20TB with the current table schema.

Which is 20 times more than the files occupy on the harddrive.

So what can you do to keep the storage space down

use the smallest possible data types possible
only index what is necessary to be index
remove columns which are not necessary, like we don't need a primary key on the ion trace table
estimate your possible amount of data stored in the database, like we won't store billions of spectra, so we can use integer instead of bigint
store intensities as integer and not as double.

To estimate the required storage we used the following function, which roughly gives us storage capacity for 4 years of netcdf files.


netcdf-repository=# SELECT pg_size_pretty(pg_database_size('netcdf-repository')/(select count(*) from netcdf)* 40000) as "estimated database size";
 estimated database size 
-------------------------
 16 TB
(1 row)

And a server for this would currently cost around 7000$

-END-

protecting jboss

2010-01-22T15:31:00.000-08:00

Protecting JBoss

in the last couple of weeks we released more and more of our BinBase Tools to make it able to actually work with the database. Which means we have to protect our data better.

Since I had no time at work for this and JBoss doesn't provide a convenient way, well I decided to write a little tool which does this for me.

So I created yet another google code project, called 'jboss-ip-filter', which basically does nothing else than providing an interceptor, which intercepts all method calls and check's if the ip is in a list of registered IP Address.

Features

protect ejb3.x services
protect ejb2.x services
ip can be defined as regular expression to support subnets

Configuration/Installation

First you need to download the latest release and copy it into the jboss library directory of your choosen configuration.

Afterwards you need to register the interceptor in the jboss configuration.

Example

vim /usr/local/jboss/server/all/conf/standardjboss.xml

Go to the part about the container configurations and register the interceptor in the first position for every ejb configuration you want to protect.

The name of the class is: 'com.blogspot.codingandmore.jboss.filter.SessionInterceptor'


<container-configuration>
<container-name>Standard CMP 2.x EntityBean</container-name>
<call-logging>false</call-logging>
<invoker-proxy-binding-name>entity-unified-invoker</invoker-proxy-binding-name>
<sync-on-commit-only>false</sync-on-commit-only>
<insert-after-ejb-post-create>false</insert-after-ejb-post-create>
<call-ejb-store-on-clean>true</call-ejb-store-on-clean>
<container-interceptors>
<interceptor>com.blogspot.codingandmore.jboss.filter.SessionInterceptor</interceptor>
<interceptor>org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor</interceptor>
<interceptor>org.jboss.ejb.plugins.LogInterceptor</interceptor>
<interceptor>org.jboss.ejb.plugins.SecurityInterceptor</interceptor>
<interceptor>org.jboss.ejb.plugins.TxInterceptorCMT</interceptor>
<interceptor>org.jboss.ejb.plugins.CallValidationInterceptor</interceptor>
<interceptor>org.jboss.ejb.plugins.EntityCreationInterceptor</interceptor>
<interceptor>org.jboss.ejb.plugins.EntityLockInterceptor</interceptor>
<interceptor>org.jboss.ejb.plugins.EntityInstanceInterceptor</interceptor>
<interceptor>org.jboss.ejb.plugins.EntityReentranceInterceptor</interceptor>
<interceptor>org.jboss.resource.connectionmanager.CachedConnectionInterceptor</interceptor>
<interceptor>org.jboss.ejb.plugins.EntitySynchronizationInterceptor</interceptor>
<interceptor>org.jboss.ejb.plugins.cmp.jdbc.JDBCRelationInterceptor</interceptor>
</container-interceptors>
<instance-pool>org.jboss.ejb.plugins.EntityInstancePool</instance-pool>
<instance-cache>org.jboss.ejb.plugins.InvalidableEntityInstanceCache</instance-cache>
<persistence-manager>org.jboss.ejb.plugins.cmp.jdbc.JDBCStoreManager</persistence-manager>
<locking-policy>org.jboss.ejb.plugins.lock.QueuedPessimisticEJBLock</locking-policy>
<container-cache-conf>
<cache-policy>org.jboss.ejb.plugins.LRUEnterpriseContextCachePolicy</cache-policy>
<cache-policy-conf>
<min-capacity>50</min-capacity>
<max-capacity>1000000</max-capacity>
<overager-period>300</overager-period>
<max-bean-age>600</max-bean-age>
<resizer-period>400</resizer-period>
<max-cache-miss-period>60</max-cache-miss-period>
<min-cache-miss-period>1</min-cache-miss-period>
<cache-load-factor>0.75</cache-load-factor>
</cache-policy-conf>
</container-cache-conf>
<container-pool-conf>
<MaximumSize>100</MaximumSize>
</container-pool-conf>
<commit-option>B</commit-option>
</container-configuration>

After this is done you need to restart your server and it should generate a property in the start directory after the next reboot. In this directory you configure your ip address. To be allowed.

For example if you started the server in the bin directory, the file will be found there


vim /usr/local/jboss/bin/ip-filter-config.properties

The ip address of the local host is always registered.

These following two lines allow it the host '128.120.136.154' to connect but refuses connections from any other hosts to the ejb's.


128.120.136.154 = true
\b(?:\d{1,3}\.){3}\d{1,3}\b = false

If you encounter any problems, please don't hesitate to contact me and I try to help with the encountered problems.

maven assembly plugin

2009-12-11T22:09:00.000-08:00

sometimes you run into the issue that you want to generate jars for a simple project. For example when you want to create an executable jar for each method with a main class.

What do you do?

you call the maven assembly plugin to the rescue and define several executions

<plugin>
<artifactId>maven-assembly-plugin</artifactId>

<executions>
<execution>
<id>MatchLibrary</id>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
<configuration>

<archive>
<manifest>
<mainClass>edu.ucdavis.genomics.metabolomics.binbase.binbase.external.tools.api.library.impl.application.ApplicationServerLibraryMatchingImplementation</mainClass>
<packageName>edu.ucdavis.genomics.metabolomics.binbase.binbase.external.tools.api.library.impl.application</packageName>
<addClasspath>true</addClasspath>
</manifest>
<manifestEntries>
<mode>development</mode>
<url>${pom.url}</url>
</manifestEntries>
<manifestFile>src/main/resources/manifest/ApplicationServerLibraryMatchingImplementation.MF</manifestFile>
</archive>
<descriptors>
<descriptor>src/main/descriptor/jar.xml</descriptor>
</descriptors>
<finalName>MatchLibrary</finalName>
<manifestFile>src/main/resources/manifest/ApplicationServerLibraryMatchingImplementation.MF</manifestFile>
</configuration>
</execution>

<execution>
<id>RegisterLibrary</id>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
<configuration>

<archive>
<manifest>
<mainClass>edu.ucdavis.genomics.metabolomics.binbase.binbase.external.tools.api.library.impl.RegisterLibraryImpl</mainClass>
<packageName>edu.ucdavis.genomics.metabolomics.binbase.binbase.external.tools.api.library.impl</packageName>
<addClasspath>true</addClasspath>
</manifest>
<manifestEntries>
<mode>development</mode>
<url>${pom.url}</url>
</manifestEntries>
<manifestFile>src/main/resources/manifest/RegisterLibrary.MF</manifestFile>
</archive>
<descriptors>
<descriptor>src/main/descriptor/jar.xml</descriptor>
</descriptors>
<finalName>RegisterLibrary</finalName>
<manifestFile>src/main/resources/manifest/RegisterLibrary.MF</manifestFile>
</configuration>
</execution>
</executions>
</plugin>

and this generates two executable jars for us.

The used assembly descriptor is

<assembly>
<formats>
<format>jar</format>
</formats>
<includeBaseDirectory>false</includeBaseDirectory>
<fileSets>
<fileSet>
<directory>target/classes</directory>
<outputDirectory>/</outputDirectory>
</fileSet>
</fileSets>
<dependencySets>
<dependencySet>
<outputDirectory>/</outputDirectory>
<unpack>true</unpack>
<scope>runtime</scope>
</dependencySet>
</dependencySets>
</assembly>

jboss maven plugin

2009-12-08T18:45:00.000-08:00

since I have issues with jboss jar and maven since I can remember I started to develop a small mojo to deal with this for me.

All it basically does is to parse the local jboss installation and depoys the files in the local repository.

Simple and efficient.

It can be found here

SQL - small mistakes, big impact

2009-11-18T01:06:00.000-08:00

the last couple of days I was fighting with a database which became slower and slower and slower and now needed close to 24h to execute a rather simple query...

After working for hours on different parameters and tuning the database, including setting up a spare server to see if a dump and restore fixes the issues I got the time down to 5 minutes.

Sounds good?

well not really since I needed an execution in the range of a couple of seconds. So I actually looked at the sql statement which caused issues and found a gigantic error in it...

the statement:


select * from BIN where bin_id not in ( SELECT bin_id from SPECTRA where sample_id = 424105 AND SPECTRA.bin_id = BIN.bin_id)

and the correct statement would be:


select * from BIN a where a.bin_id not in ( SELECT bin_id from SPECTRA where sample_id = 424105 and bin_id is not null) ORDER BY retention_index ASC

the problem?


SPECTRA.bin_id = BIN.bin_id

this little part is totally wrong in the whole statement and causes a seq scan over a table with 70 Million entries against a table with 6000 entries. It does nothing for the queries and I guess it was left over from a copy/paste accident 6 years ago when I wrote this script originally.

let's compare some numbers:

explain before:

Seq Scan on bin (cost=0.00..459513.42 rows=3044 width=1290)
SubPlan
Filter: (NOT (subplan))
-> Bitmap Heap Scan on spectra (cost=75.20..75.42 rows=2 width=8)
Recheck Cond: ((sample_id = 424105::double precision) AND (bin_id = ($0)::double precision))
-> BitmapAnd (cost=75.20..75.20 rows=2 width=0)
-> Bitmap Index Scan on spectra_sample_id (cost=0.00..18.99 rows=5696 width=0)
-> Bitmap Index Scan on spectra_binid (cost=0.00..56.18 rows=17658 width=0)
Index Cond: (sample_id = 424105::double precision)
Index Cond: (bin_id = ($0)::double precision)

explain after:

Sort (cost=1765.47..1766.23 rows=3044 width=1290)
Sort Key: retention_index
-> Seq Scan on bin a (cost=647.21..1747.86 rows=3044 width=1290)
Filter: (NOT (hashed subplan))
SubPlan
-> Index Scan using spectra_sample_id on spectra (cost=0.00..646.42 rows=3138 width=8)
Index Cond: (sample_id = 424105::double precision)
Filter: (bin_id IS NOT NULL)

in real world number this translates to 24 hours before statement compared to 4 seconds after statement...
...

maven2 - choose dependcies based on platform

2009-11-17T16:43:00.000-08:00

today I run into an old friend. I was compiling my stuff happily under oxs and building eclipse plugins with maven2 and the psteclipse plugin, which I by now more or less rewrote and adapted to my project to perform as good as possible. Anyway, basically I encountered the issue that eclipse ships with different dependencies under osx/win32 and linux.

So promptly it fails to compile under linux...

solution for this problem?

maven2 profiles, your best friend for cross platform development. They actually allow you define dependencies based on the current platform.

Example for out case would be:


   <profiles>
        <profile>
            <id>dev-windows</id>

            <dependencies>
            </dependencies>

            <activation>
                <activeByDefault>true</activeByDefault>
                <os>
                    <family>windows</family>
                </os>
            </activation>
        </profile>
        <profile>
            <id>dev-mac</id>
            <dependencies>
                <dependency>
                        <groupId>psteclipse</groupId>
                        <artifactId>org.eclipse.swt.cocoa.macosx</artifactId>
                        <version>3.5.1</version>
                        <scope>provided</scope>
                </dependency>

            </dependencies>

            <activation>
                <activeByDefault>false</activeByDefault>
                <os>
                    <family>mac</family>
                </os>
            </activation>
        </profile>
        <profile>
            <id>dev-linux</id>

            <dependencies>
                <dependency>
                        <groupId>psteclipse</groupId>
                        <artifactId>org.eclipse.swt.gtk.linux.x86</artifactId>
                        <version>3.5.1</version>
                        <scope>provided</scope>
                </dependency>
            </dependencies>

            <activation>
                <activeByDefault>false</activeByDefault>
                <os>
                    <name>linux</name>
                </os>
            </activation>
        </profile>
    </profiles>

And sets a different library for osx and linux. Sweet!

hibernate - configure caching

2009-11-11T21:43:00.000-08:00

since the hibernate tools for eclipse need to have the caching enabled, something I'm not the biggest fan off for several reason.

Well


  <property name="hibernate.cache.provider_class">
   org.hibernate.cache.EhCacheProvider
  </property>

  <property name="hibernate.cache.use_second_level_cache">true</property>
  <property name="hibernate.cache.use_query_cache">true</property>

so this makes the binbase configuration run in the eclipse tools.

Log4J and eclipse

2009-11-02T13:02:00.000-08:00

sometimes you want to use the log4j system instead of the internal eclipse provide logging framework.

The use of this is rather simple and straight forwarded. You only need to register an ILogListener at the platform object and implement it.

An example would be:


package edu.ucdavis.genomics.metabolomics.binbase.gui.swt.logging;

import org.apache.log4j.Logger;
import org.eclipse.core.runtime.ILogListener;
import org.eclipse.core.runtime.IStatus;

/**
 * used to forward eclipse logging to log4j
 * 
 * @author wohlgemuth
 * 
 */
public class Log4JListener implements ILogListener {

 static Logger LOGGER = Logger.getLogger(Log4JListener.class);

 public void logging(IStatus status, String plugin) {
  if (status.getSeverity() == IStatus.WARNING) {
   if (status.getException() == null) {
    LOGGER.warn(status.getMessage() + "(" + status.getCode() + ")");
   } else {
    LOGGER.warn(status.getMessage() + "(" + status.getCode() + ")",
      status.getException());
   }
  } else if (status.getSeverity() == IStatus.ERROR) {
   if (status.getException() == null) {
    LOGGER
      .error(status.getMessage() + "(" + status.getCode()
        + ")");
   } else {
    LOGGER.error(
      status.getMessage() + "(" + status.getCode() + ")",
      status.getException());
   }

  } else if (status.getSeverity() == IStatus.INFO) {
   if (status.getException() == null) {
    LOGGER.info(status.getMessage() + "(" + status.getCode() + ")");
   } else {
    LOGGER.info(status.getMessage() + "(" + status.getCode() + ")",
      status.getException());
   }

  }
 }

}

and to register it


  this.listener = new Log4JListener();
  Platform.addLogListener(this.listener);

Eclipse SWT 3.5 and an annoying change...

2009-10-27T12:44:00.000-07:00

after I moved a part of my software over to eclipse 3.5 to take advantage of bug fixes and new features I noticed that some of my dialogs don't work anymore.

For example the login dialog has 2 fiels. A username and a password. These fields are defiend like this:


 password = new Text(content, SWT.BORDER | SWT.PASSWORD);
 user = new Text(content, SWT.BORDER);

 user.addKeyListener(this);
 password.addKeyListener(this);

and in the key listener I validate the inputs. But with version 3.5 key listener won't work on password fields anymore and I didn't find any documentation about it. So instead you need to use a modify listener now


 password.addModifyListener(new ModifyListener() {
    
  public void modifyText(ModifyEvent e) {
  }
 });

duplicating a disk under OSX

2009-10-23T16:17:00.000-07:00

since my laptop has no cd drive anymore I have no the need of cloning all my cd's with my mac pro.

How to do this?

1. Insert CD/DVD source
2. Fire up a Terminal, you can then determine the device that is you CD/DVD drive using the following command:

$ drutil status
 Vendor   Product           Rev
 MATSHITA DVD-R   UJ-835E   GAND

           Type: DVD-ROM              Name: /dev/disk1
      Cur Write:    8x DVD          Sessions: 1
      Max Write:    8x DVD            Tracks: 1
   Overwritable:   00:00:00         blocks:        0 /   0.00MB /   0.00MiB
     Space Free:   00:00:00         blocks:        0 /   0.00MB /   0.00MiB
     Space Used:  364:08:27         blocks:  1638627 /   3.36GB /   3.13GiB
    Writability:
      Book Type: DVD-ROM

3. Umount the disk with the following command:

$ diskutil unmountDisk /dev/disk1
Disk /dev/disk1 unmounted

4. Create the ISO file with the dd utility (may take some time):

$ dd if=/dev/disk1 of=file.iso bs=2048

5. Test the ISO image by mounting the new file (or open with Finder):

$ hdid file.iso

taken from here

OSX - upgraded my macbook pro to an SSD and the differences

2009-10-16T16:28:00.000-07:00

After long considerations I updated my macbook pro to a SSD, from corsair. Since the price seemed to be not to outragios.

Now the tests are clear:

SSD

Drive Type CORSAIR CMFSSD-128GBG2D

Disk Test    173.68
    Sequential    132.15
        Uncached Write    143.67    88.21 MB/sec [4K blocks]
        Uncached Write    101.53    57.44 MB/sec [256K blocks]
        Uncached Read    94.19    27.56 MB/sec [4K blocks]
        Uncached Read    351.79    176.81 MB/sec [256K blocks]
    Random    253.29
        Uncached Write    121.43    12.85 MB/sec [4K blocks]
        Uncached Write    196.26    62.83 MB/sec [256K blocks]
        Uncached Read    1503.47    10.65 MB/sec [4K blocks]
        Uncached Read    556.60    103.28 MB/sec [256K blocks]

Drive Type ST9500325AS

Disk Test   43.26
   Sequential   80.53
       Uncached Write   78.89   48.44 MB/sec [4K blocks]
       Uncached Write   68.76   38.91 MB/sec [256K blocks]
       Uncached Read   77.68   22.73 MB/sec [4K blocks]
       Uncached Read   104.42   52.48 MB/sec [256K blocks]
   Random   29.57
       Uncached Write   10.43   1.10 MB/sec [4K blocks]
       Uncached Write   73.40   23.50 MB/sec [256K blocks]
       Uncached Read   64.02   0.45 MB/sec [4K blocks]
       Uncached Read   98.74   18.32 MB/sec [256K blocks]

I think these result's speak for them self!

my trouble with snowleopard and java5

2009-10-16T11:07:00.000-07:00

well snow leopard messed up my java installation.

Luckily there is an easy fix to it!

taken from here


cd /tmp/
curl -o java.1.5.0-leopard.tar.gz http://www.cs.washington.edu/homes/isdal/snow_leopard_workaround/java.1.5.0-leopard.tar.gz
tar -xvzf java.1.5.0-leopard.tar.gz
sudo mv 1.5.0 /System/Library/Frameworks/JavaVM.framework/Versions/1.5.0-leopard
cd /System/Library/Frameworks/JavaVM.framework/Versions/
sudo rm 1.5.0
sudo ln -s 1.5.0-leopard 1.5.0
open "/Applications/Utilities/Java Preferences.app"

and move java5 to the top.

calculating the variance - groovy style

2009-10-07T15:59:00.001-07:00

just had to calculate some variances in a flexible way and like always groovy comes to the rescue.


  static double variance(def population) {
    long n = 0
    double mean = 0
    double s = 0.0

    population.each {double x ->
      n++;
      double delta = x - mean
      mean += delta / n
      s += delta * (x - mean)
    }

    (s / (n - 1))
  }

Strange XMLParser behaviour - Groovy 1.6.4

2009-09-17T13:26:00.000-07:00

ell right now I'm stumbling over a small thing in groovy and don't understand why this is happening.

So this is my code:


new StreamingMarkupBuilder().bind({

      sop(desc: "generated for quantification") {
        transform(sizedown: 0, attributes: "height", combine: true) {
          header{
            param(value:"retention_index")
            param(value:"quantmass")
            param(value:"id")
            param(value:"spectra")
          }
        }
      }

simple enough.

And that's the code to test it


    def parser = new XmlParser().parse(source.getStream())

    assertTrue parser.@desc != null

    assertTrue parser.transform != null

    assertTrue Integer.parseInt(parser.transform.@sizedown[0]) == 0
    assertTrue parser.transform.@attributes[0] == "height"
    assertTrue parser.transform.@combine[0] == "true"

    assertTrue parser.transform.header.param[0].@value == "retention_index"
    assertTrue parser.transform.header.param[1].@value == "quantmass"
    assertTrue parser.transform.header.param[2].@value == "id"
    assertTrue parser.transform.header.param[3].@value == "spectra"

    assertTrue parser.transform.header.param.size() == 4

also pretty straight forward.

Now why is this part different...


transform(sizedown: 0, attributes: "height", combine: true)

    assertTrue Integer.parseInt(parser.transform.@sizedown[0]) == 0
    assertTrue parser.transform.@attributes[0] == "height"
    assertTrue parser.transform.@combine[0] == "true"

and my attributes are returned as an array list compared to


          header{
            param(value:"retention_index")
            param(value:"quantmass")
            param(value:"id")
            param(value:"spectra")
          }

    assertTrue parser.transform.header.param[0].@value == "retention_index"
    assertTrue parser.transform.header.param[1].@value == "quantmass"
    assertTrue parser.transform.header.param[2].@value == "id"
    assertTrue parser.transform.header.param[3].@value == "spectra"

where the attributes are returned as simple string. I has to be something obvious, but right now I'm not getting it.

XMLMarkupBuilder in groovy.

2009-09-16T12:04:00.000-07:00

Since a while i wanted to find a reason to play with the xml markup builders in groovy, since I think they are an awesome idea.
So I gave it a shot


    def document = new StreamingMarkupBuilder().bind({

      config {
        parameter {
          param(name: "java.naming.provider.url", value: "127.0.0.1", public:true)
          param(name: "java.naming.factory.initial", value: "org.jnp.interfaces.NamingContextFactory", public:true)
          param(name: "java.naming.factory.url.pkgs", value: "org.jboss.naming:org.jnp.interfaces", public:true)
          param(name: "edu.ucdavis.genomics.metabolomics.util.status.ReportFactory", value: "edu.ucdavis.genomics.metabolomics.binbase.cluster.status.EJBReportFactory")
          param(name: "edu.ucdavis.genomics.metabolomics.util.thread.locking.LockableFactory", value: "edu.ucdavis.genomics.metabolomics.binbase.cluster.locking.EJBLockingFactory")

        }
      }
    })

    println document

and the result is


<config><parameter><param name='java.naming.provider.url' value='127.0.0.1' public='true'/><param name='java.naming.factory.initial' value='org.jnp.interfaces.NamingContextFactory'/><param name='java.naming.factory.url.pkgs' value='org.jboss.naming:org.jnp.interfaces'/><param name='edu.ucdavis.genomics.metabolomics.util.status.ReportFactory' value='edu.ucdavis.genomics.metabolomics.binbase.cluster.status.EJBReportFactory'/><param name='edu.ucdavis.genomics.metabolomics.util.thread.locking.LockableFactory' value='edu.ucdavis.genomics.metabolomics.binbase.cluster.locking.EJBLockingFactory'/></parameter></config>

Neat! This was very simple and the validation of the code happens at runtime. The only uggly part is that idea shows this part in red brackets


 public:true

since public is a keyword of java/groovy. But it does compile...

Guess I have to change my api a bit to avoid this uggly keyword or at least make it optional.

Groovy 1.6.4 + IntelliJ8 + Groovy Plugin - nullpointer

2009-09-15T16:57:00.000-07:00

For soem reason I keep get nullpointer exception with IntelliJ, which have nothing todo with my code.
They only happen if I execute the project from intelliJ out.

example for this would be:


 src/example/dsl/quant.dsl
Caught: java.lang.NullPointerException
java.lang.NullPointerException
 at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:39)
 at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:40)
 at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:117)
 at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
 at edu.ucdavis.genomics.metabolomics.binbase.quantificator.dsl.QuantificatorDSL.run(QuantificatorDSL.groovy:45)
 at edu.ucdavis.genomics.metabolomics.binbase.quantificator.dsl.QuantificatorDSL$run.call(Unknown Source)
 at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:40)
 at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:117)
 at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
 at edu.ucdavis.genomics.metabolomics.binbase.quantificator.dsl.QuantificatorDSL.main(QuantificatorDSL.groovy:33)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:592)
 at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:86)
 at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:234)
 at groovy.lang.MetaClassImpl.invokeStaticMethod(MetaClassImpl.java:1296)
 at org.codehaus.groovy.runtime.InvokerHelper.invokeMethod(InvokerHelper.java:719)
 at groovy.lang.GroovyShell.runScriptOrMainOrTestOrRunnable(GroovyShell.java:262)
 at groovy.lang.GroovyShell.run(GroovyShell.java:218)
 at groovy.lang.GroovyShell.run(GroovyShell.java:147)
 at groovy.ui.GroovyMain.processOnce(GroovyMain.java:493)
 at groovy.ui.GroovyMain.run(GroovyMain.java:308)
 at groovy.ui.GroovyMain.process(GroovyMain.java:294)
 at groovy.ui.GroovyMain.processArgs(GroovyMain.java:111)
 at groovy.ui.GroovyMain.main(GroovyMain.java:92)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:592)
 at org.codehaus.groovy.tools.GroovyStarter.rootLoader(GroovyStarter.java:108)
 at org.codehaus.groovy.tools.GroovyStarter.main(GroovyStarter.java:130)

Now this exception does not occur, if I make a complete rebuild.

For some reason it is not able to initialize a class if it defined in another source file. More about this is written here

groovy 1.5.6 and delegates

2009-09-10T06:47:00.000-07:00

while trying to write a DSL I discovered an annoying bug in groovy 1.5.6, I guess. Basically if I define a delegate for a closure it has to be in the same file as the closure. If i define it in another file I always get a nullpointer exception...

example:



class MyDelegate {
 def test() {
  println "tada"
 }
}

static ExpandoMetaClass createEMC(Class clazz, Closure cl) {
 ExpandoMetaClass emc = new ExpandoMetaClass(clazz, false)

 cl(emc)

 emc.initialize()
 return emc
}

Script dslScript = new GroovyShell().parse("test{\ntest()\n}")

dslScript.metaClass = createEMC(dslScript.class, {
 ExpandoMetaClass emc ->

 emc.test = {

  Closure closure ->
  closure.delegate = new MyDelegate()
  closure.resolveStrategy = Closure.DELEGATE_FIRST

  try {
    closure()
  }
  catch (Exception e) {
    e.printStackTrace()
  }
 }
})
dslScript.run()

this works fine. Now if I move the class 'MyDelegate' to it's own file I end up with the following exception.


java.lang.NullPointerException
      at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.getDelegateMethod(ClosureMetaClass.java:207)
      at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:280)
      at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnCurrentN(ScriptBytecodeAdapter.java:78)
      at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnCurrent0(ScriptBytecodeAdapter.java:112)
      at Script1$_run_closure1.doCall(Script1.groovy:2)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:585)
      at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:86)
      at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:230)
      at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:248)
      at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnCurrentN(ScriptBytecodeAdapter.java:78)
      at Script1$_run_closure1.doCall(Script1.groovy)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:585)
      at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:86)
      at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:230)
      at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:248)
      at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:756)
      at org.codehaus.groovy.runtime.InvokerHelper.invokePogoMethod(InvokerHelper.java:778)
      at org.codehaus.groovy.runtime.InvokerHelper.invokeMethod(InvokerHelper.java:758)
      at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodN(ScriptBytecodeAdapter.java:170)
      at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeClosure(ScriptBytecodeAdapter.java:605)
      at edu.ucdavis.genomics.metabolomics.binbase.quantificator.dsl.Tester$_run_closure1_closure2.doCall(Tester.groovy:38)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:585)
      at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:86)
      at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:230)
      at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:248)
      at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:756)
      at groovy.lang.Closure.call(Closure.java:292)
      at org.codehaus.groovy.runtime.metaclass.ClosureMetaMethod.invoke(ClosureMetaMethod.java:72)
      at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:230)
      at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:912)
      at groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:946)
      at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnCurrentN(ScriptBytecodeAdapter.java:78)
      at Script1.run(Script1.groovy:1)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:585)
      at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:86)
      at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:230)
      at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:912)
      at groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:946)
      at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:756)
      at org.codehaus.groovy.runtime.InvokerHelper.invokePogoMethod(InvokerHelper.java:778)
      at org.codehaus.groovy.runtime.InvokerHelper.invokeMethod(InvokerHelper.java:758)
      at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodN(ScriptBytecodeAdapter.java:170)
      at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethod0(ScriptBytecodeAdapter.java:198)
      at edu.ucdavis.genomics.metabolomics.binbase.quantificator.dsl.Tester.run(Tester.groovy:45)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:585)
      at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:86)
      at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:230)
      at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:912)
      at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:756)
      at org.codehaus.groovy.runtime.InvokerHelper.invokePogoMethod(InvokerHelper.java:778)
      at org.codehaus.groovy.runtime.InvokerHelper.invokeMethod(InvokerHelper.java:758)
      at org.codehaus.groovy.runtime.InvokerHelper.runScript(InvokerHelper.java:401)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:585)
      at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:86)
      at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:230)
      at groovy.lang.MetaClassImpl.invokeStaticMethod(MetaClassImpl.java:1105)
      at org.codehaus.groovy.runtime.InvokerHelper.invokeMethod(InvokerHelper.java:749)
      at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodN(ScriptBytecodeAdapter.java:170)
      at edu.ucdavis.genomics.metabolomics.binbase.quantificator.dsl.Tester.main(Tester.groovy)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:585)
      at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:86)
      at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:230)
      at groovy.lang.MetaClassImpl.invokeStaticMethod(MetaClassImpl.java:1105)
      at org.codehaus.groovy.runtime.InvokerHelper.invokeMethod(InvokerHelper.java:749)
      at groovy.lang.GroovyShell.runMainOrTestOrRunnable(GroovyShell.java:244)
      at groovy.lang.GroovyShell.run(GroovyShell.java:218)
      at groovy.lang.GroovyShell.run(GroovyShell.java:147)
      at groovy.ui.GroovyMain.processOnce(GroovyMain.java:493)
      at groovy.ui.GroovyMain.run(GroovyMain.java:308)
      at groovy.ui.GroovyMain.process(GroovyMain.java:294)
      at groovy.ui.GroovyMain.processArgs(GroovyMain.java:111)
      at groovy.ui.GroovyMain.main(GroovyMain.java:92)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:585)
      at org.codehaus.groovy.tools.GroovyStarter.rootLoader(GroovyStarter.java:101)
      at org.codehaus.groovy.tools.GroovyStarter.main(GroovyStarter.java:130)

time to give groovy 1.6.4 a try and maybe migrate my project over to it.

DSL - love or hate?

2009-09-09T13:00:00.000-07:00

recently I started to work more with DSL's (domain specific language) since I wanted to go away from doing everything in XML.

The approach of XML is great, but I'm not a big fan of the overhead with it.

So the first approach was to create a DSL for the BinBase system, precisely to provide the user with a really simple way of calculating an experiment.

It should look something like this:


/**
 * our quantification dsl
 */
quantify {

  //define the name of the setup
  name "quantification for ${new Date()}"

  //define the samples for the calculations
  sample "sample 1"
  sample "sample 2"
  sample "sample 3"
  sample "sample 4"
  sample "sample 5"
  sample "sample 6"
  sample "sample 7"
  sample "sample 8"
  sample "sample 9"
  sample "sample 10"

  //defines the bins in the table
  bin 321, 22.0
  bin 322
  bin 323
  bin 324
  bin 325, 22.0

  //report definition
  report {

    sizeDown 50

    format "xls"

    replace true
  }
}

And the result should calculate the data, send them to the cluster and generates a quantification report. If this all works out the way we want it. We will than adapt it to schedule BinBase experiment's this way.

My current fazit is that I love the idea of DSL's, but hate the implementations of it.

jboss - binding to all registered network cards

2009-09-09T11:56:00.000-07:00

The JBoss Application server binds only to the local interface '127.0.0.1' since the version 4.0

Since this is not desired in a production system you can override this feature with the '-b' command.

An example to start jboss in the all configuration and bound to all registered interfaces would be


sh run.sh -b 0.0.0.0 -c all

creating a project with maven2

2009-09-03T12:28:00.000-07:00

it is kinda pathetic I'm using maven since several years now and just can't remember this one command.


mvn archetype:create -DgroupId=myGroup -DartifactId=myArtifact

so I write it down here to make it easier for me to lookup.

osx - set disk as startup disk via terminal

2009-09-02T15:11:00.001-07:00

to set a disk as startup disk is quite simple just execute:


sudo bless -mount /Volumes/"name of your startup disk" -setBoot

run-app vs run-war

2009-09-01T10:09:00.000-07:00

the main differences between these two commands is that:

run-app runs a grails applications and allows the reloading from resources
run-war generates a war file and runs the war file, so no reloading of resources.

Short during development you want to use run-app.

maven2 - define the context root of an webapp

2009-08-31T11:59:00.000-07:00

Basically I have an application (ear) based on several jars and a war which I want to easily access with a context root of my choice.

Solution:


   <plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-ear-plugin</artifactId>
    <configuration>
     <archive>
      <manifest>
       <addClasspath>true</addClasspath>
      </manifest>
     </archive>
     <jboss>
      <version>4</version>
     </jboss>
     <bundleFileName>
      binbase-quality-control.ear
     </bundleFileName>

     <modules>
      <webModule>
       <groupId>edu.ucdavis.genomics.metabolomics.binbase.quality
       </groupId>
       <artifactId>quality-war</artifactId>
       <contextRoot>/quality-war</contextRoot>
      </webModule>
     </modules>

    </configuration>
   </plugin>

Method "ApplyConfiguration" with signature "" on interface "org.gnome.SettingsDaemon.XRANDR" doesn't exist

2009-08-09T14:20:00.000-07:00

today I wanted todo nothing else than the change the primaery output of my laptop to the 40" display attached over DVI to watch movies on it.

Sadly since ubuntu 8.something I kept getting the following error message


Method "ApplyConfiguration" with signature "" on interface "org.gnome.SettingsDaemon.XRANDR" doesn't exist

So I upgraded to the newest ubuntu version. Didn't make it much better. So I could had given up here and start shouting all kinds off stuff about linux or trying to make it work.

So here is the solution:

the gui is basically broken and you need to execute the commands to adjust everything your self. Using 'xandr'

for example to turn off a monitor:


xandr --output LVDS --off

or to change the resolution


xandr --output DVI-0 --mode 1360x768 --pos 0x0

2 minutes later I had my configuration up and running.

synchornized groovy

2009-08-04T12:33:00.000-07:00

Today I discovered the question:

I want to have a websites which fires calculations to a database server, but the controller should only execute one calculation at a time

how do you do this?

The first approach was to try to queue the ajax calls, which obviously won't work if two people run calculations at the same time.

So plan b was to use synchronized groovy code.


  /**
   * does the actual calculation
   */
  def calculate_ajax = {

    assert params.database != null, "you need to provide 'database' in the params object"
    assert params.date != null, "you need to provide 'date' in the params object"
    assert params.machine != null, "you need to provide 'machine' in the params object"
    assert params.pattern != null, "you need to provide 'pattern' in the params object"
    assert params.position != null, "you need to provide 'position' in the params object"

    //needs to be synchronized so we don't kill the database server with this task
    synchronized (LOCK) {
      println new Date()
      BinBaseQualityConnector.getQualityService().calculateSingleStandard(params.pattern, params.database, new Date(Long.parseLong(params.date)), params.machine, BinBaseQualityConnector.getKey())
      render("<div class=\"after_complete\" style=\"width:${100 / BinBaseQualityConnector.getPattern(params).size()}%\"></div>")

    }
  }

  static final def LOCK = "lock"

and a possible ajax call for this looks like this


                  <script type="text/javascript">
                    YAHOO.util.Event.onDOMReady(
                            function() {
                              new Ajax.Updater('${i}_${y}_${z}', '/quality-war/processing/calculate_ajax', {
                                method: 'get', frequency: 3,parameters:'date=${date.getTime()}&database=${database}&machine=${machine}&pattern=${pattern}&position=${z}'
                              });
                            });
                  </script>

updating more than one div with grails and ajax

2009-07-28T15:49:00.001-07:00

sometimes it happens that you have to update several div's on one webpage, when you click on a form or a button.

Now you can either reload the complete page, which can take a while with a lot of data or just queue remote functions in a java script function like this


  function addStandard() {

    if (YAHOO.example.container.addstandard) {
      YAHOO.example.container.addstandard.hide();
    }

    //get our element values
    var conc = document.getElementsByName('qs_concentration')[0].value;
    var pattern = document.getElementsByName('qs_pattern')[0].value;

    //calls the remote functions
  ${remoteFunction(
                      controller: 'management',
                      action: 'addStandard_ajax',
                      params: '\'conc=\' + escape(conc) + \'&pattern=\' + escape(pattern)',
                      update: 'registered_standards')}

  ${remoteFunction(
                      controller: 'messenger',
                      action: 'showMessage',
                      params: '\'message=you should recalculate the database now!\'',
                      update: 'messenger')}
  }

javascripts and maps

2009-07-28T14:45:00.000-07:00

since java script contains no Maps, well use an object instead with dynamic properties.


var test = new Object();
test['tada'] = 'yada';

alert(test['yada']);
alert(test.yada);

distinct or group by, what's more expensive

2009-07-28T09:14:00.001-07:00

today I was asking my self about the best approach for a performance critical application.

Distinct or group by '*'



select name from table group by name



select distinct name from table

after all they both return the exact same result.

Some explain statements later it turned out that the group by clause is 18% faster than the distinct statement.

sometimes it's simpler to use plain ajax...

2009-07-27T17:54:00.001-07:00

today I was struggling for a bit with generating the update part of a remote function on the fly and after a couple of minutes. I started thinking, do I have to really use this tag? Why don't I use just plain Ajax, since grails will generate the same code anyway.

Solution:


            <div class="center">
              <img name="${bin.getId()}" src="${createLinkTo(dir: 'images', file: 'remove.gif')}" alt="zoom" onclick="new Ajax.Updater

                      ('bin-container-${db}', '/quality-war/management/remove_bin_ajax', {asynchronous:true,evalScripts:true,parameters


                              :'bin=${bin.getId()}&database=${db}'});"/>
            </div>

this also shows how to submit several parameters at once in plain ajax style.

lesson of the day...

2009-07-27T17:17:00.001-07:00

and god said you shall no duplicate div ids,
cause if you do,
you shall be send to the hell of css,
constant torture and misfortune will await you,

so do not duplicate your id's!

problem:

updating a you tab with some dynamic generated content, which generate the same div id over and over again...

Annoying maven bug

2009-07-27T00:40:00.000-07:00

well tonight I got an interesting error message from my bamboo build server


[INFO] ------------------------------------------------------------------------
[ERROR] BUILD ERROR
[INFO] ------------------------------------------------------------------------
[INFO] Error installing artifact's metadata: Error installing metadata: Error updating group repository metadata

input contained no data
[INFO] ------------------------------------------------------------------------
[INFO] For more information, run Maven with the -e switch

without actually doing any changes at the pom.xml. The problem turned out that the repository was corrupt.

So all I had todo was to delete my project from the local build repository.

Grails - using several arguments with a remote function

2009-07-23T15:46:00.000-07:00

in grails it's a bit tricky to use a remote function and pass more than one parameter, since it does not follow the map approach like the usual approach

After a bit of asking Dr Google I finally found an approach which worked


    var conc = document.getElementsByName('qs_concentration')[0].value;
    var pattern = document.getElementsByName('qs_pattern')[0].value;

    ${remoteFunction(
                      controller: 'management',
                      action: 'addStandard_ajax',
                      params: '\'conc=\' + escape(conc) + \'&pattern=\' + escape(pattern)',
                      update: 'registered_standards')}

now I just wish there would be an easier approach

grails remote function on image or checkbox

2009-07-23T13:34:00.000-07:00

Today I was in the situation that I wanted to call a remote function on an image to emulate a button, to delete an object with a specific id.

It's simple enough,


        <img name="${standard.getPosition()}" src="${createLinkTo(dir: 'images', file: 'remove.gif')}" alt="zoom" onclick="${remoteFunction(
                       controller: 'management',
                       action: 'removeStandard_ajax',
                       params: '\'standard=\' + + escape(this.name)',
                       update: 'registered_standards')}"/>

To make live easy we just save the id in the name field of the image, but you can also save it somewhere else, depending on your requirements.

Now another thing you can do is todo the same thing for checkfields.


         <input type="checkbox" id="cacheSetting" name="cacheSetting" onchange="${remoteFunction(
                 controller: 'management',
                 action: 'updateCache_ajax',
                 params: '\'cacheSetting=\' + escape(this.checked)',
                 update: 'general_config')}"/>

If you want to preserve the check state you can do this with this


          <input type="checkbox" id="cacheSetting" name="cacheSetting" <%if(cacheSetting) println "checked"%> onchange="${remoteFunction(
                  controller: 'management',
                  action: 'updateCache_ajax',
                  params: '\'cacheSetting=\' + escape(this.checked)',
                  update: 'general_config')}"/>

and an example controller function could look like this



 def updateCache_ajax = {
   assert params.cacheSetting != null, "you need to provide \"cacheSetting\" in the params object"
   BinBaseQualityConnector.getQualityConfigService().setCaching(Boolean.parseBoolean(params.cacheSetting))

   //define our model
   def model = new HashMap()

   //build the model
   model.cacheSetting = BinBaseQualityConnector.getQualityConfigService().isCaching()

   render(template: "general", model: model)
 }

scala in grails

2009-07-20T23:55:00.000-07:00

I know it's evil and I know you don't want todo this, but what the heck. This plugin allows you to compile and use scala code in grails.

I don't know why I would want to use scala right now in this, but I'm sure I find out a reason or two :)

YUI and Prototype - updating an element

2009-07-16T16:23:00.000-07:00

Well currently I'm working a lot with YUI and specially tabs. So one of the first things I wanted todo is to select a tab and this selections should do two things

show the content of the tab, happens by default
update a div on the page somewhere with some content related to the tab content

so how do you do this?

Quite simple.



<script type="text/javascript">
    var myTabs = new YAHOO.widget.TabView('demo');
    var tab0 = myTabs.getTab(0);
    
    function handleClick(e,tab) {  
        ${"div_to_update"}.update("I just clicked on my tab!");
    }
    
    tab0.addListener('click', handleClick,tab0);
</script>

and as you can see we use the 'update' function from the prototype library.