testing all urls in a sitemap.xml file for existence with groovy

While working on some projects I figured out that it would be nice to have a quick and easy way to check all registered urls in a sitemap for existence and that groovy would be the perfect tool for this.

So we wrote a simple script for this


@Grab(group='org.codehaus.groovy.modules.http-builder', module='http-builder', version='0.5.0-RC2' )
import groovyx.net.http.HTTPBuilder

//small script to verify that an url exist, which was specified in a sitemap
//args[0] is the sitemap

//simple check
if(args.length != 1){
println "Usage\n"
println "please provide one argument, which is a filename pointing to your sitemap file"
println "\n"
return
}

//load file

def file = new File(args[0])

if(file.exists() == false){
println "\nsorry your file was not found. Missing file is ${args[0]}\n"
return
}

//parsing urls
def root = new XmlSlurper().parse(file)

def urls = root.url

println "\nchecking ${urls.size()} urls\n"

def success = 0
def failure = 0

urls.each{
def url = it.loc.text()

print "success:\t"

try {
new HTTPBuilder( url ).get( path:'' ) { response ->
response.statusLine.statusCode == 200

print "true"
success++
}
}
catch( e ) {
print "false"
failure++
}

print "\t that ${url} exists\n"
}

println "\nreport..."
println "\tsuccessful: ${success}"
println "\tfailed: ${failure}"
println "\ttotal: ${urls.size()}"

println "\nchecked ${urls.size()} urls in file ${args[0]}\n\n"



and the output looks something like this...


groovy verifyUrls.groovy Downloads/sitemap.xml

checking 45 urls

success: true that http://metacore-ucdavis.appspot.com/ exists
success: true that http://metacore-ucdavis.appspot.com/services exists
success: true that http://metacore-ucdavis.appspot.com/techno1 exists
success: true that http://metacore-ucdavis.appspot.com/techno2 exists
success: true that http://metacore-ucdavis.appspot.com/techno3 exists
success: true that http://metacore-ucdavis.appspot.com/setupx exists
success: true that http://metacore-ucdavis.appspot.com/projects exists
success: true that http://metacore-ucdavis.appspot.com/staff exists
success: true that http://metacore-ucdavis.appspot.com/login_form exists
success: true that http://metacore-ucdavis.appspot.com/join_form exists
success: true that http://metacore-ucdavis.appspot.com/mail_password_form exists
success: true that http://metacore-ucdavis.appspot.com/Members/admin exists
success: true that http://metacore-ucdavis.appspot.com/services/ exists
success: true that http://metacore-ucdavis.appspot.com/services/statistics exists
success: true that http://metacore-ucdavis.appspot.com/services/protocols exists
success: true that http://metacore-ucdavis.appspot.com/techno1/ exists
success: true that http://metacore-ucdavis.appspot.com/techno1/compounds exists
success: true that http://metacore-ucdavis.appspot.com/techno1/statistics exists
success: true that http://metacore-ucdavis.appspot.com/techno2/ exists
success: true that http://metacore-ucdavis.appspot.com/techno3/ exists
success: true that http://metacore-ucdavis.appspot.com/setupx/ exists
success: true that http://metacore-ucdavis.appspot.com/projects/ exists
success: true that http://metacore-ucdavis.appspot.com/staff/ exists
success: true that http://metacore-ucdavis.appspot.com/join_form?came_from= exists
success: true that http://metacore-ucdavis.appspot.com/index_html/view exists
success: true that http://metacore-ucdavis.appspot.com/Members/ exists
success: true that http://metacore-ucdavis.appspot.com/Members/admin/ exists
success: true that http://metacore-ucdavis.appspot.com/services/index_html/view exists
success: true that http://metacore-ucdavis.appspot.com/statistics/ exists
success: true that http://metacore-ucdavis.appspot.com/services/protocols/ exists
success: true that http://metacore-ucdavis.appspot.com/services/protocols/Metabolomics%20Vol.%201%2C%20No.%201%2C%20January%202005%20%28%202005%29.pdf exists
success: true that http://metacore-ucdavis.appspot.com/services/protocols/Proteomics%202004%2C%204%2C%2078-83.pdf exists
success: true that http://metacore-ucdavis.appspot.com/services/protocols/SulfurDeprivation.pdf exists
success: true that http://metacore-ucdavis.appspot.com/techno1/index_html/view exists
success: true that http://metacore-ucdavis.appspot.com/techno1/compounds/ exists
success: true that http://metacore-ucdavis.appspot.com/techno2/index_html/view exists
success: true that http://metacore-ucdavis.appspot.com/techno3/index_html/view exists
success: true that http://metacore-ucdavis.appspot.com/setupx/index_html/view exists
success: true that http://metacore-ucdavis.appspot.com/projects/index_html/view exists
success: true that http://metacore-ucdavis.appspot.com/staff/index_html/view exists
success: true that http://metacore-ucdavis.appspot.com/Members exists
success: true that http://metacore-ucdavis.appspot.com/Members/admin/index_html/view exists
success: true that http://metacore-ucdavis.appspot.com/statistics exists
success: true that http://metacore-ucdavis.appspot.com/statistics/index_html/view exists
success: true that http://metacore-ucdavis.appspot.com/techno1/compounds/index_html/view exists

report...
successful: 45
failed: 0
total: 45

checked 45 urls in file Downloads/sitemap.xml




java6 on ubuntu

just a quick tip I found how to use apt-get to install sun-java and not open-java.

sudo apt-get install python-software-properties
sudo add-apt-repository ppa:ferramroberto/java
sudo apt-get update
sudo apt-get install sun-java6-jdk sun-java6-plugin

You may want to also add the following

sudo update-alternatives --config java

You should get the following

Selection Path Priority Status
------------------------------------------------------------
* 0 /usr/lib/jvm/java-6-openjdk/jre/bin/java 1061 auto mode
1 /usr/lib/jvm/java-6-openjdk/jre/bin/java 1061 manual mode
2 /usr/lib/jvm/java-6-sun/jre/bin/java 63 manual mode

Press enter to keep the current choice[*], or type selection number: 2
Select (2) and press enter

Now running:

java -version
Returns:

java version "1.6.0_26"
Java(TM) SE Runtime Environment (build 1.6.0_26-b03)
Java HotSpot(TM) Server VM (build 20.1-b02, mixed mode)

Taken from:

http://superuser.com/questions/353983/how-do-i-install-the-sun-java-sdk-in-ubuntu-11-10-oneric

moving a virtual machine from VMWare Workstation/Fusion to vSphere

I'm currently in the process of moving several virtual machine from my home system (VMWare Workstation) and my work system (VMWare Fusion) to my new vSphere ESXI server and running into all kinds of issues.

First of all you can't run the converter on anything else than windows, which is rather annoying. Since I only got macs at work and no Windows machines, except for mine at home. Which has no access to the work network and I refuse to carry my 50lbs workstation around.

Translation: VMWare forces me to install a windows virtual machine to convert a virtual machine from vmx to vSPhere.

Yes I am slightly irritated about this lack of support for us apple fan boys (don't forget, linuxers have todo the same)

The next issue I encountered there doesn't seem to be a possibility to define Nat/Bridged and host only anymore. Instead I need to deal with custom vlans now or possible mixup some other DHCP services in the office.

For example my rocks linux cluster for testing binbase has 3 virtual network interfaces:

eth0 = host only (fixed IP address, fixed MAC)
eth1 = host only (fixed IP address, fixed MAC, for cluster nodes)
eth2 = bridge (dynamic ip address, random MAC, so that external hosts can easily access BinBase from the world and the image has internet access)

this is rather straight forward in VMware player/fusion/workstation.

Now for vSphere it looks rather different...

Basically you need to define networks for each interface and so my best idea was to simulate this so far.

To simulate the host only network:

a) create a new switch called 'host only' which is not connected to a network device.

To simulate the bridged network:

just use the default location and rename it to public to simplify life a bit.

and assign them in the vWware image to the different network interfaces.

etho0 = host only
eth1 = host only
eth2 = public

now the big question is, if I created another virtual machine, in the same ip range as my host only network is defined, can it talk to the other machines? I think I shall test this with a virtual compute node and report back. Maybe to be on the save site, I just create a second host only network?

integration testing with the grails export plugin

currently I'm testing a lot of controllers in my miniX application and run into the issue that it's nearly impossible to test the grails export plugin. Since the setup tends to get more complicated than your actual test.

Obviously I could redesign my controllers and make them more transparent. But this is not the point right now.

So lets look at our controller:



/**
* should be a service and not a method in the controller, bad design from me. -5 points
*/
def exportStudies(def studies, def response, String format, String extension, String filename) {

//title of the file
def parameters = [title: "Study Design"]

//required labels and mappings
def label = [
"classId": "Class Id",
"comment": "Comment",
"fileName": "Filename",
"label": "Label",
"organName": "Organ",
"speciesName": "Species",
"treatment": "Treatment"
]

//required fields
def fields = [
"classId",
"comment",
"fileName",
"label",
"organName",
"speciesName",
"treatment"
]
//contains out data for the save
def data = []

studies.each {Studie studie ->

//fill our object with data
studie.classes.each {BaseClass clazz ->
if (clazz instanceof ExperimentClass) {
clazz.samples.each {Sample sample ->

StudieExportObject exportObject = new StudieExportObject()

exportObject.classId = clazz.id
exportObject.speciesName = clazz.species.name
exportObject.organName = clazz.organ.name
exportObject.treatment = clazz.treatmentSpecific.toParentString()
exportObject.fileName = "${sample.fileName}_${sample.fileVersion}"
exportObject.comment = sample.comment
exportObject.label = sample.label

data.add(exportObject)

}
}
}
}

//set the extension to xls
response.setHeader("Content-disposition", "attachment; filename=${filename}.${extension}")

//calling the export service, which is really uggly implemented
exportService.export(format, response.outputStream, data, fields, label, [:], parameters)

}

/**
* export the studie as xls file
*/
def exportStudieToXLS = {

Studie studie = Studie.get(params.id)

if (studie == null) {
flash.message = "sorry this object wasn't found!"
redirect(action: "list", controller: "studie")
}
else if (!studie.canAccess()) {
flash.message = "sorry you do not have the permission to access this object!"
redirect(action: "list", controller: "studie")
}
else {

//run the export
exportStudies([studie], response, "excel", "xls", studie.id.toString())
}
}



after spending some time with the attempt to setup the export service correctly, I just scrapped this approach and figured. I assume it works and it's well tested. So lets rather test our logic and make sure it works. Hence we just mock the service in our test.


void testExportStudies() {


Studie studie = AquisitionTableServiceTests.generateNamedStudie(20, false)

GCTof tof = GCTof.list()[0]

AquisitionTableService service = new AquisitionTableService()
GenerateFileNameService generateFileNameService = new GenerateFileNameService()
generateFileNameService.modifyClassService = new ModifyClassService()

generateFileNameService.addCalibrationSamples(studie, tof, new Date(), ShiroUser.list()[0])
generateFileNameService.addReactionBlankSamples(studie, tof, new Date(), ShiroUser.list()[0])
generateFileNameService.addQualityControlSamples(studie, tof, new Date(), ShiroUser.list()[0])


ExportStudieController controller = new ExportStudieController()

//mocking the export service, since the developer of it did a really shitty job to ensure it can be easily tested, FAIL
controller.exportService = [
export : { def a, def b, def c, def d, def e,def f, def g ->
assertTrue(a instanceof String)
assertTrue(b instanceof OutputStream)
assertTrue(c instanceof Collection)
assertTrue(d instanceof Collection)
assertTrue(e instanceof Map)
assertTrue(f instanceof Map)
assertTrue(g instanceof Map)

//some cool testing logic gets here...
}
] as ExportService

controller.metaDataService = new MetaDataService()

controller.exportStudies([studie], controller.response, "csv", "txt", "test")
}


installing vmware server on ubuntu 11.10

so I'm trying to install a ubuntu base vmware image hosting platform right now keep running into issues.

done on a system with the following kernel config:

Linux sun 3.0.0-15-server #26-Ubuntu SMP Fri Jan 20 19:07:39 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

first you need to link some files to make the script start generating the modules...

$ sudo ln -s /lib/modules/your_kernel_version/build/include/generated/utsrelease.h /lib/modules/your_kernel_version/build/include/linux/utsrelease.h
$ sudo ln -s /lib/modules/your_kernel_version/build/include/generated/autoconf.h /lib/modules/your_kernel_version/build/include/linux/autoconf.h

which now makes the installation fail with some error of missing smp.h header files

/tmp/vmware-root/modules/vmmon-only/linux/driver.c:40: fatal error: linux/smp_lock.h: No such file or directory
compilation terminated.

next step was to downgrade the kernel to the last version where I found enough info about it to make it compile with vmware.

Linking the files again

ln -s /lib/modules/2.6.38-10-server/build/include/generated/utsrelease.h /lib/modules/2.6.38-10-server/build/include/linux/utsrelease.h
ln -s /lib/modules/2.6.38-10-server/build/include/generated/autoconf.h /lib/modules/2.6.38-10-server/build/include/linux/autoconf.h

and now it fails with the following error

/tmp/vmware-config0/vmmon-only/linux/driver.c:1423:4: error: too many arguments to function ‘smp_call_function’
include/linux/smp.h:73:5: note: declared here
/tmp/vmware-config0/vmmon-only/linux/driver.c: In function ‘LinuxDriver_Ioctl’:
/tmp/vmware-config0/vmmon-only/linux/driver.c:1987:18: error: ‘struct task_struct’ has no member named ‘euid’
/tmp/vmware-config0/vmmon-only/linux/driver.c:1987:35: error: ‘struct task_struct’ has no member named ‘uid’
/tmp/vmware-config0/vmmon-only/linux/driver.c:1988:11: error: ‘struct task_struct’ has no member named ‘fsuid’
/tmp/vmware-config0/vmmon-only/linux/driver.c:1988:29: error: ‘struct task_struct’ has no member named ‘uid’
/tmp/vmware-config0/vmmon-only/linux/driver.c:1989:18: error: ‘struct task_struct’ has no member named ‘egid’
/tmp/vmware-config0/vmmon-only/linux/driver.c:1989:35: error: ‘struct task_struct’ has no member named ‘gid’
/tmp/vmware-config0/vmmon-only/linux/driver.c:1990:11: error: ‘struct task_struct’ has no member named ‘fsgid’
/tmp/vmware-config0/vmmon-only/linux/driver.c:1990:29: error: ‘struct task_struct’ has no member named ‘gid’
/tmp/vmware-config0/vmmon-only/linux/driver.c:2007:7: error: too many arguments to function ‘smp_call_function’
include/linux/smp.h:73:5: note: declared here
make[2]: *** [/tmp/vmware-config0/vmmon-only/linux/driver.o] Error 1
make[1]: *** [_module_/tmp/vmware-config0/vmmon-only] Error 2
make[1]: Leaving directory `/usr/src/linux-headers-2.6.38-10-server'

getting frustrated I went back to a bit more car research. Still at the point of 335i vs Cayman vs Boxster vs Z4. It all depends on the european delivery of if I can find a nice 1-2 year old cpo'ed car in my area.

calmed down a lot and so the next step was to rename my vmware server archive to the right extension:

mv VMware-server-2.0.2-203138.x86_64.gz VMware-server-2.0.2-203138.x86_64.tar.gz

and following the instructions here

and it worked

Starting VMware services:
Virtual machine monitor done
Virtual machine communication interface done
VM communication interface socket family: done
Virtual ethernet done
Bridged networking on /dev/vmnet0 done
Host-only networking on /dev/vmnet1 (background) done
DHCP server on /dev/vmnet1 done
Host-only networking on /dev/vmnet8 (background) done
DHCP server on /dev/vmnet8 done
NAT service on /dev/vmnet8 done
VMware Server Authentication Daemon (background) done
Shared Memory Available done
Starting VMware management services:
VMware Server Host Agent (background) done
VMware Virtual Infrastructure Web Access
Starting VMware autostart virtual machines:
Virtual machines done

The configuration of VMware Server 2.0.2 build-203138 for Linux for this
running kernel completed successfully.

Housekeeping...
Thank you for using the script!
Patch provided by:
Ramon de Carvalho Valle
http://risesecurity.org
updated by sirusdv (http://ubuntuforums.org/member.php?u=1289848)
Script author:
Radu Cotescu
http://radu.cotescu.com

now I just need to convince grub to use the 'old' kernel by default...