Wednesday, August 11, 2010

Cross Compiling OpenVPN for Windows on Linux

I went through quite a struggle to build OpenVPN and a custom installer for Windows using my Linux machine. This post describes how to achieve this. To make it easier I packaged all the actual build steps into a script.

This post works on the cross compiling environment prepared in my previous post Building a Cross Compiler on Linux for MinGW32.

If you want to build the installer exe for OpenVPN as well, you will need NSIS (Nullsoft Installer System) installed. Download and compile it from http://nsis.sourceforge.net/.

You will still be able to build the exes without NSIS. NSIS is only needed for packaging them into an installer.

Also note that this doesn't build the TAP driver. It just copies the prebuilt one. To do this would have to install the Microsoft DDK, make an amd64 cross compile environment and modify my script to build the driver instead of copying it.

First download the following:
  1. Prebuilt packages from http://openvpn.net/prebuilt/. Choose the latest -prebuilt .tbz file.
  2. Download the latest OpenVPN source code tar.gz archive.
  3. Download the build scripts from here.
Then extract the build scripts to your home directory. This will then create a directory ~/openvpn-src which contains ~/openvpn-src/archive. Copy the prebuilt and OpenVPN source code packages into ~/openvpn-src/archive.

Modify the ~/openvpn-src/env.sh script to reflect your cross-compiler environment. The variables have the following purposes:
  • PREFIX - Where to install the compiled OpenVPN files
  • TARGET - The build environment you're targeting, for example i686-mingw32msvc
  • TOOLCHAIN - The root location where the cross compiler binaries are access from.
  • MAKENSIS - The full path to your makensis variable. Leave this empty if you don't want the installer to be created or if you don't have NSIS installed.
  • The rest of them are standard autoconf environment variables.
When you're ready you can kick of build-openvpn.sh.

The resulting OpenVPN .exe files will be located in the directory your PREFIX variable points to, which by default would be ~/openvpn-dist.

This script was tested with openvpn-2.1.1 and 2.1_rc22-prebuilt.tgz.

Building a Cross Compiler on Linux for MinGW32

I was installing OpenVPN the other day and after getting everything up and running wanted to build an installer for Windows, to make it easier for the average person to be able to connect to the company VPN.

The basic idea was to prompt for a P12 cert and it's private key password during the installation and then install this along with the OpenVPN client. OpenVPN has an option "askpass", which allows you to store the private key password for a CRT or P12 cert. The build provided on openvpn.net, however, has this feature disabled. Due to this I had to recompile OpenVPN.

Not having Windows installed I knew I was going to have to figure out another way to get it to compile. I could either try and get MinGW32 running in Wine or setup a cross compiler environment. I thought the latter would be the easier option (as I was quite familiar with the existing Linux building environment). Boy was I wrong.

Either way. During the process of building the cross compiler, sorting out problem after problem I came across a script written by Paul Millar, which will extract, patch and compile MinGW32 for you as a cross compiler. This is a life saver.

To honor his hard work, even though still a bit tricky to use, I'm documenting it here.

Firstly, download and extract this script into ~/mingw-src. You can get it here. Inside this directory also create 2 other directories called Archive and Patches.

Then, get hold of the following source packages from http://www.sourceforge.net/projects/mingw/files/:
  • gcc
  • binutils
  • w32api
  • mingw-runtime
At the time of writing this, they had the following names:
  • gcc-3.4.4-3-msys-1.0.13-src.tar.lzma
  • binutils-2.19.51-3-msys-1.0.13-src.tar.lzma
  • w32api-3.14-3-msys-1.0.12-src.tar.gz
  • mingw-runtime-3.14-src.tar.gz
Then extract these archives into ~/mingw-src/Archive. The lzma archives would first have to be extacted with lzma into a temporary location, and the resulting tar archive extracted into ~/mingw-src/Archive.

Extract them one at a time, because some of them also contain patches. As you extract each of these, copy the patches into the ~/mingw-src/Patches directory, naming them {prefix}_name.patch, where {prefix} would be gcc, binutils, w32api or mingw-runtime_, depending on which archive the patch originated from.

For example, the binutils package has 2 patches, nl.
  1. 01-scriptdir.patch
  2. binutils-2.19.51-1-msys.patch
These would respectively be places in ~/mingw-src/Patches with these names:
  1. binutils_01-scriptdir.patch
  2. binutils_binutils-2.19.51-1-msys.patch.
The important part is the prefix and underscore. What comes after it doesn't actually matter. So you could name them binutils_1 and binutils_2 if you prefer.

When you're done with this, edit the parameters file, applying the following changes:
  1. Comment the LANGUAGES variable.
  2. There are 2 definitions of the DEFAULT_CC variable. The first assigns it the value gcc-4.0 and the second prefixes it with a distcc invocation. Change the value of the first to only gcc and comment the second.
  3. Comment the DISTCC_LOG and DISTCC_HOSTS variables.
  4. Change MAKE_PROCESS_COUNT to equal the number of CPU cores you have available, multiplied by 2, plus 1. So if you have 4 cores, this would be (4*2) + 1, which is 9.
  5. Comment the ROOT_TARGET variable.
  6. Comment the CLUSTER_DEPLOY variable.
  7. Check what the extensions are for your binutils, w32api and mingw-runtime archives in the ~/mingw-src/Archive directory. If it's not tar.gz for any of these, update the BINUTILS, W32API and RUNTIME variables to reflect the correct extension.
After the script has finished building the packages, change to ~/mingw32 where the packages were deployed and run the following commands:
cd ~/mingw32
cp -Rp i686-mingw32msvc/include/ i686-mingw32msvc/lib/ ./
rm -rf i686-mingw32msvc/include/ i686-mingw32msvc/lib/
ln -s ../include i686-mingw32msvc/include
ln -s ../lib i686-mingw32msvc/lib
Then you should have a working cross compiler environment in ~/mingw.

To use it, load the following environment:
PREFIX=$HOME/openvpn
TARGET=i686-mingw32msvc
TOOLCHAIN=$HOME/mingw32
BIN="$TOOLCHAIN/bin"
export PATH="$BIN:$TOOLCHAIN/$TARGET/bin:$PATH"
export CC="$BIN/$TARGET-gcc"
export LDFLAGS="-L$TOOLCHAIN/lib"
export CFLAGS="-I$TOOLCHAIN/include"
export CROSS_COMPILE="$TARGET"
Remember to update the value of PREFIX to where you want to install the resulting package, which in the example is ~/openvpn.

Once loaded you can compile a package (like OpenVPN) with the following command. This has to be run from the directory where it's source code is extracted.
./configure --prefix=$PREFIX --build=$TARGET && make && sudo make install
And that's all there is to it. The resulting package will be located in the directory where PREFIX points.

Saturday, November 21, 2009

Advanced Digi Discovery Protocol Explained

This is a continuation of my previous post.

If you haven't read the previous post I recommend you do so first, but to summarize. ADDP, or Advanced Digi Discovery Protocol is a UDP multicast protocol used for discovering and reconfiguring network settings of Digi devices (such as the Digi Connect ME) on the local network, irrelevant of their network configuration.

I've basically completed my documentation and implementation of ADDP, though like software usually is, it will never be truly complete, and the documentation of the protocol is still missing a few fields and codes. It's definitely complete enough to release so people can implement the protocol themselves.

The implementation I made is a cross platform Java version of their "Digi Device Discovery" tool from Digi International Inc. It's called JDigiDiscover and lists all Digi devices on the network (which have ADDP activated) with detailed public configuration for them, and then allows you to either restart the device or change it's network configuration. You can see some screenshots and/or download it from it's SourceForge.net project page. It's released under the Apache License 2.0.


ADDP is a proprietary protocol developed by and it's copyright owned by Digi International Inc. Use of the protocol is limited to their products and their terms. I released this purely because I noticed a need for implementing it in your own non C and non Windows applications for managing your own Digi devices. For more information see http://www.digi.com/.

ADDP
Summary
The ADDP protocol is a UDP multicast protocol, doing broadcasts on 224.0.5.128 port 2362. This is the default configuration, though it is possible to disable this for a given Digi device, or change the port.

All communications are in the form of request/response, and there exists 4 communication types which give a total of 8 packets. They are Discovery, Static Configuration, DHCP Network Configuration and Restart.

All packets consists of a header and a payload. The header format is common throughout all packet types, where the payload is common on response packets but unique for each request packet type.

The Header
The header starts with the magic bytes "DIGI", ie. 0x44,49,47,49. This is followed by a 2 byte packet type identifier (see the tables at the end of this document). After the identifier is a 2 byte short, which indicates the length/size of the payload.

The Payload
As explain, the payloads of all request packets are unique to the type of packet, where the response packets have a common format. Response packets contain information pertaining to the request. This information is given in a variable amount of fields, each field being identified by a 1 byte identifier. These identifiers are shared across all response packets, each response sending only the information relevant to the request.

So a field takes the form of a 1 byte field type ID (see the tables at the end of this document), then a 1 byte field length followed the the field's data. The field length byte indicates how many data bytes need to be read. This will always be at least one. Take for example the IP address field. It's type id is 0x02, and being an IP address it will be 4 bytes long. If the IP address was 10.0.0.5, then the field would have the following value: 0x02,04,0a,00,00,05. Response packets always have the MAC address field set, and is the MAC address of the device the packet originated from. You use this to pair responses with requests.

When a request packet had an invalid format, even though the MAC address was valid an it's obvious the packet was meant for a given device, it will be ignored and no response packet will be sent. Response packets are only sent when the request packet had a 100% valid structure. Failure responses are sent only for failure after the packet was parsed successfully.

So to parse the payload, you just read the fields in a serial manner until the complete payload has been consumed. Not all responses of the same type will always have the same fields. So it's important to read the payload based on the size specified in the header. Further, depending on the field type, an absent field could mean that it's the default value or irrelevant to the type of packet. For example, in the restart response packet the IP address field won't be set, where the absense of the RealPort number field in a discovery response could indicate it's disabled.

When transferring MAC addresses or IP addresses, they are sent as the raw binary values. Since each component of both a MAC address and IP address is a number in the range 0-255 they are each represented as a byte. This is actually what a MAC address and IP address is. Even though we write an IP address as: 10.0.0.9, it's actually a 4 byte values of: 0x0a,00,00,09. Same goes for a MAC address, which is written as comma separated hexadecimal bytes. When transferred, the host byte order is used, so you read the IP address in the byte order it appears in the packet (which is the same as the example just given).

Numbers are always either 1, 2 or 4 bytes. For 1 byte you just take the integer value of the byte, and for 2 or 4 byte you interpret it with Little Endian. The 2 byte shorts can be calculated with: "((b[0] & 0xff) <<>

Strings are never sent in request packets, though received in a number of response packet fields. The length of the string would be the field size, and it has no terminator. So to read the string value simply copy all of the field data into a character array/string. Characters are always 1 byte ASCII.

Request Packet Authentication
The Configuration and Restart request packets require authentication and a target. This is the last 11 bytes of each of these packets. The target is a 6 byte MAC address, and is the MAC address of the target Digi unit.

The authentication data seems to always be static byte sequence: 0x04,64,62,70,73. This is a 1 byte length with the value of "4", and the character sequence "dbps" (which is also the Digi Connect ME default root password). Even when changing your device's root password, this will remain the same. It's definitely used for authentication, as sending different values will result in an "authentication error" response.

Discovery/Information Request
The Discovery request packet is the simplest of all of them. It has the packet type 0x0001 and a 6 byte payload which is a MAC address. The MAC address in the payload is for the device which you request information from. So only the device whose MAC address matches the one in the packet will respond to this request. If you want all devices to respond, use the broadcast MAC address ff:ff:ff:ff:ff:ff.

This sample Discovery Request packet will request responses from all DIGIs on the network which has ADDP activated on port 2362. To request information from Digis listening on a different ADDP port, change the destination port of the UDP packet.

Data (14 bytes):
0000   44 49 47 49 00 01 00 06 ff ff ff ff ff ff        DIGI..........

Breaks down as:
            44 49 47 49  DIGI       - Magic
                  00 01  ..         - Packet type (0001)
                  00 06  ..         - Payload size (6 bytes)
      ff ff ff ff ff ff  ......     - Target MAC

Discovery/Information Response
If a Digi unit receives the discovery request packet and it's meant for either all Digis or itself (based on the request packet's target MAC address as described above), it will respond with a response packet containing details of it's configuration, such as IP address, subnet mask, number of serial ports, etc.

This packet has the id 0x0002 and is probably the largest of all the packets. See the tables at the end of this e-mail for all possible fields. The following example is a typical packet you might receive in response to a discovery request.

Data (104 bytes):
0000   44 49 47 49 00 02 00 60 01 06 00 40 9d 31 a9 0a  DIGI...`...@.1..
0010   02 04 0a 00 00 e7 03 04 ff ff ff 00 0b 04 0a 00  ................
0020   00 01 0d 0f 44 69 67 69 20 43 6f 6e 6e 65 63 74  ....Digi Connect
0030   20 4d 45 10 01 00 07 01 00 08 1e 56 65 72 73 69   ME........Versi
0040   6f 6e 20 38 32 30 30 30 38 35 36 5f 46 36 20 30  on 82000856_F6 0
0050   37 2f 32 31 2f 32 30 30 36 0e 04 00 00 03 03 13  7/21/2006.......
0060   04 00 00 04 03 12 01 01                          ........

Breaks down as:
            44 49 47 49  DIGI       - Magic
                  00 02  ..         - Packet type (0002)
                  00 60  .`         - Payload size (96 bytes)
01 06 00 40 9d 31 a9 0a  ...@.1..   - Mac address: 00:40:9D:31:A9:0A
      02 04 0a 00 00 e7  ......     - IP Address: 10.0.0.231
      03 04 ff ff ff 00  ......     - Subnet Mask: 255.255.255.0
      0b 04 0a 00 00 01  ......     - Gateway Address: 10.0.0.1
                     0d
0f 44 69 67 69 20 43 6f  .Digi Co
6e 6e 65 63 74 20 4d 45  nnect ME   - Device Name: Digi Connect ME
               10 01 00  ...        - DHCP Disabled
               07 01 00  ...        - Unknown - never seen a non-0x00 value.
08 1e 56 65 72 73 69 6f  ..Versio
6e 20 38 32 30 30 30 38  n 820008 
35 36 5f 46 36 20 30 37  56_F6 07 
2f 32 31 2f 32 30 30 36  /21/2006   - Firmware: Version 82000856_F6 07/21/2006
      0e 04 00 00 03 03  ......     - Real Port: 771
      13 04 00 00 04 03  ......     - Encrypted Real Port: 1027
               12 01 01  .          - Serial port count: 1

Static Network Configuration Request Packet
The Static Configuration Request packets have the packet type of 0x0003 and are used to reconfigure the network settings of the Digi unit. This consists of 3 values to be set: IP address, subnet mask and default gateway.

The 3 values are added in the mentioned order to the beginning of the payload. This gives a 12 byte sequence, which is then followed the MAC address of the device to be configured and the authentication data. This results in a total payload size of 23 bytes, which is a 31 byte packet.

The following sample would be to reconfigure the device with MAC address '00:40:9D:31:A9:0A', giving it the IP address: 10.0.0.9, subnet mask: 255.255.255.0 and gateway: 10.0.0.1. Since these fields are static and always in this order, to remove the gateway you would send a gateway value of 0.0.0.0.

Data (31 bytes):
0000   44 49 47 49 00 03 00 17 0a 00 00 09 ff ff ff 00  DIGI............
0010   0a 00 00 01 00 40 9d 31 a9 0a 04 64 62 70 73     .....@.1...dbps

Break down as:
            44 49 47 49  DIGI       - Magic
                  00 03  ..         - Packet type (0003)
                  00 17  .`         - Payload size (23 bytes)
            0a 00 00 09  ....       - New IP Address: 10.0.0.9
            ff ff ff 00  ....       - New Subnet Mask: 255.255.255.0
            0a 00 00 01  ....       - New Gateway: 10.0.0.1
      00 40 9d 31 a9 0a  .@.1..     - Target MAC Address: 00:40:9D:31:A9:0A
         04 64 62 70 73  .dbps      - Authentication Data

Configuration Response Packet
The configuration response packet is sent to indicate whether the configuration request was successful and if not to give a reason for failure by means of an error code and message. This packet has the id 0x0004 and uses the standard response packet format. The success is indicated by the result flag, result message, error code and configuration error code fields. See the tables at the end of this document for details on these fields.

Here is a sample packet for the above configuration request, indicating the request succeeded.

Data (44 bytes):
0000   44 49 47 49 00 04 00 24 0a 01 00 09 14 4f 70 65  DIGI...(.....Ope
0010   72 61 74 69 6f 6e 20 53 75 63 63 65 73 73 66 75  ration Successfu
0020   6c 11 01 00 01 06 00 40 9d 31 a9 0a              l......@.1..    

Break down as:
            44 49 47 49  DIGI       - Magic
                  00 04  ..         - Packet type (0004)
                  00 24  .(         - Payload size (36 bytes)
               0a 01 00  ...        - Result flag: Success
      09 14 4f 70 65 72  ..Oper
61 74 69 6f 6e 20 53 75  ation Su
63 63 65 73 73 66 75 6c  ccessful   - Result Message: Operation Successful
               11 01 00  ...        - Error code: None
01 06 00 40 9d 31 a9 0a  ...@.1..   - Mac Address: 00:40:9D:31:A9:0A

Here is a sample packet for a configuration request that failed because an invalid IP address was sent (0.0.0.0).

Data (37 bytes):
0000   44 49 47 49 00 04 00 1d 0a 01 ff 09 0d 49 6e 76  DIGI.........Inv
0010   61 6c 69 64 20 76 61 6c 75 65 11 01 03 01 06 00  alid value......
0020   40 9d 31 a3 a5                                   @.1..

Break down as:
            44 49 47 49  DIGI       - Magic
                  00 04  ..         - Packet type (0004)
                  00 1d  .(         - Payload size (29 bytes)
               0a 01 ff  ...        - Result flag: Error
   09 0d 49 6e 76 61 6c  ..Inval 
69 64 20 76 61 6c 75 65  id value   - Result Message: Invalid value
               11 01 03  ...        - Error code: 0x03
01 06 00 40 9d 31 a3 a5  ...@.1..   - Mac Address: 00:40:9D:31:A3:A5

DHCP Network Configuration Request
The DHCP Configuration Request packets have the packet type of 0x0007 and are used to reconfigure the network of the Digi unit so it requests it's IP address from a DHCP server. If a DHCP server doesn't respond it will take on a default IP address (the 169 IPs you see often when a device can't find an IP address).

The packet is very simply, in that it payload simply has a 1 byte boolean to enable/disable DHCP followed by the target MAC and then the authentication data. This will enabled DHCP. If you send a 0x00 for the enable byte then DHCP will be enabled, and the current network settings will be untouched. If you send 0x00 DHCP will be disabled and any previous network settings will be restored.

The following sample would be to reconfigure the device with MAC address '00:40:9D:31:A3:A5', instructing it to automatically get it's IP address via DHCP.

Data (20 bytes):
0000   44 49 47 49 00 07 00 0c 01 00 40 9d 31 a3 a5 04  DIGI......@.1...
0010   64 62 70 73                                      dbps

Break down as:
            44 49 47 49  DIGI       - Magic
                  00 07  ..         - Packet type (0003)
                  00 0c  ..         - Payload size (12 bytes)
                     01  ....       - Enable DHCP
      00 40 9d 31 a3 a5  .@.1..     - Target MAC Address: 00:40:9D:31:A3:A5
         04 64 62 70 73  .dbps      - Authentication Data

DHCP Network Configuration Response
This packet is the same as the Static Network Configuration response, except it has the packet type identifier 0x0008. Here is a sample response for the DHCP request packet sample above:

Data (44 bytes):
0000   44 49 47 49 00 08 00 24 0a 01 00 09 14 4f 70 65  DIGI...$.....Ope
0010   72 61 74 69 6f 6e 20 53 75 63 63 65 73 73 66 75  ration Successfu
0020   6c 11 01 00 01 06 00 40 9d 31 a3 a5              l......@.1..

Break down as:
            44 49 47 49  DIGI       - Magic
                  00 08  ..         - Packet type (0008)
                  00 24  .(         - Payload size (36 bytes)
               0a 01 00  ...        - Result flag: Success
      09 14 4f 70 65 72  ..Oper
61 74 69 6f 6e 20 53 75  ation Su
63 63 65 73 73 66 75 6c  ccessful   - Result Message: Operation Successful
               11 01 00  ...        - Error code: None
01 06 00 40 9d 31 a3 a5  ...@.1..   - Mac Address: 00:40:9D:31:A3:A5

Restart Request Packet
Th restart request is used to instruct a device to restart itself. This is useful in some cases, and needed to apply certain configuration changes, specifically the network configuration done with the Configuration packets.

This packet has the packet type id of 0x0005 and has a target MAC followed by the authentication data as it's payload. This gives a payload length of 11 bytes and a total packet size of 19 bytes. The following is a sample packet that restarts the device with MAC '00:40:9D:31:A9:0A'.

Data (19 bytes):
0000   44 49 47 49 00 05 00 0b 00 40 9d 31 a9 0a 04 64  DIGI.....@.1...d
0010   62 70 73                                         bps

Break down as:
            44 49 47 49  DIGI       - Magic
                  00 05  ..         - Packet type (0005)
                  00 0b  .`         - Payload size (11 bytes)
      00 40 9d 31 a9 0a  ......     - Target MAC: 00:40:9D:31:A9:0A
         04 64 62 70 73  .dbps      - Authentication Data

Restart Response Packet
After you send a restart request, the device will respond, reporting it's result. If successful the device will immediately initiate a restart. This packet has the packet type id 0x0006. As with all the other response packets it uses the field based packet format. The following sample packet is a restart response indicating success of the previous restart request sample.

Data (44 bytes):
0000   44 49 47 49 00 06 00 24 0a 01 00 09 14 4f 70 65  DIGI...$.....Ope
0010   72 61 74 69 6f 6e 20 53 75 63 63 65 73 73 66 75  ration Successfu
0020   6c 11 01 00 01 06 00 40 9d 31 a9 0a              l......@.1..

Break down as:
            44 49 47 49  DIGI       - Magic
                  00 06  ..         - Packet type (0006)
                  00 24  .(         - Payload size (36 bytes)
               0a 01 00  ...        - Result flag: Success
      09 14 4f 70 65 72  ..Oper
61 74 69 6f 6e 20 53 75  ation Su
63 63 65 73 73 66 75 6c  ccessful   - Result Message: Operation Successful
               11 01 00  ...        - Error code: 0x00
01 06 00 40 9d 31 a9 0a  ...@.1..   - Mac Address: 00:40:9D:31:A9:0A

Tables of Codes
The following tables summarize the different types and codes used in the protocol.

Packet Types
  0x0001: Discovery Request
  0x0002: Discovery Response
  0x0003: Static Network Configuration Request
  0x0004: Static Network Configuration Response
  0x0005: Reboot Request
  0x0006: Reboot Response
  0x0007: DHCP Network Configuration Request
  0x0008: DHCP Network Configuration Response

Field Types
  0x01: 6 byte MAC address
  0x02: 4 byte IP address
  0x03: 4 byte Netmask
  0x04: String Network Name
  0x05: UNSEEN
  0x06: UNSEEN
  0x07: 1 byte - UNKNOWN - seen in discovery responses
  0x08: String Firmware
  0x09: String Result message
  0x0a: 1 byte Result flag - see "Result Flags"
  0x0b: 4 byte IP Gateway
  0x0c: 2 byte Configuration error code - see "Configuration Errors"
  0x0d: String device name
  0x0e: 4 byte Real Port number
  0x0f: 4 byte IP address. Purpose unknown.
  0x10: 1 byte DHCP Enabled boolean flag. 
  0x11: 1 byte Error code
  0x12: 1 byte Serial Port Count
  0x13: 4 byte Encrypted Real Port number

Error codes (Field 0x11)
  0x00: Success
  0x01: Authentication Failure
  0x03: Invalid Value
  0x06: Unable to save value

Result flags (Field 0x0a)
  0x00: Success
  0xff: Error

Configuration Errors (0x0c)
  0x0000: No error.
  0x0001: Digi in different subnet than sender

Boolean flags
  0x01: True/Enabled
  0x00: False/Disabled

Some General Notes
  1. There might be more packet types.
  2. There are more error codes which I haven't seen, and thus haven't documented.
  3. Fields types 0x05 and 0x06 haven't been seen, so I'm not sure what they're for.
  4. I haven't determined field 0x07. I haven't even seen a non 0x00 value for it.
  5. It's also possible that fields above 0x13 exist.
  6. Re. fields 0x0e and 0x13 being 4 bytes. Ports can only be 2 bytes (0 to 65535). Why they made it 4 bytes is a mystery, so it could be possible that they store extra information in the first 2 bytes.

Friday, November 20, 2009

Advanced Digi Discovery Protocol Notes

This is a continuation of my previous post.

So I've come further in my analysis of the ADDP protocol. I've got what is almost a complete breakdown, only missing the definition of 2 fields and a few error codes I haven't seen yet.

I haven't formalized it yet, though all the required information is contained in this summary. With it and some common sense you can create a working implementation. I'll formalize it over the weekend for those of you who appreciate such things (edit: See my new post 'Advanced Digi Discovery Protocol Explained').

Also have a look at: JDigiDiscover. It's what I have so far for an implementation. It's not done yet, either. And *cough* excuse the 70 minute GUI job.

Here's some notes. Have fun.
++++++++++++++++++++++++++++++++++
+ Packet Examples and Breakdowns +
++++++++++++++++++++++++++++++++++

Discovery Request:
---------------------------------------------------------------------------------
0000 44 49 47 49 00 01 00 06 ff ff ff ff ff ff DIGI..........

Discovery Response:
---------------------------------------------------------------------------------
0000 44 49 47 49 00 02 00 60 01 06 00 40 9d 31 a9 0a DIGI...`...@.1..
0010 02 04 0a 00 00 e7 03 04 ff ff ff 00 0b 04 0a 00 ................
0020 00 01 0d 0f 44 69 67 69 20 43 6f 6e 6e 65 63 74 ....Digi Connect
0030 20 4d 45 10 01 00 07 01 00 08 1e 56 65 72 73 69 ME........Versi
0040 6f 6e 20 38 32 30 30 30 38 35 36 5f 46 36 20 30 on 82000856_F6 0
0050 37 2f 32 31 2f 32 30 30 36 0e 04 00 00 03 03 13 7/21/2006.......
0060 04 00 00 04 03 12 01 01 ........

Configuration Request:
---------------------------------------------------------------------------------
0000 44 49 47 49 00 03 00 17 0a 00 00 e7 ff ff ff 00 DIGI............
0010 0a 00 00 01 00 40 9d 31 a9 0a 04 64 62 70 73 .....@.1...dbps

Configuration Success Response:
---------------------------------------------------------------------------------
0000 44 49 47 49 00 04 00 28 0a 01 00 09 14 4f 70 65 DIGI...(.....Ope
0010 72 61 74 69 6f 6e 20 53 75 63 63 65 73 73 66 75 ration Successfu
0020 6c 11 01 00 01 06 00 40 9d 31 a9 0a 0c 02 00 01 l......@.1......

Configuration Failure Response:
---------------------------------------------------------------------------------
0000 44 49 47 49 00 04 00 29 0a 01 ff 09 19 55 6e 61 DIGI...).....Una
0010 62 6c 65 20 74 6f 20 6c 6f 61 64 2f 73 61 76 65 ble to load/save
0020 20 76 61 6c 75 65 11 01 06 01 06 00 40 9d 31 99 value......@.1.
0030 de .

Reboot Request:
---------------------------------------------------------------------------------
0000 44 49 47 49 00 05 00 0b 00 40 9d 31 a9 0a 04 64 DIGI.....@.1...d
0010 62 70 73 bps

Reboot Response:
---------------------------------------------------------------------------------
0000 44 49 47 49 00 06 00 24 0a 01 00 09 14 4f 70 65 DIGI...$.....Ope
0010 72 61 74 69 6f 6e 20 53 75 63 63 65 73 73 66 75 ration Successfu
0020 6c 11 01 00 01 06 00 40 9d 31 a9 0a l......@.1..

Discovery Request Breakdown:
---------------------------------------------------------------------------------
44 49 47 49 DIGI - Magic
00 01 .. - Packet type (0001)
00 06 .` - Payload size (6 bytes)
ff ff ff ff ff ff ...... - Target MAC

Discovery Response Breakdown (104 bytes):
---------------------------------------------------------------------------------
44 49 47 49 DIGI - Magic
00 02 .. - Packet type (0002)
00 60 .` - Payload size (96 bytes)
01 06 00 40 9d 31 99 de ...@.1.. - Mac address
02 04 0a 00 00 b4 ...... - IP Address
03 04 ff ff ff 00 ...... - Subnet Mask
0b 04 00 00 00 00 ...... - Gateway Address
0d 0f 44 69 67 69 20 43 ..Digi C
6f 6e 6e 65 63 74 20 4d 45 onnect ME - Device Name
10 01 01 ... - DHCP Enabled
0f 04 0a 00 00 04 ...... - SNMP Traps Host ??
07 01 00 ... - ??
08 1e 56 65 72 ..Ver
73 69 6f 6e 20 38 32 30 30 sion 8200
30 38 35 36 5f 46 36 20 30 0856_F6 0
37 2f 32 31 2f 32 30 30 36 7/21/2006 - Software Version
0e 04 00 00 03 03 ...... - Real Port
13 04 00 00 04 03 ...... - Encrypted Real Port
12 01 01 . - Serial port count

Configuration Request Breakdown
---------------------------------------------------------------------------------
44 49 47 49 DIGI - Magic
00 03 .. - Packet type (0003)
00 17 .` - Payload size (23 bytes)
0a 00 00 e7 .... - New IP Address
ff ff ff 00 .... - New Subnet Mask
0a 00 00 01 .... - New Gateway
00 40 9d 31 a9 0a .@.1.. - Target MAC Address
04 64 62 70 73 .dbps - Authentication Data

Configuration Success Response Breakdown
---------------------------------------------------------------------------------
44 49 47 49 DIGI - Magic
00 04 .. - Packet type (0004)
00 28 .( - Payload size (40 bytes)
0a 01 00 ... - Result flag (see codes below)
09 14 4f 70 ..Op
65 72 61 74 69 6f 6e 20 53 eration S
75 63 63 65 73 73 66 75 6c uccessful - Result Message
11 01 00 ... - Error code. (see codes below)
01 06 00 40 9d 31 a9 0a ...@.1.. - Mac Address
0c 02 00 01 .... - Configuration Error Code

Reboot Request Breakdown
---------------------------------------------------------------------------------
44 49 47 49 DIGI - Magic
00 05 .. - Packet type (0005)
00 0b .` - Payload size (11 bytes)
00 40 9d 31 a9 0a ...... - Target MAC
04 64 62 70 73 .dbps - Authentication Data

Reboot Response Breakdown
---------------------------------------------------------------------------------
44 49 47 49 DIGI - Magic
00 06 .. - Packet type (0006)
00 24 .( - Payload size (36 bytes)
0a 01 00 ... - Result flag (see codes below)
09 14 4f 70 ..Op
65 72 61 74 69 6f 6e 20 53 eration S
75 63 63 65 73 73 66 75 6c uccessful - Result Message
11 01 00 ... - Error code
01 06 00 40 9d 31 a9 0a ...@.1.. - Mac Address

+++++++++
+ Codes +
+++++++++
Packet Types:
0x0001: Discovery Request
0x0002: Discovery Response
0x0003: Static Network Configuration Request
0x0004: Static Network Configuration Response
0x0005: Reboot Request
0x0006: Reboot Response
0x0007: DHCP Network Configuration Request
0x0008: DHCP Network Configuration Response

Field Types:
0x01: 6 byte MAC address
0x02: 4 byte IP address
0x03: 4 byte Netmask
0x04: String Network Name
0x05: UNSEEN
0x06: UNSEEN
0x07: 1 byte - UNKNOWN - seen in discovery responses
0x08: String Firmware
0x09: String Result message
0x0a: 1 byte Result flag - see "Result Flags"
0x0b: 4 byte IP Gateway
0x0c: 2 byte Configuration error code - see "Configuration Errors"
0x0d: String device name
0x0e: 4 byte Real Port number
0x0f: 4 byte SNMP Traps host IP address ??
0x10: 1 byte DHCP Enabled flag. 0x01 = enabled, 0x00 = disabled
0x11: 1 byte Error code
0x12: 1 byte Serial Port Count
0x13: 4 byte Encrypted Real Port number

Error codes (0x11):
0x00: Success
0x01: Authentication Failure
0x03: Invalid Value
0x06: Unable to save value

Result flags (0x0a):
0x00: Success
0xff: Error

Configuration Errors (0x0c):
0x0001: Digi in different subnet than sender

+++++++++
+ Notes +
+++++++++
Authentication data in "Configuration Request" and "Reboot Request" packets. This is an oddity.
I frankly don't know why they have this in there. I doubt they're using decryption and this
information is needed for the key, since these packets are used for initial configuration. Further, this value doesn't change when you change your root user's
password. Frankly it can't, because the Digi tool doesn't prompt for a password, and this "has"
to always work, or at least I figure they meant for it to always work (unless explicitely disabled).

It's a very insecure way of doing it. They might as well have left out the password. I guess it
decreases the chance of a corrupt packet to reconfigure the device. It certainly doesn't block
hackers. A more secure option would probably have be to do a packet challenge and based on the
response encrypt a string and send it to the device. This way it's very difficult to discover,
as people would need to reverse engineer this challenge algorithm.

This design sort of indicates to me a 9th packet type which is a factory default reset packet.
Their style certainly makes this possible.

+++++++++
+ Todos +
+++++++++
1. Investigate the possibility of more packets.
2. Find the purpose of field 0x07
3. Find more error codes.
4. See if fields 0x05 and 0x06 exist and what they're for.
5. Investigate whether fields above 0x13 exist.

Tuesday, November 17, 2009

Advanced Digi Discovery Protocol

When you work with microcontrollers you most often use some type of serial protocol to communicate between devices or with a computer. This is not always suitable, especially when doing so over longer distances. To solve this you get a very useful device called the Digi Connect ME or Digi Connect WiME (which is the same, except it uses WiFi where the other uses ethernet).

This device basically converts all types of serial protocols to ethernet and is very customizable. We, for instance, have it configured to persistently attempt to establish a connection to our server. Any data sent/received to the serial port or the ethernet connection will be transferred to the other side. So it basically allows ethernet communication without the effort or complexity of implementing any of the numerous protocols needed to do TCP/IP communication.

Since you don't have a computer screen or some display, to configure/manage these devices require some extra utilities. Imagine having changed the IP address of the device, it's not configured to do any communication automatically and you forget what you configured? How would you figure this out other than doing a lengthy and complex IP scan? You can also not reset the device to it's default, which will not help you in any case, because it's default is to use DHCP and you don't have a DHCP server on your network? This is just one of the many difficulties with electronics. With a computer you can jump in front of the keyboard and configure an IP address. So what to do?

The developers of the Digi device also made a utility called Digi Device Discovery. It uses their proprietary ADDP (Advanced Digi Discover Protocol) protocol to send a UDP multicast. All Digi devices will then respond with their configured IP address and some other bits of information. It also allows you to configure the IP address using these same multicast packets. On top of this they also give a C header file and library so you can have this functionality in your own applications. I'm sure you can see the amazing benefits of this, especially if you have a distributed network of electronic devices with these embedded Digi units.

Unfortunately this application is only for Windows, and the header only for C and again only for Windows. From what I can gather the Digi folks haven't found the time to document the protocol either, and I couldn't find anything useful on the internet. I did, however, find a short description of one of the packets and a few of the fields after I completed my analysis of the first 4 packets. What timing as this could have saved me 3 hours.

I needed to use these packets though. It can be very useful to manage your devices. And obviously the fact that I didn't know how it works was unacceptable and inspired me to go to any length to learn it. So armed with Wireshark (previously called Ethereal) and using Wine to run Digi Device Discovery, I set out a few evenings and analyzed the protocol. After a few hours total I had a working Java implementation running which discovers all Digi devices and lists their public configuration (the bits available in the discovery packets) as well as being able to change their IP addresses using the same protocol. I haven't completed all the features of the configuration yet, so I'm not publishing it today. It should be done by the end of this weekend (by the 22nd November 2009), at which time I'll release it under Apache License 2.0 (edit: See my new post 'Advanced Digi Discovery Protocol Explained').

For now, here is a description of the protocol.

Advanced Digi Discovery Protocol
ADDP works with UDP multicast datagrams on multicast address 224.0.5.128 and port 2362. The same address is used for sending and receiving datagrams.

All packets whether sent/received have the same basic form. It has a header of 8 bytes and a variably sized payload which consist of a variable number of fields.

General Packet Structure of Responses
A packet has the format of header followed by payload.

The header starts with 4 magic bytes consisting of the ASCII bytes 'D', 'I', 'G', 'I'. Very creative. After the magic bytes follows a 2 byte packet identifier, and then a 2 byte integer for the payload size (in bytes).

The payload for request packets are unique to each type of request, where the payload of responses all have the same basic form, which is a variable number of fields. Each field starts with a 1 byte field type identifier, followed by a 1 byte data length, followed by the data.

And that's the packet structure. Simple.

Packet Types
So far I've identified 4 packet types, nl.
Discovery Request: 0x0001
Discovery Response: 0x0002
Configuration Request: 0x0003
Configuration Response: 0x0004

Field Types
I've also identified the following field types:
Mac Address: 0x01
IP Address: 0x02
IP Subnet: 0x03
IP Gateway: 0x0b
Network Name: 0x04
Device Name: 0x0d
Dhcp Enabled: 0x10
Some Unknown IP: 0x0f
Firmware Version: 0x08
Encrypted Real Port Number: 0x13
Real Port: 0x0e
Serial Ports: 0x12
UNKNOWN1: 0x07
Configuration Error: 0x0c

Two of the fields I haven't identified yet. I do, however, suspect the 0x0f field is the SNMP traps host as it once contained this value, though I haven't been able to reproduce this.

These are only the fields for the discovery response. I haven't documented the fields/type IDs for the configuration request/response.

Strings, Data Types and Byte Order

Byte ordering is done like it is on any standard Intel based computer. So if you've done any network programming on these architectures you should be comfortable with this protocol.

Items like integers and numbers are in Little Endian byte order, so it's most significant byte first. The number 0x0219 is decimal 537, and can be calculated as ((0x02 << 8) + 0x19).

IP Addresses are done with the host byte order, so they are read as they are in the packet. So the bytes: 0x0a, 0x00, 0x00, 0x65 is the IP address: 10.0.0.101.

Further strings are (as they are usually) just a character array and has no terminator. The field's value is it's complete data part. So a string field with length 9, will have 9 data bytes which is a string of length 9. Strings are always composed of single byte ASCII characters.

Discovery Request Payload Structure
The discovery request has a single field to specify the target mac address for which you want to request the discovery. If this has as a value the MAC address ff:ff:ff:ff:ff:ff (the broadcast MAC) then all Digis will respond. You specify a specific MAC address if you want to request a response from a specific device. This field, however, has a special structure in that it doesn't have a field ID or a size. It's simply the 6 bytes for the MAC address. This is because, as I mentioned, request packets each have their own unique structure.

Here is a sample discovery request packet (header then payload):
DIGI{0x00, 0x01}{0x00, 0x06}
{0xff, 0xff, 0xff, 0xff, 0xff, 0xff}

To explain again, the header has the magic bytes DIGI, then the packet type of 0x0001, then the payload size of 0x0006 and the single field of target mac, which is the broadcast mac of ff:ff:ff:ff:ff:ff.

Discovery Response Payload Structure
The discovery response has multiple fields for each of the configuration entries. Not all of the responses contain all the fields, so these packets have their sizes differ from device to device, all depending on the configuration of the device.

A sample packet would be as follows. I broke it down into lines to demonstrate the different parts of the packet.
Total packet size: 104 bytes (8 byte header, 96 byte payload)
44 49 47 49  DIGI       - Magic
00 02  ..         - Packet type (0002)
00 60  .`         - Payload size (96 bytes)
01 06 00 40 9d 31 a9 0a  ...@.1..   - Mac address: 00:40:9D:31:A9:0A
02 04 0a 00 00 e7  ......     - IP Address: 10.0.0.231
03 04 ff ff ff 00  ......     - Subnet Mask: 255.255.255.0
0b 04 0a 00 00 01  ......     - Gateway Address: 10.0.0.1
0d
0f 44 69 67 69 20 43 6f  .Digi Co
6e 6e 65 63 74 20 4d 45  nnect ME   - Device Name: Digi Connect ME
10 01 00  ...        - DHCP Disabled
07 01 00  ...        - Unknown - never seen a non-0x00 value.
08 1e 56 65 72 73 69 6f  ..Versio
6e 20 38 32 30 30 30 38  n 820008 
35 36 5f 46 36 20 30 37  56_F6 07 
2f 32 31 2f 32 30 30 36  /21/2006   - Firmware: Version 82000856_F6 07/21/2006
0e 04 00 00 03 03  ......     - Real Port: 771
13 04 00 00 04 03  ......     - Encrypted Real Port: 1027
12 01 01  .          - Serial port count: 1
Conclusion
This is in no way the official specification of the protocol. I discovered all of this by analyzing the protocol using Wireshark and some common sense. Like I mentioned I'm also not completely done, and even if I were there might be some misinterpretations. I'll document the other packets and have the Java implementation of the protocol and utility up around the 22nd of November 2009 (edit: See my new post 'Advanced Digi Discovery Protocol Explained').

This is a proprietary protocol developed by and it's copyright owned by Digi International Inc. Use of the protocol is limited to their terms. I released this purely because I noticed a need for implementing it in your own non C and non Windows applications for managing your Digi devices. For more information see http://www.digi.com/.