mjt.me.uk

Making the Macrosilicon MS2109 HDMI-to-USB capture device work on Linux

2020-09-26T00:00:00+01:00

I brought a cheap (£10) HDMI-to-USB2 capture device based on the Macrosilicon 2109 - but I encountered a problem where it didn't appear at /dev/video* when plugged in, despite dmesg showing some recognition of the device.

Fortunately, Nikola_F on the Arch Linux forums had already found the problem and offered a solution: The device is wrongly being treated as a USB audio device. You can confirm this by checking four new symlinks appear in ls -l /sys/bus/usb/drivers/snd-usb-audio/ when you insert the device.

To fix it, I created new udev rules (in a file like /etc/udev/rules.d/91-hdmi-to-usb-ms2109.rules ) to apply Nikola_F's fix automatically:

# For HDMI-to-USB adaptor:
SUBSYSTEM=="usb", DRIVER=="snd-usb-audio", ATTRS{idProduct}=="2109", ATTRS{idVendor}=="534d", \
    RUN+="/bin/sh -c 'echo -n $kernel > /sys/bus/usb/drivers/snd-usb-audio/unbind'"

SUBSYSTEM=="usb", DRIVER=="snd-usb-audio", ATTRS{idProduct}=="2109", ATTRS{idVendor}=="534d", \
    ATTR{bInterfaceNumber}=="00", RUN+="/bin/sh -c 'sleep 1; echo -n $kernel > /sys/bus/usb/drivers/uvcvideo/bind'"

Give it a sudo udevadm control --reload-rules then unplug and replug, and it should start working.

This disables the audio function of the HDMI-to-USB converter, but I wasn't using it anyway. More careful rule-writing could probably solve that problem.

Other hints for using the MS2109

I was able to stream video from the device at 1080p with ffplay -f video4linux2 -framerate 50 -video_size 1920x1080 -input_format mjpeg /dev/video0

The device seemed to struggle if mjpeg wasn't used - presumably a lack of USB bandwidth. If you want to use it with software you can't convince to ask for mjpeg, you can create a loopback camera and use ffmpeg to read mjpeg from the capture card and output regular video to the loopback camera, like so:

sudo apt-get install ffmpeg v4l2loopback-utils

sudo modprobe v4l2loopback devices=1 video_nr=10 card_label="LoopbackCam" exclusive_caps=1
ffmpeg -f video4linux2 -framerate 50 -video_size 1920x1080 -input_format mjpeg -i /dev/video0 -f v4l2 -pix_fmt yuv420p /dev/video10

Streaming live video over Wifi from the Panasonic Lumix GX80

2020-04-27T00:00:00+01:00

Summary & Background

The Panasonic Lumix GX80 is a mirrorless, interchangable lens micro four thirds camera. Some people have used it as a streaming webcam by connecting the camera's HDMI output to a HDMI-to-USB capture card (cost ~£120). It also has wifi connectivity, and can be controlled remotely - including a live preview over wifi.

I set out to see I could use that wifi live preview like a webcam.

In summary, I was able to get it to work - but with a lot of caveats. The live preview is 640x480 30fps and with some software trickery, I was able to use that video as a webcam in video calls.

Sample output

This image is captured raw from the mjpeg stream:

FotoForensics estimates its compression level as 49%. The video stream has a bandwidth of about 6.8 Mbps and runs at 30 frames per second (even if your camera is PAL and records video at 25fps)

Sample video

Sorry, your browser doesn't support embedded videos.

Quick Start

Install dependencies something like this (assumes you have java and maven already, if not install them too):

sudo apt-get install ffmpeg v4l2loopback-utils
sudo modprobe v4l2loopback devices=1 video_nr=10 card_label="LoopbackCam" exclusive_caps=1
git clone git@github.com:michaeltandy/lumix-wifi-webcam.git
cd lumix-wifi-webcam
mvn package
sudo sysctl -w net.ipv4.ipfrag_time=5
# This undoes CVE-2018-5391 mitigation
sudo sysctl -w net.ipv4.ipfrag_high_thresh=4194304

Turn on your camera, and connect it to wifi. I recommend using your router's WPS button so you don't have to type a password on a camera touchscreen.
Give your camera a fixed IP address in your router's DHCP settings, and note down what it is.

java -jar target/lumix-wifi-webcam.jar [your webcam's IP]

Credit where it's due

lenuisible for reporting the stream is a MJPEG video over UDP, and a Java stream viewer and Martin Pecka for keeping that alive and where I could find it. BTM_Pix and Vladimir Goncharov for additional info on Lumix remote control.

You mentioned a lot of caveats?

The resolution of 640x480 is nothing to write home about, and is quite heavily compressed.
I encountered a latency of about 130ms - 4 frames - meaning sound and video are out of sync by an amount people will notice.
The mjpeg-over-UDP protocol doesn't work very well in the presence of dropped packets or link congestion. This can lead to seemingly random performance degredation. Mitigations for consequences of this have security implications.
The GX80 can't be powered by USB while it's operating - so you need a 'dummy battery' power supply if you don't want your battery to run out (search for DMW-DCC11 if you want one)
In the course of my experiments, I managed to trigger several bugs in the camera's firmware - including one that froze the camera's entire interface, including the on/off switch, so I had to eject the battery to restart the camera. This doesn't seem like a solid foundation for a stable video system.
The GX80's wifi interface has no security, except the fact most users will never connect it.

Oddities of the mjpeg-over-udp protocol

Each frame is a single UDP datagram - making each datagram much larger than the ethernet MTU, and leading to packet fragmentation.

Here we see a 27 kilobyte image split into 19 fragments. If any of those 19 fragments are lost, the frame is lost. If you've got slow wifi, this can happen a lot.

Variable-length prefix

The mjpeg-over-UDP datagrams had a prefix before the JPEG data - usually of the same length, but occasionally varying. What's going on with that?

I reverse-engineered enough of the protocol to learn that:

The length of the header is given in the header; there's a 16-bit integer starting at index 30; add 32 to that and you've got the index of the first byte of the JPEG data.
The reason the header varies in length is it contains info about things that cause the camera to show things on the screen (like face detecton and points selected by autofocus) and the number of such things can vary.

IP fragment reassembly queue exhaustion

On Linux, the video stream would sometimes freeze for tens of seconds at a time, with UDP reads timing out as if no UDP traffic was being received - but Wireshark would indicate UDP packets were still arriving just fine.

It turns out the Linux kernel has a queue of partly-reassembled IP packets, waiting around for the missing fragments to come in. In the default configuration this queue has a fixed size of ~260 kilobytes in memory, doesn't discard stored contents for 30 seconds, and silently discards new packets when full.

With 27 kilobytes per frame, 30 frames per second and a 260 kilobyte limit on fragmented packets, if 10 frames drop a single fragment within 30 seconds the UDP socket would receive nothing until the oldest fragmented packet had aged out of the queue.

To work around this, I increased the fragment reassembly queue space - which undoes CVE-2018-5391 mitigation but I deem that tolerable as I'm not running a public-facing server. I also reduced the fragment reassembly time from 30 seconds to 5.

Blacklisting your USB Webcam's built-in microphone in Linux

2020-04-10T00:00:00+01:00

Motivation

I've got a headset with a microphone, and a webcam with a built-in microphone too; I always want to use the headset microphone, but sometimes the OS or videoconferencing tools start up using the wrong one.

The good news is Linux will let you disable your webcam's microphone without disabling its video. I've done this successfully on Ubuntu 18.04

How it's done

We're going to do it with udev-based USB device blacklisting

First, we find the USB VID and PID ('vendor ID' and 'product ID') for the webcam:

$ lsusb
Bus 002 Device 002: ID 8087:8001 Intel Corp. 
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
[snip]
Bus 003 Device 002: ID 045e:0750 Microsoft Corp. Wired Keyboard 600
Bus 003 Device 009: ID 046d:0991 Logitech, Inc. QuickCam Pro for Notebooks
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

If it's not obvious from lsusb which device is your webcam, you can always hotplug and see which device disappears - and you can view lots of details of sound devices with udevadm info -a /dev/snd/by-id/*

We then create a new text file at /etc/udev/rules.d/90-blacklist-webcam-sound.rules containing the following udev rule:

# Disables webcam audio.
SUBSYSTEM=="usb", DRIVER=="snd-usb-audio", ATTRS{idVendor}=="046d", ATTRS{idProduct}=="0991", ATTR{authorized}="0"

Then reload the rules with sudo udevadm control --reload-rules unplug and replug your device - it should now appear as a camera but not a microphone.

Loading Symantec VIP Access credentials onto Yubikeys

2020-03-29T00:00:00+00:00

Summary

You can generate Symantec VIP Access credentials and load them onto any Yubikey that supports TOTP or HOTP (i.e. any Yubikey that isn't blue).

If you don't already have a Yubikey, you might also consider a Symantec VIP Hardware Authenticator which is less than half the price of the Yubikey I used - but I already have the Yubikey and the Hardware Authenticator doesn't ship to the UK

TOTP vs HOTP

The codes generated by the Symantec VIP Access phone app (and the likes of Google Authenticator) are TOTP codes - the code depends on the time. The other option is HOTP which uses a counter instead - this is what the Symantec VIP Hardware Authenticator does, and simplifies things on devices like the Yubikey that don't have built-in clocks.

Here are the main differences:

TOTP	HOTP
Based on time - new code every 30 seconds Requires a program on your computer - cannot use Yubikey's keyboard emulation Can be backed up at creation time	Based on counter - new code every button press Yubikey keyboard emulation works - no software needed More difficult to back up, as counter changes on every code use

If you choose TOTP

Install python-vipaccess with pip3 install python-vipaccess and yubioath with sudo apt-get install yubioath-desktop
Execute python-vipaccess from wherever pip installed it (in my case ~/.local/bin/vipaccess) like so:

$ ~/.local/bin/vipaccess provision -p
Generating request...
Fetching provisioning response...
Getting token from response...
Decrypting token...
Checking token...
Credential created successfully:
    otpauth://totp/VIP%20Access:VSST89795985?secret=KVIMXW236KKUVMSVNYKZOBFPTWMKMPKZ&digits=6&algorithm=SHA1&issuer=Symantec&period=30
This credential expires on this date: 2023-03-29T16:24:42.226Z

You will need the ID to register this credential: VSST89795985

You can use oathtool to generate the same OTP codes
as would be produced by the official VIP Access apps:

    oathtool    -b --totp KVIMXW236KKUVMSVNYKZOBFPTWMKMPKZ  # output one code
    oathtool -v -b --totp KVIMXW236KKUVMSVNYKZOBFPTWMKMPKZ  # ... with extra information

This gives you the TOTP secret and token ID. If you want to back up the credential or also load it into a tool like Google Authenticator, now is the time; to generate a QR code, see the python-vipaccess README

Then simply load the secret onto your Yubikey and test as follows:

$ read -p "Enter the secret: " totpsecret
Enter the secret: KVIMXW236KKUVMSVNYKZOBFPTWMKMPKZ

$ yubioath put --name VSST89795985 --oath-type totp --touch "$totpsecret"

$ yubioath
VSST89795985  [Touch credential]

$ yubioath show VSST89795985
Touch your YubiKey...
VSST89795985      857363

$ unset totpsecret

We use read above to avoid saving the secret into our bash history. You can also use yubioath-gui if you prefer a GUI.

Test your token at Symantec's website to confirm you've set it up right.

Calling yubioath from python

If you've got a python script to log into something, and you want to call yubioath from it, you can do it like so:

#!/usr/bin/env python3
import subprocess

keyId = "VSST89795985"
pintoken = ''

while pintoken == '':
    ykResult = subprocess.run(["yubioath", "show", keyId], stdout=subprocess.PIPE)
    if ykResult.returncode == 0:
        pintoken = str(ykResult.stdout, 'utf-8').split()[-1]
    else:
        pintoken = input('Enter token manually, or press enter to retry yubikey: ')

print("Token:", pintoken)

The while loop ensures, if the code ran yubioath stage before you put your key in, you can put it in and try again.

If you choose HOTP

Install python-vipaccess with pip3 install python-vipaccess and Yubikey Manager with sudo apt-add-repository ppa:yubico/stable && sudo apt update && sudo apt install yubikey-manager-qt
Execute python-vipaccess from wherever pip installed it (in my case ~/.local/bin/vipaccess) like so:

$ ~/.local/bin/vipaccess provision -p -t FT12
Generating request...
Fetching provisioning response...
Getting token from response...
Decrypting token...
Checking token...
Credential created successfully:
    otpauth://hotp/VIP%20Access:FT1263782685?secret=KXVJPCQHMKLRP5FOPBW43OTV2424CXVY&digits=6&algorithm=SHA1&issuer=Symantec&counter=2
This credential expires on this date: 2023-03-29T18:00:44.119Z

You will need the ID to register this credential: FT1263782685

The prefix comes from Symantec's list of prefixes listed as 'event based' - Prefixes like UBHE also work.

Remember to note down the ID - FT1263782685 in the example above - as you'll need it to make use of your new credential.
The Yubikey's keyboard-emulation has two 'slots' - Slot 1 is triggered by a short press, and comes factory-configured for Yubico OTP. Slot 2 is triggered by a long press, and from the factory is empty; below, we confirm slot 2 is empty and load our credential into that slot. If a slot isn't empty, don't overwrite it unless you're happy to lose whatever credential is in that slot

$ ykman otp info
Slot 1: programmed
Slot 2: empty

$ read -p "Enter the secret: " hotpsecret
Enter the secret: KXVJPCQHMKLRP5FOPBW43OTV2424CXVY

$ ykman otp --counter 2 hotp 2 "$hotpsecret"
Program a HOTP credential in slot 2? [y/N]: y

$ unset hotpsecret

You can use ykman-gui to do the same thing with a GUI, if you prefer.

Now when you long-press (short press if you used slot 1) your Yubikey will emulate a keyboard entering a 6-digit code.
Test your token at Symantec's website to confirm you've set it up right.

How cheap digital verniers work

2017-04-01T00:00:00+01:00

Ever wondered how cheap digital verniers are able to produce good precision at very low prices? I did, so I took one apart.

I'm not the first person to look at this - Grant Trebbin has some photos of a teardown, and has done a good job of finding patents like US5068653 Capacitive displacement measuring device with t-shaped scale coatings; Nick Müller has inspected one with an oscilliscope and Anyi Instrument Co have posted some info on how the sensors work.

Turns out they use capacitive sensing - there's a thin PCB hidden under a label on the shaft of the vernier:

The green PCB on the left is for the slider. Every eighth bar in the densely striped section is connected together - i.e. the first, 9th and 17th bars are connected together, as are the 2nd, 10th and 18th, and so on.

I measured the distance between the PCBs on the slider and shaft using a cheap feeler gauge; the spacing was about 0.35mm ±0.05mm.

Based on the 600 dpi scan above, the T sections on the shaft and the densely striped section is about 250 mm², and each of the eight banks of stripes has an area of 31 mm². The return strip has an area of about 95 mm². The comb overlaps the ground plane by about 43mm²; while that's obviously the same order of magnitude as the return strip, they don't seem to have been made precisely the same surface area.

So each of the eight stripes has a capaitance of about 0.78 picofarads while the return strip has a capacitance of about 2.4 picofarads - taken in series, that's about equivalent to 0.588 picofarads. That's very small. For example, the ADC on an ATMega (like an Arduino Uno) has an input capacitance of ~14pF. It's also smaller than the input capacitance of my oscilliscope. But some capacitive sensors claim to be able to measure femtofarad capacitance changes - the AD7745 claims an accuracy of 4 femtofarads, and the AD7147 a sub-femtofarad output noise.

The stripes in the densely striped section are about 0.63mm wide, and the vernier's precision is 0.01mm. That means, if my estimate of a 588 femtofarad capacitance is correct, the capacitance measurement is precise to about 9 femtofarads.

If you attach an oscilliscope to the eight sets of bars in the striped section, and the return bar, here's what you see:

( Raw data CSV )

Of course, the oscilliscope's probes introduce a certain extra resistance and capacitance. Several tens of picofarads of capacitance and 10 MOhms of resistance, both to ground, would be typical.

Unfortunately, I didn't keep track of exactly which channel was which - but Nick Müller's oscilliscope work, linked above, shows us the inputs that are inversions of one another are four bars apart in the striped section. Also, as I used an 8-channel oscilliscope, I could only track the output and seven inputs. So the line marked "Input 5" is just input 6 flipped over.

Let's zoom in on that first section.

Nick Müller's trace shows the output spiking high and low in equal amounts - I assume the fact mine doesn't is because it's clamped to the battery voltage by a diode, or something similar. The low spikes have a width of about 0.01ms - presumably something pulls the output to the center value after the spike has been measured.

The sensing cycle appears to take differential measurements between of bars; in the diagram above a pair of lines either becomes narrow or becomes wide. The sensing cycle appears to rotate between all channels narrowing (widening); three of the four channels widening (narrowing) while the remaining channel is fixed; and three of the four channels widening (narrowing) while the remaining channel does the opposite.

A complete sensing cycle takes around 1.7 milliseconds - although the system may average multiple sensing cycles to reduce noise. The AD7147, linked above, takes 6-36ms to sample 8 inputs (depending on how precisely you want to measure).

The comb on the vernier shaft has a tooth pitch of 5mm. In other words, if the slider instantly moved 5mm, it would be undetectable, or 'ambiguous'. So what's the top speed you could go at? Depends (a) how much sample averaging you need and (b) if you can measure while moving. Assuming you can measure while moving, if a measurement takes 1.7ms and you want four per ambiguity step (i.e. a measurement every 1.25mm) you could go as fast as 0.7 m/s - but if it takes 66ms to make such a measurement, your speed would drop to 0.035 m/s. Or maybe the measurements work some other way and I'm completely wrong!

ESP8266 SPI Flash protocol

2016-09-19T00:00:00+01:00

I was wondering whether the ESP8266 SPI flash could be used for atomic (or sort-of-atomic) write operations - so I wanted to understand how the underlying SPI operations work.

TLDR

API call	SPI command	Note
`spi_flash_erase_sector`	`0x20`	Repeated status reads (command `0x05`) until erase completes
`spi_flash_read`	`0xBB`	Dual SPI
`spi_flash_write`	`0x02`	Single status read, test completed immediately

Read on to learn more...

Some of the different SPI flash chips found on ESP8266 modules

The ESP-12F on my WeMos D1 Mini uses a Winbond W25Q32BV
The SparkFun Thing uses an Adesto AT25SF041
A black ESP-01 I brought on ebay uses a Berg Micro 25q80a

These different chips seem to have enough commands in common that they can be substituted for one another.

Dual and Quad SPI

Traditional SPI has 'MOSI' and 'MISO' lines, with data travelling in opposite directions. If you don't know what MOSI and MISO mean, go review that wikipedia article otherwise the rest of this explanation won't make much sense!

The flash chips listed above support 'Dual SPI' which basically means, instead of MOSI and MISO always going in opposite directions, certain commands cause both lines to temporarily be used in the same direction. Here's an example:

The first byte on MOSI is 10111011 or 0xBB in hex - what the Winbond datasheet calls "Fast Read Dual I/O".

When that command is received, MISO changes direction for 16 clock cycles, allowing the master to output 24 bits of address data in 12 clock cycles followed by 8 bits of control options, sent in 4 clock cycles. You read the bytes MISO-MOSI-MISO-MOSI- so the address above is 001111111011000000000000 or 0x3FB000 and the control options are 0x00.

After this, MISO and MOSI change direction - both lines are used by the slave to send data to the master. In the example above, the data returned by the slave is all ones, or 0xFFFFFFFF.

Quad SPI adds an extra two lines between master and slave, and allowing for the transfer of four bits per clock cycle. When the Arduino IDE offers the choice between flash modes 'DIO' and 'QIO' flash this is what you're choosing between (whether you'll have the choice depends on the board you have selected).

The practical speed benefits of Quad SPI over Dual SPI seem suprisingly modest, according to speed test results reported on esp8266.com which reports read times reducing by only 14% when changing from dual to quad SPI.

Programs execute from SPI flash

As you may know, user programs can be up to a megabyte, but there are only 64 kilobytes of instruction memory. This means quite a lot of data gets shuttled over the SPI bus under normal operation.

I haven't looked into the details, but I assume the ESP8266 does some sort of multitasking, so it can look after all the wifi maintainance stuff in the background while the user's program runs. Anyway, in my tests I found that even when I put a long delay in my program, the SPI bus wasn't always silent during that time.

How I tested the SPI behaviour

Initially, I tried to use a 1.5MHz bandwidth two-channel scope to record MOSI and MISO, inferring the clock transitions from changes in the data lines. This didn't work very well, because you can't capture a 40MHz bus with a 1.5MHz scope; and if you slow the bus down far enough that you can capture it your program will keep resetting with watchdog timer failures; and even if you ignore those you'll most likely get a recording of code being loaded, as there's a lot of that going on over the bus.

So I moved to a 20MHz bandwidth 8-channel scope. This allowed me to also capture clock, chip select, and a pin I configured to act as a trigger when my test started.

I used an ESP-12F on a breakout board that exposes all pins (including the SPI flash pins)

Here's the program I used:

extern "C" {
#include "spi_flash.h" // Provides SPI_FLASH_SEC_SIZE (usually 4096)
}

extern "C" uint32_t _SPIFFS_end; // Usually points to 0x405FB000
const uint32_t START_ADDRESS_BYTES = (uint32_t)&_SPIFFS_end - 0x40200000;
const uint32_t START_ADDRESS_FLASH_SECTORS = START_ADDRESS_BYTES/SPI_FLASH_SEC_SIZE;

const int led = 2;
const int LED_ON = LOW;
const int LED_OFF = HIGH;

void setup() {
  pinMode(led, OUTPUT);
  digitalWrite(led, LED_OFF);
  Serial.begin(115200);
  Serial.println("Started.");
}

void loop() {
  delay(500);
  Serial.println("Testing SPI flash...");

  noInterrupts();
  digitalWrite(led, LED_ON); // We can trigger our scope on this.

  // Reduce SPI clock speed so it fits in our scope's bandwidth
  uint32_t clkbefore = SPI0CLK;
  SPI0CLK = 0x00D43002;

  spi_flash_erase_sector(START_ADDRESS_FLASH_SECTORS);
  uint32_t readBuf[1] = {0};
  spi_flash_read(START_ADDRESS_BYTES, readBuf, sizeof(readBuf));
  uint32_t writeBuf[1] = {0xF0F0F0F0};
  spi_flash_write(START_ADDRESS_BYTES, writeBuf, sizeof(writeBuf));

  SPI0CLK = clkbefore; // Restore SPI clock speed.
  digitalWrite(led, LED_OFF);
  interrupts();
}

Results

Sector erase

spi_flash_erase_sector is called to erase sector 0x3FB, producing the command 20 3F B0 00 on MOSI then repeated status checks (MOSI 05 00, MISO FF 02 and FF 03) until the status reports the operation is complete (MISO FF 00)

Data read

spi_flash_read is called to read 32 bits from start of that sector - address 0x3FB000. The scope capture is shown in the 'Dual SPI' section, as the read is performed with Dual SPI; BB 74 00 FF FF on MOSI and 00 7C 00 FF FF on MISO. A read of 0xFFFFFFFF is the expected result for NAND flash that has just been erased.

Data write

spi_flash_write is called to write 32 bits to the same address; this produced 02 3F B0 00 F0 F0 F0 F0 on MOSI and 00 00 00 00 00 00 00 00 on MOSI, indicating that dual SPI was not used. Look how many clock cycles there are compared to the write! A status check (MOSI 05 00 MISO 00 00) returned immediately.

Raw scope capture

Want to look at the data yourself in more detail? You can download it as a 15 megabyte, 2 million row CSV - but you won't be able to load it in Excel because it's too big.

Picoscope's 1,048,579 line CSV limit

2016-09-18T00:00:00+01:00

If you've got here from Google, it's probably because you've exported a CSV or TXT file from picoscope and it's come out with 1048579 lines - maybe 3 of which are headers and 1048576 are data. You can find several posts on their forum about the same thing.

Seems like Excel worksheets are limited to 1,048,576 rows by 16,384 columns, and Picoscope silently truncates CSVs to that length. I'm not sure why they decided to truncate silently, rather than giving the user the option to decimate the data or choose a window to export, or putting a note on the last line of the data saying that it had been truncated and why.

Picoscope support recommends exporting in Matlab format instead. But what if you really want a big CSV, and you don't have Matlab because it costs £1,600 a seat which is a lot of money for a single CSV? Python to the rescue!

Converting picoscope output from .mat to .csv

Here's the Python script I used to do the conversion. My install of Ubuntu 16.04 already had Python 2.7.12 installed, but I had to run sudo apt-get install python-scipy

import scipy.io
import numpy
mat = scipy.io.loadmat('20160918-0002.mat')
startTime = mat['Tstart'][0][0]
numSamples = mat['Length'][0][0]
endTime = startTime+numSamples*mat['Tinterval'][0][0]
times = numpy.linspace(startTime, endTime, numSamples).reshape(numSamples,1)
arrayForCsv = numpy.concatenate((times, mat['A'], mat['B'], mat['C'], mat['D'], mat['E']), axis=1)
numpy.savetxt('20160918-0002.csv.gz', arrayForCsv, delimiter=',', fmt='%.9f')

Note that you'll have to change the loadmat and savetxt lines with the filenames you want, and change the concatenate line if you weren't using five channels. Simply change the mat['A'], mat['B']...mat['E'] section to list the channels you want in your CSV.

ESP8266 GPIO on deep sleep and reset

2016-08-09T00:00:00+01:00

I want to do some battery powered sensing with an esp8266 - and to get a good battery life, I need to use the 'deep sleep' mode. Deep sleep requires a reset (including a bootloader and some radio calibration) to wake from. I decided to test how different pins behave when in deep sleep or undergoing a reset.

Schematic

By switching the Arduino Uno outputs between high, low and high impedance, we can test a variety of pull up and pull down values applied to the pins of the module.

Notes

These tests were performed on a 'WeMos D1 Mini V2' module carrying an 'ESP-12F' module carrying the esp8266. The modules include some pull and and down components. I've tried to note these below.
As the picoscope has a 1 MΩ input impedance, a 100 kΩ pull up to 3.3v with no other load makes a resistor divider with a 3v output voltage.
The timebase for the charts below is 100 milliseconds per division, and the vertical scale is in volts. The blue line is the reset signal and the red line is the pin being tested.
The gap between 'Reset pulled low' and 'Chip restarting' is intentional, I put it in because of the GPIO0/GPIO2 bootloader behaviour (you'll see it when you scroll down).
The two test programs are at the end of this post. I programmed the esp8266 using Arduino board support package 2.3.0 - as some of the behaviour is probably bootloader/SDK dependent, YMMV.

Stimulus circuit output without ESP8266

WeMos mini pin D0 / GPIO16

Special function - timer reset when in deep sleep

WeMos mini pin D1 / GPIO5

WeMos mini pin D2 / GPIO4

WeMos mini pin D3 / GPIO0

10k pullup on D1 mini schematic, and reset circuit connection to DTR/RTS/RST

WeMos mini pin D4 / GPIO2

10k pullup on D1 mini schematic, blue LED & 470 ohm resistor on ESP-12F

WeMos mini pin D5 / GPIO14

WeMos mini pin D6 / GPIO12

WeMos mini pin D7 / GPIO13

WeMos mini pin D8 / GPIO15

10k pulldown on D1 mini schematic

Summary

It takes 300ms from the reset pin rising edge for the user's program to start running (at least when deep sleep mode WAKERFDEFAULT is used)

Pins' values while in deep sleep, reset, and during restart are unaffected by the value the pin was set to before deep sleep.

Individual pins behave differently, and their behaviours are as follows:

D1 mini pin	GPIO	Deep sleep	Reset held low	During restart	Note
D0	16	strong high	strong high	strong high	special role - wake from deep sleep
D1	5	high impedance	high impedance	high impedance
D2	4	high impedance	high impedance	high impedance
D3	0	~500 ohm high?	~875 ohm high?	~500 ohm high?	Pull up resistor & controls boot behaviour
D4	2	~5k pull up	~5k pull up	~5k pull up	Pull up resistor & blue LED
D5	14	high impedance	weak high	weak high
D6	12	high impedance	weak high	weak high
D7	13	high impedance	weak high	weak high
D8	15	high imp & 10k pull down	weak high & 10k pull down	strong low	Pull down resistor

Program for Uno to provide timed pull up / pull down

const int led = 13;
const int reset = 7;
const int oneKohm = 6;
const int tenKohm = 5;
const int hundredKohm = 4;
const int HIGH_impedance = INPUT;

void setup() {
  // put your setup code here, to run once:
  pinMode(reset, OUTPUT);
  digitalWrite(reset, HIGH);
  pinMode(led, OUTPUT);
  digitalWrite(led, HIGH);
}

void loop() {
  digitalWrite(led, HIGH);
  digitalWrite(reset, HIGH);
  delay(5000);
  digitalWrite(led, LOW);

  cycleLoads();
  delay(20);
  digitalWrite(reset, LOW);
  delay(20);
  cycleLoads();
  delay(20);
  digitalWrite(reset, HIGH);
  delay(120);
  cycleLoads();

}

void cycleLoads() {
  pinMode(oneKohm, HIGH_impedance);
  pinMode(tenKohm, HIGH_impedance);
  pinMode(hundredKohm, HIGH_impedance);
  delay(20);

  cyclePinUpDown(oneKohm);
  cyclePinUpDown(tenKohm);
  cyclePinUpDown(hundredKohm);
}

void cyclePinUpDown(int pin) {
  pinMode(pin, OUTPUT);
  digitalWrite(pin, HIGH);
  delay(20);
  digitalWrite(pin, LOW);
  delay(20);
  pinMode(pin, HIGH_impedance);
}

Program for the esp8266:

#include <ESP8266WiFi.h>

void setup() {
  pinMode(0, OUTPUT);
  digitalWrite(0, 0);
  pinMode(2, OUTPUT);
  digitalWrite(2, 0);
  pinMode(4, OUTPUT);
  digitalWrite(4, 0);
  pinMode(5, OUTPUT);
  digitalWrite(5, 0);
  pinMode(12, OUTPUT);
  digitalWrite(12, 0);
  pinMode(13, OUTPUT);
  digitalWrite(13, 0);
  pinMode(14, OUTPUT);
  digitalWrite(14, 0);
  pinMode(15, OUTPUT);
  digitalWrite(15, 0);
  pinMode(16, OUTPUT);
  digitalWrite(16, 0);

  delay(1000);

  ESP.deepSleep(0, WAKE_RF_DEFAULT);
}

void loop() {

}

Getting prism.js to work with Jekyll from a CDN

2016-07-16T00:00:00+01:00

One option to syntax-highlight some code in a web page is prism.js. I use it in some posts on this blog.

It's fairly simple to set up - but there's a trick to getting it to work from a CDN. When you download prism.js from the project website, a configuration tool lets you choose the languages you want included, then gives you a prism.js with only those languages included.

Importing support for the language you need

The version on cdnjs doesn't have a configuration tool - but rather than making a big file with all languages included, the base version includes a small set of languages, and other languages can be imported by including additional components.

The base version includes support for javascript, css, 'markup' (which works on HTML, XML, and XML-based things like SVG) and 'c-like'.

Languages that you haven't imported support for will be left unstyled - as if prism.js wasn't working at all. You have to explicitly import the language support files you need. As an example, here's how to style a bash script with prism.js from cdnjs:

<link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/prism/1.5.1/themes/prism.min.css">

page content goes here

<code class="language-bash">
#!/bin/bash
set -e # Exit on failure
# bash script here!
</code>

<script src="//cdnjs.cloudflare.com/ajax/libs/prism/1.5.1/prism.js"></script>
<script src="//cdnjs.cloudflare.com/ajax/libs/prism/1.5.1/components/prism-bash.min.js"></script>

Getting it to work with Jekyll

Follow this stackoverflow answer to adjust your post template to support extra_css, then include the required stylesheet in the YAML frontmatter:

---
layout: post
title: "Getting prism.js to work with Jekyll from a CDN"
description: ""
category:
tags: [programming, javascript, syntax highlighting, prismjs]
extra_css:
  - //cdnjs.cloudflare.com/ajax/libs/prism/1.5.1/themes/prism.min.css
---

Then just indicate any code blocks with triple-backticks:

```bash
#!/bin/bash
set -e # Exit on failure
# bash script here!
```

and the results should look like this:

#!/bin/bash
set -e # Exit on failure
# bash script here!

Congratulations! You (probably) now have prism.js syntax highlighting.

Making an ubuntu box that's only on in business hours

2016-06-16T00:00:00+01:00

So you've got an ubuntu box you want on during business hours, but you don't need it on outside business hours. Maybe it's a build server or something similar. Obviously, you could manually start it every morning - but you might be ill or go on holiday. Let's automate it with wake-on-RTC! The instructions below are for Ubuntu 14.04.

First of all, create this file at /root/weekday-restart.sh (I use sudo pico /root/weekday-restart.sh but you can use any text editor you like!)

#!/bin/bash
set -e # Exit on failure

# For some reason, if wakealarm is already set it's an error to set it
# to a new value - unless we first clear it by writing 0 to it.
echo 0 > /sys/class/rtc/rtc0/wakealarm

# 1=Monday, 7=Sunday
current_day_of_week=$( date '+%u' )

if [ "$current_day_of_week" -lt "5" ]; then
  sleep_for_days=1
else
  sleep_for_days=$((8-current_day_of_week))
fi

# This bit determines the time of day for the next boot:
next_boot_text=$( date '+%Y-%m-%dT08:50:00' -d "+$sleep_for_days days" )
next_boot_unix=$( date '+%s' --date="$next_boot_text" )

echo "$next_boot_unix" > /sys/class/rtc/rtc0/wakealarm
cat /sys/class/rtc/rtc0/wakealarm
echo "Scheduled wake for $next_boot_text = $next_boot_unix"

# You can make a call to cronitor or healthchecks.io here if you like.

Then make it executable and edit root's crontab:

chmod u+x /root/weekday-restart.sh
sudo crontab -e

And add the following lines:

# m  h      dom  mon  dow   command
0    10,17  *    *    *     /root/weekday-restart.sh > /root/last-restart.log 2>&1
0    20     *    *    *     pm-suspend

Scheduling the restart will log the results (and any errors) to /root/last-restart.log so if you have any trouble you can debug it easily.

The pm-suspend line suspends the box at 20:00 but the wake-on-rtc will work just as well if you power off the box entirely. This can also give you a convenient chance to apply any updates - but if you're going to run apt-get, make sure your script sets the path so it works right under cron.

Certificate Transparency and the Certificate Authority Death Penalty

2016-03-23T00:00:00+00:00

Certificate Transparency (CT) is a proposal that, when a certificate authority issues a certificate, they should log the details in a public, append-only log.

This allows mis-issued certificates to be spotted by site owners - and hopefully revoked in short order. Browsers, when they know to expect a CT entry, can reject a certificate as fraudulent if the expected CT entry is missing.

But CT will have two other interesting effects I haven't heard mentioned in other public discussion.

The Certificate Authority Death Penalty

Currently, no matter how badly a certificate authority messes up, it's difficult for browsers and OSes to remove them from the trusted root certificate set because of the collateral damage when innocent sites become unavailable. This is particularly an issue with the larger CAs; in a 2010 report, 21.8% of valid certificates - 300,224 out of 1,377,067 - were signed by the same GoDaddy certificate so that would be pretty hard for Mozilla or Google or Microsoft to drop.

But there is a way browser vendors could kill a CA without such collateral damage - they can 'grandfather in' a list of certificates that should remain trusted even after the CA becomes untrusted. A list of every certificate issued by that CA that hasn't expired or been revoked.

You'd imagine every CA would maintain such a list in-house - but if the CA has messed up and they know the browser vendors are about to put them out of business, they have no incentive to cooperate and every incentive to claim the family dog ate their customer database.

By generating a public database of issued certificates forcing CAs to put all certificates into it, browser vendors gain leverage over certificate authorities because they can put the certificate authority out of business with much less collateral damage.

That sounds bad for Certificate Authorities, but it's actually good

So why would certificate authorities support this, if it puts a gun to their heads?

The first reason is because they actually care about security. They don't want to mis-issue certificates, and they'd rather their mistakes be detected and rectified. With over 650 organisations that can issue certificates, detecting and getting rid of the cowboys is good for the industry's reputation.

The second reason is browser vendors can do it with or without them - if CAs don't submit to CT logs, browsers can do it for them (at the cost of missing mailserver certificates and corporate intranet sites that are never accessed by modern browsers).

But the most interesting reason is, by radically increasing the cost of signing a bad certificate, it makes it much less likely anyone will force them to sign bad certificates. Game theoretician and nobel laureate Thomas C. Schelling describes the reason, in The Strategy Of Conflict:

There is probably no single principle of game theory that epitomizes so strikingly the mixed-motive game as this principle that a worsening of some or even all of the potential outcomes for a particular player and an improvement in none of them may be distinctly - even dramatically - advantageous for the player so disadvantaged. [...] It was reported unofficially during the Korean War that when the Treasury Department blocked Communist Chinese financial assets, it also knowingly blocked some non-Communist assets as a means of immunizing the owners against extortionate threats against their relatives still in China.

In simple terms: If you make it impossible for you to do something, nobody will try to force you to do that thing.

If a CA can issue a certificate for windowsupdate.microsoft.com and the only cost is some bad publicity and a slap on the wrist, an extortionist can threaten something medium-sized (like revaling the CEO's affair) expecting the company to cave. But if the cost is that the CA goes out of business within a week, the company is in a much better position: nobody would expect them to cave to such a threat, so nobody will bother to make such a threat.

Payoff matrix without CT:

Threat	Issues bad cert	Doesn't
Request	-10	0
Extortion	-10	-1000
Bribery	990	0

Payoff matrix with CT:

Threat	Issues bad cert	Doesn't
No extortion	-100000	0
Extortion	-100000	-1000
Bribery	-99000	0

Without CT, bribery and extortion change the optimal strategy for the CA - whereas with CT the optimal strategy is always the same, rendering bribery and extortion pointless.

Granted, bribery and extortion would work if the threat or bribe was greater than putting the CA out of business - but even then, it would only get the attacker one set of certificates which would be detected and revoked in short order. Certificate Transparency is good for CAs because it makes threat-proof.

Some software design principles I've found useful, that you might find useful too

2016-02-22T00:00:00+00:00

Here are some subjective opinions I've developed on software design - I'm "putting them out there" to see what other people think of them, and to explain to people who hear my opinions the thinking behind them.

Even the greatest dependencies can go down.

When you depend on another system, you should ask what you'll do if that system goes down. If anyone tells you "it will never go down", they probably don't know what they're talking about. Unless "it" is airliner avionics with three independent versions coded by separate companies - and maybe even not then.

Occasional multi-hour service outages are inevitable. Amazon.com and gmail can have outages, and who could have have bigger incentives, better practices or better employees than that?

Divide your dependencies into those you can and can't work without, and make the second list as short as possible.

If you're writing an service backed by an SQL database, I don't expect it to keep working if the SQL database is unavailable - but if the service also relies on a mail-sending service, you can probably still service requests if the mail-sending service has a brief outage.

Cache everything you can, and prefer to retrieve all the data to cache where that's feasible.

For any reasonably small, reasonably stable dataset - less than a few hundred kilobytes, say - you should try to retrieve the whole dataset and cache it in memory. For example, if your software needs a list of EC2 regions, you should retrieve the entire list, cache it, and periodically refresh it.

This isn't just because it'll be faster - although it will be. It's also because, if the service you're getting data from is temporarily unavailable, you'll be able to soldier on with your cached data.

The I prefer getting all data to just getting data on demand is to get simple, predictable behaviour. If you load the entire list of EC2 regions, it's either fully populated or not at all. But if you load entries on demand and some are cached and others are missing, you might end up with confusing failure behaviour where some requests succeed while others fail. It also gives you a simple way to remove items from the cache when they're removed from the underlying dataset, preventing another possible confusing behaviour.

But raise a big flag if a dependency is down when your service starts up.

Your production environment should be at least as reliable as your integration testing environment - and it's normal for integration test environments to be much less reliable, what with all the testing going on (if you have a complex system with a highly reliable test environment, I'd be glad to be corrected on this one).

If test environments have nine times as many problems as production environments have, and you launch something in your test environment that reports Amazon's production environment is down, nine times out of ten the problem's going to be on your end. Maybe you've messed up a firewall or proxy or test configuration or security group.

You should make this raise a big red flag in your test environment - something so abundantly clear the people testing can't possibly miss it. You could prevent the service from starting up at all - but that might be inconvenient because of....

Test what you fly, fly what you test

When the thing you deploy to production is different to the thing you tested, the differences will bite you in the ass. If at all possible, the file you use for final testing should be exactly the same file you're going to deploy.

Example: Your database includes a "last modified by process" column, and any time your program updates a row, it puts its name and version into that column. The version is compiled into your software, and you do a different build with for every environment, with the environment name included in the version string. The previous version wrote ProgramName-1.0-Production and the version in test writes ProgramName-1.0.1-CIT, both were tested and work fine. But ProgramName-1.0.1-Production couldn't update anything - because the version name was too long to fit into the column.

Avoid message queues wherever possible, especially for interfaces between teams.

With a HTTP API, if the sender sends a malformed message, the receiver can return an error code immediately, so an error is reported close to where it's called. Your automated integration test can fail, or your operations team get an alert about the right system.

With a message queue, such a malformed message will only cause an error at the receiver. You'll need an extra interface if you want to do automated integration testing; and if malformed messages are being sent chances are your operations team will get an alert from the receiver, not the system with the fault.

I've also seen a bunch of other mistakes and faults:

"Our consumer was down and no-one noticed for 4 hours"
"We have a queue size alert, but the alarm threshold is set to avoid false alarms from occasional surges. It takes thirty minutes of traffic to build up before the alert triggered, so our consumer was down and no-one noticed for 30 minutes"
"To detect producers going down, we have an alert if there's no traffic for a certain time - but because two separate services send to this queue, when one failed the traffic from the other prevented the alert from triggering"
"Our consumer took a batch of messages off the queue, but crashed before putting them into the database. Can you resend the recent messages, please?"

There are ways around these problems, but they're all very clunky compared to just replacing your queue with a HTTP API.

If you want to avoid vendor lock-in, you can only use the features of the second-best vendor.

Or more precisely, you can only use the intersection of the features of the first and second best vendors. If the best vendor adds a new feature no-one else has, and you start relying on it, that's vendor lock-in. And that temptation will be very difficult to resist.

I include this point also for people who are trying to convince others of the importance of avoiding vendor lock-in; some people will be difficult to convince for this reason.

Turn your problems that happen once a year into problems that happen once a day.

Code that runs thousands of times a day is easy to trust - if there were any major problems, they would have already been laid bare by the extensive use.

Software that only gets run rarely is a different matter. If it's a rare event that you have to update that huge file perhaps you'll find someone else has used the disk space you were relying on using; if it's rare you have a server failure perhaps you'll find your failover server is missing some critical configuration or your clients can't gracefully failover; if it's a rare event that you update your SSL certificates, perhaps you'll forget to do it all together.

The simplest way to make the rare events as reliable as the regular events is to turn the rare events into regular events. This is the idea behind Netflix's Chaos Monkey - but you don't have to limit it to server failures. If you want people consuming from your queues to support multiple deliveries, make sure they get multiple deliveries; and if you want your web service clients to retry when they get a retryable error, make sure they get them regularly.

Needless to say, you'll find this a lot easier to introduce at the start of a software project.

Build servers are great, but never lose the ability for developers to build locally.

If you let the build server turn into a black box with behaviour nobody else can replicate, you're going to have a bad time.

Being able to do a full build locally is super useful when you have problems with your build server - the people who aren't fixing the build server can keep working, and the people who are fixing the build server can test things locally and compare their results to those of the build server. And even if you never need to understand your build server in order to fix it, at some point you're going to want to add capabilities or upgrade it, so you still need to understand it.

The smallest 256x256 single-color PNG file, and where you've seen it

2015-10-24T00:00:00+01:00

Do you recognise these images? There's a good chance you've seen some of them before now, probably several times. Here's a hint: They all depict the same thing.

103 bytes

156 bytes

178 bytes

379 bytes

921 bytes

1,189 bytes

They all look pretty similar - so why the size difference?

A quick intro to PNG

The basic PNG file is comprised of recurring chunks. Each chunk is comprised of four parts:

Length of the data block (four bytes)
Chunk type (four bytes)
Data
CRC (four bytes)

Here's the hex content of the smallest PNG shown above - the 103 byte file:

89 50 4E 47  0D 0A 1A 0A  00 00 00 0D  49 48 44 52 | .PNG........IHDR
00 00 01 00  00 00 01 00  01 03 00 00  00 66 BC 3A | .............f¼:
25 00 00 00  03 50 4C 54  45 B5 D0 D0  63 04 16 EA | %....PLTEµÐÐc..ê
00 00 00 1F  49 44 41 54  68 81 ED C1  01 0D 00 00 | ....IDATh.íÁ....
00 C2 A0 F7  4F 6D 0E 37  A0 00 00 00  00 00 00 00 | .Â ÷Om.7 .......
00 BE 0D 21  00 00 01 9A  60 E1 D5 00  00 00 00 49 | .¾.!....`áÕ....I
45 4E 44 AE  42 60 82                              | END®B`.

The IHDR section - further specification here - marks this as a 256x256 pixel image (00 00 01 00 00 00 01 00), 1 bit depth (01), with each pixel represented as an index into a palette (03). The image uses the default deflate compression (00), basic filtering (00) and no interlacing (00).

The PLTE section is mandatory for paleted images (PNG also supports true color images, where it isn't required) - it can contain up to 256 entries, but this one only contains a single entry - B5 D0 D0 - as only a single color is needed. This is also an example of a "truncated palette" - the one-bit depth selected in the IHDR section allows 2 colors, but as only one color is used, only one is included in the palette.

The IDAT section is the actual image data. It's 31 bytes long, compressed with the DEFLATE algorithm. If you inflate the data in this example, you get 8448 bytes of zeros. Why 8448? Well, 8192 bytes correspond to the pixels at one bit per pixel ( 256*256/8 = 8192 ) and 256 correspond to one byte for each line of the image, specifying a filter to apply. In our case, the filter byte is always 0 which corresponds to no filtering being performed.

The IEND section marks the end of the image.

Why the differences in size between the images above?

Size	Explanation
103 bytes	Palettised, 1 bit per pixel, 3 byte PLTE, 31 byte IDAT
156 bytes	Palettised, 8 bits per pixel, 3 byte PLTE, 84 byte IDAT
178 bytes	Palettised, 8 bits per pixel, 3 byte PLTE, 85 byte IDAT, pHYs section (physical pixel dimensions)
379 bytes	Palettised, 8 bits per pixel, 3 byte PLTE, 307 byte IDAT (The IDAT has the same all-zeros content as the two images previously - just a less effective compression algorithm?)
921 bytes	Palettised, 8 bits per pixel, 768 byte PLTE (with padding in all but the first three bytes), 84 byte IDAT
1,189 bytes	Palettised, 8 bits per pixel, 768 byte PLTE (with padding in all but the first three bytes), 84 byte IDAT, 256 byte tRNS section (transparency data for palette entries, padding in all but the first byte)

(The last one is actually served by redirecting to a cached resource, so it doesn't work out that bad compared to the others)

Why would it take 84 bytes to represent data that was all zeros? Haven't these people heard of run length encoding?

The deflate algorithm is used to compress the IDAT section of PNGs, and it does support run length encoding - but the run length is represented as an 8-bit huffman-encoded number; the theoretical maximumn compression ratio for deflate is about 1030.3 : 1

We don't achieve that theoretical maximum in practice (there are fixed overheads for things like defining the dictionary). The compression ratios we see include 264.2:1 for the 103 byte file and 780.1:1 for the 156 byte file.

So who needs these micro-optimisations?

Here's the 103 byte image in a more familiar context:

Map tiles representing the sea!

Slippy maps traditionally serve fixed size 'map tiles' in a grid at a given zoom level - for example http://b.tile.openstreetmap.org/6/29/20.png would be zoom level 6, the 29th column, and the 20th row; http://a.tile.openstreetmap.org/19/516664/319949.png would be zoom level 19, column 516664, row 319949. The higher the zoom level, the more tiles are needed to show the entire planet - and there's a whole lot of sea within that data. Depending on the size of your monitor, you might have downloaded 10 or more all-sea tiles just to render that map above, and that's before you've zoomed in or scrolled around.

Servers could send a redirect to a cached resource - some maps do exactly that! But when you can send a tile in just 103 bytes, sending the tile is as simple as sending the redirect.

Size	Map provider	Source
103 bytes	OpenStreetMap	Example
156 bytes	Bing Maps	Example
178 bytes	Google Maps (classic)	Example
379 bytes	Skobbler Maps	Example
921 bytes	TomTom	Example (actually a redirect - so the size of the initial image doesn't matter that much)
1,189 bytes	here.com (Navteq)	Example

Got any other tricks up your sleeve?

69 bytes - if all your clients support scaling

Instead of serving a 256x256px image, you can serve a 1px image and tell the browser to scale it up. Of course, if you have to put width= "256px" height= "256px" into your HTML that adds 30 bytes to your HTML! But if you're inserting the DOM nodes programmatically using javascript (such as for a slippy map) this isn't a problem.

When you make a very small image it's actually more efficient to use a true color image instead of a palettised image, as each extra segment in the PNG file adds 12 bytes of length, header and CRC information. But that benefit quickly diminishes as image size increases; converting the 103 byte file to true color increases its size to 567 bytes.

Some slippy maps use larger tiles; a 512px tile, taking up 126 bytes, can replace four 256px tiles taking up 412 bytes between them as well as overhead for http headers. But in areas where tiles show more data, you'll use relatively more bandwidth on things outside the viewport, which won't be shown unless the user scrolls. Some versions of Google Maps use 512px tiles, as does here.com.

TomTom doesn't serve the 921 byte tile repeatedly - instead users get a HTTP 302 redirect with header "Location: http://s3.amazonaws.com/mascoma-renderer-static-san-prod/rialto/water.png" which is an 84-byte header - but it means users only have to retrieve the generic water tile once; thereafter it's served from the browser cache. The gains can be even greater if you want something more complicated than a single-color image!

Of course, no matter how small you get your PNG you'll still have to send 400+ bytes of HTTP headers in each direction!

Conclusion

You can make a PNG pretty small if you want to, and different software and settings can produce files that vary in size quite a bit.

If you're doing something that involves moving lots of small PNGs around, a hex editor is all you need to check you're using the minimum number of bits per pixel, your palette is truncated, and you don't have any data segments you don't absolutely need.

The compression in PNG files isn't as good as the source coding theorem would hope for, and large blocks of identical data don't compress down to nothing as completely as you might hope for.

Contraction Hierarchies path finding algorithm, illustrated using three.js

2015-06-12T00:00:00+01:00

Have you ever wondered how Google Maps can calculate driving directions so fast? Google can calculate driving directions in 100-500 milliseconds, whereas back in 1990s it took my father's 386sx about 30 minutes to calculate such a route.

If you've done much programming, you probably already know the basics of this. Road maps can be represented, in the jargon, as "graphs", where junctions are "nodes" and roads between junctions are "edges" and the time taken to drive down each edge is its "cost". Finding the fastest route between two locations is then a matter of finding the shortest path between nodes of the graph.

Traditional algorithms for finding such routes include Dijkstra's Algorithm and the A* algorithm. If you haven't heard of those, go read the wikipedia pages - because the rest of this post is going to assume you know them!

There are several algorithms that are a lot faster and more memory efficient than Dijkstra's Algorithm. They rely on doing a bunch of time-consuming preprocessing on the graph; once the graph is processed individual queries can run a lot faster. This post is about the "contraction hierarchies" algorithm, one of the simpler algorithms that does this.

This algorithm isn't my invention - it was proposed by Robert Geisberger, Peter Sanders, Dominik Schultes, and Daniel Delling of Universitat Karlsruhe - and I can't guarantee Google use this specific algorithm. I'm sure they use something similar, though!

Preprocessing the graph

We preprocess the graph by "contracting" the nodes one at a time.^{Wondering how we choose the order? Read on to find out} We number nodes in the order we contract them. To perform the contraction we work out every shortest path between the node's immediate neighbors and insert "shortcuts" for them, then we mark the node as processed.

Here's a simple graph, before and after a node is contracted:

To contract C we insert shortcuts from A to E and from A to B because A->C->E and A->C->B are shortest routes. We don't need to insert a shortcut from B to E because B->C->E isn't the shortest route from B to E. B->D->E is shorter.

Contracting doesn't completely delete a node - but the node can be ignored for the rest of the contraction process, because when we see an edge going toward C we know there's either a shortcut or it isn't a shortest path; either way, we don't need to visit C. If we do a search that starts or ends at C we'll still be able to find it - you'll see how later in the post.

What about when there are two routes to an end point, both routes exactly the same length? If you want to find all such routes, you should insert a shortcut whenever it's the same length as the shortest path; if you don't want multiple results, you only need to insert shortcuts when they're the only shortest path. Two routes exactly the same length is unusual with the way roads are usually laid out in the UK, but it might be relevant in cities laid out as grids, or when pathfinding for video game AIs where the game uses a grid layout.

A visualisation of the contraction process

Here's an animation showing the contraction process.

Cyan markers identify individual nodes
Node heights indicate their position in the contraction order - the highest being nodes that haven't been contracted yet
Blue lines are edges from the original graph.
Green lines mark where shortcuts are inserted.
White lines show the original graph in 2D, for comparison.

As this is implemented in three.js you should be able to scroll around with gestures, or with your mouse or keyboard - give it a go!

So how come the nodes get contracted in that order?

The algorithm will produce correct results regardless of the contraction order - but how much of a performance improvement you get depends on the contraction order.

Papers on the algorithm have evaluated several heuristics to choose node orders.

Usually we prefer contractions that reduce the number of edges, after added shortcuts are taken into account. Better to do a contraction that removes five edges than a contraction that only removes four!

We also try to keep contraction depths roughly even across the map - if you had a hundred nodes in a row and you contracted them from right to left, you wouldn't have to add any shortcuts - but to path from one end to the other you'd have to visit every node. On the other hand if you contract every other node the contracted graph will look more like a binary tree, and the search process will have to visit fewer nodes.

There are a bunch of other heuristics you can apply - see the paper linked above for examples.

Finding the shortest path

To find the shortest path between two nodes we perform two searches - one from the start node, one from the end node - then we see where they meet. Let's assume we want to find the route between node 1 and node 8.

Search from the start node

The search is similar to Djikstra's algorithm - but with one extra rule: We only follow edges going to nodes with a higher contraction order than the current node. In our visualisation, this means we only ever follow edges that slope upwards:

Search from the end node

The search from the end node works the same way. If your graph has directed edges, the search from the end node will go follow edges in the reverse direction. For simplicity, all edges in this visualisation are bidirectional.

Combining the forward and reverse searches

We take the lists of nodes visited in the forward and reverse searches and find the intersection.

From node 1:

Node 1 reached immediately
Node 2 reached at cost 1, via 1->2
Node 9 reached at cost 4, via 1->2->9 (2->9 via shortcut 2->7->9)

From node 8:

Node 8 reached immediately
Node 3 reached at cost 1, via 8->3
Node 7 reached at cost 1, via 8->3
Node 9 reached at cost 1, via 8->9
Node 2 reached at cost 2, via 8->3->2
Node 4 reached at cost 2, via 8->3->4

Putting one route after the other, we cna get from node 1 to node 8 via 1->2->3->8 at cost 3.

If the result sets had several entries in common, we would choose the one with the lowest cost.

Visiting eight nodes out of ten doesn't seem like much of an improvement?

Right, but drawing ten nodes makes for a simpler animation! Let's look at a 40x2 grid, with a total of 80 nodes:

The first search sequence visits 11 nodes, the second search 13. That's a total of 24 nodes out of 80 visited, to discover a path 39 segments long.

So what would a search of an entire country involve?

I live in the UK, so I've been processing OpenStreetMap data for the UK.

Below you can see the two searches required to find the shortest path between Albert Drive in Glasgow and Herbal Hill in Islington, London. The map is shown as a png because the interactive map usually gives me an 'unresponsive script' warning.

The search starting in Edinburgh is shown in blue; the search starting in London in red; and the shortest path in black.

Tiles © Mapbox, Map data © OpenStreetMap.

The UK map data contains 3,478,051 nodes and 7,646,738 (non-shortcut) edges. The contraction process added 6,186,168 shortcuts.

The searches from London and Edinburgh visited at 3,968 and 3,113 nodes respectively, and the solution path found passed through 3,617 nodes.

For my implementation, the search process took 18ms - plenty fast enough for the 500ms response times Google provides! Contraction for the UK took 55 minutes using a single core on a modern i7 processor, and needed about 10 GB of RAM. You could make the process more RAM- and CPU-efficient if you wanted to!

More info and similar algorithms

Hannah Bast from Universität Freiburg has a nice set of lectures on route planning - I figured out most of my implementation from lecture slides like these, because papers containing symbols like ∀ and ⊆ intimidate me :)
Robert Geisberger's 2008 thesis provides lots more details on contraction hierarchies, and comparisons to other algorithms.
Highway Hierarchies divides the road network into neighborhoods and highways. Personally I found Sanders and Schultes's algorithm harder to understand than contraction hierarchies, but they report good performance.
Transit node routing identifies important traffic junctions (such as freeway and motorway onramps) and creates a lookup table containing the shortest paths between all such junctions.
Travel networks with timetables or varying road speeds have also been addressed by Geisberger as have one-to-many travel time computations.
A google tech talk by Peter Sanders covers quite a bit about fast route planning.

Conclusion

The contraction hierarchies algorithm is actually fairly simple, and makes path finding very quick - easily below the 100ms industry standard for web pages. If you only want to know total travel time, you have to visit fewer nodes than there are in the path!

The animated 3D graphics were made using three.js - previously my only 3D experience has been with JOGL and I have to say I found three.js a lot easier!

At the time of writing, there's an OpenStreetMap fundraising drive to upgrade a database server? I couldn't have written this post without their map data, and the project has an annual budget of just $85k. Maybe you'd like to donate?

Fake £2 coin

2014-12-19T00:00:00+00:00

A few months ago I was handed some change which I didn't look at closely. Later, I realised I'd been given a fake £2 coin. I searched online and found lots of press coverage about fake £1 coins, but not much about the £2 coin. So here are some pictures!

As you can see, the fake is a slightly different colour and the detail is a lot less clear:

The fake coin bimetallic - the gold outer ring is a separate bit of metal not just painted on. But the gap between the two metals isn't very consistent.

Here's the reverse side:

The forger reproduced the texturing on the one-but-innermost of the three concentric circles, but not on the innermost circle, for some reason.

The edge includes milling and engraving In fact the text on the fake is better centered than the real coin! (Or I have two fakes, just one is much more convincing than the other)

The main things that let me spot this coin was fake were the inconsistent gap between the two metal segments, the colour, and the lack of detail in the engraving.

We know from fake £1 coins that forgers can achieve a good level of detail and colour match. So it's possible there are better fake £2 coins out there, just good enough fakes that they haven't been reported.

I gather official surveys of the number of fake £1 coins in circulation involved drawing a random sample of coins from banks and post offices and using spectroscopy to check whether the metal composition matches genuine coins.

According to FOIA requests as of December 2013 the Royal Mint only performs surveys of fake £1 coins. Another request, dated December 2014 is currently awaiting a response.

For some other reports of fake £2 coins, see here and here

Kibana Log Searching 101

2014-11-23T00:00:00+00:00

So someone has just given you access to Kibana and you're having trouble answering the kind of questions you could have answered easily with a sql- or grep-based system.

Hopefully this cheat sheet will help get you started!

How do I do a basic search?

The 'query' box works a bit Google: unstructured text search, with some special commands, and if you get the command syntax wrong it just does an unstructured text search.

Unlike Google, by default it searches for entries containing any of your search terms, and it considers hyphen a delimiter.

Example queries:

addressregistry

solution-addressregistry <-- Finds entries containing 'solution' or 'addressregistry'

What if I want to search for a string literal?

Double quotes.

'asdf-addressregistry' <-- Wrong, single quotes get ignored - this searches for 'asdf' or 'addressregistry'

"geo-address-registry" <-- That's how you quote things properly

So it doesn't tell you if you've got the syntax wrong?

I know, right? Pretty much the opposite of SQL. Takes some getting used to, and makes it harder to figure out through experimentation - hence this documentation!

How do I search a single field?

Field name, then a colon.

Example: tags:"address-registry"

What about searching for missing (or present) fields?

exists:exception

missing:message

What about searching for several things?

exists:exception AND tags:routing

exists:exception AND routing <-- Second becomes free text search

exists:exception AND ( tags:routing OR appID:geo ) <-- This does what it looks like

AND and OR are case-sensitive. Example:

tags:routing AND kjrnglkjerghljkf <-- No results, no entry tagged routing also contains free text kjrnglkjerghljkf

tags:routing and kjrnglkjerghljkf <-- Wrong, this finds anything tagged routing, or with free text 'and' or with free text kjrnglkjerghljkf

So is it it all case-sensitive, then?

Text matching is case-insensitive, but field names and AND and OR are case insensitive.

tags:routing <-- Normal query

tags:ROUTING <-- Matches the same thing (text match case insensitive)

Tags:routing <-- No results (field name case sensitive)

Why doesn't appID:geo-addressregistry-v1 do what I expect?

The hyphens are delimiters. It's searching for anything with appID containing geo, or free text containing addressregistry or v1.

Well then, why doesn't appId:"geo-address-registry-v1" do what I expect?

Check the capitalisation of that tag. Are you looking for appId when you should be looking for appID?

Well then, why doesn't appID:"geo-addressregistry-v1" do what I expect?

Because you're missing a hyphen. Try appID:"geo-address-registry-v1" instead.

So what are these filter things?

They're like query strings, except the results get cached. I think. And you can toggle them and add them automatically from that magnifying glass symbol! Pretty weird design if you ask me.

What's with the 'save' and 'apply' buttons?

The 'apply' button saves the filter and updates your results.

The 'save' button saves the filter but doesn't update the results. I'm not sure why you'd want to do that, but it's there if you do.

How do you find the distinct values for a field?

This can be done, but it's a bit odd. First you want to add an panel to a dashboard row:

Set the type to 'terms', the 'field' to whatever field you want the distinct values of, and the length to some big number.

You probably want the style to be 'table' - easy to experiment with it though.

I deleted the only filter and now it's telling me "No time filter Timestamped indices are configured without a failover. Waiting for time filter."

I think the first index on the database is on time or something. Anyway, you have to add it back in, you can do that with this dropdown:

How do I find the log lines immediately before and after a given line, as one might with grep -c20 ?

Unfortunately this isn't supported.

There's a 'share button' which will generate a permalink:

Note that if your query includes 'the last hour' or similar, in an hour the results will have all changed.

What about a permalink to a log line?

This feature might be in a future version - in the meantime you can do this:

Add a filter on _id so your query returns one line.
Mess with the timestamp so your link works when your line exits the 'last 30 minutes' window or whatever you have selected.
Then e-mail a permalink.

Not the most elegant method, obviously.

What if I want to copy a query between Prod and Dev or similar?

Permalinks are just references to a database entry, not a complete encoding of the dashboard. Export the schema as JSON and import it somewhere else like so:

Why is googling for kibana query strings so hard?

Kibana is backed by ElasticSearch so sometimes google helpfully adds elasticsearch query documentation to your search for kibana query documentation. But ElasticSearch has a bunch of features that don't work in the kibana query box.

For example, when you look at this documentation the one-liners at the bookmarked point in the page will work - but if you scroll up to the JSON stuff, that won't work in the kibana query box.

You may have better results searching for 'lucene query syntax' which is the syntax used by the kibana 'query' box

How do I get a graph with multiple lines?

So it turns out the green dot is a button!

If you set the type to 'topN' and the field to whatever you want to chart, it'll chart the frequency of the most frequent N values:

You can also add multiple queries by using the '+' to the right of the query box

Unfortunately this graph seems to count hyphenated values multiple times (e.g. 'address-registry' counts for both 'address' and 'registry') them's the breaks.

Dirty Pcbs first trial board

2014-05-14T00:00:00+01:00

After I saw Dirty PCBs on HN I decided to place a test order.

I chose a 50mm x 50mm board and paid $12.

I placed the order on the 13th April; the boards were reported as shipped on the 30th April, and they reached me (in the UK) the 14th May. Dirty boards tell me there was a hiccup at the board house, as they moved locations, so their normal turnaround time may be less.

The customs value was $5, below the threshold to be charged duty and VAT. That was good, because the Royal Mail would have ripped me off with an £8 handling fee if there had been tax to pay.

Here's the design I sent to be manufactured - yes, I could have spent more time getting the layout nicer, but this is just for testing:

And here's the board I got back:

For a more detailed scan, see here

As you can see, the solder resist between the pins of the TQFP (U1) is a bit lacking - although that might be partly due to my shitty footprint drawing skills. Edit: Confirmed as my mistake - see below.

The drills and silkscreening could be aligned more precisely - the last image above shows one of the worst boards, most of the ten are tidier than that. Even the worst board is within the advertised tolerances.

All things considered, the price is good, it's a lot less hassle than setting up tubs of ferric chloride in my bedroom, and more affordable for prototypes than Eurocircuits.

Edit: It turns out there's a setting called 'solder mask expansion' where holes for pads in the solder mask are made slightly larger, in case the solder mask is slightly misaligned. I had this set to 0.1mm - and given that my pads were 0.5mm wide and 0.8mm apart, giving a 0.3mm gap between pads, by applying a 0.1mm expansion on both sides, the solder mask was reduced to 0.1mm wide.

I ordered another board, and this time I reduced the solder mask expansion from 0.1mm to 0.05mm, and reduced the pad width from 0.5mm to 0.4mm (the datasheet lists the pin sizes as 0.37mm nominal, 0.45mm max). I also made the pads longer, as I realised I'd made them so short the pins of the chip completely covered them! Before and after footprints:

The red area here is the pad itself, and the purple border is the solder mask expansion - as you can see the gaps between pads are looking a lot healthier now!

Amazon collection locker

2014-04-06T00:00:00+01:00

I tried collecting some stuff from an Amazon locker, to see what the customer experience is like.

The locker address is presented as a normal delivery address in your address book.

When your item is available, you get an SMS: "Hello. Your parcel with (Item Name Which May Be Truncated) was delivered to Amazon Locker. Use pickup code: ABCD1A for this parcel"

The message arrived quite quickly - I assume the locker triggers sending the SMS when the parcel is deposited, rather than the courier company triggering it when the driver marks the item as delivered.

This locker is indoors. The power and 3G connections can be seen at the top.

There's a camera above the screen. Presumably this monitors customers picking up goods and drivers dropping off goods.

A customer service contact number is marked on the lockers.

When no-one's picking things up, the screen shows information on locker delivery:

An access code is sent by e-mail / SMS when your item is in the locker. On the bottom left is a button marked 'carrier entry' which is presumably what the delivery drivers use to add items to the locker.

The code unlocks the locker like so:

The locking mechanism looks like a rotary latch lock.

I had no need for this barcode scanner - presumably it's used by the driver dropping packages off:

The parcel looks like this - my guess is the driver presents the 'locker bar code' to the locker scanner and it opens an appropriate locker for the package.

Getting jd-gui to work on Ubuntu 13.10 x86_64

2014-02-22T00:00:00+00:00

So you've just downloaded the latest version of jd-gui for linux but it doesn't start.

TLDR:

sudo apt-get install libgtk2.0-0:i386 libxxf86vm1:i386 libsm6:i386 lib32stdc++6

More details:

Here's the process I went through:

$ ./jd-gui 
./jd-gui: error while loading shared libraries: libgtk-x11-2.0.so.0: cannot open shared object file: No such file or directory
$ sudo apt-get install libgtk2.0-0:i386
$ ./jd-gui 
./jd-gui: error while loading shared libraries: libXxf86vm.so.1: cannot open shared object file: No such file or directory
$ sudo apt-get install libxxf86vm1:i386
$ ./jd-gui 
./jd-gui: error while loading shared libraries: libSM.so.6: cannot open shared object file: No such file or directory
$ sudo apt-get install libsm6:i386
$ ./jd-gui 
./jd-gui: error while loading shared libraries: libstdc++.so.6: cannot open shared object file: No such file or directory
$ sudo apt-get install lib32stdc++6
$ ./jd-gui 
Gtk-Message: Failed to load module "overlay-scrollbar"
Gtk-Message: Failed to load module "unity-gtk-module"
(jd-gui:4746): Gtk-WARNING **: Unable to locate theme engine in module_path: "murrine",
...And it works!

Happy decompiling!

Sending breaks to serial devices from the terminal, using termios

2014-01-04T00:00:00+00:00

I'm playing around with an mbed LPC1768 for a project I'm working on. When you upload a new binary to the LPC1768 you have to perform a reset for it to run the updated binary; you can do this with a reset button on the board, or by triggering a soft reset by sending a break signal to a virtual serial port. To speed up my compile-upload-test cycle I wanted to be able to do this from the terminal, on OS X.

You can send a break manually using screen, but I wanted something noninteractive so I could trigger a compile-upload-test with a single command. Turns out it's fairly simple in C:

Compile with gcc -o sendbreak sendbreak.c then run with ./sendbreak /dev/tty.usbmodem622 where /dev/tty.usbmodem622 is the device you want to send the break to.

Don't have gcc? I think for me it came with xcode.

Barcode reader test cases

2013-12-01T00:00:00+00:00

I've had an interest in barcodes and image processing for some time, and as a side project some time ago I put together a software barcode reader. One of the things I did for that project was start collecting photos of barcodes that didn't scan well - either to use the images for my unit tests, or to order the products for physical tests.

You can use these photos of degenerate barcodes to test your own barcode scanning software, or to order the products yourself for testing. If you want the images at a higher resolution, drop me an e-mail.

Low constrast barcodes

I don't know if these ones are just low contrast, or if the blue causes problems for red-laser scanners.

Seriously?

Product packaging in grey:

Low quality printing

Barcodes on flexible material

Had to stretch this one out to get a scan:

Barcodes on transparent material

Size & Detail

Which is a dark band, which is a light band?

Bad quiet zone

Reflective packaging

Difficult to see in the photo but this was hard to scan with a presentation scanner:

Falsehoods programmers believe about addresses

2013-05-29T00:00:00+01:00

Perhaps you've read posts like Falsehoods Programmers Believe About Names and Falsehoods programmers believe about time. Maybe you've also read Falsehoods programmers believe about geography.

Addressing is a fertile ground for incorrect assumptions, because everyone's used to dealing with addresses and 99% of the time they seem so simple. Below are some incorrect assumptions I've seen made, or made myself, or had reported to me. (If you want to look up an address for a UK postcode or vice-versa to confirm what I'm telling you, try the Royal Mail Postcode Finder)

An address will start with, or at least include, a building number.

Counterexample: Royal Opera House, Covent Garden, London, WC2E 9DD, United Kingdom.
When there is a building number, it will be all-numeric.

Counterexample: 1A Egmont Road, Middlesbrough, TS4 2HT

4-5 Bonhill Street, London, EC2A 4BX
No buildings are numbered zero

Counterexample: 0 Egmont Road, Middlesbrough, TS4 2HT
Well, at the very least no buildings have negative numbers

Guy Chisholm provided this counterexample: Minusone Priory Road, Newbury, RG14 7QS

(none of the databases I've checked render this as -1)
We can put those funny numbers into the building name field, as no buildings have both a name and a funny number

Counterexample: Idas Court, 4-6 Princes Road, Hull, HU5 2RD
When there's a building name, there won't be a building number (or vice-versa)

Counterexample: Flat 1.4, Ziggurat Building, 60-66 Saffron Hill, London, EC1N 8QX, United Kingdom
A building number will only be used once per street

The difference between 50 Ammanford Road, Tycroes, Ammanford, SA18 3QJ and 50 Ammanford Road, Llandybie, Ammanford, SA18 3YF is about 4 miles (Google Maps).
When there's line with a number in an address, it's the building number.

Counterexample: Flat 18, Da Vinci House, 44 Saffron Hill, London, EC1N 8FH, United Kingdom

You also get suite numbers, floor numbers, unit numbers, and organisations with numbers in their names.

Adrien Piérard contributes an address from Japan with fifteen digits in six separate numbers (five if you count the zip code as a single number). The format is: 980-0804 (zip code), Miyagi-ken (prefecture) Sendai-shi (city) Aoba-ku (ward) Kokubuncho (district) 4-10-20 (sub-district-number block-number lot-number) Sendai (building name) 401 (flat number).
OK, the first line starting with a number then

Counterexample: 3 Store, 311-318 High Holborn, London, WC1V 7BN
A building will only have one number

Benton Lam offers this address from the Hong Kong Special Administrative Region - it has both a number on its road (14) and in its group of buildings (3): 15/F, Cityplaza 3, 14 TaiKoo Wan Road, Island East, HKSAR
The number of buildings is the difference between the highest and lowest building numbers

Tibor Schütz points out building numbers may be skipped - for example, on a street where even-numbered buildings are on one side, odd numbers on the other; multiple buildings sharing the same number (such as where a new house has been built) and buildings with more than one number.

Cyrille Chépélov and Sami Lehtinen tell me in Antibes, France and rural Finland some buildings are numbered based on the distance from the start of the road - such as Longroad 65 for the building 750m from the start of longroad.
If the addresses on the left of the road are even, the addresses on the right must be odd

Cyrille Chépélov points out that in places, Boulevard Théophile Sueur, Montreuil, Seine-Saint-Denis, France has evens-only on both sides. The two sides are also in different cities and Départements.
A building name won't also be a number

Ben Tilly reports on Ten Post Office Sq, Boston MA 02109 USA - which is not, reportedly, the same as 10 Post Office Sq, Boston MA 02109 USA.
Well, at least you can omit leading zeros

Shaun Crampton reports living at 101 Alma St, Apartment 001, Palo Alto - where apartments 1 and 001 were on different floors.
A street with a building A will not also have a building Alpha

Douglas Perreault reports he lived in a block within a condo association; it was a large association, with blocks A through Z then Alpha, Beta, Gamma, Delta, and Theta. Mail and deliveries were often misrouted from block Alpha to block A and vice-versa. His address at the time was: 14100 N 46th St., Alpha 39, Tampa, FL 33613

A street name won't include a number

8 Seven Gardens Burgh, WOODBRIDGE, IP13 6SU (pointed out by Raphael Mankin)
OK, but numbers in street names are expressed as words, not digits

Jan Jongboom reports streets can be numbered in the Netherlands - for example, Plein 1944 in Nijmegen.
When there's a numbered street and a house number, there will be a separator between them

Another from Jan Jongboom: Gondel 2695, Lelystad, means area Gondel, street 26, number 95
Street names always end in descriptors like 'street', 'avenue', 'drive', 'square', 'hill' or 'view'

They don't always - for example: Piccadilly, London, W1J 9PN
OK, but when they do have a descriptor there will only be one

A street name can be entirely descriptors: 17 Hill Street, London, W1J 5LJ or Avenue Road, Toronto, Ontario.
OK, but when they do have a descriptor it will be at the end

French addresses use prefix descriptors like 'rue', 'avenue', 'place' and 'allee'.
OK, but if there's a descriptor it'll be at the start or end of the street name.

Or the middle, like 3 Bishops Square Business Park, Hatfield, AL10 9NA
OK, but at the very least you wouldn't name a town Street

Actually there's a town called Street in Somerset, UK.
Street numbers (and building numbers) don't contain fractions

Dan, Fred Kroon, David Underwood and Daniel Dickison submitted examples of fractional street numbers like 43rd ½ St, Pittsburgh, PA, and of fractional building numbers. These can be written in unicode (43rd ½ St), as a fraction with a slash (43 1/2) or as a decimal (43.5)

Gene Wirchenko reports a fractional building number: 1313 1/2 Railroad Ave Bellingham WA 98225-4729
Street names don't recurr in the same city

Here's a map of the following addresses:
- High Street, London, W3 6LJ
- High Street, London, W5 5DB
- High Street, London, N8 7PB
- High Street, London, SE25 6EP
- High Street, London, E13 0AJ
- High Street, London, E17 7LD
- High Street, London, NW10 4LX
- Islington High Street, London, N1 9TR
- Shoreditch High Street, London, E1 6PG
- Camden High Street, London, NW1 0JH
- Kensington High Street, London, W14 8NL
- Lewisham High Street, London, SE13 6AD
- High Street Wimbledon, London, SW19 5DX
- High Street Wanstead, London, E11 2AJ
- High Street Colliers Wood, London, SW19 2AE
- High Street North, London, E6 2HJ
But street names don't recurr in close proximity

Julian Fleischer provides an example from Bocholt in Germany showing several roads in close proximity all called Up de Welle.
An address will be comprised of road names

Kirk Kerekes spent several years using an address of the form "2 mi N then 3 mi W of Jennings, OK 74038" which regularly got successful deliveries. Mike Riley used to mail the Very Large Array radio telescope at "50 miles (80 km) West of Socorro, New Mexico, USA"

Sam pointed me to Menomonee Falls where houses are addressed using Milwaukee County's grid system instead of house numbers - giving addresses like N88 W16541 Foobar St.

Andy Monat sent the following address example, from a semester abroad program at Tulane University : CIAPA, 50 meters north of the Hypermas/Walmart of Curridabat, San Jose, Costa Rica. Adrien Piérard and Luke Allardyce point out street names are seldom used in Japan - instead, districts and blocks and lot numbers are used (more info on the Wikipedia entry for the Japanese addressing system). A 2002 World Press Review report gave this sample address: From where the Chinese restaurant used to be, two blocks down, half a block toward the lake, next door to the house where the yellow car is parked, Managua, Nicaragua. Shaun Crampton sent an article with more details and examples of the Nicaraguan system. Stig Brautaset pointed out a BBC article about post in Kabul gives this example: "Hamid Jaan, behind Darul-Aman palace". Nathan Fellman reports similar addressing is used in Nicaragua and Costa Rica.

Paul Puschmann and Tibor Schütz pointed out the city of Mannheim in Germany is sometimes called Quadratestadt (City of Squares) as the city centre is arranged in a grid, with blocks assigned a letter (along the north-south axis) and a number (along the east-west axis) then buildings numbered by block number. So an example address at numbers 6 to 13 on block R 5 would be: Institut für Deutsche Sprache, R 5, 6-13, D-68161 Mannheim

Leoni Lubbinge gives an example of a South African address: Part 84, Strydfontein 306 JR, Pretoria which means the 84th plot of the farm Strydfontein 306 JR.

A road will have a name

Plenty of roads like driveways, onramps and the aisles of carparks don't have names. Some roads in Japan also don't have names, as the prevalent addressing system works on districts, subdistricts, blocks, lots and lot numbers.

Peter Kenway points out in America some homes are addressed as Rural Routes, where numbers are allocated to boxes on a route covering multiple roads. For example: Box 1234, R.R. 1, Winthrop, ME 04364.
A road will only have one name

Many different roads, from Goswell Road in London to Regent Road in Edinburgh, make up the 410 mile A1. And while there may only be one "1 Goswell Road" and only one "1 Regent Road" there are multiple buildings numbered 1 on the road designated A1.

Roads may also be named in multiple languages. For example, in Ireland roads may be named in both English and Irish
Addresses will only have one street

The Royal Mail have what they call a 'dependent street' - for example: 6 Elm Avenue, Runcorn Road, Birmingham, B12 8QX, United Kingdom (Runcorn Road is the street, Elm Avenue is the stubby 'dependent street' and isn't unique within the city. Google Maps )

Another counterexample: Rogue Hair, 1 Hopton Parade, Streatham High Road, London, SW16 6EP (Streatham High Road is the street. Hopton Parade is a little row of shops on the road - Google Maps )
Addresses will have a street

The Royal Mail will deliver to certain premises by name, such as farms and cottages. Example: Oakland, Fairseat, Sevenoaks, TN15 7LT, United Kingdom (Fairseat is the town - this is actually on Vigo Road. Google Maps )
An address will include a state in the US sense.

Counterexample: Any address in the United Kingdom.
Addresses will have something other than the organisation and city name.

Large recipients of mail are sometimes addressed by organisation alone - for example: Electoral Reform Society Ltd, London, N1 1RS, United Kingdom
An address will have a county

Iain Parris and Naath both pointed out that, while many websites ask users for a county, not all countries use them in addressing. For example, the Royal Mail stopped using postal counties in 1996.

Yves Daoust reports that in Belgium an address only requires a street, postcode and city; example: Boulevard Frère Orban, 27, 4000 Liège. Erik Engheim reports that in Norway Oslo is a By (city), Tettsted (urban area), Kommune (municipality) and Fylke (county) but it usually only appears once in a written address.
An address require both a city and a country

jzwinck points out Singapore is a city-state, leading to addresses like Singapore, Singapore - or even Singapore, Singapore, Singapore if you demand a city, county and country.
You can't have two towns cities with the same name in the same country

Sure you can - for example the UK has three towns called Newport.

Jan Jongboom reports from the Netherlands, where there are two cities called Eursinge - in the same province!
OK, but those cities won't have duplicate street names
- 10 High Street, Newport, PO30 1SS
- 10 High Street, Newport, NP20 1FQ
- 10 High Street, Newport, TF10 7AN
An address will have a postcode

Richard Fletcher, Lee Hosty, Paul O'Nolan, Simon Diab, Tony Finn, Donal Maccarthy, mark lynch and Donovan all pointed out countries like the Republic of Ireland have addresses without postcodes, or only have postcodes in certain parts of the country. Danny Angus points out this is also the case in Hong Kong.

Jessica Enders points out the post-paid address of the AEC: Australian Electoral Commission, Reply Paid 9867, your capital city

The user will know their postal code/zip code.

Most users will, of course, but I've seen incorrect postcodes on corporate letterheads! Misreading handwritten postcodes seems like a common one.
A single postcode will be larger than a single building

Although zip codes in the US usually cover very large areas, Anthony Elizondo points out some buildings have their own zip codes - like the Empire State Building (10118)

In the UK, alphanumeric postcodes are typically much more precise than US zip codes.
OK, but you don't get multiple postcodes per building

Graham Lee points out DVLA Swansea (that's the whole address), where different departments are identified by postcode: V5Cs are processed at SA99 1BA, driving licences at SA99 1AB and so on.

Malcolm Gilbert points out this example, with five postcodes for five departments:
- London Borough of Enfield, Civic Centre, Silver Street, ENFIELD, EN1 3ES
- Returning Officer, London Borough of Enfield, Civic Centre, Silver Street, ENFIELD, EN1 9SA
- Edmonton, London Borough of Enfield, Civic Centre, Silver Street, ENFIELD, EN1 9SB
- Enfield North, London Borough of Enfield, Civic Centre, Silver Street, ENFIELD, EN1 9SD
- Enfield Southgate, London Borough of Enfield, Civic Centre, Silver Street, ENFIELD, EN1 3ZW
- But the Enfield council website contact page lists their postcode as EN1 3XY - which the Royal Mail think is a PO Box at the sorting office.
A single postcode will only cover a few tens of addresses / customers

Mostly this is reasonable in the UK, but there are certain exceptions. For example CV4 7AL is the postcode of the University of Warwick - one postcode for 6,000 students living on campus, and academics/administrators working on campus.

Also, if your customers get things delivered to them when they're on holiday, lots of customers may have the same holiday parks on their accounts.

Some addresses correspond to 'flexible office spaces' and organisations that offer PO Boxes that sound like fancy offices. The Royal Mail lists more than 90 organisations operating out of Tower 42, 25 Old Broad Street, London, EC2N 1HQ. Holiday parks and cottages may also appear on many customers' accounts.

Victor Nicollet contributes the example of French postcode 75015, representing the XVth arrondissement of Paris, which covers over 230,000 people.
A zip code corresponds to a single city

Mike Cohen reports zip code 33334 covers 3 cities: Oakland Park, Wilton Manors, and Fort Lauderdale, all in Florida.
Zip codes don't start with a zero

Some Brazilian zip codes do according to speeder, as do Israeli postcodes for army units. Jessica Enders and Frank Malcolm pointed out postcodes in the Northern Territory of Australia start with 08; for example the postcode for the city Darwin is 0800. Morsillo Lindsay, Thomas Norris and Jerry B. Altzman point out American addresses in the north east have zip codes starting with zero, for example Ten Post Office Sq, Boston MA 02109, USA; and some zip codes start with two zeros. Mikael Vejdemo-Johansson reports the zip code of Jena in Germany is 07737.

Antti Vikman and Johan Myréen tell me all postcodes in Helsinki, the capital of Finland, all start with two zeroes. Some special codes even start with four zeroes (00002 HELSINKI). Andrew Jones reports the initial digits of French postcodes are the départements*, and may use a leading zero. Post addressed to 06130 Grasse is correctly delivered to the town in district 06, Alpes Maritimes - but post addressed to 6130 Grasse is first routed to department 61, Orne. klez reports Italian Codice di Avviamento Postale (CAP) numbers can have a leading zero.
- Except Corsica - Cyrille Chépélov reports it's split into départements 2A and 2B, but the Post Office kept the former single-number 20 (Ajaccio, 20000; Bastia, 20200)

Addresses will have a reasonable number of characters - less than 100, say.

Not when organisation and department names can be included in addresses! For example: Department For Environment Food & Rural Affairs (D E F R A), State Veterinary Service, Animal Health Office, Hadrian House, Wavell Drive, Rosehill Industrial Estate, Carlisle, CA1 2TB, United Kingdom

Another example: The Gynaecology Cancer Research Unit, Department of Obstetrics & Gynaecology, St. Bartholomews & The Royal School of Medicine & Dentistry, Charterhouse Square, London, EC1M 6GR, United Kingdom
But street names will be reasonably short - certainly less than 50 characters

Susanne Schmidt provides the longest street name in Germany: Bischöflich-Geistlicher-Rat-Josef-Zinnbauer-Straße in 84130 Dingolfing, Bavaria

Graham Rhind suggests this 89-character street name in Bihac, Bosnia: Aleja Alije Izetbegovića Prvig Predsjednika Predsjedništva Republika Bosna i Hercegovina
Five lines and country will cover all cases.

You'll need 8 lines and country to deliver to: GB Technical Services, Unit W7a, Warwick House, 18 Forge Lane, Minworth Industrial Park, Minworth, Sutton Coldfield, B76 1AH, United Kingdom
Addresses don't contain commas (so I can replace newlines with commas then commas with newlines and get back to where I was)

Addresses can contain organisation names, and organisation names can contain commas. For example: Society of College, National & University Libraries, 102 Euston Street, London, NW1 2HA
But they don't contain commas, brackets, apostrophes, hyphens, ampersands, dots or exclamation marks

St. Judes & St. Pauls C of E (Va) Primary School, 10 Kingsbury Road, London, N1 4AZ

1 Acre View, Bo’ness, EH51 9RQ

1 Highview Terrace, Westward Ho!, Bideford, EX39 1AQ (exclamation mark is omitted in some databases)

Flat 1.4, Ziggurat Building, 60-66 Saffron Hill, London, EC1N 8QX, United Kingdom

Kirkland, Lane, Mathias & Perry, North Muskham Prebend Church Street, Southwell, NG25 0HQ

Mark Wallace tells us The Hague in the Netherlands is has multiple official names, one of which is 's-Gravenhage (not a plural, literally a city name starting apostrophe s and with a hyphen)

Signposts for Station Road East, Grantham, NG31 6HX render it as STATION ROAD (EAST).

More generally, addresses can contain organisation names, and organisations can have names like Yahoo! with an exclamation mark. Legislation on company sets out the allowable characters for the UK and it permits company names including ! LTD (company 08209948); @ LTD (company 08209882); $ LTD (company 08209885) and % LTD (company 04487680) as well as a variety of other names I don't have examples for as the companies house website won't let me search for them.

An address will exist in the country's postal service's database

Simon Westcott points out databases are only released periodically, so it's not unusual for new buildings to be omitted.
An address in the country's postal service's database will exist

Royal Mail and Ordnance Survey data can contain demolished buildings, and buildings currently under construction. The Royal Mail even have a database product called Not Yet Built. Temporary postcodes can even be assigned to building sites!
The address from the postal service database is sufficient to deliver

Christopher Allen points out that people in new blocks of flats and houseboats in boatyards sometimes need to prefix their official address with a boat name or flat number.
Every address can be expressed in a way that will pass all validators

XaspR8d must have been exasperated by the fact his road is variously known as "S Hwy X", "Highway X" and "South County Rd X" - and related a story of being unable to buy a product as the only addresses that passed a vendor's address validation wouldn't pass his bank's address validation and vice-versa.

Jon Peterson lives in an apartment community that is divided into Quail Ridge Cir, Quail Ridge East Lane and Quail Ridge West Lane. Only the USPS and the city electric utility seem to recognize the "Lane". Everyone else requires it be shortened to "Quail Rdg E" (or W) and reportedly when UPS turns a package over to the USPS it gets returned as undeliverable-no such address.

Susannah Fleming lived on the road the Royal Mail call Top O'The Lane, Brindle, Chorley, PR6 8PA. She reports representations in different databases include:
- Top o' th' Lane
- Top o'th' Lane
- Top oth Lane
- Top o' the Lane
- Top of the Lane
- Workhouse Lane (a historical name of the road)
- Denham Lane (name of the road continuation)

Customers will have a fixed address with a fixed location

A developer just a few seats from me recently brought a house boat with the intention of using it as her primary residence. Needless to say, boats can move - including between towns and even countries!

fr0sty points out the State of Illinois catch-all approach to addresses: "IF YOU HAVE NO STREET ADDRESS, below describe your home: list the name of subdivision; cross streets; roads; landmarks; mileage and/or neighbors' names."

Sharon Freas has dealt with systems supporting "Snowbird" clients, who alternate between addresses with the changing seasons.
But written addresses don't change

Addresses can move between counties and administrative districts. For example, I used to live in the county Gwent, but that no longer exists. Addresses may also be assigned new postcodes.

Susanne Schmidt points out cities, streets and entire countries were renamed in eastern europe - for example, when Lenin fell out of favour as a street name. Addresses can be in disputed territories, or even war zones.

Douglas Perreault reports owning a condo that changed address three times; first it was 14100 N 46th St., Alpha 39, Lutz, FL 33549; then a new post office was built and it became 14100 N 46th St., Alpha 39, Tampa, FL 33612; then the ZIP Code changed giving 14100 N 46th St., Alpha 39, Tampa, FL 33613; then the condo association changed to a less error-prone block naming scheme, giving 14410 Hanging Moss Circle, #101, Tampa, FL 33613.
Postal, traditional and administrative counties all line up, right?

As you can learn from wikipedia's page on postal counties, even when the Royal Mail used postal counties, they didn't always line up with administrative counties. And of course administrative regions come and go with changes to local government structures - for example, the county I was born in in no longer exists.

Military addresses are just like regular addresses

Peter Bailey points out several countries have special military mail to deal with the complexities of delivering to soldiers deployed to other countries, ships at sea and similar; and their addresses don't always follow conventional address formats. For example, the address BFPO, BF1 4FB is the address of the navy vessel HMS Example.

Ed Schiebel reports the postcodes allocated to Israeli army units roam around with the units.
An address corresponds to the recipient's location.

Addresses such as PO boxes are often only as precise as the recipient's city or sorting office. Jessica Enders tells me the Australian post service Reply Paid addresses (no stamp needed); PO Boxes (Post Office Boxes); GPO Boxes (General Post Office Boxes, in the middle of capital cities only); locked bags; private bags; parcel lockers; parcel collect; "Care of Post Office"; CMA (Community Mail Agent); CPA (Community Postal Agent); CMB (Community Mail Bag) and Mail Service (MS)!

Tibor Schütz points out many post offices have novelty handling of mail to Santa Claus, even going as far as to allocate special postcodes. For example, in Germany: Santa Claus Nordpolen, Julemandes Postkontor, DK-3900 Nuuk; in Canada: Santa Claus, North Pole, H0H 0H0; in the UK: Father Christmas, Santa’s Grotto, Reindeerland, XM4 5HQ
An address can be expressed with a single country

Matthieu Valleton got in touch to point out his address on Kerguelen Island (Google Map), a French territory in the Indian Ocean, his address was District de Kerguelen (island), Terres Australes et Antarctiques Françaises (territory), via la Réunion (indicates where the mail should be routed through), France (country)
Overseas territories aren't (or are) always included in the postal code system

Monty points out that uncommon political hierarchies can lead to uncommon postal addresses. For example, the entire of the Falkland Islands shares postal code FIQQ 1ZZ. On the other hand, the British Virgin Islands have their own postal code system.
All addresses with a box number are PO Boxes

David Kuder pointed me to a 1990 article: Risks Digest correspondent Tim Kay had problems getting mail sent to his university campus pigeon hole: Timothy L. Kay, Box 256-80, Pasadena, CA 91125. Reportedly automatic systems changed his zip code to 91102. David Kuder identified this was because all Pasadena PO Boxes were in box 91102.

Addresses will be written in ASCII or at least Latin characters

Alastair Houghton reminds us the Greek tax office's address is Χανδρή 1 & Θεσσαλονίκης, Τ.Κ. 18346, Αθήνα

Wikipedia has a photo of a parcel where a Russian/Cyrillic address was displayed on a computer with the wrong character encoding, and transcribed from that. Reportedly a russian postal worker was able to reverse the mapping and deliver the parcel.

@shyhoof wrote an poem about an address label with ó converted, via latin1 and two rounds of HTML entities, into &AMP;ATILDE;&AMP;SUP3;
Addresses will be written in the character set of the destination country

Alastair Houghton points out addresses may be written in the character set of the source country.
But people at least use the same character set for the entire address?

International mail may specify the country (and possibly other details) in both the source and destination countries' character sets, so it can be read by postal workers on both ends.

Addresses will be written from most to least specific

Alastair Houghton provided this example of a Japanese address: 〒100-8994 (zip code), 東京都 (Tokyo-to, i.e. Tokyo prefecture or state) 中央区 (Chuo-ku, i.e. Chuo Ward) 八重洲一丁目 (Yaesu 1-chome, i.e. Yaesu district 1st subdistrict) 5番3号 (block 5 lot 3), 東京中央郵便局 (Tokyo Central Post Office). Thanks to Norman Diamond for telling me which parts of the address are which!
OK, but they'll either be in either ascending or descending specificity

Erik Engheim and Jan Jongboom report in Norway and the Netherlands the building number (within street) comes after the street name, but before the town name.

Douglas Perreault reports units within American condo associations may be written below the street name. For example: 14100 (position of condo association on street) N 46th St. (street name), Alpha (block within condo association) 39 (unit within block), Lutz (city), FL (state) 33549 (zip). Likewise, when mailing an individual at a company some people put the person's name after the company name, but before the street name.
OK, but at least the same address will always be written in the same order

György Farkas tells me Hungarian addresses can be written in different orders depending on how many lines are available. If you write the address on one line, it's expressed from less specific to more specific:

{zip} {town}, {street} {buildingNr}.

If there are more lines available, the address starts with the street, and if the country is specified, it comes after the town name:

{street} {buildingNr}.
{zip} {town}, {country}

And if there are no set number of lines, like on an envelope, it's a bit different again:

{street} {buildingNr}.
{town}
{zip}
{country}

Gene Wirchenko tells me in some parts of Canada, suite 123 in building 456 on TheStreet would be written: 123 456 TheStreet whereas in other parts the common formatting is 456 TheStreet #123
Building numbers appear before street names

In some countries this is reversed - such as in the Netherlands. For example, Plein 1944 85 D (where Plein 1944 identifies the street, 85 the building and D the flat/apartment).

Sami Lehtinen provides this example from Finland: Kornetintie 6 A II krs (Kornetintie is the street name, 6 the building number, A the staircase, II krs indicates the second floor.
Flat names/numbers names appear before building numbers

Toni Cornelissen points out addresses in the The Netherlands, such as: Vroomstraat 1a Rood, 2021HL Haarlem where Vroomstaat is the street, 1a is the building number, and Rood (literally translated as Red) indicates the upper part of the building.

A building will be within a few hundred meters of a public road

Buildings like farms and country houses can be at the end of a private road or driveway several hundred meters long.
An address with a street name is always closer to that street than any other

Lots of examples of this. For example: The National Museum of Computing, Bletchley Park, Sherwood Drive, Bletchley, Milton Keynes, MK3 6EB. 70m from Roche Gardens, 346m from Sherwood Drive, but only accessible by entering Bletchley Park from Sherwood Drive.

Real place names won't contain rude words

It's not just Middlesex and Scunthorpe you have to check for; Gary Gale has compiled a map of rude-sounding place names.
A customer will only want reminders mailed to single address

John Dye reports that many doctors' offices, dentists and so on are unable to mail both of a child's divorced parents.
Each person has exactly one address

Tibor Schütz points out people often have a different home and work address.

There are other interesting examples in the discussions at Hacker News and Reddit Programming.

Graham Rhind pointed out the website Practical International Data Management which contains other edge cases (and has added some of the examples from above).

Miscellaneous

2012-12-29T00:00:00+00:00

Not so much a blog post as things I want to remember the URLs for!

TLS (SSH) Server Name Indication patch for OpenJDK

Webrev standards-compliant HTML patch

Here's a robot:

There should be a video here.