Blogish - Tom O'Connor

That DMG ate my System Preferences

Well, that's certainly another strange problem.

We have a tendency to build our own DMG images for certain bits of software we roll out here. Sometimes we'll incorporate our own patches, other times it's just to make the application structure more FHS compliant, and stop it "shitting all over the filesystem", as we so charmingly term it.

In the past, we've used InstallEase to build a DMG and PKG installer for OSX. It's been pretty good up until last week, when one of our engineers built a PKG that had a nasty side effect of destroying the System Preferences once installed. We initially pegged this as "another weird thing about Lion", and rebuilt the image, and so on.

We tested the installation again on a different computer, all the time using Munki for package deployment.

These are the important test findings:

1) If you install the package interactively, it's fine.

2) If you install the older version, it's fine.

3) If you install the new version with munki, it breaks everything.

4) If you install anything else with munki, it's also fine.

That points to a clear difference between installing Interactively (with a person doing it) and an automated deployment with Munki.

I actually had a look at the package today, and gave it a good hard poking. I found 2 really rather worrying things.

1) Inside the Resources directory, there's an Universal Binary called DeleteObjectHelper (Not off to a good start here..), and a file called DeleteObjectList.plist

DeleteObjectList contains:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
	<key>objectList</key>
	<array>
		<dict>
			<key>filePath</key>
			<string>Library/Preferences</string>
		</dict>
		<dict>
			<key>filePath</key>
			<string>Library/Receipts</string>
		</dict>
	</array>
</dict>
</plist>

2) Inside the postflight (think postinst for debian) file was the following:

#!/bin/bash
"$1/Contents/Resources/DeleteObjectHelper" "$1/Contents/Resources/DeleteObjectList.plist" "$HOME" "$3"

So here's what happens if you install it by hand. Postflight runs after installation, and replaces $HOME with /Users/YourName. DeleteObjectHelper goes off and deletes the files in ~/Library. I assume this is to delete older versions, or something similar.

I suspect that if you run it non-interactively, with munki, then it might run as root. It might also not have the environment that you'd expect as an interactive user. That means that if $HOME was '/', or nonexistant, it might default to '/'.

Very Bad Indeed.

The pointer to all of this was something system.log complaining about Preferences files being missing, when trying to authenticate to our wireless. I've worked with *nix systems long enough to pretty much start looking in /var/log/(syslog|messages|system.log) as a matter of principal.

So, for whatever reason, the process that builds the package is incorporating a thing for cleaning up after itself. Not a bad thing, but it does have a fantastic bug in it that seems to make it delete the System Preferences if it can't find $HOME. As it's running as root (or whomever powerful), it just goes ahead and deletes everything it can find.

The questions that remain on my mind are these:

1) Why did it put that in there anyway, as none of our other hand-rolled packages have it.

2) Can we build PKG and DMG files with some kind of GNU toolchain on our Jenkins environment to stop this nonsense from happening again (We know exactly how the build process works, etc..)

3) What on *earth* went through those developer's minds when they wrote that Evil Little Binary?

Puppet, Apt and our very own Thundering Herd

Puppet really is great. Don't ever get me wrong there. It's saved me masses of time and money over the last few years, and allowed me to do my job quickly and efficiently.

That said, it really does have issues with scalability. After about 20-30 clients using WEBBrick, everything kinda falls over a bit.

We had this problem at Baseblack. We've now got ~60 workstations and rendernodes all using Puppet for configuration management and software deployment. It's great. It far simplifies the process of rolling out updates and upgrades to new build machines.

The problem we had was most clearly shown by the frequency with which the PSON error comes up.

"err: Could not retrieve catalog from remote server: Could not intern from pson: Could not convert from pson:"

And so on.

This was always a transient error, and would go away if you ran puppet 2-3, or more times, and then it'd work fine. This isn't really a valid long-term solution. It's alright now and again, but it brings up a problem of reliability, and "how do you know when it's last run, if it can't be guaranteed to run every time".

I wrote a dirty wrapperscript that would re-run it if it failed, and so on. This kinda worked a bit better, but sometimes, stuff would just not run anyway.

So today, I'd had enough of this problem, and decided to do the Apache2/Mod_passenger thing. Back in the days of Puppet 0.24/0.25x this was a bit more of a pain in the arse than it seems to be now. Back then, the server of choice was Mongrel, now it's Passenger/mod_rack.

Just follow this guide: http://docs.puppetlabs.com/guides/passenger.html

It's actually a pretty good explanation of the steps, and I don't see any need to replicate the information here. I made a couple of slight modifications.

The config.ru file is horribly outdated in the given form, and I used this one instead: https://github.com/puppetlabs/puppet/blob/master/ext/rack/files/config.ru
I changed the apache2 defaults for mpm_worker so that it could handle a shitload more requests than the default.

Incidentally, this thing is cool. Some guy's written an insanely simple "calculator" spreadsheet for OpenOffice and Excel that allows you to calculate decent settings for MaxClients.

<IfModule mpm_worker_module>
ServerLimit 150
StartServers       5
MinSpareThreads    5
MaxSpareThreads    10
    ThreadLimit         5 
    ThreadsPerChild      5
MaxClients        750
    MaxRequestsPerChild   0
</IfModule>

I wanted to be able to handle our own thundering herd of workstations and rendernodes, so that meant that the default ServerLimit had to go, and that it had to be able to handle *many* more threads than the default.

I also moved the puppetmasterd "application" from /usr/share/puppet to /srv/puppet because in my mind (and the FHS), it makes more sense.

There's a bit of a caveat in the process of moving that directory, in that wherever it is, it must be chowned puppet:puppet. After that, it's all fine.

The problems really started for us after that. It worked fine with one workstation testing it, but throw 2+ at passenger, and apache tended to kill off the ruby children.

The big hint was in /var/log/apache2/error.log:

[Tue Feb 21 15:50:12 2012] [alert] (11)Resource temporarily unavailable: apr_thread_create: unable to create worker thread

[Tue Feb 21 15:50:15 2012] [error] (12)Cannot allocate memory: fork: Unable to fork new process

So, our puppetmaster runs on Proxmox as a VM. Proxmox is an OpenVZ virtualisation host, and as a result, it has hardlimits based around the content of /proc/user_beancounters. This is the thing that's basically stopping Apache from spawning threads to handle the requests.

There's a page here about how to remove OpenVZ's limits:

clear; cat /proc/user_beancounters
vzctl set 101 –tcpsndbuf 999999999:999999999 –save
vzctl set 101 –tcprcvbuf 999999999:999999999 –save
vzctl set 101 –numtcpsock 999999999:999999999 –save
vzctl set 101 –numflock 999999999:999999999 –save
vzctl set 101 –othersockbuf 999999999:999999999 –save
vzctl set 101 –numothersock 999999999:999999999 –save
vzctl set 101 –numfile 999999999:999999999 –save
vzctl restart 101

Where 101 is the #id of your VZ container.

I also had to bump the allocated memory up from 512M to 2GB, and push the swap (and why not) up to 1GB.

After a quick restart of the Puppet container, and restarting apache one last time, I sucessfully ran 56 puppetd. One on each of the nodes, all at once, without a single error.

Sounds like success to me.

The next problem we've got that's currently holding back speedy software deployments, and apt-updates, is our apt-cacher-ng server. That too is a Proxmox VM, and I initially thought that the same problems might be true, with having a connection limit on OpenVZ, which would be preventing stuff from getting a connection.

If I run apt-get update on 50+ nodes simultaneously, the probability that some of them will error, something about connection to http://apt failing, is pretty close to P(1).

W: Failed to fetch http://ppa.launchpad.net/ubuntu-x-swat/x-updates/ubuntu/dists/lucid/main/binary-amd64/Packages.gz  Unable to connect to apt:3142:
W: Failed to fetch http://ppa.launchpad.net/webupd8team/java/ubuntu/dists/lucid/main/binary-amd64/Packages.gz  Unable to connect to apt:3142:
E: Some index files failed to download, they have been ignored, or old ones used instead.

The 3142 bit is because we've got apt-cacher-ng running on port 3142, and a line in /etc/apt/apt.conf.d/ containing

Acquire::http { Proxy "http://apt:3142"; };

This is the most fool-proof way to make sure that *everything* gets cached, PPAs, and the whole kitchen sink. This is what we want to do, because having a full debmirror is a) wasteful of disk space, something we're always at a premium with, working in VFX, and also, disk space is expensive; especially after the Thailand floods.

So, we're using apt-cacher-ng, and I don't personally see that changing anytime soon.

Right. So I bumped up the limits in openVZ's configuration, and there's still a problem that means that the apt requests aren't getting handled.

I suspect that because Apt's protocol is just HTTP, that it might be possible to use something like Varnish or HAProxy and a bunch of apt-cacher-ng backends.

It doesn't appear that apt-cacher-ng can run with multiple threads for handling lots more requests/second.

root@apt:/# netstat -anp|grep 3142|wc -l
2964

Yah.. That could be a problem.

That said, I've just tested hammering it with ab as follows:

tom.oconnor@charcoal-black:~$ ab -n25000 -c550 -X apt:3142  http://ppa.launchpad.net/mozillateam/firefox-stable/ubuntu/dists/lucid/main/binary-amd64/Packages.gz
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking ppa.launchpad.net [through apt:3142] (be patient)
Completed 2500 requests
...
Finished 25000 requests

Server Software:        Debian
Server Hostname:        ppa.launchpad.net
Server Port:            80
Document Path:          /mozillateam/firefox-stable/ubuntu/dists/lucid/main/binary-amd64/Packages.gz
Document Length:        420 bytes
Concurrency Level:      550
Time taken for tests:   3.127 seconds
Complete requests:      25000
Failed requests:        0
Write errors:           0
Non-2xx responses:      25018
Total transferred:      20990102 bytes
HTML transferred:       10507560 bytes
Requests per second:    7994.07 [#/sec] (mean)
Time per request:       68.801 [ms] (mean)
Time per request:       0.125 [ms] (mean, across all concurrent requests)
Transfer rate:          6554.54 [Kbytes/sec] received
Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0   16 208.9      1    3002
Processing:     1   19  97.9      9    1478
Waiting:        1   19  97.9      9    1477
Total:          4   35 230.9     10    3023
Percentage of the requests served within a certain time (ms)
  50%     10
  66%     12
  75%     13
  80%     13
  90%     16
  95%     20
  98%    128
  99%   1056
 100%   3023 (longest request)

25k requests, at 550 concurrency, and I still can't make it return errors.

So it looks like the problem isn't with serving files from the cache, it's downloading new stuff at the same time, and serving simultaneously.

So it's blocking. Well that's an epic stack o' fail.

Here's some more evidence to back up my findings. They're using Squid.

They're also using Fabric for orchestration. Intriguing.

More here on this one later on.. When I've actually figured it out.

Not Storing Files In A Database

Originally a comment here http://creativedev.in/2012/01/storing-a-file-in-database/

In the above article, Bhumi gives a method for storing files *in* the database, using MySQL and PHP.

My personal distaste for PHP aside, I don't think I could ever find a reason to store files *in* the database, rather than *on* the filesystem.

I'm also primarily talking about RDBMS type databases, not NoSQL, which tend to have a mechanism for storing files a little bit more sanely than "old-fashioned" databases.

Let's take a look at this idea in a bit more depth.

If you read the original article, there's the very basic bare bones of a web application. There's a form. There's a MySQL table definition. There's a bit of PHP for handling uploaded files.

Personally, I write pretty much exclusively in Python. This isn't going to be a technical article with codesamples however, more just a look at comparing the methodologies of storing in a database to storing in a filesystem.

Why would you choose a BLOB database type over a Filesystem?

I think there's gotta be about one use-case for storing files in a database. Hang on. I take that back. There aren't any.

Filesystems are optimised for File Access. Databases are best optimised for row/column based tabular data access. The two should not be confused.

Here's how I store uploaded files:

1) Upload the file to .../uploaded_files/...

2) Rename it to something sensible.

3) In the database, store the original filename, and the path to the uploaded file.

3a) Possibly also store some neat metadata, call magic() on it, store the size, the MD5Sum, and so on.

It greatly depends on the application, but it's conceivable that some of the file metadata might need to be retrieved in the file's lifespan within the application. It's a lot quicker to stat the file once, and store the result, than it would be to stat it many times whenever the information is requested.

This makes a few drastic differences to database-based file storage.

1) The database contains data of a predictable size. I mean, it's easier to calculate and predict the size of the Table based on the known bit-widths for each column. Once you start introducing BLOBs into your database, all bets for size are off.

2) Database Backups are smaller. - Not having arbitrary binary data in your database means that gzipping a SQL dump is likely to be more deterministic than it would if you already had gzipped (for example) binary data stored as a BLOB.
When you attempt to compress already compressed data, the output is frequently larger.

3) You can scale easier by having a shared/clustered filesystem, such as Gluster.

4) If it's a website, loading files can happen externally of your webapp, simply have a media subdomain and handle files with a lightweight webserver such as LigHTTP or Nginx.

This means that your webapplication isn't a bottleneck for loading every damn file that's requested, into memory, stripping the slashes or base64_decoding it, before streaming it back to the user.

This is actually one of the most important points. If the data is stored in a proprietary format in the database, you can't use regular filesystem utilities to access it. You need to do all that in your application.

Why reinvent so many wheels when the OS-provided Filesystem tools are all so outstanding?

How would you stat the file inside a database? Write it out to /tmp/tmpXXXXX and then stat that, before deleting the temporary file?

Sounds a bit slow, to me.

What if the file is Huge? How long could that take? What if the uploaded file is bigger than your system RAM? Surely it'd make sense to be able to handle multiple large files...

What if your application breaks? Could it silently corrupt files on their way in / out? What if Something Bad Happens, and your database is partially corrupted. Would all the files potentially be corrupted? What about recovery on a non-application-serving system. Could be tricky potentially.. Certainly more tricky than just rsyncing files around.

See what I mean?

You *can* store files in a database.

Doesn't mean you should.

Building updated packages for sun-java6 6u30

Firstly, welcome back.

It's now 2012, and there's lots more to write about.

Recently, Oracle withdrew the ability for Linux distributions to repackage Java and distribute their own packages. This has been widely regarded as a bad idea. I tend to agree.

So, let's re-roll an old sun-java6 deb file, with a new content to contain the latest 6u30 java release.

You will need:

1. A set of build packages (I've got a set for lucid, so if this goes away, I'll find some way to host them.) from http://archive.canonical.com/ubuntu/pool/partner/s/sun-java6/
The latest Java packages: http://download.oracle.com/otn-pub/java/jdk/6u30-b12/jdk-6u30-linux-i586.bin and http://download.oracle.com/otn-pub/java/jdk/6u30-b12/jdk-6u30-linux-x64.bin
dch. just install devscripts package to get this.
Some idea of how packaging on debian/ubuntu works.

Let's get started.

mkdir package-build
cd package-build
wget http://archive.canonical.com/ubuntu/pool/partner/s/sun-java6/sun-java6_6.26-2lucid1.dsc
wget http://archive.canonical.com/ubuntu/pool/partner/s/sun-java6/sun-java6_6.26-2lucid1.debian.tar.gz
wget http://archive.canonical.com/ubuntu/pool/partner/s/sun-java6/sun-java6_6.26.orig.tar.gz
wget http://download.oracle.com/otn-pub/java/jdk/6u30-b12/jdk-6u30-linux-i586.bin
wget http://download.oracle.com/otn-pub/java/jdk/6u30-b12/jdk-6u30-linux-x64.bin
tom.oconnor@charcoal-black:~/package-build$ ls -1
jdk-6u30-linux-x64.bin
jdk-6u30-linux-i586.bin
sun-java6_6.26-2lucid1.debian.tar.gz
sun-java6_6.26-2lucid1.dsc
sun-java6_6.26.orig.tar.gz
tom.oconnor@charcoal-black:~/package-build$ dpkg-source -x *.dsc
gpgv: Signature made Tue 13 Dec 2011 22:31:53 GMT using RSA key ID CC559573
gpgv: Can't check signature: public key not found
dpkg-source: warning: failed to verify signature on ./sun-java6_6.26-2lucid1.dsc
dpkg-source: info: extracting sun-java6 in sun-java6-6.26
dpkg-source: info: unpacking sun-java6_6.26.orig.tar.gz
dpkg-source: info: unpacking sun-java6_6.26-2lucid1.debian.tar.gz
tom.oconnor@charcoal-black:~/package-build$ cd sun-java6-6.26/
tom.oconnor@charcoal-black:~/package-build/sun-java6-6.26$ ls
debian  jdk-6u26-dlj-linux-amd64.bin  jdk-6u26-dlj-linux-i586.bin
tom.oconnor@charcoal-black:~/package-build/sun-java6-6.26$ rm *.bin
tom.oconnor@charcoal-black:~/package-build/sun-java6-6.26$ ../jdk-6u30-linux-i586.bin jdk-6u30-dlj-linux-i586.bin
tom.oconnor@charcoal-black:~/package-build/sun-java6-6.26$ ../jdk-6u30-linux-x64.bin jdk-6u30-dlj-linux-amd64.bin
tom.oconnor@charcoal-black:~/package-build/sun-java6-6.26$ vim debian/rules

Head down to the block "# check if the sources are the "same"

Then find and comment the block following it, out, so you get this.

#       : # check if the sources are the "same"
#       set -e; set -- $(all_archs); a1=$$1; shift; \
#       unzip -q -d tmp-$$a1/src $$a1-jdk/src.zip; \
#       for a2; do \
#         unzip -q -d tmp-$$a2/src $$a2-jdk/src.zip; \
#         echo "Comparing sources: tmp-$$a1/src tmp-$$a2/src ..."; \
#         echo "    diff -ur $(diff_ignore)"; \
#         diff -ur $(diff_ignore) tmp-$$a1/src tmp-$$a2/src; \
#       done

Save that file, and then run:

dch -v 6.30

This will create a changelog entry for version 6.30, and open $EDITOR to edit the changelog entry.

Enter a stub entry..

I put something like

* Updating internal contents to 6u30

.. There's some output, but you can ignore this.

dch warning: New package version is Debian native whilst previous version was not
dch warning: your current directory has been renamed to:
../sun-java6-6.30
dch warning: no orig tarball found for the new version.

tom.oconnor@charcoal-black:~/package-build/sun-java6-6.26$ cd ..
tom.oconnor@charcoal-black:~/package-build$ cd sun-java6-6.30/
tom.oconnor@charcoal-black:~/package-build/sun-java6-6.30$ dpkg-buildpackage -b -uc

... LOTS OF STUFF ...

tom.oconnor@charcoal-black:~/package-build/sun-java6-6.30$ cd ..
tom.oconnor@charcoal-black:~/package-build$ ls
ia32-sun-java6-bin_6.30_amd64.deb     sun-java6_6.26-2lucid1.dsc  sun-java6-6.30                sun-java6-bin_6.30_amd64.deb   sun-java6-fonts_6.30_all.deb   sun-java6-jdk_6.30_amd64.deb  sun-java6-plugin_6.30_amd64.deb
sun-java6_6.26-2lucid1.debian.tar.gz  sun-java6_6.26.orig.tar.gz  sun-java6_6.30_amd64.changes  sun-java6-demo_6.30_amd64.deb  sun-java6-javadb_6.30_all.deb  sun-java6-jre_6.30_all.deb    sun-java6-source_6.30_all.deb

Woo. Debs.

What you want to do with them now is up to you. Next blogpost, I'm going to go over creating a package repository with reprepro.

Thanks to @mibus for his similar article, which this is based partially on.

2011: Personal Retrospective

This post has been abridged / redacted to preserve the identity of individuals mentioned. If you know me well enough, ask and I might show you the unredacted version. Or you might be able to figure it out yourself.

My, what a year it's been. Mostly good, but pricked throughout with sadness on occasion.

Such a lot has happened, I've changed jobs twice. I left a good Infrastructure Engineer job in July for a promise of a better job at another company. I did 3 months of hard graft on a new infrastructure project there, only to find they'd chosen to make me redundant, and terminate my contract at 3 months. Bit of a bugger that, and the way it was phrased to me made it feel like it was a personal attack. The line I was given by the CTO was "You're not a good fit for the team", but that was actually bollocks, because everyone in the team confirmed my feelings, that I was actually a good fit for the team. A little stubborn, but that's a good quality in an engineer, I feel.

That was on the 14th of October..

Turns out, that my "redundancy" / "contract termination" / "whatever" came at a time when they also got rid of 2 product managers and a senior developer. Such is life.

So, I spent about 3 weeks job hunting, various interviews ranging from "yes, that sounds really cool" to "eugh. Massive company, sounds too boring for words". I picked up a week-odd's work of contract work related to my former infrastructure job, and that kept me entertained for a while long enough.

Soon enough, a good offer at a decent company turned up. I heard from my former colleague at the 3 month place, that a cool VFX company were looking for a DevOp engineer of my type of background.

I applied, and was interviewed twice. Turns out that I'd previously been recommended to them by a guy who'd interviewed me in the past, so it felt like it really was all meant to be.

I started there in the first week of November.. It feels like it was much longer ago, for some reason. Probably because I've been so impossibly busy. Still, that beats the original infrastructure job where I was impossibly bored.

There's lots of new infrastructure stuff to do there, and we're doing lots of stuff with cool tech. I'm not entirely sure what I can talk about, so I'll just say that it's all insanely cool.

This is good, because I was starting to feel quite bored of working for the same kinds of Web2.0 companies. Scalability, yeah, very buzzwordy for 2011, and important nonetheless, but once you've solved the problem, it feels very self-similar when you come to do it again for a different client or a different employer.

There will always be the same problems, namely Time, Money and Knowledge. Scaling a system takes time, the hardware and servers cost money, and the developers need to know how to make their code performant.

I don't think I'd go so far to say that I'm bored of Web2.0, but I'm certainly bored of the cheapskate mindset. I've said numerous times, both online and IRL, that if you want to play with the big boys, it's gonna cost you. There's some things you just can't do for free. Building decent software is definitely one of them.

So. I'm working for a VFX / post-production company. They did all sorts of insanely crazy VFX for some recent films. It's truly cool. There's plenty of *new* challenges, and very little of the same old web stuff. This is good. Sometimes all you need to keep you interested is a sea change. It certainly worked for me.

Until the 24th of October, my partner and I had been in a relationship (and also sharing a flat in Chiswick). It had become pretty apparent that we both wanted our own space, and the relationship in its current format had become pretty much impossible. I think it was all brought home to me when I wanted to start dating again, and to some extent, it felt like he already had. I got home from a trip a few hours earlier than planned, and there was some random guy on the sofa. I jumped in the shower, and by the time I got out, both of them had vanished, without saying a word. Definitely an uncomfortable moment.

I spent a very long time trying to rationalise the feelings that I'd once had, and whatever was left of them. I came to the eventual conclusion that what i really wanted was a companion. Someone I could chat to now and again, but not have any requirement to y'know, bugger them.

When it came down to it, this was pretty much what the relationship had become after a couple of years. We'd done Open Relationship Rules from the outset, initially because he was living in Spain, which made everything a bit tricky, but even when we were living together, in the same flat; sharing the same bed, we'd kept those rules because they worked better for us.

I think that should probably have been a bit of a better warning sign. I suspect in future, I'll be more aware.

So we broke up, and went our seperate ways. I moved out, and found a new flat.. A somewhat bigger flat, and a new flatmate. An incredibly nice guy, and we do get on well. And I'm glad that I'm not living alone. I think I'd have ended up feeling pretty damn lonely. I mean, there's nights when Alex isn't home, and it's quiet, and I think that if I were living like that all the time, I'd end up pretty bored and probably self-destructive. So yeah, it's good to have a new place, and a new flatmate.

I gather my ex is sticking in Chiswick in the old flat. He took over my share of the rent, but I suspect he'll fulfil his long-held dream of living around the corner from his office.

I tried dating a guy back in January, whilst I was still technically in a relationship, albeit an open one, with my ex. That turned out a bit weird in the end, as I was falling head-over-heels in love with him, and he wasn't feeling the reciprocal feelings. At least, not to the same extent. That was weird. I don't think I've been in that position for a very long time, and such, I'd forgotten just how hard it was to deal with. One of the things that makes us human, I suspect, is the ability to reflect on past experiences and imagine how alternative paths might have turned out.
In some regards, this is a good ability. Truth be told, it's a bitch, and hindsight is a killer when it comes to that kind of thing.

Still, he seems happier now with his new boyfriend. I suppose at the end of the day, that's all you can really ask for, for the best for your friends.

After my long-term ex and I broke up, I tried dating again, again. Possibly too soon, as I still found that I was craving solace in being alone. Not the best thing if you're trying to make a new relationship (albeit, only dating) work. I don't know what that experience has taught me. Possibly that I'm a busy bugger, and I'm happiest when I'm busy, and that potentially higher-maintenance individuals are unlikely to be a good fit for me. Perhaps it's nothing of the sort, and I was just trying too hard, too soon.

So far, it sounds like it's been a pretty depressing and gloomy year, perhaps.

But I've done some insanely cool stuff. I went up the BT Tower, and took photos. Photography has played a pretty big part of my life in 2011, too. I got back into film photography with a Nikon F301 and a Nikon F5. Perhaps next year I'll buy a video camera, and branch out in that regard too.

I went to a couple of awesome Winter parties, in a stunning tuxedo (that I now own). I've been blogging and writing technical articles a lot more in 2011 than ever before. I've got a Most Valued Blogger award and republishing arrangement from an online journal, and I'm looking to write some guest articles for some technical magazines in 2012. I'd also like to think about writing a book in the new year.

I'm going to summarise 2011 as a number of small setbacks, but a persistent push forward against adversity, between job problems and the woes of the end of a long term relationship. It's not all doom and gloom, and I know it could have been a lot worse of a year. But from my reasonably comfortable life, some changes that might seem small to others are quite large and affect me in different ways.

I find myself reminded once again of the phrase from Ulysses.

"To strive, to seek, to find, and not to yield"

I first saw this phrase gracing the doorway of the Mechanical Engineering department at Birmingham University, and for me, it's always struck a chord.

So that's about it. The personal reflection and retrospective on 2011. Here's to a better year in 2012.

Twitter and their REST blunder

Hah. So it's New Year's Eve. Twitter is down, and has been for about 3-4 hours. That's because the NYE celebrations have already started. Somewhere on the other side of the world.

I include Twitter on my personal website. The one you're reading. There's a template tag that displays my five latest tweets.

About 2-3 hours ago, I got some error reports about XML parse errors on that template tag. I use api.twitter.com and pull in the XML feed for parsing. No problems there, it's always worked.

It relies on testing that the status from Twitter was sensibly formed. In order for that to happen, the request has to return HTTP 200 OK. That's cool. That's easy, and it's one of the principle tenets of RESTful APIs.

Until tonight, it seems.

curl -I 'http://api.twitter.com/1/statuses/user_timeline.xml?screen_name=metacheetr&count=5'
HTTP/1.1 200 OK
Content-Type: text/html; charset=UTF-8
Content-Length: 4968
Set-Cookie: k=217.146.95.120.e7f9484aa1ec178d; path=/; expires=Sat, 07-Jan-2012 16:25:06 UTC; domain=.api.twitter.com; httponly
Date: Sat, 31 Dec 2011 16:25:06 UTC
Server: tfe
Twitter is currently down for maintenance.
We expect to be back soon. For more information, check out Twitter Status »
Thanks for your patience!

So what we find is that Twitter are returning HTTP 200 OK, for a status which is blatently not what I asked for.

This is BAD. This is bad for two reasons, mostly.

Web developers and engineers rely on HTTP Status codes to actively represent the output of the service, and also to provide a machine-readable representation of the webpage returned.
One of the principle tenets of RESTful APIs is that the status returned should accurately represent the state of the service.

There's nothing to stop you returning a HTML response with a HTTP 5xx status code. It's better to do that, in fact. The user sees something pretty, and the machine readable representation says "Er, this service is fucked.

But if you return 200, and encapsulate a non-standard error message, inside an arbitrary block of HTML, it makes machine parsing incredibly difficult.

So I'm turning off the latest_tweets template tag for a bit. At least until Twitter read this, apologise, and start returning decent HTTP status codes.

Come on guys, you're one of the flagship startups of Web 2.0, if we can't look up to you to set a good example, what hope does everyone else have?

A manifest for Agile DevOps

I’ve decided. We need to start doing points poker here at Baseblack if we’re going to carry on this Agile DevOps thing.

I’ve got to admit, the first time I came across the Agile methodology was quite late in my career. In the past, prioritisation of “operations” projects was reasonably first come first serve, or by order of priority (frequently, business need, and seldom operational requirement).

For software development teams, Agile is a pretty good, native fit. The concepts embodied by stories and sprints fit a development team very cleanly. When it comes to systems administration and engineering, or what I’ve come to refer to as DevOps, Agile can be a bit more awkward initially.

Operations teams across the globe will tell you that their tasks are intrinsically more “sprawly”, and that interconnections between tasks are frequently more complex.

The truth of the matter is, that frequently there is no simple and sensible way to break up a task into entirely unconnected subtasks. Something which can bugger up Agile, if you’re too hard and fast with the requirements and rules by which you play the game.

Pretty early on in this new job, I started looking at the previous DevOp Engineer’s puppet manifests. They were *mostly* ok, but with some absolute crazy meatballs thrown in for good measure.

It’s actually a common fault of Sysadmins to want to throw out the previous team’s work and start afresh, but in this case it actually was easier to start fresh than repair the foibles and cockups of the old code.

Given that I’d already spent 3-4 days reading and trying to interpret the state of the system, and it was blatently apparent that there were too many bits of “wouldn’t it be cool if we hacked this in to make it do X”, and not enough actual hard and fast config to make things work.

I’ll put that one down to my predecessor not being very puppet-savvy.

One of the big reasons this sprint overran was that the discovery process (first 3-4 days) was mostly involved with exploring the state of the systems, and what we wanted to accomplish. In the old manifests, there were huge chunks of code that installed numerous applications, which would be easier to manage and integrate if modularised.

A good proportion of time in the implementation phase went into creating lots of individual modules for various applications and packages.

As I was saying earlier about interconnected tasks, this wasn’t just a Fix Puppet sprint.

The background to fixing puppet was to enable the faster building of new machines from unboxing to users logging in.

There were some massively weird problems with the internal DNS, using Bind9, and the old DHCP server was prone to some peculiar lease issues, and it was running on a physical VM host, when it probably ought to have been a VM guest. Fixing DNS would best be done whilst fixing DHCP. Fixing DNS meant installing PowerDNS, which in turn means installing Postgresql. Setting up DNS Slaves means installing PowerDNS on multiple servers and configuring Postgres replication.

There’s no way that I’m building out multiple copies of anything without Puppet, so there’s the first bit of recursive loop.

The way to untangle this is to realise that puppet doesn’t need a puppetmaster to run manifests. All you need to do is write the puppet configs and then use the puppet agent itself to run the manifests from files. You can then use that to bootstrap a puppetmaster, or a DNS server, or just get a sense of how it will all fit together when you do the final server buildout.

I’m going to leave this here. I think the general conclusions to draw are the following.

1) Agile is great.. It doesn’t fit all teams, and it’s worth trying. If it doesn’t fit, no worries. If it does, cool.

2) Planning is the biggest stage of any project, or at least, should be.

3) Infrastructure projects shouldn’t be forced into the traditional Agile Sprint, because they tend to become a lot more sprawly on investigation of the actual problem than they look at first glance.

I’m about to post the articles on Postgres replication, and the technical portion of this article.

Low-level Infrastructure: Puppet, DNS and DHCP

Right. Let’s have a look at the massive technical implications of the Fix Puppet idea.

As I mentioned in my earlier blogpost, in order to fix puppet in a sensible way, we’ll have to review all, and overhaul some of the underlying infrastructure that allows it all to run.

The interlinks and dependencies between all the parts are a little tricky to visualise. So, here’s a picture.

Anything in red needs attention, and the stuff in green *just works*. Things in blue are install stages, and these are what we’re working on making perfect.

Right, so we’ve basically got a directed graph, representing the steps and stages that have to happen to a new machine before users can log in.

The steps taken to build a machine, roughly look like this:

Unbox.
Plug in.
Configure Netboot.
Hand MAC Address to DHCP server and assign a hostname.
Client PXEBoots.
Client downloads a preseed file.
Client installs itself.
Client Reboots.
Puppet runs on First Boot.
Puppet completes.
Client Reboots again.
Users login

That’s about it, really. The first 4 steps are a hell of a lot easier with the support and co-operation of the supplier. It’s nice to have systems preconfigured to PXE boot as the BIOS default, and even cooler if they can send the MAC addresses as labels on each physical machine.

If we’re going to build out a new infrastructure, we’re going to need to review and reinstall the servers that provide this infrastructure, before we can build any workstations.

I’m a massive massive fan of puppet, and believe that it should be used for the configuration of all servers and workstations. As such, I didn’t want to rebuild anything without using puppet, so the first step, had to be getting puppet working again.

So, without further ado, let’s take a look at the Puppet portion of this, well, one of them.

My predecessor saw fit that all nodes should be defined with puppet-dashboard, which is itself, a fine piece of software, but I think more for reporting than specification.

Initially, at least, I rebuilt the puppet manifest from a known-good configuration. Namely the base configs I wrote for a blogpost about a year ago; base configs that I’m going to update soon.

I’m a bit of an old fashioned puppet user. I like my nodes defined in nodes.pp, not some External Node Classifier service.
Reason being, I like to be able to look in one place and find exactly what I want. It’s not a massive ballache to clone down the puppet git repo, make a change and push it back up.

In fact, it’s better than having a web interface for your node classifications, because git provides you with an intrinsic log of what was changed, and it’s easy to revert to an old version, because everything’s stored in source control.

You can also test what you’re about to do, because again, it’s just a source control repo. I’m a fan of having Jenkins run a few sanity checks on your puppet repo, but that’s a digression for another blogpost.

I’m not going to go into great depth about how to install DHCP and DNS, and how to make it work with puppet, at least, not here.

What I will say, though is that Puppet Module Tool is the most fantastically easy way to generate boilerplate modules for puppet.

All you need to do is run

puppet-module generate tomoconnor-dhcp

and you get a full puppet module folder called tomoconnor-dhcp which contains all the structure according to the best practice guidelines.

Excellent.

As part of the review process, it became quite apparent that Bind9 has no sensible admin/management interface, or at least, there wasn’t one installed, and frankly, anything that has such horrific config files should be shot.

Having had good experience and results using PowerDNS in the past, we decided that this would be a valid upgrade from BIND.
PowerDNS relies on a SQL backend for storing the record data in.

You can use either MySQL or PostgreSQL, or possibly some others. Since MySQL can be a bitch, and is, to all serious purposes, a toy database, Postgres seems like a better choice. 9.1 is stable, and there are deb package available for it. 9.1 also does hot-standby replication, which is a miracle, because Postgres replication used to be a massive pain in the testicles.

There were, initially some mysterious problems with the TFTPd server being generally crappy, mostly regarding timeouts, which was because the storage of the TFTP data was on a painfully slow disk. Moving it from there to the NFS mount dramatically increased performance and stopped TFTP going crazy.

In the TFTP'd config, there's a block for configuring the boot options of the preseed install. This is how PXE hands over the details of the preseed server, and the classes of preseed file to run (basically, which modules)

label lucid_ws
        menu label ^2) Auto Install Ubuntu Lucid WorkStation
        text help
        Start hands off install of a workstation.
        endtext
        menu default
        kernel ubuntu-1004-installer/amd64/linux
        append tasks=standard pkgsel/language-pack-patterns= pkgsel/install-language-support=false vga=normal initrd=ubuntu-1004-installer/amd64/initrd.gz -- quiet auto debian-installer/country=GB debian-installer/language=en debian-installer/keymap=us debian-installer/locale=en_GB.UTF8 netcfg/choose_interface=eth0 netcfg/get_hostname=ubuntu netcfg/get_domain=installdomain.wibblesplat.com url=http://autoserver/d-i/lucid/preseed.cfg classes=wibblesplat;workstation DEBCONF_DEBUG=1

Initially, the Preseed files contained all sorts of crazy hacky shit in the d-i late-command setting.

late-command is cool. It’s basically the last thing to run before the first reboot when you build a new debian/ubuntu system. You can tell it to do all sorts of stuff in there. You probably shouldn’t, though. Especially when what you’re doing in there is better done elsewhere.

The previous Preseed file contained a whole bunch of “inject these source files into /etc/apt/sources.list”, which is utter bullshit, because you can do exactly the same thing with d-i local repositories, which does the same thing, only far far cleaner.

That’s not to say that my refactored preseed files don’t use late-command at all.

I’ve chosen to insert some lines into /etc/rc.local on the freshly built system that ensures a puppet run at first boot.

On the preseed server, there’s a file called “firstboot.sh” which gets dropped into /usr/local/bin by way of a wget command in late-command.

The next thing that happens in late-command is a line to remove “exit 0” from /etc/rc.local and replace it with a thing that calls “/usr/local/bin/firstboot.sh”

When firstboot runs, it runs puppet, checks for sanity, and then removes itself from /etc/rc.local.

The code to actually do that looks like this:

d-i preseed/late_command string  \
wget -q -O /target/root/firstboot.sh http://autoserver/d-i/bin/firstboot.sh && \
chmod +x /target/root/firstboot.sh && \
sed -i 's_exit 0_sh /root/firstboot.sh_' /target/etc/rc.local

This relies on having something on http://autoserver that is basically just apache hosting some files for the preseeder to retrieve during installation.

Cool huh?

That ensures that the first thing that happens once the new machine has been built and rebooted, is a puppet run.

Some stuff we do here relies on our hand-rolled deb packages, which are stored in our own, internal APT repo. We’ve also got an APT cache, created and maintained by apt-cacher-ng, which at least means that when you’re rebuilding systems frequently, that all the packages you would otherwise download from archive.ubuntu.com come straight over the LAN.

The major problem initially with this was the speed, or lack of. It certainly wasn’t performing anywhere near speeds you’d expect from a 1GE LAN, and the reason was again, slow disks. Moving the apt-cache files to the NFS highspeed storage again helped performance. If we struggle in future, I’m going to look at a SSD cache for this, but I think that the performance of the SAS/SATA disks on massively parallel storage provided by our NFS servers will be adequate for the forseeable future.

Next up, the Puppetmaster. Again, I was pretty keen on building this from scratch, but using puppet itself to configure it’s own master. Sounds pretty counter-intuitive, right? But the puppet client can bootstrap the master quite easily by using files as it’s source.

The first step is to clone down the latest puppet manifests from git, so you either need to git export elsewhere, or install git-core. Your choice.

Once you’ve got those, all you need to do is install puppet-client, and run:

 puppet apply /path/to/your/manifests/site.pp

If you’ve written the manifests right, and you’ve got your master defined as a node, you should find that puppet will install puppetmaster, and so on, and then you get a ready and working puppetmaster that just configured itself.

I used puppet-module tool to generate modules for the following services/items: “applications” - which actually contains a bunch of custom/proprietary application install rules, a declassified example is there’s a googlechrome.pp file that installs chrome from a PPA.

Other modules: dhcp, kernel, ldap, network, nfs, nscd, ntp, nvidia, postgres, powerdns and ssmtp.

As is the trend with puppet, and modern DevOps, a vast majority of the code in the entire manifest repository has been gleaned and researched from other puppet modules on github. Acknowledgement is in place where it’s due, and the working copies we’re using are frequently forked on github from the original.

It’s great, this, actually. If you search on PuppetForge http://forge.puppetlabs.com/ the array of modules available is staggering. It makes bootstrapping a new manifest set remarkably quick and easy.

The NFS module contains a bunch of requirements for mounting NFS shares, and the definitions for an NFS share to be mounted. All pretty simple stuff, but modularised for ease of use.

I’m particularly proud of the postgres module which has a master class, and a slave class, which installs and configures the required files and packages to enable streaming hot-standby replication on Postgres9.1

I will release the declassified fork of this soon.

I’m going to wrap this post up here. It’s a massively long one, and there’s still lots more left to write.

Postgres Replication on 9.1

Our new PowerDNS cluster (of 2 nodes, so far).. Is backed by Postgresql.

In the past, I’ve found that Postgres performs far better for a PowerDNS backend, than MySQL, and certainly better than the BIND, LDAP or SQLite backends.

Until version 9.x, Postgres replication was a pretty sorry state of affairs. There were a few options for replication.

Slony was commonly used, if not very good.. You’d tend to get a horrific SPoF around the single master. In total, there were 9 or 10 different third party solutions for Postgres replication and clustering. They all had their pros and cons, and some were great, and some were downright awful.

In 2008, the Postgres core team started to bring replication and clustering into the fold with the rest of the features of Postgres, and now, in 9.x, the option of hot and warm standby are both available, and stable.

There’s a comprehensive writeup of the history of Postgres replication here: http://wiki.postgresql.org/wiki/Replication,_Clustering,_and_Connection_Pooling

One of the things I adore about the hot-standby replication mode is that the basic configuration (/etc/postgresql/9.1/main/postgresql.conf) is identical between master and standby.

This makes puppeting insanely easier than it would’ve been if the master and standby had to have largely different configuration files.

I changed about 5 config settings in the main config file.

listen_addresses = '*'
wal_level = hot_standby 
max_wal_senders = 5
wal_keep_segments = 32
log_destination = 'syslog'

^^ I only changed the log_destination to make centralised logging easier in future.

There’s a limited change to pg_hba.conf to allow host-based authentication of the standby to the master.

Add a line like:

host    all     all     $SLAVE_IP/32      trust

I actually did this as a Puppet template file.

On the standby server, you drop a file called “recovery.conf”, into /var/lib/postgresql/9.1/main

Yes, the *DATA* directory. Yes, that makes no sense. Yes, It should by rights be /etc/postgres.... but it isn’t.

In that file, you have 2 lines.

standby_mode = 'on'
primary_conninfo = 'host=<%= psql_master -%> user=<%=replication_user -%> password=<%=replication_password -%> '

That’s copypasta’d from a puppet template.

The interpolated lines are more like:

standby_mode = 'on'
primary_conninfo = 'host=192.168.100.100 user=replicant password=wibblewibblewibble '

Then all you’ve gotta do is instantiate the standby with pg_basebackup, and then restart the master, and the standby, and they should come up, connect to each other, and start streaming replication updates.

It’s pretty magical.

pgbasebackup lives in /usr/lib/postgresql/9.1/bin/pg_basebackup

and should be run (as postgres user)

/usr/lib/postgresql/9.1/bin/pg_basebackup -D /var/lib/postgresql/9.1/main/ -x -h $Master_Hostname -U postgres

You should start the standby first, so that the master doesn’t have a chance to get out of sync.

The standby will start accepting read-only connections as soon as it’s up to date with the master.

Baking Certificates into OSX Lion for 802.1X

This is tricky. No question about that.

In order to configure OSX Lion to use 802.1X authentication over WiFi, to login, and also connect (without prompting for credentials), we need to generate a .mobileconfig parameter file (plist).

These files are a bugger to craft by hand, so what we'll do is use the Enterprise iPhone tool, to build one which can be used for a deployment to an iPhone, or OSX Lion laptop/desktop.

Apple have a bunch of stuff about Enterprise Deployment, here.

The file you want, however, is the iPhone Configuration Utility 3.4 for Mac OS X.

You'll need to run this on an Apple device, Macbook Air, or MBP, or iMac, etc.. As far as I know, you can't do this from an iPad.

1. Download and install from the DMG.

Run the Configuration Utility .

Click "Configuration Profiles" in the selector on the LHS.

Select "New", and you should get a blank new profile.

Enter some details.

"Identifier" is a reversed format of your profile, in a kinda java package-style notation, ie, wifi.wibblesplat.com becomes com.wibblesplat.wifi; Simples!

As part of the Profile, you can configure all sorts of settings that will be installed on the target device. Scroll down through General, Passcode, down to "Credentials".

When you hit "Configure", you can choose a certificate file.

At this point, we're going to pause here, and quickly recap how to create self-signed SSL certificates.

Open Terminal, and create a new directory that we can shove all the SSL related gubbins into.

cloud-white:~ tom.oconnor$ mkdir wibblesplat

cloud-white:~ tom.oconnor$ cd wibblesplat/

Next, we need to generate a private key.

cloud-white:wibblesplat tom.oconnor$ openssl genrsa -des3 -out wibblesplat.key 1024
Generating RSA private key, 1024 bit long modulus
...++++++
...........................++++++
e is 65537 (0x10001)
Enter pass phrase for wibblesplat.key:
Verifying - Enter pass phrase for wibblesplat.key:

You should enter a passphrase here, but we can strip it off later.

Now we've got the key, we'll use that to generate a Certificate Signing Request (CSR)

cloud-white:wibblesplat tom.oconnor$ openssl req -new -key wibblesplat.key -out wibblesplat.csr
Enter pass phrase for wibblesplat.key:
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:GB
State or Province Name (full name) [Some-State]:England
Locality Name (eg, city) []:London
Organization Name (eg, company) [Internet Widgits Pty Ltd]:Wibblesplat Ltd
Organizational Unit Name (eg, section) []:R&D Department
Common Name (eg, YOUR name) []:*.wibblesplat.com
Email Address []:
Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:

Of course, you need to fill in the CSR with your *own* information, but that goes without saying, doesn't it? Do you sign your cheques with "Signature" in a cursive hand?

Next, we'll strip the passphrase from the key, because it makes it a bugger if you use this certificate for Apache, or whatever, and it will always block and wait for the key if you've not stripped it.

cloud-white:wibblesplat tom.oconnor$ openssl rsa -in wibblesplat.key -out wibblesplat.unprotected.key
Enter pass phrase for wibblesplat.key:
writing RSA key

Now we've got the key and the CSR, we can generate a SSL Certificate. You can specify anything from 1 day to 7304 days (20 years) for the validity. For CA Roots, it's probably best not to use 1 day ;).

cloud-white:wibblesplat tom.oconnor$ openssl x509 -req -days 900 -in wibblesplat.csr -out wibblesplat.crt -signkey wibblesplat.unprotected.key 
Signature ok
subject=/C=GB/ST=England/L=London/O=Wibblesplat Ltd/OU=R&D Department/CN=*.wibblesplat.com
Getting Private key

Now we've got the Certificate (.crt), the Key (.key), the unpassphrased key (.unprotected.key), and the Certificate Signing Request (.csr)

cloud-white:wibblesplat tom.oconnor$ ls
wibblesplat.crt                 wibblesplat.key
wibblesplat.csr                 wibblesplat.unprotected.key

Let's jump back to the main theme of this evening's symposium.

Now we've got a generated certificate, we can continue with profile generation.

We were here.

Navigate to wherever you left those SSL certificate files, and select the .crt

When you click "Open", the right hand side of the credentials pane will display the signed certificate.

Excellent.

Now we can configure the Wifi settings to use that certificate.

Scroll back up through the Profile settings, up to "Wi-Fi".

Hit "Configure", and the right hand pane changes to another profile builder screen.

Enter the SSID of your Wi-Fi, and select Security Type "WPA / WPA2 Enterprise"

Scroll down the Right hand side down to "Enterprise Settings" and click some boxes.

Click the "Trust" tab, and select the Certificate that we added to the Stored Credentials.

Under "Trusted Server Certificate Names", hit the [+] button, and add whatever matches the CN of your certificate. In this case, it's "*.wibblesplat.com".

Nearly Done!

Along the top button bar, hit "Export", and you get the Export Dialog:

For "Security" ensure "None" is selected, then hit "Export..."

Save the file with the .mobileconfig extension

Right. That's the OSX bit done.

The next thing I did, was to jump back over to my Ubuntu desktop, and fire up Meld.

In case you've never used it, Meld is a great, interactive, diff tool. It supports 2 and 3 way diffs, and you can shuffle bits of code between the two panes of it easily.

We're going to open someone else's mobileconfig file, and sanity check our own.

Ronald Ip over at iphoting.com has published his configuration profile for accessing the wireless at Singapore Management University.

It's an interesting read, and the link to the .mobileconfig file is at the bottom of his blogpost, also here.

Open up Meld (you might need to apt-get install meld).

Create a New Diff, and select the file you downloaded from iphoting as the Original, and your generated mobileconfig file as the "Mine".

Now all you need to do is Sanity Check them. Make sure that, side by side, the files look *similar*. Of course, they can't be identical, but you want some idea that the keys and values are in the same order (this is *important*), and that yours has got most of the same information as the master.

**Important: **

If you wish to use 802.1X to authenticate to Radius for logins, then you'll need to configure a "Login Window" profile.

This means you need to add a "macloginwindow" user account to your LDAP (or whatever your Radius server looks up against), and then configure the username and password for that in this file.

To do that, edit the .mobileconfig file in a decent text editor, and add the lines

<key>UserName</key>
<string>macloginwindow</string>
<key>UserPassword</key>
<string>000INSERTSecurePasswordInHERE000</string>

Those lines need to go *just* after the block:

<key>TTLSInnerAuthentication</key>
<string>MSCHAPv2</string>

Then after

<key>SSID_STR</key>
<string>wibblesplat-wifi</string>

Insert

<key>SetupModes</key>
<array>
	<string>System</string>
	<string>Loginwindow</string>
</array>

Which will define that you're doing System settings, and Login Window settings.

If you don't want to do Login Window stuff (but frankly, why wouldn't you), then you can safely remove the LoginWindow key.

Somewhere near the bottom of the file, there's a Key marked "PayloadType", with Value "Configuration".

One line above that, insert the two lines:

<key>PayloadScope</key>
<string>System</string>

That should be it for manual changes. As soon as I figure out how to do those from the OSX iPhone configurator, I'll update this. I suspect that because LoginWindow isn't actually an iPhone option, but more pertains to OSX on non-mobile devices, that it's not actually covered as a Thing in the Configurator.

Once you're pretty happy, you can get on with the next step. On Lion, you can load the files with the omnipotent "open" command from Terminal. We used HTTP to distribute the files, but you could equally just scp them across to your Lion clients.

You need to do the profile load as an Admin User, so in Terminal, do something like:

su - adminuser open wibblesplat.mobileconfig

Some box might appear asking if you want to apply the settings, say yes.

You've come this far, it'd be foolish to say no.

Then all you've gotta do is reboot.

Technically, you might not need to, but at least rebooting should clear any saved session state, and you'll get a more representative idea of what ought to happen.

Done. Congratulations. You've just baked in configuration details for WPA2 Enterprise and 802.1x

Renewing a SSL Certificate on OSX Server

This article relies on having a soon-to-expire SSL certificate on an Apple OSX Server. Ours are running Snow Leopard, and I’m yet to try the whole thing on Lion.

I’ve got to admit, I went through a bit of a rigmarole to do this.

To generate a new certificate, you need a key, and a CSR.

To get the key, you need to export a PKCS12 file from KeychainAccess as ROOT. Yes, Root. Yes, OSX = Toy operating system. No, another admin user won’t cut it. Yes it’s a pain in the arse.

For an imaginary wibblesplat.com:

Open Terminal.
Run

sudo /Applications/Utilities/Keychain\ Access/Contents/MacOS/Keychain\ Access

Unlock the System keychain.
Locate certificate (Category -> Certificates) , Control + click => Export...
Export to /tmp/wibblesplat.p12
Feed it a password for the p12 archive. Do NOT forget this. You’ll need it.
Go back to the Server Admin panel, grab the expiring certificate, and hit the Gearwheel, then select Generate Certificate Signing Request.
Save that to a file. (/tmp/wibblesplat.csr)

Next, we need to split the PKCS12 archive, to get the old private key out.

tom.oconnor@cloud-white:~$ cd /tmp
tom.oconnor@cloud-white:/tmp$ openssl pkcs12 -in wibblesplat.p12 -nocerts -out wibblesplat.key

> Enter Import Password: *****
> MAC verified OK
> Enter PEM pass phrase: *****
> Verifying - Enter PEM pass phrase: *****

Strip the passphrase from the key (otherwise you have to enter it lots when you restart services.)

tom.oconnor@cloud-white:/tmp$ openssl rsa -in wibblesplat.key -out wibblesplat.unprotected.key

> Enter pass phrase for wibblesplat.key: *****
> writing RSA key

Export the old Certificate from the p12. You might as well.

tom.oconnor@cloud-white:/tmp$ openssl pkcs12 -in wibblesplat.p12 -clcerts -nokeys -out wibblesplat.old.crt

Generate the Certificate from the CSR from earlier, and the freshly exported key.

tom.oconnor@cloud-white:/tmp$ openssl x509 -req -days 7300 -in wibblesplat.csr -signkey wibblesplat.unprotected.key -out wibblesplat.new.crt

> Signature ok
> subject=/CN=wibblesplat.com/C=GB
> Getting Private key

Now, you go back to Server Admin. Re-select the expired certificate, and hit Gearwheel -> Replace with new signed certificate.

Find the file "wibblesplat.new.crt" in Finder, and drag it into the Server Admin "Replace Screen"

You don’t need to replace the Key, because of the above steps, we used the old key.

Head back over to Keychain Access,

Find the newly updated certificate, and you should find that the new expiry time is somewhere about 20 years from now (7300 days, which is the longest you can set a certificate Valid To date)

Then double click the new certificate, and under the Trust dropdown/treeview thingy, set

"When using this certificate, to Always Trust"

Congrats. You've just replaced an expired certificate with one that won't expire for 20 years (well, near enough.)

Analysis and Comment: Why Point of Sale is a POS

A little background, perhaps:
Last night I attended a Winter Party at a bar called The Sterling, in the ground floor of the Gherkin.
I have absolutely no problem with the organisation of the Party itself, but more complaints about the Venue.

I drink in various bars quite a lot. I've worked in a few bars. I observe how bar staff operate, and how their tills work. There's a few longstanding massive problems with pretty much every till/billing system I've ever seen in a bar.

Some are downright terrible, and who knows what the designers/integrators were thinking. These typically have a full 103 button QWERTY keyboard, and you've gotta type shit in, to get stuff to come up on the tab. It's slow, poorly designed for purpose, and not ideal to use a keyboard in a wet/food preparation environment, so the keyboard gets filthy and broken, so the staff hammer on the keys harder and harder.. Yeah. You can see where this is going.

There's the generic Touchscreen POS system, that's been adapted for bar use. Generally a better lot, but the biggest problem is traditional POS relies on having barcodes on everything, and a scanner on the till. Doesn't work very well for bar things, where things aren't quite as fixed-format as a traditional shop.

One of my favourite haunts in Brighton gave all the staff a very small barcode reader, and everything was barcoded.

You ask for a double G&T, the barman swipes the Gordons shot twice, and a bottle of tonic, and you're done. It worked *perfectly*. This is the right kind of idea. Sadly, it's about the only place I've ever seen it.

While we're vaguely on the topic of Brighton, there was a particular favourite cocktail bar down there which had the most remarkably unfriendly till system. @PoBK and I persuaded the barman to let us have a look at the interface after it took him nearly 10 minutes to enter the contents of a custom cocktail. The conclusion we came to was that some idiot designer had melded the keyboard entry system with the touchscreen system, but failed to recognise the actual *modus operandi* of a bar. Every time you wanted a new ingredient, you had to re-search the database, then enter the quantity, in Millilitres on the keyboard.

So here's an idea. Cocktail bars need a till system that's designed for their use. Frequently used ingredients in frequently used sizes have more accessible buttons than other, less frequently used things. Ergo, Gordons Gin has a big button, Anchovy paste, a smaller one, at the end of a list.

Also enter a bunch of cocktails (matching the menu) into the database, so you can ring up a mojito with one click, rather than say, entering 2 shots of rum, mint, lime, etc etc etc.

I'm going to change theme here, momentarily.

One of the biggest complaints I've heard of (mostly) last night's venue, is the concept of tab inflation.

A number of people have reported on Twitter and IRC that their tab receipt contains/contained drink items which were not their own.

I can come up with a few reasons for this. All of them are avoidable, and yet were not avoided by design/implementation.

Having observed the staff using the bar tab feature, this is the typical pattern:

Customer orders drinks.
Customer presents tab card, (a business card sized piece of paper with a number on)
Barstaff hit "Add to Tab" button, which displays a screen of small (1 square cm) touchscreen buttons with numbers 1 to 500, or so.
Barstaff select matching button to tab card, and receipt is printed.
Customer gets drunk.
GOTO: 1

There's a number of problems with this.

Let's start with Authorisation and Authentication, or "How does the bar know you are who you say you are". Well, under the above system, they don't.
I could grab/steal/replicate/forge anyone's card, and rack up a massive amount of money on their tab, because there's no checking mechanism in the system to be sure that the tab belongs to me.

In the past, I've seen systems (often in hotels) where you sign the paper against your room number, the bar staff keep the paper, then when you check out, you can see the list of receipts.

My local pub asks for the name on the payment card that is securing the tab.

There's a pub in Hammersmith that uses a slightly more upmarket solution where the tab card is actually a key to a lockbox that holds your debit/credit card. A bit better, but that also leaves much to be desired in the whole card security theatre.

Authentication is a bit of a bugger. Many of the ideas that work well, aren't ideally suited to the fast paced realm of bar service. Many of the ideas that are currently used are massively insecure.

From what I've heard, the above problem isn't the actual vector for tab inflation, or at least, if it is, it's of considerably smaller incidence and volume.

The biggest problem is threefold. Primarily, the bar staff have to manually read the number from the card, and secondly, match it with a list of small numbers from 1 to 500 (ish), thirdly, and from an User Experience (UX) point of view, most importantly, the buttons are small, closely spaced, and there doesn't appear to be a molly guard (something to stop you from incrementing the wrong tab, in short, "Are you sure you mean tab 123?").

On more than one occasion last night, I heard the admission "I might've put that on the wrong tab, I'm not so good with numbers." - Fair enough. I'm not brilliant with numbers myself, but I can design that point of failure out of your system.

Here's the simplest solution. It's so simple you're gonna kick yourself. Print some computer-readable representation of the tab card number, on the card itself.

A few options, in ascending price might be: Barcode, QR Code, Mag-stripe, Punched Card, RFID contactless technology, SmartButton, Fingerprint recognition (This'd be cool.)

Yep, that's it. It's slightly more expensive, but you'd soon find that you'd be losing less revenue through unclaimed drinks. You'd get more money from repeat business, because you've not alienated customers by saying "you lied, and you did actually buy this drink" when they, blatently, haven't.

Add a simple short authorisation code that the customer sets when they start the tab, and you've got primitive Authorisation as well as Authentication.

[1]

eBuyer's Black Monday

It's only a few days after Black Friday, and eBuyer have experienced their very own Black Monday.

I'll set the scene. This morning, I had an email mailshot advertising a £1 sale of clearance range items on eBuyer. Sounded like an ideal plan, especially if there were any hard disks up for grabs.

I never actually got that far though. I followed the "sneak preview" instructions, duly "liked" eBuyer on facebook. 10:30 rolled by, and oh look. Error 500 from eBuyer. Refresh. Try again. Same error. Connection terminated. Session reset. Page returned no data.

Let's have a look at the Facebook page for eBuyer. Seems to be some problem here.. Reports coming in from all over the web that the site's down. *gasp* Surely.. No? Oh my god.. They didn't anticipate the extra load on their servers that this clearance sale would cause?

What a surprise. Another company suffering *seriously* bad PR. There's enough bile and vitriol about the entire fiasco split between their Facebook page, and #ebuyer on twitter .

So what actually happened?

Well, eBuyer effectively started a DDoS against themselves at about 10:30 this morning. It's fairly safe to assume that there's two main problems.

1) Their site is almost entirely dynamic content that has to be generated on every page view, especially as the clearance prices are only visible if you're logged in. So there's cookies involved, so the content can't be cached. This means that every page has to be "built" from scratch by the web-server(s), it's gotta make requests to the backend databases for prices and stock levels. Imagine this happening for every user.. Then every user's click, then every user's click in multiple tabs. No wonder the site's fucked.

2) More importantly, the majority of their connectivity has been saturated by people trying to access the site. There's commenters on Facebook and Twitter both saying if you chain-refresh the site, basically keeping your finger held down on F5, then you'll get to the site quicker. Probably not guys.

The problem is that you've got the audience of the email, the people from twitter advertising, anyone who's seen the Retweet and anyone from Facebook all piling down to go and have a look at the eBuyer deal.

Then there's the others.. The rubberneckers. Those are the types who slow you down on the highway by leaning out of the window of their car watching the ambulances wheel off the bodies. They're down there too. Looking for the charred remains of eBuyer and the scorchmark of their PR agency.

So how did Amazon survive their Black Friday sales?

Well, that's fairly straightforward. Amazon are *vast* and incredibly intelligent when it comes to business analysis. Amazon have been running sales for a lot longer, on a massive scale, but they've learnt from their mistakes. I can remember the Amazon sites falling over at peak time over Christmas. It's happened. It used to happen quite often, but they scaled up, and they scaled out. That's the only way they got to be one of the top eCommerce sites active on the internet.

I'm not saying that eBuyer need the same level of architecture as Amazon have, but the key point is the Business Intelligence that's missing. This is how the conversation should've gone:

Date: 01/11/11
Place: Business Intelligence Dept, Ebuyer HQ.

Alice: "Hey Bob.. On our mailing list, we've got a reader coverage of, hey, let's say 750,000 people, right?"

Bob : "Sure Alice, Why?"

Alice: "When we launch the £1 deals on the 28th of November, I bet they're all gonna visit at 10:20, so they can sit there and refresh until we launch the deals."

Bob: "Oh Alice, you're so wise. We need to tell the Operations team so that they can get some extra server power in for that day."

Alice : "Correctamundo, Bob. Imagine if we launched these deals and our site went down. Boy would our faces be red!".

I can speak with experience when I say that I've worked in places where they fail to plan for scalability during sale season, or before a product launch. It's impossibly stressful to be given less than a week's notice of this kind of event, and have to be in the position to ensure the site's stability and continued uptime. In some cases, it is *actually impossible*, especially without a massive amount of planning, scaling up the number of servers, increasing the bandwidth to the routing core. Sorting out load balancing, particularly eCommerce aware load balancing.. It's tricky.

This is where "cloud" services and infrastructure comes into it's own. Numerous cloud IaaS providers are offering scalable bandwidth, pay as you go, scalable servers, add as many as you like to their scalable cloud load balancers. eBuyer would only have had to pay for a couple of weeks usage, probably. A few days for testing, then 2 days either side of today to iron out bugs. I'm sure that it could have been handled more gracefully.

Somewhere inside eBuyer HQ, someone forgot to tell the Operations team. Perhaps one day, advertising and marketing teams will come down from their high horses, and realise that without full support from the entire company, an advertising campaign such as this is actually seriously detrimental to the company.

Live situation update: It's 11:44 on the 28th, and eBuyer's site is still down. @ebuyer on Twitter are attempting to make up for their technical faux-pas by promising better servers for future sales. Yes, well done.. That's called "Locking the door after the horse has bolted".

I last wrote about a similar problem in 2009, when Derren Brown advertised his website on his TV show, and it was down for 2-3 days. Since then, I'm still seeing the same problems, companies failing to anticipate server load caused by advertising. Nothing's changed. It's still embarrasing. It's still unacceptable, and in this case, for eBuyer, it's going to prove very costly. Not only have they lost the potential of more sales from their £1 sale, but they've also lost the regular traffic buying stuff for their day-to-day needs.

Black Monday for eBuyer, Stunningly good day for their competitors.

So, You Wanna Be a Sysadmin?

So you wanna be a good sysadmin? I don't blame you. It's fun, and it's lucrative. Especially if you do it right.

The difference between a good admin and a bad one are many and varied.

Most importantly, it boils down to a level of devotion to the company. And a knowledge that is expansive.

Not that it has to be all-encompassing. One of the important things to know is the limit of your knowledge. Know your comfort zones, and know your limits.

There is nothing wrong with not having an instant answer, but there is something with not being able to find an answer. Powers of inference and deduction are key to your success, and you should exercise them frequently.

I'd much rather work with someone who knows what they do not know, than someone who will blag it.

You've also got to recognise your weaknesses, and work hard to ensure that they don't impact your work.

Among sysadmins, the biggest weakness I see is ego.

Ego, especially in an enterprise or commercial environment is a killer. Egotistical sysadmins will act as if the system is their Property. Often failing to document important infrastructure, or trying to make sure that "the company can never fire them" as they've engineered themselves into a critical position.

This is a massive, yet all too common anti-pattern among admins.

As I mention earlier on, a good admin needs to be devoted to the success of the company. With or without them.

One of the biggest problems with Sysadmins with an ego the size of a planet, is the inability to accept personal responsibility. This is a real double-edged sword. There’s some things which, at some point in time, will be, or will have been your fault. It’s up to you to do two things. One, you’ve gotta take the rap for it. Admit you were wrong, and apologise. Secondly, and more importantly, it’s down to you to debrief the rest of the company about what happened, why it happened, and what you and your team will do to ensure that it doesn’t happen again. You’re allowed to make some mistakes, everyone does, but it’s a fool who doesn’t learn from past mistakes.

You also need to consider what happens if you're unfortunate enough to get fired/made redundant?

Bad times, but you'll only make them worse for yourself if you're petty enough to have installed logic bombs or backdoors.

Let me tell you this. There are some very skilled sysadmins out there. Some better at computer forensics than me. Many with better toolkits and detection algorithms and hardware to recover deleted files.

We will catch you. You will get found out. You will get into trouble, and you will never work in IT again.

You could spend 10 years building systems and making a perfect CV, but if you let your ego get the better of you, you might as well have not bothered.

I work hard to ensure that the changes I make are made for a reason; and they're well documented. The thing I fear most in any system, is a Single Point Of Failure. Engineers have worked for years to eliminate SPOFs from a number of systems. The important SPOF that must not be forgotten is the human factor.

If you were hit by a bus tomorrow, what would happen? Are there passwords that only you know? Scripts that are only on your Mac?

Citing ‘security’ as a reason not to disclose information is stupid and childish. Sure, don’t put passwords on a public wiki, but do put them in a password management utility, and do share access to that to your team. It’s the same as that old adage, “There’s no I in TEAM”, but without a solid team, your skills are of no use to the company.

Empire building is a common trait among sysadmins. Often in a new job, an admin will seek and destroy existing systems because they're not perfect in their eyes. Only in their eyes, mind you.

An existing system could have been running non-stop for 20 years on AS/400, but that doesn't warrant a costly move to x86 blades, based on a pitch from an amply bosomed Booth Babe.

No, again, you need to think in terms of what is best for the company. "The Greater Good", not flashy new toys.

The flashy new toys thing is massively dangerous. There are companies out there who don’t shop around, who aren’t interested in actually researching what they’re being sold, and where mis-selling of infrastructure happens a lot. It’s down to the sysadmin (or engineer) to actually test out stuff, instead of going with whichever has the best looking hardware, or the most promised (but invariably, individually licensed) feature sets. If that means going with slightly less bleeding edge technology, because it’s more tried and tested, then so be it.

There is absolutely nothing wrong with being a lazy sysadmin. Being irresponsible, on the other hand, is a massively concerning trait, and should be addressed as soon as possible.

I frequently find myself doing things to make my life easier. I’m a big fan of automated installations, Puppet, Kickstart/debian-installer, and centralised logging and monitoring.

The reason for this, is that I don’t want to repeat myself. I find building systems very enjoyable, but the minutia are very tiresome. I’d rather do something once, and repeat the action, rather than do something a dozen times on different servers.

Being a lazy sysadmin effectively means that you can concentrate on the important and interesting stuff, rather than spending all day working on something boring and trivial.

Being irresponsible would be ignoring logs, and being obstructive with documentation, and going out of your way to piss people off. This is the “media” view of many sysadmins, not helped by the wicked and evil character Denis Nedry in Jurassic Park. You remember, he took power systems and security offline so that he could steal stuff un-noticed, then initiated logic bombs and so on to cover his tracks and prevent his detection.

Don’t be like that. You might end up like him, getting eaten by a Dilophosaurus.

(You’ll probably just get fired, and never work in IT again)

Coming Out - My story

I came out roughly of my own accord, I think it was sometime in November 2000, but I can't be that precise on the date. It seems a long time ago now. I say 'roughly of my own accord', because as the story will unfold, some might say I was outed.

As a mere technicality, I came out to my internet group of friends a while before this. I remember that a lot more clearly as being a few days before my 13th birthday, or was it my 14th? Anyway.

I found a huge group of other guys, all the same age (apparently(!)) who were all going through the same things.

In the actual process of my coming out in reality, I'd been planning my thoughts on paper, as I knew that the conversation itself would come up soon, and when it did, I'd wanna be ready. I wasn't the best secret-keeper at that age. I told a bunch of "friends" at school, some were supportive, some gave up my truths to the kids who bullied me anyway, and that gave them more ammunition. Anyway.

Writing things down on paper proved to be the catalyst for my eventual coming out. One of my teachers got hold of this notebook. Fuck knows how. Perhaps I'd left it somewhere in a lapse of personal security.

Anyway. She phoned my parents. And told them Everything.

I got back from a youth group, and Dad said "One of your teachers called earlier, and we need to talk". My mind rattled through a bunch of alternative possibilities. Was I in trouble? What on earth had I done that could possibly cause that kind of parent-teacher interaction with such urgency.

We sat down in the lounge, and Mum asked me outright. "Tom, are you gay?". I admitted, and I don't remember a lot after that. I remember crying in her arms, more out of relief than anything else.

Turns out my parents had known all along. As they always seem to do.

My parents and I have a very healthy, open dialogue about my homosexuality. They've met almost all of my boyfriends, they've been out clubbing with me in Birmingham, for my 20th and 21st birthdays.

I only wish that everyone could have such open and accepting parents, but I'm afraid that the truth of the matter is, that not everyone is as accepting and modern-thinking as my folks.

To that end, if you're coming out today, then rest assured, the community will support you, and the organisations that exist will do so too.

Further Reading: My It Gets Better story

Endnote:
The next day, I had an interesting conversation with the headteacher of my school. Apparently the teacher in question had acted outside of school policies, and was pretty severely reprimanded in the coming weeks. I don't begrudge her actions today, but it's certainly not the best way to go about these things.

For help and advice on coming to terms with being gay, you can call the Lesbian & Gay Foundation helpline on 0845 3 30 30 30 (local call rate), between 10am and 10pm or use their online contact form to receive a reply within 72 hours.

Stonewall also offer advice on coming out.

A Sensible Java Build Tool

I've been writing Java in one sense or another for a few years now. I learnt stuff at university, then used it in a few jobs. I've written Beans and Applets, and various bits of stuff in between.

It's fairly safe to say, that I like Java.

One thing however, that's been a pretty consistent bugbear in all the time I've been writing Java, has been the classpath, and dependency resolution. Luckily, all that can now change.

I used to think that Ant was a pretty neat build tool. All the IDEs supported it, it kinda worked most of the time, but sometimes, building was a bit of a ballache - Some stuff had to be in your lib/ folder, sometimes in Ant's lib/ too.

Lately though, and this week in particular, I've been playing with Maven.

Maven is a pretty fucking cool build tool for Java applications. I suspect it probably works with other languages, but it's "designed" for Java.

I don't think I really have the expertise or knowledge to explain how Maven works, partly because I haven't studied the inner workings that deeply, but also, because it's far better explained here (http://maven.apache.org/what-is-maven.html).

Instead, I'm going to dive right in, and explain what I've been working on this week.

The company I work for currently, is making a pretty radical shift away from using PHP for everything. Instead, we've been investigating Java for creating a middleware layer that everything can talk to.

I'm pretty chuffed with this, but I do wish that it had come a lot earlier on. If it had, I might not have been so decisive to leave when offered a better job.

Basically, when we came up with this project, I insisted that we do it properly, for a change.

I suggested that a good workflow would be something like: Netbeans IDE -> Maven Project -> Git SCM -> Jenkins CI -> Maven Repository (We chose Artifactory, but I did test Sonatype Nexus too, but didn't like it).

This is a good pattern for the Joel Test's "Can you make a build in one step?"

I basically wanted to create a demo project that can be used as the basis for all future FR projects, I do the R&D to make the initial POM work, then everyone else can clone this, or inherit from it..

This decision was twofold, I also wanted to figure out JPA/Hibernate and have some clue how that works for reverse engineering the classes from an existing database, the answer to that is: Pretty well, actually. - But that's another story.

My IDE of choice is Netbeans. I've been using it since I was at university, except for a small android-related foray into Eclipse, and an experimental nosing around IntelliJ IDEA.

Stuff I did:

Created a new Netbeans Maven project from the quickstart archetype.
Added the Dependencies on Hibernate (all the sub-dependencies get resolved, and added)
Added the <scm></scm> and <ciManagement></ciManagement> lines to the POM
Added maven-shade-plugin to allow us to build a fat JAR, which makes the jar bigger - it includes all the dependency JARs, but does make deployment a damnsight easier.
Configured <distributionManagement></distributionManagement> to contain the url of the repository we're using.

That's pretty much it. Here's the finished POM, with various bits of secret removed.

When I edit something in Netbeans, and commit a change, there's a post-commit hook (post-receive) that calls the Jenkins API, and builds the project. Jenkins then deploys the artifacts (a fat JAR and a POM) to the Artifactory.

Epic.

Deeply Concerning

Right. This is important. I want you stop what you're doing and read this. It won't take long.

I've just witnessed another act of homophobic bullying amongst school children.

Sadly this time, they weren't in uniform, so tracking down the school responsible is going to be somewhat harder. What I can tell you is that there were 4 boys, three black, one white, and all very troubling.

The thing that troubles me most, as gay man living in london, is that if these kids are to be believed, then we are all under threat.

As the kids boarded the bus, three of the kids goaded the other one with chants like "dirty queer" and "fuckin battyboy". This alone troubles me.

I thought that now that Section 28 has been repealed, that schools are supposed to encourage a level of tolerance and acceptance. This is evidently not true of these kids.

I think we need to do something about this. Together. Not as a group of gay people trying to right the wrongs of the school curriculum, but as members of society.

I genuinely feel bad for the kid they were harassing, that kid being merely an echo of my former self.

One of the problems I experience when I see these types of incident is that I don't always feel comfortable to intervene. Last time was a different case, the kids were all in Holland Park School uniforms, so if any of them tried anything funny, I'd have at least some idea where they were from.

These kids today looked and acted a lot tougher. I tried my best to listen in to their further goading and haranguing over their tinny R&B sodcasting, but didn't really get very far.

The general high-level overview of it is "You're different, we don't like that. You'd better die before we kill you". I don't know about you, but I'd say that kind of talk is pretty bad for teenagers who by that age, really should know better.

So. Here's an interesting idea. I'd like to know what the actual problem is.

Are schools not handling homophobia, when it comes up, do they brush it casually under the rug, like The Chase did?

Do they have any out gay members of staff who are prepared to act as a positive role model for students? I know as sure as hell that this would have helped when I was having similar problems.

What exactly are kids taught these days with regard to homosexuality? I mean, when I was at school it was very glossed over. Something along the lines of "There are these people called homosexuals. What they do is bad."

I would like all of you to write a letter to the Headteacher of your local school. If it's one that your children go to, then that's all the better. I want you to ask them the following questions.

What does the school do to combat homophobic bullying?
What level of teaching is there with regard to homosexuality?
Do you have any out gay members of staff who act as outreach to gay kids growing up?
If not, why not?

Usually, I'm pretty proud of London. It's a great city.

Sadly today, I feel let down by the city and its future generations. It's deeply concerning, and up to us to do something about it.

Bored Engineer

So there's this saying, "There's nothing more dangerous than a bored engineer"; I tend to think that it's true. I've had very little to do at work lately, which has been in equal parts frustrating and annoying. I like having stuff to do. I like having plans for the future, but at the moment, there's very little.

Anyway. I popped into Westfield the other day, and caught a Free BBC Prom. Very cool. Then I had a poke around on my mobile and realised two things.

1) There is no mobile proms website.

2) There is no mobile proms app!

So I thought I'd have a go at writing one. Given that I have a Google Nexus S, and all of the SDK bits.

I wrote a parser/scraper for the BBC Proms website (about 100 lines of python, using BeautifulSoup and simplejson) - I might Github this later on.

I then set about writing a thing for the Android that would let me browse the list of proms, give me information about where they are, what time, etc. It's pretty close to being complete. Another day of development and it'll do most stuff.

If you've got a 2.1 or better Android, then you can have a play with the v0.2 Beta of PromGuide by clicking here or scanning the following QR code:

If you find it useful, or broken, or totally hopeless, leave me some feedback.

Cloud Backup Strategy

It has recently been brought to my attention that a number of users of cloud-based hosting services tend to use an "integrated" backup solution provided by the cloud host. This is probably some form of snapshot-based backup of a server's state.

I quite like the idea of doing this, especially if there's no impact to the server being backed up whilst the snapshot is taken.

However, I can immediately see one big problem with it.

At least one scenario I can see that would require me to restore a backup is failure of the server host. Under this circumstance, it might be possible that a) you will be unable to get hold of the backup, which is probably stored somewhere on their storage cloud. or b) You can get access to the storage, but the backup is a proprietary format, either a raw snapshot, or a VMDK disk image which might be difficult/impossible to transfer to a different host.

I'd be especially scared of using snapshot-backups for a database server, because in the unlikely event that the restore target is different to the backed up server, you might have some compatability problems, especially if you're using x86 MySQL and go to a x86-64 host.

For this reason, I think it's probably best to have a couple of different backup strategies.

I suggest having a snapshot backup is a good thing, and will allow a very fast restore process, but is only useful while your server's host is online.

In the event that your host has gone down, it's important to have an offline/offsite backup. This alternative backup should also be as platform agnostic as possible.

In other words, any databases should be exported as SQL files, and as far as possible, the system state should be backed up. I tend to keep track of what packages have been installed by storing `dpkg --get-selections > /var/lib/backup/dpkg-state` or similar. This means that if I have to rebuild a server, i can just use that file, and restore the state of package installations really quickly and easily.

That, and a copy of /etc, and restoration should be pretty easy.

On the other hand, the concept of trying to restore a VMware, KVM or Xen snapshot (which might be inaccessible, or otherwise unavailable for export/download) onto a different system entirely, frankly fills me with a little bit of fear.

Given the choice, a snapshot restore is almost certainly preferable, but it'd be prudent to have a backup strategy for your backup strategy. ;)

Britannia Country House Hotel, Manchester.

I cannot begin to understand what the General Manager of the Britannia Country House Hotel, Manchester (BCH) is thinking when he runs the hotel day-by-day.

Never before, have I found such incompetance among public-facing hoteliers. Firstly, a Disabled room was requested at the time of making the booking. We were given a non-disabled room on the 4th floor.

*pester*

Now a disabled room on the 1st floor.

4 hours later, the main lift packs up. Stops working entirely, because *apparently*, 5 days ago, it was full of 8 american tourists, with 8 biiiiig bags, all crammed in so tightly they exceeded the weight limit and had to be freed by the fire service.

Did the BCH have the lift fully repaired since then? Apparently not. We got in and it made a worrying *groink* noise before the doors closed.

Anyway, 4 hours or so after checkin, the lift freezes and goes to the 4th floor, where it locks itself down, and won't move for love nor money.

Leaving my disabled roomie trapped on the first floor.

On investigation, it's revealed that they don't have an Evac-chair. Nor a service lift, not any means of getting people from Disabled Rooms (on the first floor), down to safety.

Fuck knows what their fire contingency plan is. (Actually, my roomie spoke to the duty manager who was equally unaware of any fire contingency plans.

Apparently there wasn't a lift engineer available to fix it within 24 hours, so he had to be carried down a flight of stairs on a chair, by 4 heavy barmen. Not exactly dignified.

So after a *lot* of jiggery-pokery with some more unhelpful, rude and incompetant hoteliers, we now had a room on the ground floor, save for being up 3 steps, which have a plywood/carpet ramp. Which moves and flexes with every step. It's also quite a steep ramp, probably impossible to navigate with a wheelchair, based on its narrowness too.

The room is also smaller than the previous room, with no disabled handrails, or pull cord.

I also discovered, much to my infuriation (being someone who requires a hot shower in the morning, lest I be an evil cunt all day), that the shower didn't actually work. Some kind of mixer tap with a pull-up knob to divert the water flow, actually doesn't pull up at all.

Secondly, and more worryingly, The in-room phone doesn't work. Combine this, with the room not having a disabled pull alarm, means that if Sam did fall in the bathroom, or anywhere else, he'd be completely stranded. Can't rely on mobile phone signals in these rooms, the windows seem to be lead-glass and the walls forged from bricks of Depleted Uranium.

The hotel manager sent a maintainance man around, who poked at the shower, and the phone, and came to the conclusion that the shower was *so* old, and full of limescale that it was shagged, and the phone was more frustratingly "fucked". Apparently this hotel is made of 2 sections. The front bit is newer, and more modern, and the back bit is a converted block of flats. Apparently none of the phones in this back bit work.

I managed to get 2 decent showers out of the shower, before it returned to original form, and stopped being functionally usable, and was just a dribbly tap. Great.

I would mention it to the hotel staff again, but I don't think they really give a fuck.

So, it's Monday morning, and we're not due to check out until Tuesday morning. *knock knock*. Oh, it's you. Head of Housekeeping. I think I'll name you Chardonnay for the duration.

Yes, we're not checking out until Tuesday.

*4 hours later*

*sounds of key in lock*

I go and answer the door, before she has a chance to unlock it further, and see middle aged cleaning woman (Helga, perhaps?), doesn't speak a fucking word of english, other than "my boss say this room empty"

me: "Well, we're not checking out till tomorrow"

her: "My boss say room empty"

me: "Well, your boss is an idiot".

her: "I go away now"

5 minutes later, Chardonnay, Helga, and some guy turns up, and says "Yes, the cleaning lady doesn't speak much english"

me: "Yes, I told you earlier, we're not checking out today."

*sighs*

Seriously. Would it be too much to ask for people who are employed in the UK to be able to speak passable english?

Would it make sense if they knew what Do Not Disturb means?

I wonder how many times Helga has caught someone in flagrante delicto whilst trying to service their rooms? Is she some kind of voyeuristic cleaning-pervert?

"I wash your sheets, you make them dirty!"

FFS, BCH. I'm used to far better customer service. Far better staff, far less rude cleaning staff, and generally, not being fucking disturbed when i leave a DND sign on the knob. What part of that is so fucking difficult to grasp?

The available food at the hotel is similarly gash. Apparently they have an on-site pizza place. I'm yet to actually see anyone eating in it though. Someone asked at reception about the hotel pizza place, and they got given a Dominos menu. Insert comment here about dogfooding (or is that the toppings on the pizza?)

There was a "Light Bites" menu available, which seems to have been mostly microwaved ready-meals, except for the "Stuffed Potato Skins", which were skins filled with tomato puree (tube quality), topped with cheddar, and microwaved.

Eugh. So so so acidic.

Sam ordered the Bruschetta, and we were both surprised that it was Ciabatta, untoasted, cut end-ways, rather than length ways, so it was 6 slices, each with a surface area of about 3 square inches, and coated with a thick layer of Margarine, topped with some raw onion, raw peppers and raw tomatoes.

Perhaps we're spoiled, and London really is the paramount of global cuisine, but something tells me, that this isn't the case, and the cooks at the hotel are as incompetant as the rest of the fucking staff.

On the day we checked in, Thursday, there was a "Carvery", which was actually just some lukewarm roast pork, and palid apple sauce, where I got a paltry 4 small slices of pig, and could have quite happily devoured 4x that amount, but apparently that wasn't an option. For this, we paid £13.50.

On other days, there was one of three options, Something meaty, and tasteless, something fishy, and smells funny, and something vegetarian and cold.

Nothing particularly appetising, or nutricious. I am reminded at this time of school dinners, for a similar calorific value, and flavour level.

Which reminds me. Further to the aforementioned disability problems, out of a possible 6 bars, only one was at ground level, with no steps to get to, but this wasn't open anywhere near as often as any of the others. The main lobby bar is down a flight of 3, quite deep, stairs. The bar in their built-in "nightclub" is down a flight of 4 steps, then up a further flight of 5. The bar in the back "bistro" area, requires climbing 6 steps, and descending 4.

Basically, if you're unfortunate enough to be disabled, and unable to use stairs, you'd better be either tee-total or not thirsty, because your chances of getting a drink are pretty much nil.

I'd hate to have to navigate the hotel in a wheelchair, many of the doors are seriously weighty, including the one to the corridor for our room, and that one doesn't open fully, because there's a mysteriously placed sticky-out-bit of wall, which makes opening the door past about 60 degrees, completely impossible.

It's almost as if the floor plans were designed by Goebbels himself, as a disabillity assault course, designed to weed out the less capable.

Perhaps a word of praise, now. Although only a brief one. The beer is cheap, cold and plentiful, and the bar staff are cute. However, they seem to hate the rest of the hotel staff as much as I do. A fantastic insight, for which I am deeply grateful to see that they have absolutely no faith in their management either.

Overall review. Shocking. Don't stay here at any cost. If you do find yourself here, Run like hell.

I keep finding "quirks" about this place that leave me aghast and open-mouthed. The lift/disabled access thing being fairly prominent in my mind.

Oh, and I saw a rat in the lobby.

Photos of this hellish establishment can be found here: https://picasaweb.google.com/tom.bioinf/HotelFromHell?authkey=Gv1sRgCO-O3fCkvsG83AE&feat=directlink

Desktops as Servers

Personally, I hate the idea of using a desktop as a server in a production environment. I'm going to define the term "production environment" first. If you've got an environment, any environment where the service provided is relied on by anybody, for any reason, then that's a production environment. If it's just for you, and you don't mind when it all goes wrong and the shit hits the fan, then that's fine.

Case in point: I've got 2 re-appropriated desktops as a pair of Domain Controllers for testing a domain deployment. Each desktop is running Windows 2008 R2 server, and provides Active Directory, DHCP, DNS and Windows Deployment Services. This was fine for testing, and playing around with building workstations, but the problem comes when people find out about this, and want to rely on it. For about a week, I was experimenting with using Windows' DHCP and DNS servers for the entire office. This was fine and dandy until there was a powercut, and neither of the desktops came back on automatically. This is because, unlike most servers, the default ACPI configuration is to start "off", and not "last setting" or "on".

So the desktops didn't boot up, and nobody could get a new DHCP lease. Bit of a bugger that, but easily fixed.

In the event that I ever do get this kind of scenario in the office in production, where people are reliant on the availability of the Domain Controllers for login and file sharing, then I've already got some HP Proliant servers specced up and ready to order.

There's other problems too. Desktop hard disks aren't designed for 24/7/365 operation, and aren't designed for a high duty cycle like that of a server. What disk manufacturers call "Enterprise Disks" are much more sturdily built than "Desktop Disks", they're designed to work harder, at higher temperatures, with higher duty cycles, and are generally designed to be always on.

There's also a running trend amongst high capacity desktop hard disks, where they're "Green" or "Energy Efficient". One of the ways that manufactures implement this, is having the disk stop spinning when it's not in use, or send the entire unit to sleep. If you have a RAID set built out of Green Disks, then you'll probably find at some point that the array ends up degraded - "broken" in layman's terms. There's probably nothing wrong with the disk, but the disk firmware has shut it down, or put it to sleep because it's not immediately being used. The RAID controller, software or hardware, sees this as a disk failure, and all hell breaks loose. Especially if you have 2 of them that go to sleep, in a RAID 5 array, then you're really screwed.

Desktop motherboards are also a different breed, they're generally designed with Athlon or Intel Core processors in mind, which have a very different fetch-execute cycle to a Server-grade Opteron or Xeon. They're kinda not really designed with server operation in mind, and are sorta "slower" or less performant than an equivalent speed server processor.

On the topic of Desktop Motherboards, they're also less built for high memory configurations, typically with 2 or 4 DDR3 slots, and their capability to accept ECC (Error Correcting Code) RAM is very variable. Some do, some don't.

I like build-in redundancy, and defence-in-depth, especially when building server solutions. I like having ECC RAM, it's more expensive, but does protect against bit-flip scenarios, those which could cause kernel oops, panics and blue screens of death. I also like having more than one of things, like multiple disks, and so on. I visibly squirm when I find SMEs using desktops as servers, in production, and then find that the "server" (or desktop) only has one hard disk.

Server motherboards also often have neat features built in, like more PCI slots (and 64 bit width ones -- handy for RAID cards). There's also iLO/DRAC/IPMI for remote management built in, but remember, if you have remote management, make sure it's configured before it's too late.

They also tend to have better BIOSes, which are designed for headless operation, no more "Keyboard not found - Please press F1 to continue" messages, which prevent your headless server from booting.

Servers that are built as servers, on server hardware, cost more than a desktop, but last far longer. You get a much greater Return On Investment by not having to replace disks and memory that have failed in the first year, because they've simply worn out.

As with any electronic equipment, the bathtub curve of failure rates applies, but the entire graph length is much shorter for consumer-grade hardware.

If you look at the cost of a server, along side the cost of a desktop, then the cost of a server really is quite a lot higher. The rub is that the cost of downtime can be enormous, especially if the services provided by the server is core to the business, or it's core to the operations, such as logins, and file sharing (in the case of an office domain).

Hardware is cheap, Downtime is damn expensive.

Perhaps, along side everything else, the old adage is truer than ever:

You really do get what you pay for.

mod_rewrite is killing social media.

This is a little ranty, but it's really pissed me off lately.

That’s right. It’s you. The ones with image hotlink protection, and the ones who rewrite URLs to do strange and special SEO things, but who don’t actually think about what happens when you send someone a link to something.

(For the uninformed, hotlink protection is that thing where you get sent a link to an image, but the site owner is being draconian, and redirects you to google, because your referer wasn’t their own site, so the image must have been stolen, and put on another webpage (!))

Here’s what happens. Someone makes a blog, and posts a funny image of a kitten. We all like kittens, so I copypasta the link, and send it to my friend.

Problem is, the site owner is being a twat. They think that we’re still in the 1990s, and bandwidth is expensive. They set cookies when I visit the site, and then they look for those when I look at their images.

I post a tweet like “Hey, check out this cute kitty! http://you.are.killing.the.inter.net/kitty.jpeg”

I have the cookies, so it looks fine. My friends, however, do not, so they redirect to google, or something equally stupid.

Here’s the result. Either I look stupid, or they look stupid, or both. Neither of these are particularly good things.

I can’t save the image, and host it somewhere else, because that would be stealing it from the site owner / copyright holder, adding a dose of further legal problems, and also a massive layer of effort on top.

Here’s what site owners should do. Stop being a twat. If you’re concerned about bandwidth usage from your assets, host them on Amazon’s S3 cloud, and shovel it all through Cloudfront. Set up a CNAME to your Cloudfront Distribution point, like “media.killing.the.inter.net”, and serve your static assets through there. You’ve got enough bundled bandwidth in the Free Tier to last more than long enough, and you also leave a sensible system by which I can share your media files on the social web.

The first time I found this today, was on some guy’s site where he claimed to be “the self appointed curator of the internet”. Hell, I think I’m better for the health and wellbeing of the internet, personally. I don’t protect against hotlinking, because it’s stupid. It’s like anti-right-click scripts on websites. Those are fucking dumb too.

By “protecting” your images with some mod_rewrite trickery, you’re actually diminishing the traffic to your website. I’m never going to link to you again, because you’ve got crap policies. You’ve also lost the inquisitive organic traffic sources, the people who go “ I wonder what else is on that site”, because you bounce them through to google, instead of your homepage, or the page that the image was originally on. That would be smart, that would mean you’d get more traffic in general, more adword hits, etc etc.

But no, you’re all still living in the past, back in the days when bandwidth was an expensive commodity. Wake up and smell the megabits. We’re not in that world any more. If your host is threatening to cut you off for costing them a fortune in bandwidth, tell them to fuck off, and find somewhere else. There’s no shortage.

Hell, go it alone on an EC2 micro instance on the free tier. I’ll even tell you how to do it.

Secondly, if I’m visiting webpages on your site, I’d like you to do 2 things. They’re really simple, and you should have been doing them for years.

1) When I click a link, I’d like the Address bar to change accordingly. Or you can show a permalink link. One or the other. I’d like to be able to share your website with my friends on twitter, or IRC, or Facebook. I can’t do this if you don’t give me the links to share. All I end up sharing is an invalid link that then bounces them to a HTTP Err 500 page, or a 302 Redirect to google. Yeah. Smart move there. NOT.

2) This is the biggie. I’d really like it if once you’ve generated an URL, then it doesn’t change. Ever. You could make my life immeasurably easier if I can keep a bookmark to your site for 10 years, and never have to wonder “where’d that page go? I’m sure that URL is right...”

Oh, and read this: http://www.w3.org/Provider/Style/URI

Seriously, What?

Sometimes you read something on the internet and think "Huh? Really?". When I read this, I swear, you could almost hear my brain go *boggle*.

When I first started using Java, I remember reading something in the EULA (yes, I read it), about not using it for mission-critical or life-critical circumstances. Something about avionics and nuclear power stations.
Specifically "You acknowledge that Licensed Software is not designed or intended for use in the design, construction, operation or maintenance of any nuclear facility."

The thing is, we all click-through these, because we all suspect that nobody would actually use Java for a nuclear power station, or say, host a mission-critical service on the cloud.

However, tonight, that is exactly what it appears someone has done. I've also archived the page as a PDF, should it get deleted from sheer terror.

I am honest-to-FSM scared by the concept that there could be no built-in redundancy to that system. (Part of me wants the CTO from them to contact me WRT systems consultancy, the other part wants me to run around screaming)

I think the commenters say it best, but I'll still add my $0.02 here.

While Amazon EC2 may be compliant to a number of standards, and have previously had no major issues, this latest incident should serve as a reminder to all users of cloud infrastructure.

It's no different to any other system. It can go down, you can lose your data, and shit can hit the fan.

Have lots of redundancy built-in from day one. Have lots of different layers of security and redundancy, like Defense-in-depth for nuclear reactors.

Plan for the worst case scenarios, because in systems engineering, we deal with the when, not the what if.

The Name Game

This is real-life Social Engineering.

(If you've just read for the first time today, you should read all of it.)

The current meme is the "Royal Wedding Name".

BOHICA.

Again, It seems that some of you aren't understanding how these things work. The Royal Wedding Name asks for

Your grandparent's name (first name, male or female)
Your first pet's name
The name of the street you grew up on.

Right, you lot. Stop this now. I hate these name game memes, because as you should remember from last time, they're a crafted attack to reveal bits of information about you. Remember what I did to someone's facebook profile based on this info?

This one's been going on for a lot longer than I thought.. And a lot of you will be using your real grandparents' names, and the real names of your pets, and the real streets you've lived on. That's just silly. And dangerous.

And it's your fault if you get your identity stolen because of that.

Previous Memes:

March 4th 2011

The current meme is the "Pornstar Name"

This meme is asking for your First Pet, and your Mother's Maiden Name.

Seriously, These are two of the most common security questions used on a very large number of websites. By publicly tweeting the answer, you are handing over all the details a nefarious hacker needs to compromise your account, and steal your identity.

I cannot stress this highly enough. Do NOT tweet your Porn Name / Pornstar Name, or any other of these Name Game memes.

There’s often a meme going around on facebook/twitter/etc.. One of these note things, you do it, you tag your friends, they do it, and so on, or it proliferates on twitter.

These bug me enormously, because they ask for a fair bit of information. Here’s a brief summary of the answers you give.

Your Full Name
Your Mother’s Middle Name.
Your Grandfather’s Name.
Your favourite: Colour, Animal, Drink, Ice cream flavour, Cookie
Place of Birth
Street where you live
Street you grew up on
Name of your Pet

I recognise some of those as secret question/answer pairs from a number of websites. I’m really only kicking the tyres on this one, but what if someone designed these memes to gather data about people, including data about their past, place of birth, residential address, pet names, other stuff that’s commonly asked for sample questions on “Secret Question/Answer” credentials online.

I decided not to participate in this one unsurprisingly. In fact, I recommend that everyone who has done the “Name Game” note looks closely at their note privacy settings, just to make sure they don’t mind everyone knowing this information about them.

Where are your eggs stored?

When I was growing up, one of the things that particularly interested me about the English language were idioms and proverbs.

I think today, whilst many are still suffering the effects of the week, we should look a little more closely at one particular proverb, and perhaps its effective meaning today.

"Don't put all your eggs in one basket" :- This phrase is commonly (and some might say, incorrectly) attributed to Miguel Cervantes (in Don Quixote), but some sources have reported its usage as early as 1600. Also of little surprise is that many other historical cultures had similar phrases.

OK. We've established that historical peoples knew about having redundancy in their Ova storage and distribution methods, so pray-tell, why has this fantastic tradition been forgotten?

I am, of course, talking about the recent (21/04/11) Amazon EC2 and related services outage. Reddit, Foursquare and Quora are the big 3 companies who've been very public about their outage, but I wonder how many smaller companies and startups who rely on Amazon services for their server needs are also ending up out of pocket (due to lost revenues), or simply offline.

So the problem is this. Amazon are fucking cheap, in comparison to pretty much any other VPS solution. This is a royal pain in the arsehole, from a systems engineering point of view, because Amazon also price all of their other services similarly cheap. S3 is Seriously Cheap Storage, (they should have called it SCS perhaps). There's also the Load-balancer and cloudfront CDN frontend, again, incredibly cheap. There's a real movement towards building ones entire infrastructure around the Amazon cloud, and I think this is the problem. Amazon even offer a DNS service (Route 53), so you can serve your website's DNS records from the cloud too.

Can anyone see the problem with this? The architecture of the intrinsic scalability of the Amazon cloud does certainly allow you to create lots of small servers for things, so you've got a webserver basket, containing a half dozen server-eggs; and another basket for database-eggs. There's a massive problem here. Enormous problem. All of your baskets are inside one enormous basket. One incredibly big basket called "the Amazon cloud".

What appears to be happening to Amazon's cloud at the moment is one of two things:

1) People have built crap websites, or have only one egg. If you've only got one server, and it goes down, you're screwed again. You might as well have a dedicated server from anywhere else. You've still got a massive Single Point of Failure, and when the worst case scenario happens, you're fucked.

2) People have lots of inter-cloud redundancy, but no intra-cloud redundancy. This is akin to having lots of small baskets of eggs, in one big picnic hamper.

This is actually very common. It's trivially easy to construct a pretty big network on the Amazon Cloud, you add more EC2 compute nodes, then add some S3 storage, EBS block stores, Cloudfront CDN, oh, maybe Route 53 DNS, how about Simple Payments Service for micropayment, maybe Simple Message Queue, and that's before I get onto their database offerings.

Amazon have gone a long way to making sure that everything you could ever need for this kind of system building architecture is there, at one place.
They're like Home Depot, only there's a greater chance of Amazon having what you want.

ERRR.
There's a problem here. I feel the same way about people who buy a 50 disk SATA array, and fill it with disks with the same batch number. It's no surprise that if one fails, you're probably going to get another failure, caused by the same bug or hardware problem.

If you're going for true redundancy in the face of real adversity, then you need to start putting your eggs in many separate baskets. Globally distributed baskets. Baskets held by many different people.

I generally approve the use of S3 for system backups, because by and large, it's fast, cheap, and pretty secure (especially if you encrypt it). It's *really* fast if you're uploading from inside Amazon's network. There is an epic problem though. Say you take nightly snapshots, and upload them to S3. One day, your server goes down, either Amazon's fault or one of a number of other reasons.

I can see 2 enormous problems here. Primarily, if it's a fault on the Amazon network, it may affect your snapshot storage, and the ability to access them in a timely fashion, so while your Disaster Recovery Plan may say "Download the disk image and redeploy", you may not be able to download the disk image. Then you're screwed.

It's also possible that a disk error on the Amazon side corrupts your snapshot images, in which case, again, you're screwed. In a subtly different way.

Secondly, and this is a far more "doh!" problem, you may be able to locate and download your disk images, but not decrypt them, because the encryption key is stored on the primary server (also inside the backup image, encrypted). This is easily solved. Copy the key, print it out, and store it in an envelope in the company safe / bank deposit box / other secure location.

The biggest problem with all of this, is that there doesn't seem to be a straightforward way to share data and server instances across diverse cloud providers. I'd like to build an universal image, and then deploy it to the Rackspace Cloud, Amazon EC2, Flexiscale, and so on, and be able to

a) interchange data between them easily (not too hard, but would require some API glue)

b) have a global system for GSLB between them, so that if EC2 is offline, then all traffic is mopped up by the other two clouds.

c) Have a sensible "in-one-place" billing system (more API glue)

Physically, and from an engineering point of view, the biggest challenge of that lot is b. You'd need true global redundancy, and that don't come cheap. However, I think that's the topic for another blogpost.

In the meantime, perhaps you should evaluate where your eggs are, and how many baskets you have.

You should worry somewhat when all of your eggs are in one superbasket.

Then I think it's time for an ovum redistribution exercise.

ISC DHCP and PowerDNS

Lately, I've been playing around with a pair of domain controllers in the office, trying to figure out a good way to implement a domain. See, the problem is, this kind of thing is a "nice-to-have" rather than a core requirement. At least as far as the business directors are concerned. Their argument is something like "It worked fine with just a bunch of PCs connected to a switch".

I do like things manageable, and planned, and certainly now as we're approaching 50 desktops in the office, plus mobile devices, plus laptops, and FSM knows what else, that there's a real need for a bit more structure and management.

I ditched the Draytek's DHCP ability to allow me to test out Windows 2008R2's DNS / DHCP server, which interoperate fabulously, but do have a few limitations when it comes to specifying static leases (outside of the dynamic range). Bit annoying.

It does however do the dynamic dns updates, whenever a client gets a new lease, the DNS gets updated automatically. This is cool indeed.

I've been thinking of a way to replace this DNS and DHCP functionality with a bit of open-source goodness, because it's a nice thing to have, and even nicer to have for free.

I chose PowerDNS, because, well, I like it, and it's pretty scalable. Apparently it's the DNS of choice for the Wikimedia foundation, and i've used it before in a couple of other tasks. It's got a pretty nice MySQL backend, and also one for Postgres. For the time being, i'll be using the MySQL one, because that's what we tend to use around here.

So.. DHCPd, I chose ISC's DHCPd, because it's easily installed in Ubuntu. Always a winner there.

After a considerable amount of googling around, I figured out how to use the dhcpd.conf file to trigger an event to happen on commit, release and expiry hooks. https://lists.isc.org/mailman/htdig/dhcp-users/2011-February/012753.html and http://invalidmagic.wordpress.com/2010/03/27/magic-dhcp-stuff-isc-dynamic-host-configuration-protocol/ were pretty useful.

Then all I had to do was write a bit of python that would interact with the database, and update the records table.

Two major things caught me out.

1) Don't forget to COMMIT the data to the database, PowerDNS uses InnoDB on MySQL, so you'll need to commit the transaction, or bugger all happens.

2) apparmor on Ubuntu prevents dhcpd from using the exec() syscall. This is easily resolved by setting apparmor from enforcing to complaining for dhcpd.

Here's a couple of bits of code, one is the python updater, and the other shows how this all fits into the dhcpd.conf file.

https://gist.github.com/931984

https://gist.github.com/931988

Policing the Tweet-waves

On Saturday morning, I noticed a particularly dangerous meme (for want of a better word), making the rounds on Twitter.

Basically, this image was being retweeted over 25 times a minute.

After some digging around, I managed to trace the source of the image (that is, it's first known posting) to the /x/ board on 4chan.

Tweetmeme tells us that it was first reported as being tweeted by @ryphons, who still hasn't contacted me for further information, WRT the image source.

Tweetmeme also reports over 400 retweets, but I'm certain that the actual figure is much much higher.

So, the main thing that I take issue with, is that the image was being interpreted as fact, rather than a simulation, or prediction of what might happen. There is mass panic in Japan already, the last thing we need (as humanity) is the panic and hysteria to spread to the west coast of the USA.

While I am not a nuclear physicist, and cannot directly comment on the state of the nuclear reactors. I can definitely say that there is something seriously wrong with taking this kind of "map" as fact.

One of the most interesting things about this, aside from @ryphons not stating the original source (Was he the original creator?), is that the company whose logo is on the image "Australian Radiation Services" have no record, on their website, or on google's index of their site, of the image being theirs.

That should be raising red flags for you already.

Combine that with the fact that the apparent source of the image was twitter, and prior to that, 4chan, that most reputable news agency (!), and it should be fairly clear why I made the decision to attempt to stem the spread of this image across twitter.

I came under quite a lot of flak from a number of tweeps, who were concerned that I was playing down the situation. Rightly so. I am / was trying to prevent undue panic and the spread of misinformation.

Twitter is an incredibly powerful tool, allowing fairly free transfer of information between large groups of individuals. Sadly it's also got lots of wankers, spreading Fear, Uncertainty and Doubt. It's this that we (as the clever people on Twitter) need to stop.

There are many sources of valid information about the situation in Japan, notably from @arclight and blogging scientists like (http://morgsatlarge.wordpress.com/2011/03/13/why-i-am-not-worried-about-japans-nuclear-reactors)

I am a scientist. I don't believe in "god", I don't believe in "karma", and I certainly don't approve of trash media. I believe in factual information, and interpretations thereof by qualified individuals.

I don't read the Daily Mail for exactly the same reason. What you've got to bear in mind is that the media outlets are in this for the money. They'll continue to print uncertain and scaremongering drivel, because that's what people buy, out of uncertainty, or purely morbid curiosity. Sad fact of the matter is, that tabloid quality news outsells fact and science by quite a large margin.

That alone, as a fact is quite sad.

Proposal: Increasing Facebook Security

As I proved in my last blogpost, it's actually trivial to compromise a facebook account given a very small amount of personal information. After talking to a number of other geeks on Friday night, two things became quite apparent.

Facebook security is poor, at best, and the ability to change the user's contact email address is shocking.
Security questions and secret answers are easily exposed by social engineering, thus, these questions only work effectively if you have a completely different identity which you only use for secret questions and answers.

I don't approve entirely of having secret questions that aren't related to you directly.. I mean, if you had a secret question which was "What is your mother's maiden name?", and you gave an answer which wasn't true, you'd have to do two things. a) remember that you lied, and b) always use the same one, or you'd be forever confused.

Anyway. The real point to tonight's blogpost is that Facebook Security is gash. Seriously, even I was surprised that I was able to change my friend's contact email address, and sucessfully change his password.

The only good thing about all of this, is that Facebook lock the account for 24 hours, and email the other email accounts, This was the only way that my friend was able to regain control of his account.

I propose that facebook implement two-factor authentication for password resets, and possibly logins too. Given that Facebook already has and retains your phone numbers, it would be trivial, both in cost and implementation to produce a mechanism of 2-factor authentication for advanced profile control.

User story:

Alice wants to reset her facebook password.
She clicks the Forget Password link, and correctly identifies her profile.
She selects one of her registered phone numbers for 2-factor authentication.
She then selects whether she is to recieve a voice call, or SMS message.
Facebook send a validation code to the number, either as a SMS, or a short voice call, reading out the code.
Alice enters the validation code, confirming her identity.

This system would only work if you couldn't change the numbers that Facebook could contact you on (like you can currently change your contact email address), and you had already confirmed your phone numbers with Facebook in advance (on registration, perhaps, it could authenticate your phone number)

I don't suppose anyone who works for Facebook reads this, do they?

Interesting sidenote:

It appears that it is not possible to change a Facebook Security Question, for "security reasons".

"To protect account security, it is not possible to update your account’s security question once you have added one. "

Why the buggery not? This seems unusual. Surely these kinds of events (twitter memes, facebook notes for these Name Game things) expose users' security questions and answers, and most important thing to do after a data breach, is to change the credentials in question.

Most peculiar...

Identity Theft

To prove a point about the latest "Pornstar Name" Meme that's currently going around Twitter. Basically, the meme asks for you to tweet your Pornstar name which is comprised of the name of your first pet, and your mother's maiden name.

I'm furious about this. Those two names are the two most common answers to security questions found on a number of websites.

So. A theory: "Given just a user's facebook name, and their Pornstar name, it should be possible to compromise their facebook account".

I did this test with the full permission of the real account holder. I do not condone the use of this information for nefarious or illegal purposes, it is presented for educational use only.

Proof:

Open facebook, and click the "Forgot Password" link.

1) Identify the target account:

2) Confirm the account, but click "No longer have access to these"

3) Provide a new email address:

4) Go check that email account for further details on how to proceed.

4b) There is a missing step here. I forgot to screencap the bit where it asks your secret question, which may or may not be one of the ones referred to in the Meme, but I bet it is. Mother's maiden name and the names of pets are the most common questions.

5) You can then create a new password :O (For an account you don't own. .. Yeah, it's pretty bad, this, isn't it?)

6) There is, however a problem. Facebook by default will lock the account for 24 hours. This does however protect the user, as it sends them a load of emails to their other email accounts, basically saying "OH SHIT, WHAT ARE YOU DOING?!!"

IMPORTANT:

I'm presenting this information as proof of the theory that the Pornstar Name meme is damaging, and provides enough information to compromise an account.

Again, I did this test with the full permission of the real account holder. I do not condone the use of this information for nefarious or illegal purposes, it is presented for educational use only.

isdisconnected.info

New Project: isdisconnected.info (or How to build an application in 5 days)

About a week ago, my good friend @Moof asked the question “Is there a website out there monitoring if countries currently in revolt have full connections to the internet? Is eg Bahrain disconnected?”

I thought this sounded like a challenge too good to pass up, and set about coming up with a way to figure out how we could programattically determine the state of a country’s internet.

I’ve lately come up against the problem that when faced with a new idea, the hardest problem is getting it created, and working fast enough to ensure that your idea isn’t stolen by another like-minded individual.

With this in mind, I started work as soon as i’d finished $dayjob at about 5pm on the 14th, and didn’t stop until 3am. Putting together a week of 5pm - 3am development time, and calling in a favour from a very good designer I know, meant that we were able to launch the site by early friday afternoon.

isdisconnected.info is a simple at-a-glance view of the world’s internet connection status. Every country has a button, with their name and flag, which is either Green, Orange or Red, depending on the status of their internet.
Green is a Systems OK, all checks passed state, Orange indicates that some of the country’s server are inaccessible, OR there are no servers registered for that country, and Red indicates that the country is Offline, ie, all servers registered against that country returned a false check status.

The application is written exclusively in Python/Django, and backed onto a PostgreSQL database, with a hint of memcached in there to accelerate the page load-times. In the hour or two before go-live, I was experimenting with diferent caching settings.

Using no page caching at all, the time to load the index page was about 4s (down to page generation, more than anything), rising to 8-10s whilst handling 20 concurrent connections. Moof expected a viral response to the site, especially if it ended up on Linklog, or reddit, so fast performance was a high priority.

Due to the way the pages are generated, some of the data doesn’t lead itself to caching. Static assets are already served from Nginx, so that’s pretty fast and well behaved. The individual country pages (http://gb.isdisconnected.info) don’t lend themselves to caching, because some of the data is very changable.
In spite of that, the service that provides the data for the graph, does heavily cache the stream. Given that the resolution of the graph is on a scale of hours, the caching time reflects that, so that concurrent hits to a page will get cached graph data.

We can also anticipate that more hits will occur to a country which is Offline or Unstable, as people will want to find out what’s going on, so having some level of caching on those pages is very important.

I experimented with a site-wide cache of all pages generated, but discovered early on that cache invalidation was a big problem, basically country statuses weren’t updating quickly enough, based on the lifetime of the cache object, so as a trade-off of having more up-to-date information, against not quite caching so much, having a correct view of the global internet won out, naturally.

The index page, now that the list is cached for 10 minutes, loads roughly 1600% faster than before. There’s two tiers of caching taking place on this page, firstly queries are cached with Memcached (transparently by Django), and sections of the index are template-cached.

I’m very aware that the site is currently prone to false-negatives, that is to say, sometimes countries appear Unstable or Offline when they’re not, but we’ve also seen good reporting of positives, such as Saturday morning when Libya was disconnected.
This is a beta service, at best, still under active development, and still very much reliant on the power of crowd-sourcing to visit out website, get the word out about the application and the project, and ideally submit IP addresses for us to check.
The more IP addresses we’ve got, the more accurate the check data will be, and then the more accurate the site will be.
It’s very difficult to perform accurate statistical functions on a very small dataset, and when you do, the margin for error is vast.
We’re actively improving the site to make it more feature rich, as well as more accurate by determining servers to register against countries more intelligently.

We’ve got a reasonably good idea of what’s required to make the data even more accurate, and we’re working on that at the moment.

Monitoring with Munin

One of the things I’m massively fond of when it comes to systems administration, is logging and monitoring. I love munin, and still prefer it over Cacti and Zabbix. I think the main reason is that it allows plugins to be configured with absolutely no browser interaction.
Creating a new graph on cacti and zabbix both require a considerable number of clicks. It’s easy to install new munin plugins with things like Puppet. So.. Munin. Let’s take a bit of a closer look.

There’s two parts to a munin installation. Munin server, and munin-node.

Munin server doesn’t really do the cool stuff, just data aggregation and graph creation.

I’ve included an example munin.conf here.

There’s only a couple of quirks here.

I’ve found for the majority of installations, that you can leave the vast majority of settings in-place as they are from the version installed by apt / yum / $package_manager_of_your_choice.

So, the actual munin documentation suggests that use_node_name is a dodgy thing to do, but it’s actually pretty useful, especially when you’re defining SNMP hosts.

use_node_name tells the not to grapher to use the hostname that’s in [brackets], but instead to use the name in the connection banner (you can see this yourself, once munin is running, to telnet (or nc) to localhost:4949, and the line “#munin node at <your host>”)

SNMP hosts.. are without doubt the coolest thing that Munin can do. by default, the auto-configuration of SNMP hosts will allow you to monitor some interesting things about routers, switches and windows hosts. So.. the only major quirk about this, is that because the snmp plugins run on one of your munin-node instances, so you have to set that as the address in the host definition. In the example, I’ve done this on the munin server.

Munin-node. Very extensible, but as far as config goes, the default configuration that comes in the installation is more than capable.

Here’s mine.

If you have multiple munin-servers, or want to retrieve munin-plugin data from Nagios servers, then you can add multiple “allow” regex lines.

So.. Munin plugins. This is the Really Cool Stuff.

You can write munin plugins in any language you like. The vast majority on Munin Exchange are written in Perl or Bash. I prefer writing in Python, and the munin-python module is gorgeous.

Basically, you need to handle two things, “config” and “run” modes.

Munin-run is the thing that handles the plugin, and runs “your-plugin config”. This is what defines the format of the RRD files that munin uses to generate graphs. OK, so let’s look at a simple munin plugin. I think we’ll monitor... the number of files in /tmp (well, why not?)

https://gist.github.com/813813

If we run that with python tmp_files config, then we get:

graph_title Number of Files in /tmp

graph_category system

graph_args --base 1000 -l 0

graph_vlabel files

files.info The number of files in /tmp

files.warning 10

files.critical 120

files.min 0

files.type GAUGE

files.label files

and if we run it without “config”, we get:

files.value 18

So, that works. :)

Now if we copy that into /usr/share/munin/plugins, and chmod +x, and symlink it into /etc/munin/plugins.. and restart munin-node..

$ sudo mv tmp_number.py /usr/share/munin/plugins/tmp_number

$ sudo ln -s /usr/share/munin/plugins/tmp_number /etc/munin/plugins/tmp_number

$ sudo chmod a+x /usr/share/munin/plugins/tmp_number

$ sudo /etc/init.d/munin-node restart

 * Stopping Munin-Node                                                   [ OK ]

 * Starting Munin-Node                                                   [ OK ]

$ munin-run tmp_number

files.value 18

Cool. Right.. now that’s done, and munin-node’s been restarted, all we have to do is wait a while, and the new graph will get created. This can take a while, 5-10 minutes is a good guesstimate, but it can be longer.

This is the graph produced by the plugin:

Clever, eh?

If you find that you’ve waited ages, and still have no graphs, take a look at /var/log/munin on the munin-server and munin-node. There’s plenty of non-cryptic logging there, and it’s all pretty self explanatory.

Dedicated, Dedicated, Dedicated, Dedicated

After answering this question, I reconsidered my answer a number of times, and I’ve finally decided to rewrite it as a longer version as a blog/essay on my website. One of my fellow sysadmin types on Serverfault wrote an answer from a blog-post, and I intend to do the opposite.

Right.
I see a lot of questions which are basically, “I want my blog/social network/niche site/new product launch site to handle a whole bunch of traffic, how do I do it?”.

That’s pretty much what a lot of these questions boil down to, eventually.
I’m going to make a few assumptions too.
Given that someone’s taking the time to ask, I’ll assume that they’re actually concerned about uptime of the site. For whatever reason, whether it’s because their employer is telling them that they must have 5 nines uptime or better, or the site’s actually making money for them. Whatever the reason, we can generally accept that these websites are business oriented, and *should* have a reasonable budget assigned.
After having worked for a few different companies now, I can also fully accept that this second assumption might be a bit great, and not everyone has a good concept of how large, or encompassing the budget should be.

Let’s start at the bottom, with the really basic stuff.
One server will be OK for a certain level of uptime, but at some point, you’ll near the sharp increase of the bathtub curve, and the probability that the hardware will fail goes up rapidly, and when it does, which it will, and if Murphy’s law is anything to go by, it’ll fail when you’re out of town, at a wedding, or in the pub.
It’s for this exact reason that as a Systems Engineer, I can’t count any lower than two. What I mean by this, is that everything should come in pairs. Two servers, containing >2 hard disks, 2 power supplies, and so on.
So, let’s build a server, based on the above theories.
Disks fail lots. They’ve got moving parts. So let’s concentrate on those. If you’ve only got one disk, and it fails, you’re screwed. So let’s put 2 disks into this server.
You’ve got a choice again between hardware and software RAID. Linux software RAID is pretty good these days, but in some cases, hardware RAID is still preferable. I’m a massive fan of 3ware and Adaptec cards. Hardware RAID, if you get a good card, is invaluable. FakeRAID, as typically found on motherboards, or low-end raid cards is a bit of a ripoff. It’s actually a form of software RAID, and utilises the main CPU. On a hardware RAID card, the onboard CPU takes a massive load off from your main CPU, and is more efficient at processing nested RAID levels than the software RAID is, which uses the main CPU, which probably should be doing the really cool stuff that your server is designed for, not low-level stuff like disk processing.
There’s also something to be said for hardware RAID when it comes to non-linux operating systems. I gather that hardware RAID on windows platforms is a lot more stable than software RAID on the same.
So, basically, if you value your sleep, and your uptime, then you’re going to need to protect yourself from these failures.

That’s disks out of the way for the time being, let’s talk about power.
Most good servers (and by good, I mean ones I’d consider in a high-availability infrastructure), have the capability of dual, or multiple power supplies. These are brilliant, and protect against PSU failure, and power rail failure. Be warned however; if you connect the PSUs to different phases, you’ll probably see a very pretty, yet expensive fireworks show, and possibly set off the fire detection systems in the datacenter. Not a great idea.

In spite of the benefits of multiple PSU servers, they are more expensive, and to some extent, don’t offer a massive benefit, if the multiples are all plugged into the same power source, then you’re really only protecting against PSU failure.
The biggest problem I’ve seen in a datacenter, related to power, is the rack monkeys unplugging or rebooting the wrong server. As far as mitigating this goes, accidental unpluggings can be cut down with locking C13 cables , and remote-hands reboots can be avoided by using iLO/DRAC or an IP-PDU (Power Distribution Unit).
Whilst we’re on the topic of ancilliary rack hardware, things worth having:

IP-PDU (APC are very good)
IP-KVM (Raritan and Avocent both seem to be leaders in this market, Startech are ok but the interface is a bit clunky.)
IP-Serial Console (Raritan, Avocent, etc. )

I’ve rarely seen a 1U keyboard/monitor shelf in a rack. There actually is little point, you’d be better off with a good Dell laptop, an USB->Serial cable, and perhaps stow a keyboard and monitor seperately in your rack somewhere.

Wow, I really digressed there. Sorry about that. Where was I? Disks, Power, let’s look at the network.

Good servers have multiple NICs. You need to design your network to make use of this. Having one server/one NIC is good until your switch dies, or the NIC dies, or similar. Then your server goes down, and people get shouty.
But again, a pair of NICs is no good if they’re only connected to a single switch. Not only will it intensify any Spanning-Tree problems you may have, but it also provides no protection against switch failure.

So, a pair of switches. Or similar multiples of two, thereof.

Not only do you want a pair of switches on the network infrastructure, but you also want them to connect to a HA pair of firewalls, I quite like Cisco 55xx series, the 5510 and better offer good Active/Standby pairing, so switching between the two is simple, and they each monitor each other, and will share a virtual IP between them, with HSRP.

Next up, routing. You’ll want a couple, if not more connections to the internets, for really nice stable connectivity. I’ve argued this point over with a couple of colleagues, former and current. When you’re relying on someone else for your network connectivity, and you only have a single connection to their network, regardless of how diverse their network may be, you still only have a single connection, and that’s your biggest point of failure. You can harden your servers as much as possible against device and part failure, and I still highly recommend that you do, but if you don’t have resilliency and redundancy at every level, then there’ll still be a single point of failure somewhere on your network.

It’s actually perfectly acceptable to have multiple IP Transits from different providers and provide different IP addresses. As long as your application can cope with that.

The ultimate solution however, requires having a couple of powerful routers at the edge of your network.
These connect to a couple of transit providers and advertise your IP addresses over BGP. You get a full internet routing table, and the rest of the internet sees the routes to your netblock. The real bugger is though, that you’ve gotta have a reasonably large netblock to get noticed. At the moment, this is a /24 or bigger. That’s 255 IP addresses, and given the current rate of IPv4 depletion, getting justification for one of these is getting harder and harder.
Your carrier/transit provider can help you with the paperwork to get it all sorted out, though. To stop the size of the full routing table becoming enormously massive, ISPs filter out netblocks smaller than a /24, so a certain amount of route aggregation takes place.

Having BGP capable routers, and maintaining your own connections to the internet isn’t a walk in the park. It’s a specialist task, and requires a skilled systems engineer to operate and maintain it. Not a task for the unprepared.
There are other advantages of course, if for example, you use the Akamai CDN lots, and you’re paying a lot for traffic to their network, then you may be able to enter into a peering agreement with them, where the traffic to their network is delivered cheaply, or for free, because it’s mutually beneficial for both parties in the agreement.

So I think we’ve covered most of the major points on hardware and network resilliency and availability. Let’s look at how to put that all together and build a hosting cluster.

For the purposes of brevity, and clarity, let’s assume for the time being that you’ve chosen to use a dedicated host provider. I can personally recommend Melbourne. I use them at $dayjob for some dedicated server needs.
If you’re choosing to do the owned hardware / rented rack space thing, then there’s not a great deal of difference in the actual configuration of the servers, but there’s more complexity involved, etc.

For most small/medium size hosting needs, I tend to recommend a 4 node hosting cluster. This is based on having 2 web servers, and 2 database backends.
When you start off with a single webserver, with the database server on the same physical box, the fastest way to increase performance is to split and have 2 physical boxes, one for the webserver, one for the database. The biggest problem with this is, that speed and scalability/redundancy/resilience tend to go hand in hand. I personally don’t like single points of failure, so having a pair of everything has become something of a personal motto.
“I’m a systems engineer, I can’t count any lower than two”

So if you’re concerned about uptime, then having a group of servers is a great thing.

This is a rather quick and dirty sketch I knocked up in mspaint.

I’ve been asked to provide the configuration information about this cluster, so I’ll copy that from my VM server later on. I tend to use my powerful desktop to build test infrastructures quite a lot, so knocking up and provisioning 4 VMs is no massive stress.

A note now on configuration management. I have been in the position in the past where the “How to build a server” information is in a wiki page somewhere. This is alright, but you do tend to end up with a difficult to document process.

Instead, I prefer a combination of Preseeding and Puppet. For no other reason than it’s what I prefer, we’ll be using Ubuntu 10.04 LTS in this article, although the processes for any other distribution aren’t too different. When it comes to Puppet, both Debian/Ubuntu, and RHEL/Centos work well. I haven’t tested Puppet with any other Distros, but I gather that it’s fairly well supported across the board.

Preseeding is the process of automating the steps of installing the operating system. It’s basically an unattended installation process, that tells the installer what you would have selected, had you been in front of every machine whilst installing. As you can no doubt imagine, when you’re building a farm of servers, preseeding is a massive bonus, and a timesaver.
Ubuntu, Debian, and Redhat-like distributions all have a mechanism for Preseeding a machine from bare metal. I suspect that other distros do similar things, but I’ve never used them in a production environment. Preseeding is a fairly broad topic, so I’ll cover that in a separate blogpost.

One of the things I adore about Puppet is the community contributed packages that are available at the touch of a button. I’ve built a fairly comprehensive puppet infrastructure from a majority of contributed modules and packages. I tend to just search Google for “puppet-” like puppet-apache and so on.
I, like many other sysadmins and systems engineers, am quite lazy, and have lately started to reap more benefits from the open-source puppet community. You can pretty much build an entire infrastructure for a simple LAMP stack, based solely on other people’s puppet configs.

Puppet is lovely, it really is very easy to get going with, just start off with a server as your “puppetmaster”. I’ve tended to go towards using Amazon EC2 micro instances for these, for small deployments. When I’m working on my VM network, i just use my workstation.

I use bzr for my source control for puppet. I like bzr, and it’s one of the best VCS tools i’ve used. It doesn’t matter what you use, as long as you use something.
But if you’re not using source control, then there’s bigger problems, and you need to rectify those first. Really.

Luckily, the vast majority of peoples contributed puppet modules and classes interoperate pretty well. Once you’ve got a decent setup for the /etc/puppet directory, and configured the puppet configuration itself, then the next bit is really easy.

A very basic guide to setting up puppet can be found here:
Bitfield Consulting - Puppet

Software we’re going to use:

Ubuntu 10.04 LTS
Apache 2.2
Varnish 2.1.4 (although, in this article, i’m using Pound, rather than Varnish, but I’ll detail Varnish in another blogpost)
MySQL 5.2.something
PHP 5.3.something
Wordpress? I think WP will be OK, actually.
Memcache 1.4.5

All of it, nice free, open-source goodness. We like open-source.

This is the full working puppet config that I use . It’s all pretty much gleaned from other puppet-* repos on github, and a few other places.

So.. In theory, you should be able to checkout a copy of the above, and put it in /etc/puppet, and you should get a working puppetmaster, and be able to initialise 4 nodes.

You will need to do some individual config, such as the loadbalancer setup, and adding vhosts for apache. I’ve found that if you’re building biiiig farms, with lots of the same stuff, then adding the vhost config to the puppet manifest is a good thing to do, but for 2 servers, this kind of manual config is very easy to do by hand.

So that’s about it. I think.
I know I've digressed momentarily from the main stream of this evening’s symposium [extra points if you know where this is from], but I think it’s for the better. There’s a lot of bits and bobs that I left out, and perhaps shouldn’t have, and some stuff I left in, that perhaps shouldn’t have been. I’ve been meaning to write this up for a Very Long Time, and hope that it might be of some use to some of you, albeit under a somewhat bizarre set of circumstances [and this..].

I’ll just recap briefly and say that when it comes to the server design for new projects, that a VPS server isn’t a total writeoff, but I have found in a number of instances that the IO performance is the biggest bottleneck on these virtualised systems.
That isn’t to say that every new project and infrastructure desires a 4 node LAMP server system, not by a long way, but if you actually have the traffic and requirement, and also the budget to do it, then having dedicated servers (or colocated/owned servers), then you’ll probably find considerably better performance than you would with a VPS.

A final side-note on general price and suitability, and something that’s more relevant to the first part of this article.
All servers are not created equal. A 1U server from HP, might set you back as little as £600, or as much as £2500. On the other hand, you could build your own 1U servers from a 1U case, and off-the-shelf parts, but the build quality will be lower.
Combine that with the fact that you don’t get any kind of parts warranty to the same extend that you do with business grade hardware, and that consumer parts aren’t designed for a 100% duty cycle.
It is possible to make a desktop machine into a server, and do everything on the cheap, but I highly recommend against it. Things will fail, you won’t get warning, you won’t get warranty and it won’t be pretty.

If you’re making money from your infrastructure, or the systems that sit on it, then you’ve got some mechanism of getting the money back from the outlay of “doing it right”. If you’re just building a lab environment, or playing with toy servers in your parents’ basement, then good luck to you, enjoy everything you do, but don’t host other people’s data on your toys.

High-availability isn’t something to look at lightly, it’s a pretty hardcore branch of systems engineering. You’re playing with the big boys now, and you need to have invested a similar amount of money in your hardware as they have in theirs.

Unless you’re Google, Facebook, or Twitter. But you’re not.

(If you are Google, Facebook or Twitter, then please leave me some insightful comments ;)

Velleman 8055 Drivers

About 5 years ago, I wrote some "drivers" to interface a Velleman K8055 USB interface card . After a recent request, I have decided that they should be brushed up a little, and reinstated with a download link.

I'm open to bug reports, but I can't promise that I'll be able to fix them quickly.

Instructions to compile/install

You'll need the following dependencies:
libusb-dev libqt4-dev qt4-qmake (ubuntu deps)

1) run qmake -o Makefile qcontrol.pro
2) cd lib
3) make all; make install
4) cd ..
5) make
6) plug in your 8055 usb board
7) sudo ./qcontrol
8) enjoy!

I'm reconsidering writing some python bindings for the libueib.so driver, stay tuned!

Legacies

The legacy of a nobody

I work hard, and try not to let the future bother me. I don't make 10 year plans, hell, I'm lucky if i know where I'll be in 30 days time. I don't make massive future plans, because I've found that everything can change massively week-to-week, and hastily made plans are often proved unservicable. I suppose this is par for the course when working in IT. There's a sheer unpredictability when working with the internet, long hours and late nights along the way, but basically all computer systems listen to Murphy's Law.

You will be interrupted mid-holiday, mid-wank, or if you're really lucky (or unlucky...) mid-shag. When you have a network to maintain, I like to think it's like a parent with a child. Especially a toddler. There's always the haunting suspicion that something will happen, requiring emergency attention, although these things tend to be a lack of disk space, rather than a pea shoved into an unsuitable orifice.

All that aside, these days, I find myself wondering more and more; "what will my legacy be?". That is to say, when I've died, how will I be remembered?

I'm gay, and genetically messed up enough to be unable to have my own children, this alone is something that troubles me occasionally. Not the being gay bit, but being unable to continue my family's genes. Under the circumstances, that might be a good thing.

In my father's and my own eyes, my grandfather was a great man. We both learned a great deal from him, both about engineering, and life in general. A great deal of my skill with metal and wood, and design comes directly from his influence. My interest in computing is mostly my dad's influence, at an early age, with an Apple IIe. I guess I'm just trying to say that my ancesters are greater than me. Greatness being totally subjective, of course, but I still get the massive feeling that there's something epic missing.

For a very long time, I looked at my peers who had had children with a sense of disdain. Perhaps a something wasted, by having spawned so early on, but I now realise, at least they have done it. At whatever stage of life, at least they had the chance.

Perhaps I could term my computer systems as my children. If that were the case, then I'd have had more than two dozen, over the last 10 years. Under this metaphor however, there's no longevity involved. Aside from some minor things which I know are still there, the majority of the systems with which I've worked have now ceased to be. Servers and Networks which I've designed and implemented have been replaced by smarter and faster ones. It's like a destructive evolution. One that leaves no fossils, no trace of earlier systems, and their designer.

I haven't published any papers, written any journals, sown my seeds of academic greatness (ha!), or computational excellence. If I died tomorrow, there would be very little to mark my place in history.

N's Story

After receiving such a great response to my own article (thanks everyone!), a good friend of mine asked whether I'd publish his similar story here.

If any of these stories give enough hope to just one teenager (or anyone) to let them survive the hardship of coming out, and homophobic abuse, then that's enough.

For various reasons which will become apparent as you read, N has decided to use this moniker to protect his identity.

So, without further ado, here is N's Story.

It Gets Better (No, really)

Reading Tom’s article (credit where credit’s due) inspired me to write my own It Gets Better story, to which as you’ve noticed I had to add an allusion to the fact that were it not for most of my friends (i.e. true friends, yes, again we come back to that careful wording, in some ways it’s even more important in my case) I’d be looking very lonely in my corner.

You see, being gay was unthinkable... literally.

I grew up with two parents who’ve both decided that I’m a poor excuse for a human being, as first my father and more recently my mother decided that disowning me was the better part of parenting. Now don’t get me wrong, I deeply love and admire my mum. I want to stress that, because you’re not going to like her much and I do want to insist that she has redeeming qualities.

For example, she was very supportive financially and always pro-active in standing up for me when I was bullied at secondary school (something I’ll come back to later in this article).

She also proved to be surprisingly OK with other people’s sons being gay... although in that respect she deserves to be cited as an example of what a NIMBY (Not In My Back Yard) is, because she certainly wasn’t OK with me not being straight.

One of the things that I found moving in Tom’s article, in respect to my own experience, was how similar our experience of growing up gay was.

We both had our first big crush at the same ages, i.e. 6-8 years old, in my case, my best friend’s boyfriend.

It’s ironic that her dad was openly homophobic and that the words “poofs” and “queers” being his very disgusted words on the subject, already hurt when it came up for whatever reason in conversation.

I suppose the difference in my upbringing and Tom’s was that no sooner did I realise that I Liked Other Boys at the innocent age of 6-8 (I only learned the word for it when I was about 10), that I was immediately hit by a wave of hostility, disgust and disapproval.

Things became even worse when my parents separated, as I was stuck in the middle and both of them saw me as an extension of their ex-spouse.

This meant that while sticking up for my mum at his house I was getting “you’re just like your father” thrown at me (and occasionally fists, frozen bread and on one occasion I was threatened with homelessness... well I was 16 by the time that last incident happened), and on the other hand I was getting “you and your mother” from my “dad” who then disowned me two years down the line when the divorce came through (I was 15 at this point).

I smothered my feelings the same way a rat eats her young if you disturb her. It was a way of protecting myself from the hurt that I was getting, and was partly a conscious decision, partly a defensive reflex.

Of course, this set me up for trouble, and just how much will soon become apparent.

My first head-on encounter with this concerned a massive crush on my then best friend when I was ten, a tall blonde lad (yep, another rugby/ football player) who had my heart skipping beats every time we spent time together and then broke it by saying “Oh God, you’re not homosexual are you?” A question provoked by a completely unrelated “I have something to tell you” that referred to a message from someone else.

I stammered a “No... No. I’m not.”

Yes, that is correct. I denied my sexual orientation and my love three times... and then the break time ended. Oh the irony.

There was a moment I’ll never forget concerning him though, this time for happier reasons. During a school trip, one of the other boys was being very very nasty about my obvious feelings for our friend and the fact that I was always trying to please him. And quite frankly my dear, I didn’t give a fuck.

I felt like my chest was going to explode with that warm feeling that radiates out of you when you’re happily in love and you don’t care who knows. And yes, there was a slight sexual element to what I felt (use your imagination).

Now the trouble that was brewing began when I started secondary school. Suddenly I went from being an unassuming pupil among others and occasionally a teacher’s pet, to being a teacher’s pet and a target for name-calling, stone throwing, being blamed for things I hadn’t done and a few things I just prefer to forget now.

As in Tom’s case, it lasted all of five years and seems to be what people I went to school with mostly remember me for. I had very briefly explored my sexuality with a couple of other boys my age in primary school (sorry to disappoint, but apart from the usual stuff most little boys do, such as showing each other our penises behind a wall, it didn’t go very far) but this was now out of the question because I was already a loner and made to feel it.

Noticing a girl in my class when I was 11 gave me an opportunity to try out romance, that might dare to speak its name, and I repeated this a couple of times over the next few years until I was 16.

In every case it was timid, furtive, and ultimately purely platonic. What I tried to convince myself were, crushes turned out to be a lonely teenage boy trying to A) be straight and B) make friends.

As they were always well out of my “league” in at least one way or another, and as I never got anywhere it was easy to think that it was just down to my isolation or for more noble motives (if they had boyfriends for example).

The fact that when one of them actually made a sexual pass at me I didn’t like it and ran away should of course have made it obvious that I wasn’t that kind of boy.

Unfortunately, the overtly homophobic context I was wading in, had started influencing me to the point that I went to great lengths to deny my feelings for other boys.

Given that I was already trying to hide my socially acceptable feelings for girls, you’ll appreciate why self-harming was only a step away whenever I so much as got a kick out of a hot Sixth Form boy or (male) fellow pupil smiling at me.

I don’t want to give too many ideas to anyone who’s potentially fragile, but the mildest example happened when I was 12. A Sixth Form boy whose name I didn’t know but for whom I was carrying a torch of Olympic proportions ran past and ruffled my hair with a friendly smile. I spent the rest of the day grinning from ear to ear with a happy look that lasted until I got home.

As soon as I got to my bedroom, I consciously realised that I was in love with another boy. Something snapped and I started crying and cut off all the hair he’d touched as if he’d somehow “made” me gay. I’ll draw a veil over the more painful and dangerous things I did to punish and “cure” myself over the years, but you get the idea.

You may have noticed that I kept pushing my attraction to boys to the back of my mind and that convincing myself that it was as dirty, wrong and even dangerous as my parents, grandparents, teachers (thank you Section 28) and classmates said it was, took up every brief second of the time during which another boy would catch my eye.

Obviously I had platonic relationships with other boys, and I had a few very loyal friends willing to risk being stoned in the Biblical sense of the word, which was the danger for anyone who ventured outside with me on the side of the school grounds by the playing fields.

But there were of course Other Boys (men didn’t interest me yet, unlike boys in my own year up to and including Sixth Formers) who I’d see helping their parents on market stalls on Saturdays or at the swimming baths (particularly in the changing rooms obviously) and for whom I’d have romantic feelings with a sexual element to them. This was something I definitely wasn’t equipped to handle; and it eventually led to my nearly having a nervous breakdown just before my coming out at 17, to a friend who’d literally come out to me a minute before.

Bisexuality seemed the obvious term to define my sexuality at this point, as the only reason I’d admitted to liking boys was that I couldn’t hide or suppress it, and liking girls sexually was never called into question because it was assumed you did.

I’d also spent years trying to fantasise about girls, and succeeding (use your imagination if you must) and on those occasions in which a boy I liked was involved. This (very unhealthy) pattern carried over into two encounters with young women, both of which involved me reacting to being pounced on and kissed by kissing back and letting my imagination get me aroused.

I’d spent years getting turned on by imagining straight guys I fancied doing things with girls, so it was only after a random conversation with my father-in-law and wife that I realised just how much of a mess I’d let my life become when it dawned on me, after years with the second woman (and paternity) that I’d been barking up the wrong tree since 11 years old.

I came out to my mother before I was ready, because the friend I came out to was threatening to tell her if I didn’t.

A word on this – A) please don’t submit to blackmail, and B) don’t let other people’s prejudices shape the way you see yourself.

My mother’s reaction to me telling her I was bisexual can be split into three stages.

Her first reaction was an encouragement to try with girls and the ludicrous suggestion that I spend more time with her ex-husband, followed by a very hurt and disappointed order not to talk to anyone else, and particularly my sister about it.

Her second reaction a few days later was to shove a leaflet on blood donation into my hand and to order me to read it *very* carefully. I think you get the message too. (Queer = AIDS = you die...) Her third reaction was a combination of avoiding the subject, one very brief “And you’re not gay” (nice to know she was listening) at the end of a comment on an unrelated subject, and a steady stream of homophobic comments in which the word “gay” was always used as a provocation and as an insult.

At the end of my first year of university she even told me that I was very selfish because there were rumours about my sexuality and she had to live with what the neighbours thought. So, on meeting a woman who loved me and whom I cared about deeply (and still do), I thought it only natural to be faithful and to build our relationship together. Whenever doubts began to accumulate, I dispelled them by telling myself that I was with someone who would please my family and that I was doing my duty both to her and to them. I didn’t realise to what extent I’d internalised my “education”, but that fact of course was going to open my eyes in its own time.

I got a lot of homophobia while at University, including having stones thrown at me by a couple of other students. Don’t be put off by this, because it doesn’t invalidate the fact that universities take this very seriously and other students will support you just they did in my case.

I should also add that your true friends won’t reject you and any gay or bisexual men among them will be more than happy to know that you’re being yourself.

One straight friend once said as much to me when I apologised for a crush on one of his friends from home. To be precise, he said “Nah, don’t be sorry. It’s best to be honest about your feelings. If he wants to be a woman about it that’s his problem!”

My father-in-law and my wife are both Christian.

I’m not.

They both believe that homosexuality is a sin, and so for reasons unrelated to me, when the topic came up during dinner, my father-in-law aired his view that “It’s obvious that homosexuality is a form of deviant behaviour”.

I replied that people who are homosexual aren’t attracted to the opposite sex and that, although I respect the right of other people to hold the same view as he does on the subject, I disagree.

What hit a nerve I think, was when I insisted that it’s in a person’s nature and not just a question of preference, as almost every gay man I know (I don’t know as many lesbian women) has zero attraction to women.

I’ll spare you the theology, but I was left with the nagging sensation of having accidentally breached a wall I’d put up at the back of my mind and it coincided with my attraction to women being questioned.

A few friends, bi, gay and straight alike had expressed surprise at me being with a woman, as I’d never shown any interest in girls before. Now this in itself didn’t bother me, because I believed, and still believe, that there are bisexual people out there. I know a fair few, and they’re very visibly attracted to people of both sexes.

What I realised, as I sat down with pen and paper and worked my way over my private life properly, was that I was looking at myself objectively for the first time since I was 10.

Assuming that I liked girls was an obvious conclusion to draw as a teenager, but as the final pieces fell into place, it hit me like a sledgehammer that my defensive reflex against any form of sexuality had impaired my judgement. The prejudice I’d picked up that love without desire was pure had led me to the conclusion that I liked girls and to hate my romantic and physical feelings for other boys. And of course hating myself for not being able to change, was an inevitable side effect and conveniently stopped me from exploring my feelings, let alone my sexuality.

And so the floodgates burst.

Years of sabotaging budding relationships with other young men from the age of 17 to 22 and over and running away from any gay or bisexual man who made a pass at me suddenly became painfully fresh in my memory. So did years of avoiding any kind of contact with LGB organisations beyond professional relations while doing “governmenty” work, as one friend who I later learned is gay referred to it, for the Union of Students.

Even my attraction to him was unrequited, as I decided not to ask him out (it turns out my hunch was right and that our attraction was mutual) to concentrate on sorting things out with my ex-boyfriend who’d recently been subjected to homophobia of life-threatening proportions... and in no shape to be in a relationship of course and therefore unattainable. And I’d accepted the break-up in the first place because my mum had just had a serious car accident... something I’ll always “kick” myself for, particularly now.

And at no point did girls interest me other than as friends. A point of which I was reminded by the memory of a speaker at an event in one of the Union bars when he said that at our university there was a ratio of something like 5 girls per boy. One young guy not far from me said “Shiiiiit....” in awe. As you’ll have gathered, I was unfazed by this piece of information, not being interested in girls but very much under the spell of one of my very tall, muscular and athletic (and straight, so nothing happened) flatmate.

Suffice to say I’ve managed to unwittingly come out several more times than I should have done, given that I came out to myself properly at about 10 and again at 14-15 and then again at 15 to a boy I fancied, or more precisely outed myself by replying “Depends. Do you?” to his hostile “Do you smoke helmet?”.

I then came out at 17 to a few close friends. Naturally the entire Sixth Form got to know and were almost unanimously OK with it. As Tom mentioned in his article, and as my then boyfriend pointed out, girls are generally very supportive friends (if not among the most supportive) when you come out.

Wives, understandably are not so supportive. It’s not a situation you’d wish on anyone, but as I can’t properly fulfil my role as a husband now that I’m conscious of what really pushes my buttons, all the arguments about my sexual orientation being an “abomination” and something “ I can change” if I really want to become meaningless.

I don’t hold with running away and abandoning your dependents, but staying together on the grounds that I signed a contract despite the consequences for my wife and child would be just as irresponsible.

The irony is that it was my wife who convinced me of that, by arguing exactly the opposite. Not least by equating the relationship I had with my (now ex-)boyfriend to the “marriage” that some people have with their dog, or comparing any form of LGB group with Sodom and Gomorrah. By using the kind of ignorant comment I hear so often when you tell “them” that you’re gay or bisexual - “You don’t look it.”

Now I’m not a coward when it comes to physical dangers, and I’ve survived enough people trying to grievously hurt me to be able to make that statement. Homophobic violence hurt me less than having to live with homophobic attitudes from people I actually care about, but conforming to what people expect of you doesn’t require courage even though it’s painful.

Living your life according to what’s right for you and others is much braver and also much more responsible. Not that I really stood a chance, but I did make several bad decisions, and trying to be straight when I thought I was bisexual was one of them.

When you get into your twenties (or any age) and realise that you’re living with the decision of a scared teen with no apparent support to turn to, it’s time to stop running and repair the damage before it gets worse.

If this sounds familiar, go back and read Tom’s article, that’s where to go from here.

Given my current situation it’s not easy to be optimistic, but that’s where my friends come in. I have to admit to feeling jealous when I see how accepting some parents are, but the people who love me for the person I am will always be supportive.

My best friend very tactfully let me know that he’d known for years and that he understood how hard it had been for me to get the words out.

One of my other close straight friends who’s built like a rugby player even asked “And how is that offensive?” when I warned him that it was common knowledge on campus that I liked guys and that some people seemed to think we were a couple. I really hope he meets a girl who deserves that much sweetness. Of course, friends of “other” persuasions who’ve been through the same stuff as I have can read this and smile now and again as their own memories suddenly become more vivid.

And while I’m building a future that takes into account the fact that I have the right to be happy too, friends like that are the ones who remind me that it does get better.

Oh, one more thing for my friends. Cheers!

It Gets Better

I can clearly remember the reactions of most of my friends when I came out. I've worded that carefully, notice.. Friends. All of my true friends were supportive, one guy, a bit of a rugby lad, put his arm around me and said "Well done mate".

Mind you, that was when I'd chosen to come out, of my own accord.

Truth be told, I'd actually been outed years before that. Let's take a look back at that.

I've known that I'm gay for a very long time indeed. I had a crush on a guy I knew while growing up in America, a kid called Evan. Let's see.. that would have been 1992-1994, so I'd have been ages 6-8.

Evan's folks had this massive "ranch-like" place a bit out in the sticks, or at least, that's how I remember it. He had this "secret" hiding place that only he knew about, and he took me there, and we did things, and I Liked It. (I'm being vague on details here, use your imagination if you absolutely must.)

In the very early years of exploring my sexuality, it was defined by points in time, with other boys my age, and "exploring" with them too. If I look back now, I can count about 6 guys who were similarly explorative, and this is still in the America Years. The strangest thing, perhaps because of our collective youth, was that nobody made a snide or derisive comment about any of this. It was just fun. It was never gay, queer, faggottry. Perhaps we didn't know the words, but I think that's not true. Perhaps innocence just wins out over all, and we don't know that it's "wrong" or "bad".

I think part of it is that we were mostly expat kids, at good schools, with good parents, and that kind of bullying and behavior isn't really part of the close community that had been built up. I don't remember any instance of persistent bullying at the Montessori School I went to. I suspect that might be because everyone lived in fear of the Headmistress, a seriously scary woman, who is still Headmistress there to this day.

But I digress.

In 1994/5, dad's overseas job ended, and we moved back to a smallish town in Worcestershire, called Malvern. I started school at the "West Malvern Primary", mid-way through the term, I think, and was widely accepted with wonder and confusion. A softly spoken, somewhat gangly kid with an american accent and foppish ways. I know this for a fact. I did not fit in.

I knew nothing of football, or well, anything that defines the british boyish primary school group of friends.

As a result, I did a better job of making friends with the girls, well, some of them. One friend I made then was Robyn, who has been my friend ever since, and was really supportive when I came out in 6th Form.

To the typical british schoolboy, apparently making friends with girls automatically makes you gay. I have to admit now, I've never followed the logic behind this one. I was a massive teacher's pet, and not particularly bothered about it. I enjoyed learning, and would have happily spent my lunchtimes pestering with the aging BBC Micros or reading away in the library. In fact, as time wore on, this is exactly what I did.

Around about 1996, I made a friend, a very boyish boy, he was on the school football team. Somehow (and I really wish I could remember exactly how this came about), we ended up "exploring together" too. First at his house, watching Neighbours while we did it, then at my house ,there was some element of computer gameplay rewarded with a wank, and this carried on for some time.

I once asked him, "What happens if I win the game?", and his reply has stayed with me for all these years, "I'll fucking suck your cock", he said.

Actually, there was another guy with whom I had some brief encounters, or perhaps another two, but they were fleeting, and largely unremarkable. (I never did win that game.)

The problem came a little later on, when we all started at a Secondary school (High School, for my american audience). Pretty much everyone who had been at the primary school went to one of two secondary schools. The bible-bashing Dyson Perrins, or slightly less mad, and much better The Chase. For obvious reasons, I went to The Chase.

This is where the trouble really started. Going from a small school in a good bit of town, to a much bigger school, with a massive variety of kids from different backgrounds and upbringings. This was terrifying. I still lacked the social skills to make new friends, and for a long time, was still riding on the old friends I had at primary school. The problem was, our timetables were largely different, and we didn't often see each other aside for a short period at lunchtimes.

Having a hard time making friends was made worse when some of the boys from my primary school had said something to the older kids, or the other kids in my year, about the experiences we'd had the year before. Apparently now it was wrong, and very bad, and I was queer, and gay, and I had big ears, and i was weird and not quite right.

Incidentally, whenever I got bullied, and told my parents, I always mentioned it based on the big ears fact. I still wonder whether anything would have been handled differently if it had been known homophobia at that stage. I suspect not, not least from the school's point of view. I just wasn't quite ready to admit it to my parents. Dunno what i'd have said under further questioning. It seemed better to base it on the tangible fact that, yes, I do have big ears.

I suspect my parents already knew that there was more to this story, but there it is, that's the truth.

To me, the scariest part of this story was that the bullying didn't stop for FIVE years. There was always somebody willing to make a jab at my alleged sexuality, or with some cover story, and make comments about the size of my ears, or the way I walk, or that I'm a geek, or a nerd, or any other derisive and derogatory remark they could invent.

Not all schools are created equal, I understand this now, but to the 14-year old me, this wasn't obvious.

The general pattern was this: Get bullied, tell someone, the bully gets a light ticking off, you get bullied for a) the original reason, and b) being a grass.

Now, the thing that annoys me most about all of this, retrospectively (although, it annoyed me then, too), was The Chase had a lot of "threats" against bullies. "We'll do this, and we'll make an example of you in assembly, or we'll name and shame you to the local newspaper". None of this was ever done, not in any of my time there, and I suspect little has changed. A look at the school's website today, they seem to have a clearer code on equal opportunities, and the bullying code hasn't changed. I do wonder if the prevalence of homophobic abuse at the school has changed.

I'm not afraid or ashamed to admit that on a number of times, I considered suicide, but ultimately, the reason I'm still here today, is because I didn't want to devastate my family. I don't want to go into detail on that for a number of reasons.

Let's jump forwards 2 years, to 6th form (or college), and I'm largely more comfortable with my friends group. The people around me are those who I want to be around me, and the folk who bullied me are now elsewhere, working in dead-end jobs where they will be for the rest of their lives.

I joined a youth group in 2000, Malvern Young Firefighters, and met a great group of people, from outside school, some I knew from primary school, others were unknowns, but the group, and leadership allowed me to develop a new sense of self-confidence that had formerly been destroyed by years of bullying.

There was rarely any bullying in this group, I think, for two reasons. 1) The group leaders ruled strictly, and 2) the peer group was much more tightly knit.

Everything changes again at University.
I went to Birmingham, and I spent the better part of 4 years there. It's funny, you go from having hidden your sexuality all through school and highschool, to an environment where not only are there a few people who look like you, and think like you, and fuck like you, but there's hundreds of them. Birmingham has one of the finest, and friendliest LGBTQ societies that I've ever found.

These people made me feel welcome, they protected and educated me about what it's actually like being gay.

It's all different, and it gets better. It really does.
You too will find people like you at university, and in bigger cities, and in liberal arts colleges. You will find your first "real" boyfriend, and you'll go through everything that your peers at school went through at age 12 with a girl behind the bike-sheds.

I'd never have met all the wonderful people I've met in the last 10 years, if I'd let the bullies win. I'd never have met my wonderful boyfriend if I'd died when I was 14.

First loves, first lovers, first boyfriends are all things that will happen, and can happen, but you have to give them the chance.

Here's the important point, they're all right, Joel Burns, Tyler Oakley, the numerous others on the Trevor Project and youTube. It does get better. It got better for me, and I promise you, it'll get better for you.

Sometimes you have to make a pro-active stand, and get yourselves out of the situation, other things just change over time, like people's attitudes to homosexuality. It's a continuum, and it's changing all the time. You just have to give it time to change. The vast population aren't perfect like us, they can't see the world the way we do, but the reason people are homophobic is because they're also ignorant, and they're afraid.

If you've been reading this, and thought at any point, "Hey, that's me!" or, "That's what they do to me", and you're being bullied, for whatever reason. Please don't suffer in silence, there's no need. Times are changing, and we live in a progressive world. There is somebody out there willing to listen. There are people who've been through the same things. There is support for gay teens, hell, there's support there for anyone.

Please give yourselves a chance for things to get better.

It gets better. It really does.

If you're in the USA:

http://www.thetrevorproject.org

If you're in the UK:

http://www.lgf.org.uk/

http://www.samaritans.org/

London Gay & Lesbian Switchboard:

http://www.llgs.org.uk/

List of local LGBT support groups / helplines:

http://queery.org.uk/StaticPages/Advice.asp

Mysterious Tiles

I recently acquired some ceramic tiles, and after a good bit of cleaning, they're all presentable and nice. There's 30 in total.

Problem is, I'd like to know what the pattern is, who the designer/manufacturer was, and also, do they have any value.

Some friends and family have suggested that they: "Look over 50 years old", "Look like a morris pattern", "look handmade", "look valuable", "mediaevally beautiful", "worth finding out about".

Pictures on Flickr:

http://www.flickr.com/photos/tom_twinhelix/4704457478/

http://www.flickr.com/photos/tom_twinhelix/4703819869/in/photostream/

Zen and the Art of Speccing Servers

Say for example you want to build a new Virtualization cluster. You've chosen the CPUs you want, and know you want 32 GB of fast shiny RAM.

The next thing to decide on is how the hell you're gonna store your VMDK (or otherwise) images, and then store the backups and snapshots too.

So. A typical VM Host server might be one of three choices.

For sake of argument, i'm using Dell as a vendor.

Option 1:

Dell R805, Dual AMD 2425HE, 6 cores per CPU, 2 CPUs.

32G of fast DDR2 ECC RAM.

Ah. Hard disks. Bugger.

You can have only 2 disks, in the R805 chassis. Bugger.

I'll have 2 fast SAS 300GB 6Gbit 15K 2.5" drives, in RAID 1.

Bugger. Only 300GB of storage. That's about enough for 3 small servers, or one big one.

Bugger.

Approx Cost: £3100

So, If i want to use the R805, i'm gonna need some kind of backend storage, be it NAS, or SAN, or an Unified Storage Device, providing NFS and iSCSI.

Option 2:

Dell R815

Dual or Quad CPU, also AMD, 8 or 12 cores per CPU.

32 G of RAM, again

More disks!

Split volumes, R1 / R5 (shame it's not R6, but there we go.)

2x300GB SAS + 4 x 500GB SATA

Giving 300GB + 1.3TB

A bit better, but prohibitively expensive.

Dual 8 Core CPU = £6208

Quad 8 core CPU = £6698

Dual 12 Core = £7408

Quad 12 Core = £8608

Bugger.

Option 3:

Dell 2970

Dual 2425HE, again

32 GB RAM

Option A (8x2.5" disks)

2 x 300GB SAS + 6x500GB SATA

= 300GB + 2.3TB

Total Cost : £ 5125

Option B (6x3.5" disks)

2 x 300GB SAS + 4 x 2TB SATA

= 300GB + 5.7TB

Total Cost : £4705

2 x 300GB SAS + 4 x 1TB SATA

= 300GB + 2.8TB

Total Cost : £4145

Right. Now. The interesting part is that this last server, the cost of storage alone, is only £191/TB.

One of the biggest problems associated with having large disk storage on the actual VM host itself, is the problem of not being particularly able to free up pockets of unused disk space.

Alternatively, a separate storage node would effectively allow better distribution of the storage, and exporting disks across the network.

So let's price that up.

From Broadberry.co.uk

(Because I like their up-front pricing, and shiny configurator)

Supermicro chassis, Intel server mobo, Intel Xeon E5504, dual CPU, 24GB RAM

6x300GB 15K SAS = 1.3TB

6x 2TB SATA = 9.5TB

Total Storage: 10.8TB

Total Cost: £6528

That's about £605 per TB. Not ideal.

But there's no real doubt that using iSCSI (or NFS) would provide masses more flexibility for the provisioning of storage for this project. Because the initial plan involved high-availability, using IP-based network storage protocols would also allow the disk-traffic to be routed across the public internet, using some kind of VPN technology.

My gut feeling is that the best solution is a cheap(-er) server, backed onto a more expensive disk storage unit.

I did consider pricing up a DAS array, and connecting it to one or other of the VM Hosts directly, either by FC or SAS, but then in the remote case of the failure, the disks aren't easily exportable to another server. Especially as SAS traffic can't be directly routed over the network.

The Cost of Forward Thinking

In the last two weeks, I've seen at least two websites fall off the internet because of a distinct lack of forward planning

Firstly, there was Derren Brown's blog

After Derren did his "The Events" trick with the lotto balls and dark magic, the number of fans hitting his page daily looking for clues, news, and gossip, caused the server to fall over. It even caused some of the channel 4 servers some traffic troubles (and they've got a lot of nodes!)

Derren's blog was down for at least 2 days, as far as i could see. If his producers/agents/IT manager had said "hey, this stunt might turn out to be popular, let's move onto a cloud infrastructure, with a CDN cache, we might have to invest a bit of money now, but we'll have better uptime than if we're just serving from a single 1U Dedi in a rack" then the site may have remained up and serving for far longer, to endure the wave of traffic generated by the publicity on tv.

The second one of these, was caused tonight by Dragons' Den Online, a cut-down version of the popular Dragons' Den format.

The final segment was dedicated to a web startup, introducing Yet Another Social Network for families.. Something about sharing photos, videos, calendars and wishlists
Personally, I do all this with Flickr, Google Apps, and Amazon Wishlist.

It was remarked to me at least once, that this could be breaking down the nature of the family unit, because everyone spends their time in front of the computer instead of actually interacting with each other

But I Digress

About 20 minutes ago, I was looking at their site, Family Fridge, and noticed that it winked out of existence as soon as the web address was mentioned.

Yes, they got Slashdotted by the BBC

I've seen many a site get taken down by getting a FryTweet, that's a pretty effective way to kill a webserver, when 50,000+ followers all open the site at once, it's not good for any website

I suppose there's that old adage about "no such thing as bad publicity"... I can't help but apply the same scenario as before
"If we spend a little money now, get a cloud computing services infrastructure, then we can use the Dragons' Den as advertising and get a whole stack of new members in one night"
Sure, upgrading the platform isn't free, but the potential in increased revenue from such a "publicity stunt" is significant, and should be enough to offset the cost of the new infrastructure.

Moreover, I think it proves to some extent that the investment might not be quite so sound

Scalability is something of a buzzword of the times we live and work in, but it's also very important, the moment you launch a product on twitter or facebook, you've instantly got a far wider audience than perhaps you initially anticipated

In my opinion, it looks kinda bad on the developers of this site, that either they never anticipated that this would happen, or they don't care.

On a technical note, they probably wouldn't need to go as far as a cloud-computing infrastructure, or even a CDN.
Simple page optimisations and front-end caching can make a world of difference to generating a new dynamic page for every single visitor.

Knowing my luck, someone on Twitter or Facebook will pick this up as "Interesting" and i'll get a hundred requests a second, and my poor overworked hosting account at Streamline will get overwhelmed.

The True Age Test

A few weeks ago, I wrote about this facebook meme, “The Name Game” and I hypothesised that this wasn’t a meme, but actually a data gathering exercise, possibly started by scammers.
I’ve found another one. One of my friends took the “True Age Test”, and came out younger than their actual age. I’ve just had a brief flick through the questions.

Starting off with fairly harmless, questions which are related to the app, “What is your actual age, what race are you, how much exercise do you get” etc…

Rapidly progresses into “Have you ever had any heart conditions, did anyone in your family die before the age of 60 from coronary related illnesses”

Later, “Do you have diabetes. Do you have any Digestive problems, Do you use drugs, How depressed do you feel, What is your relationship status” and so on.

Now, not only are these questions a bit personal, but there is no obvious information on how your data will be stored, or used, or archived. Given that facebook already shares a good proportion of your personal data with these applications, what is the probability that you’ve just answered enough data to build up a probability report of how much a risk you would be to a) a future employer, b) a bank, building society, etc or c) an insurance salesman.

It also doesn’t state (nowhere that I saw, anyway) what they’re gonna do with the data, Is it transient, or stored in a file somewhere. How long is it stored for? Do they plan to sell the data? Domestically, or overseas?

Also, without a comprehensive code review, it’s not very easy for people to see whether the data is going to be exported through a backdoor in the code, so even if they say “Oh no, the data isn’t stored, or identifiable”, there doesn’t seem to be any easy way to prove that.

IIRC, Facebook don’t ask to see your sourcecode to the application, so it might be quite easy for an individual with malevolent intent to gather a vast amount of potentially sensitive information quite easily.

The motivation for people to participate in this application is simple “I want to prove that my ‘real age’ is younger than my biological age, therefore I feel good about myself”.

We all want to feel good, don’t we?

But at what cost?

Drabble

I wonder if you’ve heard of a Drabble?

A drabble, simply put, is a story, normally science fiction or fantasy that is exactly one hundred (100) words in length. No more, no less.

Here is mine:

It was a slow day in the spaceport.

“These rocket cowlings aren’t going to fix themselves”, Simon thought to himself, wistfully.

It was 4 days since the incident, nobody said a word after it happened, not until this morning, that is.

Simon knew exactly what to do; he lifted the great copper mallet above his head, and struck the cowling with all his might.

The resonance shook the entire rocket, the mallet, his arm and the rest of his body. “Damnit”, Simon swore, just as a shadow appeared over Simon’s left shoulder.

“I owe you a pint, for this”, the shadow said.

The Wiki Problem

I love collaborative websites. Wikipedia, Blogs, community oriented stuff like Stack Overflow and ServerFault

There is however, the lingering problem of vandalism, and it’s one that seems to crop up on pretty much ever collaborative website i’ve ever seen. Wikipedia has a lot of newbies contribs which are utter nonsense, advertising, spam, page blanking and so on. There’s a hefty team of people on Wikipedia however who go around reverting this kind of stuff. I’m one of them. I use mediawiki at work also, so I’m pretty confident around the entire wiki platform, and IMHO, mediawiki is the best wiki software out there.

Anyway, on Saturday, I was quite pleased to discover that the Science Museum in London has now got a collaborative object wiki.

I love the idea of having visitors add their own memories of stuff that is on exhibition. It seems that it’s mostly household items that are well commented on, for example Frigidare Refridgerators.

It was on this site, on saturday that I discovered that they had fallen to the terrible plague of edit vandalism, and the homepage of the wiki was now some statement about some girl called Louise and her love of turkey and cannock. It seemed she had also discovered her User Page, and decided to spread the nonsense to the public home page.

I created an account, reverted her edit, left her a message on her talk page (Sometimes these passive-aggressive things are all you can do!), and then had a rather nice thank you message from one of the administrators.

I think that might have been my 8 or 9th visit to the science museum. I’m forever discovering new stuff there.. That, and they keep adding new stuff :-). I’m quite looking forward to the future “Biker Tribes” exhibition, as I’m rather mad about motorbikes these days (more on that soon! [Sidenote: Anyone following me on Flickr might be interested in my Motorbikes Collection]).
There’s much more I could say in praise of the Science Museum, but I haven’t time, or pixels left.

Uncrackable Passwords

I got an email today from some software company.. Trying to sell me a password management tool. I used to use KeePass which was pretty effective. This one is considerably more expensive. Among its features, it boasts:

Generate uncrackable passwords using the integrated Password Formulator
Maximum protection of your sensitive data thanks to the security algorithm Rijndael 256-Bit!
Instead of passwords like “toothbrush” or “Rover”, which can both be cracked in a few minutes, you now use passwords like “g\/:1bmV5″£$p’}=8>,,/2¬%`CN?\A:y:Cwe-k)mUpHiJu:0md7p@<i” (with a 1-GHz-Pentium-PC, it takes approx. 307 years to guess this password!).
Password lists on the internet: Place your encrypted password lists on the Internet and enjoy access to all of them, no matter where you are!
Protection from keylogging (intercepting of keystrokes) – All password fields are internally protected from keylogging.

I’ve got issues with all three five points above.

That’s a pretty bold statement to say that your passwords are uncrackable.. I suspect they really mean that they haven’t been able to crack them, or somebody hasn’t been able to crack them YET.
Another word for Rijndael… Yep, AES. Really nothing that sophisticated. Under closer inspection they’re really no better than the free alternatives.
While “g\/:1bmV5T$x_sb}8T4@CN?\A:y:Cwe-k)mUpHiJu:0md7p@<i” may be long, secure, mixed cases, characters, alphanumeric, and symbols, it’s certainly not memorable. So what happens if you generate this password for XYZ internet banking service, and then you go on holiday and forget to pay a bill, or need to move some money about.. You don’t have your password safe with you. Bugger.
Does anyone else think this is potentially asking for trouble? Assuming XYZ company is hosting them, “securely”, how can you prove they don’t have a backdoor to decrypt the files. Do you trust them? Considering you’ve paid €30 for this package, it’s not really as binding as a really expensive legal SLA.

The other thing that’s at the front of my mind now, is what password do you use to lock the password safe? Do you use a long, complex, difficult to break one, which you’ll probably never remember, and will need to write it down (therefore making it totally pointless anyway), or a simple short password like your first pet’s name, and some thoughtful numbers after it.

Sidenote to point 3. 307 years on a 1GHz Pentium.. What about a dual-quad core Pentium Xeon. Or a distributed attempt across 256 nodes of dual-quad core Xeons. Still, it’s reaching a bit far, but it doesn’t mean that this password is unbreakable. Not by a long way.

Uh, right.. So this software is going to prevent me from putting a PS2/USB hardware keylogger between the PC and the keyboard? I think not. And if it claims to protect against software keylogging, how could you prove that it wasnt a keylogger itself. It would be a pretty ingenious way to harvest credentials, make the user believe they’ve just bought a security enhancement, really they’re buying a back door. (I’m not saying that’s what they’re doing, but it’s certainly enough to make me want further verification of the publisher’s honesty.)

I really don’t like the sound of this software, actually, I’m not keen on this “credentials management” type thing at all. There’s too many unanswered questions. And that’s before we get onto the rather open question of the use of biometrics for passwords. There seems to be a growing trend at the moment where biometric data (fingerprints, webcam images, iris scans) provide the password data, as opposed to the identity data that is then confirmed with a password.

Private keys and passwords are easy to change when compromised, but how do you change your fingerprint, facial shape, or iris detail when your credentials are compromised?

Epic Fail

Another lesson learnt by a company who really should know better.

Raid != Backup.

This might be widely regarded as old news, but it’s not too late IMO for me to add my $0.02.

I picked this up on Slashdot about 20 minutes ago, and there’s a few things that strike me as odd about the whole malarkey.

Before I go any further though, I’ve never heard of Journalspace until this article arose, then again, they’re not really in my general field of view, I’ve always had my own blog, on my own space.. so it’s not really my ‘thing’.. Anyway, one thing that is my ‘thing’ is data security and assurance.

Journalspace is no more.

DriveSavers called today to inform me that the data was unrecoverable.

Here is what happened: the server which held the journalspace data had two large drives in a RAID configuration. As data is written (such as saving an item to the database), it’s automatically copied to both drives, as a backup mechanism.

Anyway… A few things strike me as odd.

Was that RAID really their ONLY backup?.. for a site that had been going for 6 years, and probably had >1000 users, I’m surprised that they didn’t write backups into their disaster recovery plan, and/or their plan to scale their site to meet their user’s needs.

Blaming OSX? That’s a bit of a low blow. I’ve never used OSX Server for webhosting, but it seems reasonably unlikely that this is the cause of their problems. And even if it was, that’s still NO excuse not to have some form of backup.

Disgruntled Employee Syndrome. While it’s not always possible to keep 100% of employees happy for 100% of the time, it is reasonably easy to revoke root keys on servers, delete user accounts, remove privileges, etc when the employee leaves the company. It’s like not taking their office keys from them when you escort them from the building. “Come back and steal our data, we’re practically leaving the whole office open”.

On a slightly different note, RAID is not Backup, it’s part of the solution.

Well, actually, it’s closer related to high availability and redundancy, but that is a different story.

Off-machine backups are the key here. While they’re costly, and time consuming to set up, they’re also an essential part of the plan to maintain and scale.

Mirrored disks on a RAID array will restore data fine if one of the disks fails, but if you run rm -rf /path/to/raid/folder, then the RAID will just mirror that command on each of the disks. Bye bye data

I think it’s somewhat unfair to expect all users to keep full backups of their own data. Not outrageous a demand, but you kinda expect at least some form of data storage, so that your 6 years of precious thoughts and feelings aren’t lost into the ether one day.

Given that each user might have 6 years of blogposts, maybe 2 a week?

For 10,000 users, I make that about 600GB, averaging 100k per blogpost.

(6*2*52*100*10000)

Still, that’s not a vast amount.. Could fit that on a fairly small DLT tape.. Could easily replicate that across a few servers, seperated geographically, you could write it to a shitload of DVDs, few dozen blurays, Shove it on a portable USB HDD and stick it in a fire safe in the CTO’s basement.

Hell, it’s still less than a terabyte of data.

Chip, Pin, Password...

The list goes on. It really doesn’t end there.

Anyone who uses internet banking these days will find themselves handing over a vast array of numbers and passwords, authentication tokens and browser cookies. You have a card, this has a chip, you have a Challenge/Response card reader, and you have a pin.

There’s at least half a dozen banks in the UK that I can name who use the Challenge/Response type card readers.

To log into my online banking, I need my Passwords, Pins and if i want to do “advanced functionality” I need my card and challenge auth reader.

Now.. this is all very cool, I don’t mind the CR device. No, my beef is with SecureCode.

MasterCard / Natwest have licensed this “extra level of auth” for online transactions.

Dominos pizza.. one of my favourite food retailers on the web have a requirement that I use my SecureCode password to authenticate that I’m not a thief when i want to eat pizza.

This is not helpful. The SecureCode password can’t be any of the ones you already use for phone or netbanking

and has to be >8 chars alphanumeric. no symbols it seems ..

Ok, so you can’t remember it.. no problem, you just enter your DOB, some card details and it lets you through..

How is that any more secure than just plain card details auth? IF anything.. isn’t it less secure because it’s loading a seperate site in an iframe on the retailer’s website?

Why can’t i just use my Challenge/Response card reader and have everything work together?

And secondly, Why can’t I use that to login to online banking?

If you work for natwest, mastercard, or any other monetary establishment, do pop a comment in and explain why the system is so archaic, and to be frank. SUCKS.