About Matt

Cloud Systems Engineer at Reed Elsevier; cloud computing advocate, rock climber, swing dancer, amateur photographer, professional idiot....

2015: Yet another year in review

Sometimes it seems that these ‘year in review’ posts are the only contents in my blog; a once a year tradition to justify the continued existence of my blog, domain name, and hosting. I get the feeling these are more for my benefit than anyone else, so forgive me if they become ever more narcissistic.

I usually start writing these posts on the train home after Christmas; a journey that is usually more protracted than the train journey out! The fact is, this is often the only time that I have to catch my breath and think about what I have been up to the past year. I think the fact that I find little to no time to compose blog posts any other time; is a good sign of a busy and exciting lifestyle… or perhaps just that the daily grind of London commuting drains me of the will to write.

Home is where the heart is

Although I purchased my flat last year, I was not able to move in until early this year. I absolutely hate moving house, for me it really is one of the most stressful and nerve shredding experiences. The run up to my moving date was wrought with anxiety and  stress, not help by the fact that:

Sky broadband disconnected me 10 days early

Just a side note on this: seriously, you call up a company to tell them that you’ll be moving out, and they say “no problem, but your contract runs to the ends of the month so we won’t disconnect the line until then” (1 week after moving out); you expect to have no issues. Well Sky, much like the rest of Murdoch’s bullshit empire, like to talk nonsense and take the piss! Instead of honouring this statement, they initiated the cut-off process summarily, leaving me without internet. My complaints yielded nothing but empty apologies and explanations as to why they could not fix their mistake. One to avoid, don’t be tempted by their ads and special offers.

Anyway, I digress! After using my mobile to tether the internet for a week and a half, I moved in on January 24th.

The flat is far from finished; I have big dreams for the kitchen, bathroom, and garden; but such dreams are not cheap. They will all come in due time. I did, however, spend almost a month in the loft, scraping out the antiquated, paper mulch insulation and replacing it with modern glass wool, which British Gas then topped up for free.

So finished it might not be, but it is at least very cozy and warm.

Summer swinging back-to-back

What would my summer be without swing dancing?  I had little time off work during the start of the the year, so instead I used the bulk of my leave during the summer. I returned to the azure, clear waters of Halkidiki in northern Greece for my second time at Swing ‘n Swim dance festival. I knew less people there this time around but that simply meant I met more new friends. This time around I did not leave the country right after the festival, instead spending two days in Thessaloniki; an interesting place, though watch out for the wild dogs!

When I did leave Greece, I did not head home, I went to Como for another week of swing dancing by the waters of Italy’s great lakes. It was good to be back, although by the end my dancing was subordinate to my gelato eating.

After two whole weeks back-to-back I was shattered, but I regret nothing and would do it all again.

September swing wedding

Two very dear friends of mine got married in September this year. I met them both swing dancing several years ago, and was honoured to be invited to their wedding; where the happy couple performed a phenomenal first dance which turned into a “snow ball” dance with the brides maids, myself and the other swing dancing guests.

Congratulations to Matt and Jenna Joan Haggar!

Time for a change (again)

Once again I found it time to push the envelope with my career; I was perfectly happy where I was, the people, the work, and the office were all great… especially the office with its 31st floor view of London. I will miss that view, but one cannot arrest ones career for the sake of a beautiful vista.

Currently, if you work in IT in London (or possibly elsewhere too) you will be familiar with the feverish and constant solicitations from recruiters; phone calls, emails, LinkedIn messages and invites almost daily. I usually look over these with disinterest if at all, however, one such email caught my attention, and let me to a new job at Reed Elsevier at the end of November.

Cars, swans, and soldiers

I never thought that I would enjoy the ballet, but it turns out I really do. I went to see Matthew Bourne’s The Car Man (not strictly a ballet itself, but based on a ballet version of Georges Bizet’s opera Carmen) at the Sadler Wells Theatre. It was a very contemporary interpretation, but thoroughly enjoyable.

I also went to see two classic ballets; Swan Lake at the English National Opera, and The Nutcracker at The Royal Opera House. Both were fantastic, though the latter was my favourite; I found the story and settings a little more interesting. I love the whole experience of going to an opera house, the dancing, and the music too. I’m sure I will attend some more next year.

Czech please!

I couldn’t leave it at just two swing dance festivals for the year. So, rather short notice, a good friend and I signed up for the weekend workshops for Prague Swing Xmas.

I had never visited the Czech Republic before, though I had heard great things about it. I was not disappointed; the classes were great, the venue for the parties was decadent, and the dancers were all lovely. Oh and the beer was wonderful and also cheap!

Lets party like its 1926

Sound familiar? Well the venue was different, though the theme was much the same as last year. A 1920’s band playing to a crowd wearing pencil dresses with tassels, headbands with feathers, tuxedos, and bow ties.

This year we were at the Booking Office at St Pancras Renaissance Hotel. The band played a little fast for some of us Lindy Hoppers, though the experienced Balboa dancers didn’t seem to mind. Either way, it was a lovely atmosphere, and a great way to end the year.

Synology Disk Station failed to update network config with new router

I have a Synology Disk Station DS411slim, which has been working flawlessly for almost two years; that is to say, that it has never crashed, corrupted data, failed to keep up with streaming demands etc.

The one issue I had was after I switched internet providers, and thus replaced my router. Everything seemed fine at first, the box used the same static IP address and was visible to my desktop devices. I only noticed something was wrong when I attempted to check for updates in DSM.

Connection failed. Please check your Internet connection

I logged into the box via SSH and found the issue after a quick poke around the system. The new router used a different IP to the old one (192.168.1.1 instead of 192.168.1.254) and the box had not updated this for either gateway or DNS. There may be a way to do this from within the DSM itself, but it was easy enough to resolve by hand.

Firstly, fix resolve.conf

Then fix the default route and make sure it uses the correct gateway even after a reboot

This should be all that is required to fix the problem, however, there are quite a few other files that hold a reference to the old router, so I updated these as well just to be neat.

After that the box was able to resolve hostnames and access the internet again, even after a reboot, and DSM updates worked correctly

DSM 5.2-5644 is ready to update

 

 

2014: Where did the time go?

¿Qué lenguas hablas?

What languages do you speak? Before this year, my answer to that was… “English”. Now, I can say “English and a little Spanish” (in at least two languages). Learning a new language was a big deal for me, I was always appalling at French in school. In a world [London] where almost everyone I know speaks several languages, I was determined to no longer be a mono-lingual Englishman. After a year of Spanish classes, I am hardly ready for translating Don Quixote, though I can manage simple texts and hold a conversation. Many thanks to the excellent teachers at Cactus Learning!

Heading south

With my new found linguistic skills, I went travelling in the south of Spain, not once but twice, this year. This was partly to practise Spanish, but also just to experience Spain and part of it’s rich culture. During my two trips I managed to visit Madrid, Córdoba, Cádiz, Sevilla, Málaga, Tarifa, Ronda, and Granada. I met many interesting people in the hostels; a pair of Argentinians who helped me with my Spanish; some crazy Israelis who taught me how to cook a traditional middle-eastern dish while on a beach; quite a few Germans; and even a swing dancer I knew from London! I visited lots of castles, cathedrals and museums; walked along the beach by the Atlantic and the Mediterranean; went hiking in the Sierra Nevada; got just a little bit sunburned; ate lots of amazing food… oh and went swing dancing!

Swing dance addiction

Speaking of swing dancing. I did quite a lot of that this year too (big surprise!). This years highlights included; the mainstay, The London Lindy Exchange, as fun and well organised as ever; The Ahoy Ball, dancing on a boat on the Thames; and Swing & Swim! A week long dance camp at a gorgeous beach-side campsite (with bar and restaurant) in Greece. I did the workshops here too, the teaching and general level of dancing was amazing. Kudos to Lindy Hop Greece for organising it so well!

Climb the highest mountain

Ok, so I didn’t climb any actual mountains but I did get introduced to the wonderful sport of Bouldering. That is, indoor climbing without the aid of ropes or safety harnesses. Intense physical exercise is important in life. Some enjoy jogging, others enjoy doing reps at the gym. For me these are not exciting, but climbing really hit the spot and although my fingers, knees and feet absolutely hate me now, I have become somewhat addicted to this activity.

Buda-ful Budapest

I know, my puns are still terrible, I’m sorry! I missed out on Lindy Shock, in Budapest this year, but I went on a work trip instead. I still got to explore, take in the impressive architecture, and bathe in the famous public baths. The food and wine in Hungary is divine, I enjoyed it all. Although I missed the festival, I did manage to find a local swing dance night and met the local dancers.

Home sweet home

So here’s a first: I bought a flat!

I’ve only been looking for about 15 years but I finally made it. I have spent a lot of my time in the past month or so stripping paint, sanding, painting, and fixing. All of which is surprisingly satisfying when you know it’s yours.

Lets party like its 1925

As the final hurrah for this year, some dancing friends and I went to the 1920s themed, Prohibition Party in Bloomsbury. I always love these sorts of events, as everyone is dressed up to the nines, suitably merry with gin and fizz cocktails and generally having a great time. It gets even better when we all hit the dance floor, which never fails to impress the uninitiated. I won’t lie, my ego loves being the centre of attention, but I also love seeing people getting interested.

A perfect end to a year that, for me at least, has been truly great. Thanks to everyone of my friends who have been a part of it.

Automatically remove dead AutoScale nodes from Chef Server

So you’ve created an autoscaling group in Amazon AWS, when it scales up you’re nodes automatically authenticate with Chef, configure and deploy successfully. That’s great! At least until you need to scale down again. Chef nodes are good at telling the server they exist, but not so great at telling it they’ve checked out. This becomes a real problem if you try to use knife to issue commands to all servers of a particular role for example. You’ll end up with ever increasing stale data about nodes that no longer exist. So what can you do to fix this?

Luckily, AWS provides most of the tools for you, you just need to stitch them together.

What you will need

  • An AWS autoscale group
  • An AWS simple notification service (SNS) topic
  • An AWS simple queue service (SQS) endpoint
  • A script to consume the queue

Topics and Queues

First of all, create your notification topic in SNS, giving it a title (I used AutoScaleDown) and a display name. You can do this in either the web console or the command line tools. Next create a queue in SQS, giving it a name ( I used DeregQueue) and leave all other options as defaults.

SNS Topic Create Subscription

Now you have the notification topic, and the queue endpoint, you need to “subscribe” your queue to the topic, so that any notifications that come into that topic are passed to the message queue. The easiest way to do this is to note the queue’s ARN (Amazon Resource Name) and then “Create Subscription” for the topic in the SNS section of the console.

You can test that the subscription is functioning by sending a test message to the topic, and checking that your message gets onto the queue. If you do this, do not forget to delete the message, as it will interfere later on.

AutoScaling Notifications

Autoscale notificationNow that you have a notification topic, you should be able to create a notification for your autoscaling group to send to this topic. This can be done in the EC2 section of the AWS Console, or with the command line tools. Either way, ensure that the notification happens only when an instance terminates.

 Consuming the queue

So at this point, when your autoscale group scales down and terminates an instance it should trigger a message to your topic, which should then end up on the queue endpoint. This is great, but there needs to be something consuming these messages. You could use any viable scripting language you like, although I found that Ruby was the easiest tool for the job as it has decent AWS libraries and execution of shell commands is a breeze.

The messages are in JSON format, and contain various details about the scale down event that just occurred. The only important bit of information for this example is the AWS instance ID (which doubles up as the chef node / client ID).

The script needs to do three basic tasks; check for messages on the queue, extract the instance ID for each message it finds, and then run the necessary shell commands to remove the nodes from the chef server.

I installed my script on the chef server itself, although any system that has knife configured correctly would suffice. I then created a cron job to run the script once every 30 seconds. Works like a charm!

Here is my script in full:

 

2013: Of dreams and nightmares

The year that triskaidekaphobes had been dreading for decades has finally come and gone. No longer will they be reminded of their peril every time they have to write down the current date. The rest of us can draw a line for a new start and reflect on another year past. Not soon enough for some, and tragically too soon for others.

An ode to Judith Pote

The year didn’t get off to a good start, with the news that my aunt, who had been bravely battling cancer for several years, was finally succumbing to its vice grip. Having lost my own mother to cancer many years ago, I can empathise deeply with my cousins and what they have been through this year.

My aunty Judy had always the kindest of heart; always determined to see the best in people; to do right by those she loved and to avoid sides in any conflict. Not an easy accomplishment where I was concerned in particular, though she managed all the same. She never had much voice for anger, though she was brave and would not be trifled with. She was brave, kind, and genuine and loved by everyone. She has been missed by everyone this year.

I have always been close to her sons, who are an immense credit to her. My thoughts will be with them all later this month.

Turning over a new leaf

Most people who know me are aware of my long standing affinity for a certain herb, one which for some time had been a hindrance upon my life. I was aware of the problem and resolved to face it for a while. In February I finally determined to turn my back on it for good. It was not easy, I could not do it alone, but [so far at least!] I have managed it and feel so much the better for it. Much of what I went on to do this year might not have been had I not. My sincerest thanks a gratitude to all those who helped and supported me.

Iceland: The land of snow and fire

March saw the first of several epic holidays I took this year, to the frozen, surreal landscapes of the most volcanic country in the arctic circle. I went alone, hiring a 4×4 and driving through the tundra, although I was advised to avoid the mountain passes as a lone, inexperience icelander! I saw the great Geyser, along with many other geothermal pools and fissures in more secluded spots away from the tour buses. I saw the glacial waterfalls of Gulfoss and swam in hot springs. I also managed to see a bit of the northern lights, although my photographic skills failed to capture it successfully. I also sought out the local swing dancing scene and made some new friends at a social event in Reykjavik.

Fred Astaire eat your heart out

I briefly dipped my toes into the world of tap dancing this year. I went along  with a friend for a laugh, but really enjoyed it. I invested in a cheap(ish) pair of tap shoes and went back every week. I can’t claim to be brilliant; I cant even claim to be competent; truthfully, I am an awful tap dancer, but I enjoyed it all the same. Sadly our tap teacher had to return to Australia to look after family, I’m not sure if I’ll find another class that suites and fits into my rapidly filling schedule this year.

London Lindy Exchange

It was a highlight of my year last year, and this year was even better. Epic performances from bands such as Echos of Ellington, Steve Coombes Hot Six and artists Gordon Webster and Naomi Uyama really made the event. I stayed to the end of every evening, including the blues after party on sunday, where the outstanding Gordon Webster came along and played for free way into the early hours. There was an afternoon party on the Monday bank holiday too. I was shattered by the end of it all, but so worth it and cannot wait for LLX 2014.

Moving on from the Lonely Planet

My fixed term contract at the Lonely Planet expired in June and I decided to take a job elsewhere. I left to work for a much smaller company, where I would basically be in charge of creating and managing the cloud computing infrastructure. A challenge that had me nervous to start with. I pleased to say that I managed, and now have a solid infrastructure which scales and manages itself. I will be adding a few posts about some of the neat tricks I have employed in achieving this.

Sun, Pasta, Gelato and Swing Dancing

In June I jetted off to beautiful Como in the Italian lakes for a week long swing dance festival. Half of the London swing dance scene was there, many of whom I knew others were new acquaintances. We were sun seeking, gelato eating, photographing tourist by day; swing dancer extraordinaires by night. We explored the towns that pepper the lake’s waterfront, sampled way too many flavours of gelato, the local food dishes and vino!

The main piazza had been converted into a stage and dancefloor and every evening saw live bands and swing DJ’s playing while the crème of Europe’s swing scene congregated on the dancefloor.  Every evening, at midnight, the scene moved down to a club where swing dancing continued until 4am. After 6 days of dancing for around 8 hours a night I was exhausted but certainly feeling fit!

Sadly this festival will not be returning until 2015, so I will need to seek an alternative for 2014.

Hearts and Seoul

I was also honoured to be invited to a friends wedding in Seoul in South Korea in July. My first jaunt into Asia, I found it a slightly unsettling culture shock at first, but after I acclimatised I had a great time. As we were there for a wedding, there was obviously a stag party, where we went on a bar crawl through the notorious Itaewon area of Seoul, well known for its debauchery. No goats were involved, however.

The wedding was a traditional Korean wedding, which was an interesting experience for me, involving fantastic traditinal robes (known as Hanbok) and hats, some wooden geese, and quite a lot of bowing.

We also explored the palace and other sights of down town Seoul; rode on the Subway (I call it the Seoul train); danced gangnam style in Gangnam (yep, its a real place); gorged ourselves on the local cuisine and booze, although for me Makkoli was a bit too much; and of course, I sought out the local swing scene and tried my best to keep up with the frighteningly good dancers.

It’s the little things that count

I also made a few new friends this year, unexpectedly, whom I met at the odd little event throughout the year. These are sometimes the friendships that can become the most valued when there was such little expectation to begin with. I am truly honoured to know these people and that have helped me out, in all those little ways that make such a big difference. Thank you my friends!

And finally…

My last hurrah for this year a night out at the NYE masked ball for the Last Tuesday Society; a decadent and debauched affair where face masks must be worn and effort on your outfit is required.. assuming you wear an outfit at all! Everyone was glammed up; there was an orchestra, dancing, and cabaret acts; live nude drawings and a naked feast, where gold pained naked people fed bemasked revelers food from gold platters. This was certainly an eye opening, sometimes eye popping experience. The perfect way to end the year!

Getting started with Amazon Python AWS CLI tool

The new Amazon AWS CLI tool coded in Python is far superior to the old Java implementation. The official docs can be found at https://github.com/aws/aws-cli but here’s a quick run down of how I install and configure:

Installation

or if you’re not using a virtual environment

You can also use easy_install in the same way if you prefer

Configure Environment Variables

Add the following to your .bash_profile, .bashrc, or .profile:

The key ID and secret key are required, the default region means you do not need to specify the region for every command, and the default output, changes the output type; it is set to JSON by default, which is fine for scripting, but not the best for reading in a terminal.

You can also just specify an environment variable to point to a config file and add the credentials and options there (see the github wiki)

Enable auto-complete

Add the following line to your .bash_profile, .bashrc or .profile

Finishing and test

To apply the changes to your environment and test, run the following:

The first command will apply the environment changes, the second should display all instances in the default region, using your default output type ( I prefer text). The final line should prompt to display all options, and then list all available AWS sub-commands.

iTerm2 Keybindings for OS X

Word Jumping

If you use a mac and tried the iTerm2 application, you may have noticed that the keybindings for word jumping and deletion are not set up correctly. Here’s how to fix that:

From the iTerm2 menu select Preferences > Profiles and from the right select Keys. Now add the following key bindings by clicking (+) for each:

Alt + Left
Send Escape Sequence, Esc + b

Alt + Right
Send Escape Sequence, Esc + f

Alt + Backspace
Send Hex Code, 0x17

Split Terminals: Creating and Navigating

I also like to bind keys to split the terminal vertically and horizontally and move focus between them.

From the iTerm2 menu select Preferences > Profiles and from the right select Keys. Now add the following key bindings by clicking (+) for each:

Alt + Up
Split Horizontally with <profile>

Alt + Down
Split Vertically with <profile>

Cmd + Up
Select Split Pane Above

Cmd + Down
Select Split Pane Below

Cmd + Left
Select Split Pane on Left

Cmd + Right
Select Split Pane on Right

A simple cloud-init trick

If you use Amazon Web Services or other cloud providers that make use of cloud-init, you may occasionally have reason to re-trigger the initial cloud-init run; usually because something didn’t work quite right the first time round. Rather than having to kill the cloud instance and booting a fresh one, you can try this simple trick to trigger cloud-init on reboot:

 

The Cliff

I found myself standing on the edge of a cliff,
The turbulent sea crashing at its rocky base,
I had been on an adventure through beautiful countryside,
The simplest things bring the greatest pleasures.

As I travelled I came across a beautiful butterfly,
She danced in front of me and settled for a moment,
Waving her beautiful wings, which shimmered in the sun,
Before she fluttered off again away from me.

I followed her as best I could, dancing among the trees,
But she fluttered ever onwards towards the cliff,
There I could follow her no longer; she was free but I was not,
I could only watch as she flew away from the cliff into the distance.

I looked out from my perch on the cliff,
Far away I could see even more beautiful scenes,
I could only imagine what wonders lay within them,
Places I longed to go, but the treacherous sea was in my path.

I looked back the way I had come and saw smoke rising,
Last night’s campfire still smouldering offering promises of warmth,
Should I brave the journey and try to cross the sea?
It would surely be much easier to just turn back…

 

In life you can never turn back the way you came,
So when you find yourself on a cliff you have but one option,
No matter what lies between you and your goals,
You must be brave; you must be fearless; you must JUMP!

 

You must JUMP

You must be brave; you must be fearless; you must JUMP

Sleep Paralysis: Of Hags and Horses

Last week I experienced sleep paralysis for the first time; I am not sure what brought this about, but I had been sleeping poorly all week which probably didn’t help.

Sleep paralysis is a phenomenon which causes the sufferer, usually when transitioning between sleep and wakefulness, to experience hallucinations and paralysis with a sensation of weight pushing down upon them. Many sufferers report seeing an intruder in the room to whom they cannot react due to paralysis.

I had heard of this before, several of my friends had described their experiences to me. I had no idea how real and terrifying it was. This is what happened to me:

I had been unable to sleep again that night; I remember looking at the clock, 3:20am. The next thing I knew I heard my front door opening and someone coming up the stairs. At first I thought “oh good my house mate is home, about time”, it took a few seconds for my thought to catch up with me as I heard the footsteps come closer and open my bedroom door when I sudden thought “wait, I live alone! I don’t have a housemate, so who is this who has just come into my room?!?!”.

I was lying face down on the bed, I could see a shadowy figure out of the corner of my eye, I tried to turn to see but before I could, the figure lunged on top of me and pressed me down hard onto the bed. For that instant I was in complete panic “HELP! Someone has broken into my home and they’re attacking me!” I thought. I struggled to move but I couldn’t; I was completely pinned, with the intruder’s weight pressing down on me and I could not breathe.

The sensation only lasted a few moments, after which I found I was able to move; at first just my arms, which I flailed about and then managed to flip myself violently on to my back, nearly leaping clear out of my bed! I looked around me; nothing there, the door still closed, everything dark and silent except for the sound of blood surging through my body; my heart was beating so hard I could see it thumping in my chest. I looked at the clock again 3:25am

I knew right away that this was sleep paralysis; based on accounts I have heard from friends I got off lightly; some have reported seeing demons, high pitched screaming, having to concentrate intensely in order to move a single finger before they could get out of that state. I certainly hope this isn’t going to become a regular occurrence for me!

The Nightmare

The Nightmare - John Henry Fuseli's iconic depiction of sleep paralysis