Synology Disk Station failed to update network config with new router

I have a Synology Disk Station DS411slim, which has been working flawlessly for almost two years; that is to say, that it has never crashed, corrupted data, failed to keep up with streaming demands etc.

The one issue I had was after I switched internet providers, and thus replaced my router. Everything seemed fine at first, the box used the same static IP address and was visible to my desktop devices. I only noticed something was wrong when I attempted to check for updates in DSM.

Connection failed. Please check your Internet connection

I logged into the box via SSH and found the issue after a quick poke around the system. The new router used a different IP to the old one (192.168.1.1 instead of 192.168.1.254) and the box had not updated this for either gateway or DNS. There may be a way to do this from within the DSM itself, but it was easy enough to resolve by hand.

Firstly, fix resolve.conf

Then fix the default route and make sure it uses the correct gateway even after a reboot

This should be all that is required to fix the problem, however, there are quite a few other files that hold a reference to the old router, so I updated these as well just to be neat.

After that the box was able to resolve hostnames and access the internet again, even after a reboot, and DSM updates worked correctly

DSM 5.2-5644 is ready to update

 

 

Automatically remove dead AutoScale nodes from Chef Server

So you’ve created an autoscaling group in Amazon AWS, when it scales up you’re nodes automatically authenticate with Chef, configure and deploy successfully. That’s great! At least until you need to scale down again. Chef nodes are good at telling the server they exist, but not so great at telling it they’ve checked out. This becomes a real problem if you try to use knife to issue commands to all servers of a particular role for example. You’ll end up with ever increasing stale data about nodes that no longer exist. So what can you do to fix this?

Luckily, AWS provides most of the tools for you, you just need to stitch them together.

What you will need

  • An AWS autoscale group
  • An AWS simple notification service (SNS) topic
  • An AWS simple queue service (SQS) endpoint
  • A script to consume the queue

Topics and Queues

First of all, create your notification topic in SNS, giving it a title (I used AutoScaleDown) and a display name. You can do this in either the web console or the command line tools. Next create a queue in SQS, giving it a name ( I used DeregQueue) and leave all other options as defaults.

SNS Topic Create Subscription

Now you have the notification topic, and the queue endpoint, you need to “subscribe” your queue to the topic, so that any notifications that come into that topic are passed to the message queue. The easiest way to do this is to note the queue’s ARN (Amazon Resource Name) and then “Create Subscription” for the topic in the SNS section of the console.

You can test that the subscription is functioning by sending a test message to the topic, and checking that your message gets onto the queue. If you do this, do not forget to delete the message, as it will interfere later on.

AutoScaling Notifications

Autoscale notificationNow that you have a notification topic, you should be able to create a notification for your autoscaling group to send to this topic. This can be done in the EC2 section of the AWS Console, or with the command line tools. Either way, ensure that the notification happens only when an instance terminates.

 Consuming the queue

So at this point, when your autoscale group scales down and terminates an instance it should trigger a message to your topic, which should then end up on the queue endpoint. This is great, but there needs to be something consuming these messages. You could use any viable scripting language you like, although I found that Ruby was the easiest tool for the job as it has decent AWS libraries and execution of shell commands is a breeze.

The messages are in JSON format, and contain various details about the scale down event that just occurred. The only important bit of information for this example is the AWS instance ID (which doubles up as the chef node / client ID).

The script needs to do three basic tasks; check for messages on the queue, extract the instance ID for each message it finds, and then run the necessary shell commands to remove the nodes from the chef server.

I installed my script on the chef server itself, although any system that has knife configured correctly would suffice. I then created a cron job to run the script once every 30 seconds. Works like a charm!

Here is my script in full:

 

Getting started with Amazon Python AWS CLI tool

The new Amazon AWS CLI tool coded in Python is far superior to the old Java implementation. The official docs can be found at https://github.com/aws/aws-cli but here’s a quick run down of how I install and configure:

Installation

or if you’re not using a virtual environment

You can also use easy_install in the same way if you prefer

Configure Environment Variables

Add the following to your .bash_profile, .bashrc, or .profile:

The key ID and secret key are required, the default region means you do not need to specify the region for every command, and the default output, changes the output type; it is set to JSON by default, which is fine for scripting, but not the best for reading in a terminal.

You can also just specify an environment variable to point to a config file and add the credentials and options there (see the github wiki)

Enable auto-complete

Add the following line to your .bash_profile, .bashrc or .profile

Finishing and test

To apply the changes to your environment and test, run the following:

The first command will apply the environment changes, the second should display all instances in the default region, using your default output type ( I prefer text). The final line should prompt to display all options, and then list all available AWS sub-commands.

iTerm2 Keybindings for OS X

Word Jumping

If you use a mac and tried the iTerm2 application, you may have noticed that the keybindings for word jumping and deletion are not set up correctly. Here’s how to fix that:

From the iTerm2 menu select Preferences > Profiles and from the right select Keys. Now add the following key bindings by clicking (+) for each:

Alt + Left
Send Escape Sequence, Esc + b

Alt + Right
Send Escape Sequence, Esc + f

Alt + Backspace
Send Hex Code, 0x17

Split Terminals: Creating and Navigating

I also like to bind keys to split the terminal vertically and horizontally and move focus between them.

From the iTerm2 menu select Preferences > Profiles and from the right select Keys. Now add the following key bindings by clicking (+) for each:

Alt + Up
Split Horizontally with <profile>

Alt + Down
Split Vertically with <profile>

Cmd + Up
Select Split Pane Above

Cmd + Down
Select Split Pane Below

Cmd + Left
Select Split Pane on Left

Cmd + Right
Select Split Pane on Right

A simple cloud-init trick

If you use Amazon Web Services or other cloud providers that make use of cloud-init, you may occasionally have reason to re-trigger the initial cloud-init run; usually because something didn’t work quite right the first time round. Rather than having to kill the cloud instance and booting a fresh one, you can try this simple trick to trigger cloud-init on reboot:

 

VNC at graphical login for Ubuntu 12.10 or 12.04

It sounds trivial; create an upstart script to start on boot, but this had me googling for a little while.

I used x11vnc as the vnc server. The main issue was that x11vnc needed to bind to an Xauthority file. In order to get it running for graphical login, the Xauthority needed to be for the desktop manager (in this case, lightdm) . It took me a while to find the right location ( ⁄var ⁄run ⁄lightdm ⁄root ⁄); once I had found the answer, the script was easy. I created a file in the upstart init script directory,  ⁄etc⁄init ⁄x11vnc.conf

Lets take a look at what this does; lines 1 and 3 are self explanatory, line 5, ensures that the vnc server is started at the right time (not before lightdm), otherwise the Xauthority file will not exist. Line 6 stops the vnc server on shutdown or reboot.

Lines 8 and 9 ensure that the vnc server is restarted if it fails, but if it respawns x times in y seconds it stops (10 times in 5 seconds in this case). This makes sure it is available but if there is a more serious problem, it stops trying. Line 10 ensures that the process has the correct permissions.

Finally, line 12 starts the x11vnc server. The -forever flag makes sure the server does not exit when a user logs out. The -auth flag is the Xauthority file, which lightdm stores in  ⁄var⁄run⁄lightdm⁄root⁄, usually :0, which indicates the display number lightdm binds to. The -display flag binds to display 0, though it should match the display number used for the Xauthority file. The -rfbauth flag is optional, used only if x11vnc has been configured to use a password file.

Reboot the system, and there it is. Simple but effective.

 

Voyage: Coping with read-only file systems

I am a user of Voyage Linux, a lightweight Debian flavour designed for integrated and low-powered computers. I use it with an ALIX 3D3 board which uses a Compact Flash card for it’s main storage. One of Voyage’s features that makes it ideal is that it operates with a read-only filesystem (a good idea if using a CF card). This introduces some interesting challenges; what about logs? what about home directories and command history?

Well Voyage attempts to resolve the first of these questions by mounting ⁄var⁄log as a tmpfs volume, which works ok but obviously useless if you restart or the box crashes. By default, home directories are stored on the main disk which makes them read-only; so no bash history and no meta files from applications such as vim or ssh. I also noticed sudo was outputting some warnings whenever I sudo’d and refused to remember that I had recently authenticated. Fixing these things are not very difficult, but I hope this helps someone out who has similar issues. Here is what I did.

First of all…

As Voyage mounts the root file system as read-only, it need to be remounted as writable in order to modify the system. Luckily there is a helpful script provided, so I just:

Now home directories

The obvious solution here is to mount an external drive (I used a USB pen drive, though I plan to use a NAS eventually). First I mounted the USB drive and made copies of my users’ home directories. Then I added the following line to ⁄etc⁄fstab

I then unmounted the pen drive and tested the config:

The first command will unmount, the second will trigger parsing the ⁄etc⁄fstab file and report any errors and the last lists all mounted volumes which included this line:

After a quick check that users could still read ⁄ write to their home directories I was good to go…. except for one thing. I use SSH keys, which shouldn’t be left on a USB pen drive. So I put them somewhere appropriate on the main disk and made symlinks back to them from ~⁄.ssh/id_rsa.

Next to fix the logs

Using the same USB volume I created a .log directory. I used the dot to prevent confusion with user home directories. I then copied all log files to it and created a symlink:

Now sudo

Whenever I sudo’d I was getting this mesage:

Sudo keeps information about a users authenticated state. I don’t really need this if I reboot, so I simply created a 1M tmpfs volume mounted on /var/lib/sudo by adding this line in the ⁄etc⁄fstab file:

Note the mode setting, this is important or sudo will keep complaining. Testing the mount again gave me this line:

After this sudo worked normally.

And Finally

Now that I had these things in place and everything running smoothly, there was just one bit of tidying required:

That remounted the root file system as read-only again and I was done.

I ♥ Python

Oh what a wonderful language it is!

Whenever I start up my Windows desktop, it is programmed to wake up my Linux server where I keep all my music and movies. I often make the mistake of just opening up Winamp and expecting it to play whatever I choose from the media library. The reason this does not work, fundamentally, is because Windows is shite (on several counts in this case); The only way to automate tasks at startup is through the autoexec.bat batch file which means that if it tries to reconnect to a mapped drive at that time, it obviously fails as the server takes just over 40 seconds to boot. So I do not do this, instead I have to connect by opening the drive.

Having to remember to do this every time has been pissing me off. Today I’d finally had enough and decided to find a way to make the autoexec.bat file check if the drive is available, if not wait 60 seconds and set up the mapping. After looking into how to simply wait 60 seconds in Batch, I shuddered and heard a voice (which was actually George) say “If you’re gonna script, do it in Python!”. Excellent advice as the code to do it in python is import time, time.sleep(60). I had another problem though, again due to Windows’ inadequacies;

Unlike *nix, Windows does not have comprehensive command line tools. It is supposed to sometimes but its attempts are feeble! Try running net use <drive> <location> <user> on a letter already assigned and you will be in a world of pain! Even if you don’t it still seems to completely b0rk the gui created link to the drive and so still screws everything up. Excellent work! Luckily pyWin32 with its array of API extensions can handle this flawlessly.

One final problem here though is that using this script meant having my username and password for my server in plain text on a Windows machine. Not a good idea! Oh well, a distutil extension called py2exe sorts that out. It isn’t perfect as it turned a 500byte script into a 3.5Mb .exe file (its includes a python interpreter and deppendancies so it can run on systems without a python install) but still, a small price to pay to have a 2 line autoexec.bat file which wakes up the server, waits 60 seconds and connects the drive.

I ♥ Python!

BOINC – volunteer grid computing

After discovering that running my computers (modestly used as they are) cost over half of what I pay for electricity every year, I started thinking about how my computers could be put to better use.

I remembered a little known yet well established concept; volunteer computing, which I used to partake in years ago (I can’t think why I actually stopped being involved before). Which basically involves installing a program on your PC which uses your space cpu cycles to work on scientific projects. A central server for a project hands out small chunks to each volunteered computer over the internet and collates the results.

It would appear that this technology has come along in leaps and bounds since I last looked into it, as there are now dozens upon dozens of these projects, all of which have settled on a common framework known as BOINC (Berkeley Open Infrastructure for Network Computing).

I have a very powerful desktop machine (quad-core Phenom 2.5Ghz) which means during daily use I barely scratch the surface of my computers ability.

With BOINC installed however I am constantly using nearly 100%, though despite this I do not notice any loss in performance while running applications.

This is because all other processes take priority over the BOINC client, it simply uses up whatever CPU power you have spare.

It makes me wonder how much quicker, big scientific problems could be solved if every computer in the world was running this software.

The BOINC client is available on most standard platforms and operating systems and is open source so can be modified to run on potentially any system.

Most linux distributions have the software available from their repositories via package management tools such as Yum and Aptitude. If you have a powerful computer that does not use 100% of it cpu time, can you justify not having this software installed?

Full details and install instructions can be found at http://boinc.berkeley.edu

The Cost of Computing

Owing to the soaring cost of energy, I decided to invest in a plug in power meter. That is a handy little device that monitors how many volts/amps/watts and most importantly watts/hour is being drawn from a power socket. I decided to plug this into the socket which supplies my entire computer rig to see what my computing is costing me.

I had running my desktop PC, my file server, LCD monitor and speakers. One hour later and the meter was reading 0.34kWh which means that to keep this running for 24 hours would use 8.16kWh. It doesn’t look like much but lets do the maths!

Putting these figures into pounds and pennies is a convoluted process, and different for everyone, but here is how I worked it out. I assumed that the more expensive units (first 182 kWh per quarter) were used up by general household usage leaving just the daily and night time rates (17 hours @ 12.75p/kWh and 7 hours @ 5.4p/kWh). This means if my usual setup uses 0.34kWh every hour it will use (5.78kWh @ 12.75p + 2.38kWh @ 5.4p) meaning the cost per 24 hours is (73.695p+12.852p) = 86.547p, which is £315.90 over a whole year.

Now that is a LOT! Even though I do not run my computers 24/7 any more, the meter still read 5.14kWh after 24 hours. By the same calculations, that is still £200.18 per year that I spend just on running my computers.

The fridge/freezer has often been cited as the most wasteful and costly home appliance, however mine uses just 1kWh every day, which is just £38.72 per year. This means that my computers are over five times more expensive to run. I consider my own setup and usage modest among my peers. I know people who run several home servers 24/7. I wonder what sort of bills they are racking up. Furthermore, what must it cost to run a warehouse-sized data centre these days?!

The cost of computing? Very expensive indeed