rowtheboat - George Palmer

Lately I’ve been working on a stock trading program that automatically trades and saves the results to a database (more details to follow in the not too distant future…)  Naturally the choice of language was ruby, and for the front end, a rails application.  Once coded, I bundled the stock trading program into the rails lib folder and reused the rails model for saving to the database.

I ran into a slight problem doing this though as ActiveRecord needs configuring manually outside the rails framework but I wanted to follow best practices and keep my code DRY. After a quick search I found a clean solution that allows you to re-use the database.yml configuration file from the rails application:

dbconfig = YAML::load(File.open('config/database.yml'))
ActiveRecord::Base.establish_connection(dbconfig["development"])

This snippet got me thinking about YAML which I use a lot but have never played with in Ruby. It turns out the YAML class is basically a hash that uses strings (gotcha not keys) to access a simple hierarchy:

dbconfig = YAML::load(File.open('config/database.yml'))
dbconfig["development"]["adapter"] # = mysql

All very simple stuff but if its never crossed your mind before, you yet again appreciate the elegance of ruby.

No comments

This week I’ve been working on a client project where I needed to import a massive XML dataset into a database.  The XML was non-standard and broke its own rules in several places.  Consequently my rake task quickly become very complicated and I needed some tests to ensure I wasn’t breaking previous work.  This leads to a rather interesting question: how do you test rake tasks?  After a bit of googling I found a rather neat solution of simply pulling out the rake code to a class. For example, rather than:

desc "Import the XML"
task :import => :cleanup do
  File.open(XML_FILE) do |file|
    # Do importing here...
  end
end

do something more like

desc "Import the XML"
task :import => :cleanup do
  File.open(XML_FILE) do |file|
    XMLImporter.parse_lines(file)
  end
end

XMLImporter can then be tested in the normal way. Good ruby - clean and effective.

No comments

I had a few issues getting a Capistrano file to work with Git and frozen MERB the last few days. The script below is how I finally cracked it and also has starters and stoppers for Memcached and my forked backgroundrb for MERB.

# Basic deploy details
set :application, "myapp"
set :deploy_to, "/var/www/apps/#{application}"
set :mongrel_conf, "#{deploy_to}/current/config/mongrel_cluster.yml"

# Repository details
set :repository,  "git@ipaddress:reposname.git"
set :scm, "git"
set :scm_passphrase, ""
set :set_branch, "origin/master"
set :git_enable_submodules, 1

# Required to get git password prompt
default_run_options[:pty] = true

# Set the user account to use for deployment and running
set :user, "deploy"
set :use_sudo, false

# Server details
role :app, "mysite.com"
role :web, "mysite.com"
role :db,  "mysite.com", :primary => true

desc "Deploy to the production server"
task :production do
role :app, "mysite.com"
role :web, "mysite.com"
role :db,  "mysite.com", :primary => true
end

desc "Change the database configuration file"
task :after_update_code do
run "mv #{release_path}/config/database.yml.production #{release_path}/config/database.yml"

# Remove code we don’t want on the server
run "rm -rf #{release_path}/autotest"
run "rm #{release_path}/config/deploy.rb"
run "rm -rf #{release_path}/coverage"
run "rm -rf #{release_path}/spec"
run "rm -rf #{release_path}/stories"

# Make config directory just readable
run "chmod 700 #{release_path}/config"
run "chmod 400 #{release_path}/config/*"
run "chmod 700 #{release_path}/config/environments"
run "chmod 700 #{release_path}/config/initializers"
run "chmod 400 #{release_path}/config/environments/*"

# Make frozen-merb bin file executable
run "chmod 755 #{release_path}/framework/merb-more/merb-freezer/bin/frozen-merb"
end

# Override db:migrate as it calls rails specific stuff
namespace :deploy do
desc "Migrate the database"
task :migrate, :roles => :db do
run "cd #{release_path}; rake db:migrate MERB_ENV=production"
end
end

# Override the default deploy options to use backgroundrb, memcached and mongrel
namespace :deploy do
namespace :backgroundrb do
desc "Start backgroundrb"
task :start, :roles => :app do
invoke_command "cd /var/www/apps/bablo/current && script/backgroundrb start — -r production", :via => run_method
end
task :stop, :roles => :app do
invoke_command "cd /var/www/apps/bablo/current && script/backgroundrb stop", :via => run_method
end
end

namespace :memcached do
desc "Start memcached"
task :start, :roles => :app do
run "memcached -l 127.0.0.1 -d -m 96 -p 17898"
end
task :stop, :roles => :app do
run "killall -s TERM memcached"
end
end

namespace :mongrel do
desc "Start mongrel"
task :start, :roles => :app do
invoke_command "cd /var/www/apps/bablo/current && #{release_path}/framework/merb-more/merb-freezer/bin/frozen-merb -d -e production -c 5", :via => run_method
end
desc "Stop mongrel"
task :stop, :roles => :app do
invoke_command "cd /var/www/apps/bablo/current && #{release_path}/framework/merb-more/merb-freezer/bin/frozen-merb -K all", :via => run_method
end
end

desc "Custom restart task for mongrel cluster"
task :restart, :roles => :app, :except => { :no_release => true } do
deploy.backgroundrb.stop
deploy.memcached.stop
deploy.memcached.start
deploy.mongrel.stop
deploy.mongrel.start
deploy.backgroundrb.start # Doesn’t work if straight after the stop
end

desc "Custom start task for mongrel cluster"
task :start, :roles => :app do
deploy.backgroundrb.start
deploy.memcached.start
deploy.mongrel.start
end

desc "Custom stop task for mongrel cluster"
task :stop, :roles => :app do
deploy.backgroundrb.stop
deploy.memcached.stop
deploy.mongrel.stop
end
end

I’ve still not managed to get Git caching working with submodules, so any input on this would be appreciated (it seems to work fine without submodules).  Updated: Fixed in Capistrano 2.4

No comments

I presented on MERB last weekend at Barcamp London 4 I’ve been using the technology the last two months at Bablo so it was good to share my experience with other users. With it being quite a bleeding edge technology there was just a small gang of us there but there was plenty of good discussion. The talk I gave had a purposely provocative title but ended with a great quote by Ezra

more choices make the Ruby ecosystem a better place. So let’s just stop with the Rails VS Merb stuff. How about people choose what framework they want to use based on the frameworks merits and features rather then religious arguments about how my framework can beat up your framework.

I was then flicking through some slides from the recent RubyConf and was mightily impressed by a slide from the end of the MERB talk:

It’s consider a bug if:
* It’s not documented
* MERB gets slower
* There’s a public API change without prior deprecation in a timely manner

It seems the guys at EngineYard have a great attitude, not to mention a really well thought out framework. I’m looking forward to using it fulltime.

No comments

Two weeks ago my hard drive failed and as if by coincidence Ubuntu Hardy was due out the next day. This seemed like an ideal opportunity to take a day off (the now nearly finished) idlasso and install the newest Ubuntu. I have skipped a few versions of Ubuntu so was mightly impressed when everything worked out the box, no problems with wireless, VoIP phones or flash in the browser! I now believe that the distribution is sufficiently easy to use for a novice that if the Ubuntu team made it look really cool by default they could grab some good Windows/Mac market share. To confirm this point I refer to my sister who upon seeing my new setup was “like that’s so cool I don’t want windows anymore” - I fear she’s not the only one.

Introduction

Anyway this is just a round about way of saying two years ago I made my Ubuntu look like a mac and, after several requests, it’s about high time I blogged how to do it. It’s not the shortest tutorial in the world so here’s a finished screenshot to give you a little motivation to follow it through to the end :)

Getting started

We’re going to use the Mac4Lin project as the basis for the changes so head over there and download the main package, icons and wallpapers. Once downloaded move them to a new folder called macforlinux (this is just a temporary folder that’ll make cleaning up at the end much easier. I’ll refer to it throughout though so change the name if you use something different). Got those? Great, let’s start with the icons.

Icons

First of all you’re going to need to extract out two of the compressed files we’ve downloaded:

cd macforlinux
tar -zxf Mac4Lin_Part1_v0.4.tar.gz
tar -zxf Mac4Lin_Wallpapers_Part3_v0.4.tar.gz

Now goto the Gnome menu and choose System -> Preferences -> Appearance. Click the install button and navigate to the macforlinux folder, choosing the Mac4Lin_Icons_Part2_v0.4.tar.gz file. Dismiss any popup asking if you wish to change to the new theme. You can now click Customise, select the Icons tab and choose the Mac4Lin_Icons_v0.4. At this point you should notice some changes to icons on your windows and desktop (if there are any individual icons you don’t like you can replace them by finding the icons in your home directory under .icons)

Mouse pointers

Next up is the mouse pointer. Again click the install button and this time navigate to the macforlinux/Mac4Lin_v0.4/GTK Cursor Theme folder. Here there should be a tar.gz file that you can install. Again dismiss any popup and click customise. This time select the pointers tab and choose the Mac4Lin_Cursors_v0.4. With any luck you’ll see the change straight away.

Window theming

Now lets get the infamous red, yellow and green circles on the windows. Choose customise as before and this time navigate to macforlinux/Mac4Lin_v0.4/GTK Metacity Theme. You can install all 4 files and when clicking customise and going to Window Border choose the one that suits you best (my preference was Mac4Lin_GTK_v0.4, the menu version just gives a different coloured menu bar). This also installs additional options for the controls tab. You can choose the same option as for the Window Border but I found the really dark windows too much so went for the Ubuntu Human choice (I’m sure there are themes that work better with this look - leave a comment if you find a good one). Under the colour tab I also change the Selected Items colour to a blue to be more in keeping with the mac theme (try a RGB of #C2D9F8).

If you don’t like the strange bar along the bottom of the window (I didn’t) you can change its size by editing the theme file:

gedit ~/.themes/Mac4Lin_GTK_v0.4/metacity-1/metacity-theme-1.xml

Change the distance tag with name “bottom_height” to have value 2 (of course you can customise other stuff whilst you’re in here). You’ll have to go back to the appearance options dialog choose another window theme and then go back to the theme you just edited to refresh your windows.

So now our basic theme configuration is complete let’s save it. On the main appearance screen the Custom theme is probably selected as we’ve made changes to the default. So lets click Save As and give it a name. This keeps these settings safe and you fine tune theme to your needs or revert back to the Ubuntu defaults if you change your mind.

Fonts

Next up is the fonts - I’m going to extract both the OSX and MSfonts as its useful to have both around on your system (note you can install the MSfonts via synaptic but I’m doing both the same way here for demonstration ease):

cd macforlinux
cd Mac4Lin_v0.4/Fonts
tar -zxf OSX_Fonts.tar.gz
mv OSX\ Fonts osxfonts  # folders with spaces in are bad in the linux world
mkdir msfonts # the msfonts don't have a folder so lets create one and move into there
cd msfonts
mv ../msfonts.tbz .
bunzip2 msfonts.tbz
tar -xf msfonts.tar
cd ..

Now we just need to move the fonts to the system font directory and refresh the cache

sudo mv msfonts /usr/share/fonts/truetype
sudo mv osxfonts /usr/share/fonts/truetype
fc-cache -f -v

If you wish to change your system to use the windows fonts by default (over the standard Ubuntu one) you can do:

bunzip2 fontconfig.tbz
tar -xf fontconfig.tar
cd /etc/fonts
cp alias.conf alias.conf.bck # Some of these may error but you should try so you have backups incase anything goes wrong
cp local.conf local.conf.bck
cp misc.conf misc.conf.bck
cp msfonts-rules.conf msfonts-rules.conf.bck
cp pathtomacforlinx/*.conf .

At this point you’re going to need to reboot. It’s the only point where you will need to, so I’ll see you in a few minutes.

Back? Cool now let’s configure our fonts. Below is a screenshot of the preferences that I used, although you may wish to tinker this slightly if running on a different resolution to 1280×1024. Go to the Appearance dialog again and select the fonts you want as appropriate:

Background

To complete the appearance section let’s change the backgrounds. First we’ll install the backgrounds to the system directory:

cd macforlinux/Wallpapers
sudo cp * /usr/share/backgrounds

Now goto the Gnome appearance dialog as used in changing the fonts & window and select the background tab. Click add and goto /usr/share/backgrounds. You can select all the backgrounds and click ok. I then chose the default leopard background.

The launch bar

Now its time to get setup with a cool launcher bar. First off you’re going to need remove the gnome bar along the bottom, right click on it and select ‘delete this panel’.

We’re going to use Avant Window Navigator as a replacement, which can be installed by:

sudo apt-get install awn-manager

Launch the avant window manager by going Applications -> Accessories -> Avant Window Navigator. Once loaded you’ll see a funny shaped object in the bottom middle of the screen, possibly with a few icons depending on what programs you have open. Right click on this and select preferences, goto Themes and click add. Locate your macforlinx folder then Mac4Lin_v0.4, AWN Dock Theme and select the tgz file. You can then choose the leopard theme by selecting the radio button and clicking apply (although I had to exit the preferences box and come back in to see the newly installed theme option). If you now have problems getting to the right click preferences menu aim for the back left of the dock, its easy to pickup there. Personally I like my icons a bit bigger so I changed my bar height to 50 and the arrow offset to 5 in the general preferences.

The Avant bar works like a mac one - if you have an application pinned to the bar it shows an arrow underneath to show its loaded. If you load an application that’s not on the bar it pops up on the right side of the bar until it’s closed. Also clicking an icon brings the window to focus rather than opening another instance. There are also a ton of extra options on what the icons do when you hover over them etc - it’s all rather neatly done.

OK we’re not too far off complete now. The new bar looks quite bare though - it needs some launchers. For these I used a combination of the mac icons we installed earlier, Nuove XT, d3a icons (which I can no longer find a working link to) and the icons that come with the application. For the mac, nuove XT and d3a icons I extracted copies to the /usr/share/icons folder. To add a program select launchers from the Avant bar preferences, click add, fill in the details and click the icon to change. You’ll be able to build up a bar of your most used software like my screenshot below:

To make sure the bar loads at startup goto the Gnome menu bar System -> Preferences -> Sessions, click Add and type ‘Avant’ as the name and ‘avant-window-navigator’ as the command.

The terminal

I’m a rails developer by day so like a good terminal to use. To roughly match the mac settings open a gnome terminal and go edit -> profiles. Create a new profile and under the colours tab choose white on black. Then change the background to around 75% transparency on the effects tab, save and change your new theme to be the default.

3D effects

To finish off it looks really cool if you can use the linux 3D effects. To do this you first need to install native graphics drivers - I won’t document how to do that here, there are plenty of good articles on the subject. Once they’re installed though you can do:

sudo apt-get install compizconfig-settings-manager

This adds ‘Advanced Desktop Settings’ option to the System -> Preferences menu which you can click on to configure your desktop effects.

All done!

At this point I’m going to stop because that’s all the changes I like on my desktop. However if you check out the Mac4Lin project details there are several more tweaks you can make to the Gnome logon screen, Gnome splash screen and even the boot screen. Additionally you can get some plugins for Avant that change the icon depending on the state of the program (for example the very cool exaile plugin shows the cover art of the song currently playing).

All you need to do now is cleanup your macforlinux folder and you’re all done. Hopefully your desktop is looking something like these screenshots:

If you love your new desktop then why not add it to the flickr group I’ve just created? I’ll endeavor to post a follow up article with the best desktops and any other bits people find out/I’ve missed. Thanks to the Mac4Lin guys for all their hard work on the project (it was much harder doing this 2 years ago) and thanks again to idlasso for allowing me the time out to do this.

Now who said Linux couldn’t be cool?

11 comments

Back in Feburary 2007 myself and Andy went to BarcampLondon2 on an exploratory trip to bash around a few ideas for what would later become idlasso. I presented on scaling web applications, the presentation for which can be seen at slideshare or below (hmmm not working on new wordpress):

[slideshare id=24597&doc=how-to-scale-your-web-app-3792&w=425]

No comments

I’ve just found a great video of Google’s CIO on innovation. There’s a fantastic part where he states you need to reward innovation and risk taking and not punish failure. In my experience this seems to be the polar opposite from most companies these days!

No comments

I’m really into rails performance (and now merb but that wasn’t around when this post was written) so decided to look at a comparison between the standard rails versions - 1.1, 1.2 and 2.0. In these tests the excellent railsbench was used to drive the tests - I favor this over the standard rails benchmarking tools is it measures the raw performance of Rails request processing, ignoring the time spent passing the request from the web server to the Rails application. Additionally it is consistent across rails versions and I’m aware there have been changes in the benchmarking area in the rails 2.0 release.

The hardware

The tests were run on an old machine of mine dedicated to performance testing. It’s an AMD Athlon 1.4GHz, 256 MB RAM, Ubuntu 6.06 LTS server edition with latest security patches applied, Ruby 1.8.6 and Gems 1.0.1 (installed from source) and a dedicated crossover ethernet cable to a much more powerful desktop machine. It’s not the aim of these tests to find reasonable performance values for a modern server but more the most efficient rails version to use.

The tests

The tests compose of several pages designed to stress different areas of rails. These are:

Page	Action	Area
front1	Renders current time without a session	Designed to hit as little of rails as possible
front2	Renders current time with a session	Designed to show the difference in using a session
users	Lists of users (100 in test database)	Designed to test the database driver
showuser	Shows the user details (also shows the user’s posts)	Designed to behave like a typical test page with a user who has_many posts
edituser	Thed edit user page (posts aren’t included)	Tests page rendering and form helpers
updateuser	Updates the user details via a POST	Accessing/Updating 1 database row

In all tests (unless otherwise noted) 5000 requests were performed 3 times and the average request time taken.

In the first tests I performed it was noticed that creating actions in a web service style (respond to different MIME types) caused a significant performance hit. EG in rails 1.1 the code:

def index
  @users = User.find(:all)
end

was changed in rails 1.2 to:

def index
  @users = User.find(:all)

  respond_to do |format|
    format.html
  end
end

As a result two versions of the test appliaction were created, one with responds_to blocks and one without.

Test name	Rails setup
performance11	Rails 1.1
performance12	Rails 1.2 without responds_to
performance12	Rails 1.2 with responds_to
performance20	Rails 2.0 without responds_to
performance20	Rails 2.0 with responds_to

There were some changes in the edit view in each of the versions above. The first change was introduced in rails 1.2 due to a new way of using routes as a result of RESTful applications. So rather than:

<%= form_tag :action => 'update', :id => @user %>

in rails 1.2 we use:

<%= form_tag(user_url, :method => :put) %>

Testing showed this made no noticable difference to performance.

The second change was introduced in rails 2.0 as a result of a new way of constructing forms. So rather than:

<%= form_tag(user_url, :method => :put) %>
    &lt;%= text_field(:user, :notes) %&gt;
    &lt;%= submit_tag %&gt;
&lt;%= end_form_tag %&gt;

in rails 2.0 we use:

<% form_for(@user, :url => user_url, :method => :put) do |f| %>
    <%= f.text_field(:notes) %>
    <%= submit_tag %>
<% end %>

This is only really a style change from the rails 1.2 approach and testing showed it made little difference to performance.

The results

Showing requests per second:

	front1	front2	users	showuser	edituser	updateuser	all
rails11	390.49	179.45	111.62	122.61	180.11	107.24	149.53
rails12	306.51	192.66	127.29	129.27	160.43	106.72	151.33
rails12ws	258.02	172.83	117.16	118.31	144.60	101.01	138.04
rails20	292.12	186.00	93.63	119.84	142.77	101.93	134.41
rails20ws	261.40	173.68	89.94	113.96	134.22	97.03	127.03

Or in graph form (click for the large image):

So what can we see from the results?

• Rails 1.1 seems to excel when rendering a simple page with no session, but then you would sort of expect this as the codebase will be smallest. Still the margin was quite significant.
• Rails 1.1 and rails 1.2 share the honors for fastest at processing the page depending on the test (indeed their averages are very close).
• The web service versions of the rails 1.2 and rails 2.0 applications are on average 9.6% and 5.8% slower respectively
• If you’re jumping from Rails 1.1 to Rails 2.0 with web services (as we are going to at idlasso.com at some point) then you could see a performance decrease of 17.7% on average

Conclusion

So does this mean rails 2.0 is the slowest release yet? Well no. In order to keep the comparison fair the session store default in the rails 2.0 tests was changed to pstore which is the default used by rails in versions 1.1 and 1.2. However in rails 2.0 the default was changed to use a cookie stored in the user’s browser. A quick test using Apache Bench shows that rails 1.2 averaged 48req/s whereas rails 2.0 averaged 60req/s on the front2 page. However this was just that - a quick test. Only 1000 requests were sent, with no averaging or warmup. I’ll explore sessions more fully in a later article but for now we can safely say use cookie based session if using rails 2.0

So from this series of performance tests it seems it’s sensible to recommend:

• Avoid the use of the respond_to statement unless your action actually responds in more than one way
• Use client side sessions if using rails 2.0 or later.

1 comment

PC-Rower was a piece of software produced for my dissertation at Durham. The idea was simple, plug rowing machines into a computer and create images of boats racing down a river that can be projected in front of the rowing machines. It was created using SWT from the eclipse project and communicated with the rowing machines through a serial port on the back of the PM2+. It worked really well and the Durham University boat club found it helped performance during training.

Unluckily as I was finishing the project Concept2 produced a new rowing machine with a USB performance monitor, a company founded and produced similar software (albeit only on windows), the Java support for USB sucked and I was off traveling for 3 months. Consequently the project died a premature death. I decided to open source the project and if nothing else has it provides a good implementation of a (reversed engineered) serial protocol.

No comments