Zen and the Art of Programming

Ruby on Rails for Microsoft Developers available for pre-order on Amazon.com

2008-08-05T21:58:37Z

Ruby on Rails for Microsoft Developers is now available for pre-order on Amazon.com. Contrary to what the Amazon page states, the publication date is planned for the middle of December, not January. Similarly, the description is quite messed up and loosely based on the first outline. It almost makes it sound like a book on Ruby as opposed to one on Rails. In reality, only two chapters are dedicated to Ruby, while all the others are Rails specific. A more accurate description should be up soon though.

This is the cover, with the picture of yours truly. Ladies, easy on the emails, I’m happily married. Jokes aside, I’ll share a bit of trivia about the picture with you. The few decent photographers in the area quoted me outrageously high numbers just for the one shot I needed for the cover. Being passionate about photography, I ended up shooting it myself, so it’s a self-portrait. I wonder if it’s a first in the world of Wrox covers.

Page-wise, I’ve written roughly half the book, which is going to be about 450 pages, and I expect to finish writing sometime in October.

This Week in Ruby (July 26, 2008)

2008-07-26T08:16:10Z

This is the 13th episode of This Week in Ruby, please consider subscribing to my feed so as to not miss any weekly installments. Also, if you enjoy the series and this blog in general, please consider recommending me on Working With Rails.

JRuby 1.1.3 has been released. This version includes several bug fixes and major speed and memory improvements.
Satish Talim of RubyLearning has announced a new course dedicated to the subject, and also interviewed Charles Nutter for the occasion, who provided some suggestions for RubyLearning participants. This week, Satish also interviewed Guy Naor, the CTO of Morph Labs, a prominent cloud computing Rails hosting company.

On the .NET side, things are moving just as quickly. Some great news emerged from OSCON 2008 regarding IronRuby, including the first binary release and the setting up of a project called ironruby-contrib on GitHub. This already includes the Rails plugin for Silverlight. Meanwhile, Peter Cooper published a great set of IronRuby tutorials to bring C# developers into the Ruby fold.

In the alternative framework world, Mack Framework 0.6 was released, which includes DataMapper 0.9.2 and RSpec support, transactional tests, internationalization and other improvements. The roadmap to Merb 1.0 was also posted on the official blog.

I had previously mentioned a few well known issues with Ruby and XML. Well, it appears that there is hope regarding a libxml-ruby resurrection after all. RedCloth 4 was also released this week.

Two interesting articles were: Don’t forget about RubyForge, which covers the issue of mass migration towards GitHub, and Modules underuse by Jay Fields.

Toronto’s sponsor-less conference RubyFringe is over and according to its participants it was fantastic. I truly regret not being able to participate in it. For those who were there, feel free to share your opinions in the comment section.

One last thing before you go; I must give my “caught-red-handed Ruby Community award” this week to Thiyagarajan Veluchamy. This dude thought it was a good idea to lift the entire content of one of my most popular articles from more than a year ago, even hotlinking the image, and then attributing the post to himself. Did he really think that no one would notice? Its link became popular on Del.icio.us for the Rails tag, and I got all sorts of emails from people who recognized my old (and now somewhat outdated) article. After a brief investigation, it turns out that Thiyagarajan has a habit of stealing content. Other articles appear to be copied verbatim from various blogs. Thiyagarajan, a word of advice, gathering inspiration from a certain post is fine, copying it is not. Especially if you try to pull it off by copying from someone who reports on the most interesting and popular posts in the community. That’s just a really dumb move.

To keep the good times rolling, the fourth edition of This Week in Rails is available on the official Rails blog.

Goodbye Randy

2008-07-25T17:58:27Z

My farewell to Randy, on Be Life Savvy.

Be life savvy: my new non-programming blog

2008-07-20T16:12:58Z

This is a short post to let you know that I’m starting a new non-programming blog. It’s called Be life savvy and it’s going to be about practical tips for modern living, without the typical mumbo jumbo of self-help blogs.

As you probably know I don’t have much spare time, but this is a project that I’ve thought about for a long time and it’s good to take an hour, once in a while, to think and write about a different topic. Starting this blog, and other blogs in the past, has taught me that the writer ends up learning as much or more than their readers. I started the Be life savvy to share my ideas, opinions, experiments and findings with other bright people.

If you follow this blog and you are somewhat interested in these topics, feel free to head over to belifesavvy.com and subscribe to its feed.

Rogers has reached a new low

2008-07-19T02:53:27Z

If we exclude resellers, there aren’t too many Internet Service Providers (ISP) available to residential customers in Canada. It is in fact my understanding that the market of ISP who own their own infrastructure in Toronto is limited to a duopoly between Rogers and Bell.

I’ve been with Rogers for the past two and a half years and I must say their service has always been quite satisfactory in terms of speed and reliability. At $54.95 per month they are not exactly cheap, and they place a cap on the amount of monthly traffic, but otherwise they offer up pretty decent service.

Rogers gained a bad rep though, because they do traffic shaping. This means you can forget about making Skype calls in peace, legally using non-throttled P2P systems or legitimately download, using BitTorrent at a decent speed (saving encryption workarounds). More recently, they also came under fire for proposing the second most expensive iPhone plans in the world.

More importantly though, a while ago they started rewriting the pages that their customers visited, in order to warn them of their high internet usage.

Now however, they’ve reached a new low. They figured out that hijacking their customers’ user experience can be a very lucrative thing. Today I discovered that if you type a non-existing URL, Rogers will automatically redirect you to a page chock-full of ads, and a few possible matches (in reality they are mostly unrelated results).

They’re like the evil twin of OpenDNS, making money off of less Internet-savvy users who may not be aware that they can disable this “feature”. Clicking on “Learn more about this page” at the bottom of the page we’re brought to a “Click here if you would no longer like to receive the Rogers Supported Search Results service.” link, where any Internet-savvy user will opt out.

This being an oligopoly, there aren’t many choices available, but I’m a big believer in “voting with your money”, so not only will I switch to a different ISP, but I’ll also let them know why. At this stage the most likely candidate is Tech Savvy, but they’re slower than Rogers and throttled by Bell anyway. Other ISP suggestions are definitely welcome.

A close look at three Rails 2.1 bugs

2008-07-14T12:43:00Z

Ruby on Rails 2.1 has been out for six weeks now. Let’s take a closer look at three database related bugs that affect this release.

1. SQLite’s db creation generates false warnings

This is an innocuous bug, and if you work with SQLite I’m sure that you encountered and safely ignored it. When you create a Rails application, the default adapter in use is sqlite3, unless you specify otherwise with the -d option. The config/database.yml will reference three databases by default: db/development.sqlite3, db/test.sqlite3 and db/production.sqlite3. Now if you try to create the databases through the db:create or db:create:all rake tasks, the database will be created, but you’ll also get a warning:

$ rake db:create:all
db/development.sqlite3 already exists
db/production.sqlite3 already exists
db/test.sqlite3 already exists

That warning message is not actually true. The three databases didn’t exist before you ran rake, and were created by the task instead, as you’d expect. This is a small annoyance, but one that pops up way too often for my taste. Let’s see where things went wrong.

The db:create and db:create:all tasks invoke the method create_database defined within the file railties/lib/tasks/database.rake in the repository (or the frozen vendor/rails folder). This is the definition of the method:

def create_database(config)
  begin
    ActiveRecord::Base.establish_connection(config)
    ActiveRecord::Base.connection
  rescue
    case config['adapter']
    when 'mysql'
      @charset = ENV['CHARSET'] || 'utf8'
      @collation = ENV['COLLATION'] || 'utf8_general_ci'
      begin
        ActiveRecord::Base.establish_connection(config.merge('database' => nil))
        ActiveRecord::Base.connection.create_database(config['database'], :charset => (config['charset'] || @charset), :collation => (config['collation'] || @collation))
        ActiveRecord::Base.establish_connection(config)
      rescue
        $stderr.puts "Couldn't create database for #{config.inspect}, charset: #{config['charset'] || @charset}, collation: #{config['collation'] || @collation} (if you set the charset manually, make sure you have a matching collation)"
      end
    when 'postgresql'
      @encoding = config[:encoding] || ENV['CHARSET'] || 'utf8'
      begin
        ActiveRecord::Base.establish_connection(config.merge('database' => 'postgres', 'schema_search_path' => 'public'))
        ActiveRecord::Base.connection.create_database(config['database'], config.merge('encoding' => @encoding))
        ActiveRecord::Base.establish_connection(config)
      rescue
        $stderr.puts $!, *($!.backtrace)
        $stderr.puts "Couldn't create database for #{config.inspect}"
      end
    when 'sqlite'
      `sqlite "#{config['database']}"`
    when 'sqlite3'
      `sqlite3 "#{config['database']}"`
    end
  else
    $stderr.puts "#{config['database']} already exists"
  end
end

At first glance, it looks OK: it tries to establish a successful connection, if it succeeds, it executes the else clause and prints a message stating that the database already exists; otherwise Ruby creates it in the rescue clause with the sqlite3 command line tool (or sqlite if using the sqlite adapter).

What truly happens though, is that the rescue clause is never executed (short of Ruby raising an unrelated error). The reason for this is that the two lines of code between begin and rescue, not only attempt to connect to the given database, but will create the database if this doesn’t exist already (well, the SQLite’s Active Record adapter creates it). This means that regardless of whether the database existed or not, those two lines will be successfully run and won’t raise any errors. The final outcome is that the database that didn’t exist, gets created, and the else clause gets executed, wrongly warning us that the database already existed.

I opened a ticket for this and submitted a patch. This is the new create_database method that works correctly:

def create_database(config)
  begin
    if config['adapter'] =~ /sqlite/
      if File.exist?(config['database'])
        $stderr.puts "#{config['database']} already exists"
      else
        begin
          # Create the SQLite database
          ActiveRecord::Base.establish_connection(config)
          ActiveRecord::Base.connection
        rescue
          $stderr.puts $!, *($!.backtrace)
          $stderr.puts "Couldn't create database for #{config.inspect}"
        end
      end
      return # Skip the else clause of begin/rescue    
    else
      ActiveRecord::Base.establish_connection(config)
      ActiveRecord::Base.connection
    end
  rescue
    case config['adapter']
    when 'mysql'
      @charset   = ENV['CHARSET']   || 'utf8'
      @collation = ENV['COLLATION'] || 'utf8_general_ci'
      begin
        ActiveRecord::Base.establish_connection(config.merge('database' => nil))
        ActiveRecord::Base.connection.create_database(config['database'], :charset => (config['charset'] || @charset), :collation => (config['collation'] || @collation))
        ActiveRecord::Base.establish_connection(config)
      rescue
        $stderr.puts "Couldn't create database for #{config.inspect}, charset: #{config['charset'] || @charset}, collation: #{config['collation'] || @collation} (if you set the charset manually, make sure you have a matching collation)"
      end
    when 'postgresql'
      @encoding = config[:encoding] || ENV['CHARSET'] || 'utf8'
      begin
        ActiveRecord::Base.establish_connection(config.merge('database' => 'postgres', 'schema_search_path' => 'public'))
        ActiveRecord::Base.connection.create_database(config['database'], config.merge('encoding' => @encoding))
        ActiveRecord::Base.establish_connection(config)
      rescue
        $stderr.puts $!, *($!.backtrace)
        $stderr.puts "Couldn't create database for #{config.inspect}"
      end
    end
  else
    $stderr.puts "#{config['database']} already exists"
  end
end

I removed the creation of the SQLite (2 and 3) database from the case statement inside the rescue clause, but the actual deal is in these lines:

if config['adapter'] =~ /sqlite/
  if File.exist?(config['database'])
    $stderr.puts "#{config['database']} already exists"
  else
    begin
      # Create the SQLite database
      ActiveRecord::Base.establish_connection(config)
      ActiveRecord::Base.connection
    rescue
      $stderr.puts $!, *($!.backtrace)
      $stderr.puts "Couldn't create database for #{config.inspect}"
    end
  end
  return # Skip the else clause of begin/rescue    
else
  ActiveRecord::Base.establish_connection(config)
  ActiveRecord::Base.connection
end

If the specified adapter is sqlite or sqlite3 then Ruby checks if the database file exists or not. If it exists, the warning message is printed; if it doesn’t, Ruby creates the database automatically by invoking the connection method after running establish_connection with the configuration as an argument. If there is an error during the creation of the database — or the connection to the database, if this already exists — the error and its backtrace are printed. Finally, we exit the method so as to not execute the else clause of the begin/rescue statement, which would erroneously print that the database already existed. If the adapter in use is any other adapter, then we just run the two lines as we did in the non-patched version of the method.

Notice also that there was a second problem with the original code. I couldn’t replace those connection lines with `sqlite3 "#{config['database']}"` because they would open an SQLite3 shell which remains open, so the rake task waits forever for that shell to exit (you can actually see this if you use system rather than “).

2. Preloading has_and_belongs_to_many associations generates non-standard SQL

This second bug is quite serious and relates to the generation of non-standard SQL in has_and_belongs_to_many associations. Imagine that you have two models and a finder as follows:

class Project < ActiveRecord::Base
  has_and_belongs_to_many :developers
end

class Developer < ActiveRecord::Base
  has_and_belongs_to_many :projects
end

p = Project.find(:all, :include => :developers)

That finder will generate a query like:

SELECT developers.*, t0.project_id as _parent_record_id
FROM developers
INNER JOIN developers_projects as t0 ON developers.id = t0.developer_id
WHERE (t0.project_id IN (1,2))

Notice that _parent_record_id? That’s problematic because an SQL identifier cannot begin with an underscore, and this is true for all three versions of the standard (92, 99 and 2003). You can verify the non-conformity of the generated SQL through this validator. That single underscore broke support for DB2 and Oracle, and possibly other standard compliant databases.

I submitted a patch for this some time ago and it was immediately applied by Jeremy Kemper, so Edge Rails and the next version of Rails aren’t affected. Meanwhile the IBM_DB gem worked around the issue and I believe the Oracle enhanced adapter did the same.

3. DEFAULT NULL NULL

The third bug is somewhat similar to the second one and it’s caused once again by non-standard SQL. This too, like the previous bug, breaks support for DB2, Oracle and possibly some other databases. Imagine that we have the following migration file:

class CreatePosts < ActiveRecord::Migration
  def self.up
    create_table :posts do |t|
      t.string :title
      t.text :body

      t.timestamps
    end
  end

  def self.down
    drop_table :posts
  end
end

When the up method gets invoked by a rake task, the following query will be generated (this is the SQLite version of it):

CREATE TABLE "posts" (
    "id" INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
    "title" varchar(255) DEFAULT NULL NULL,
    "body" text DEFAULT NULL NULL,
    "created_at" datetime DEFAULT NULL NULL,
    "updated_at" datetime DEFAULT NULL NULL
)

Exclude for a moment the primary key, whose syntax is entirely adapter specific. The rest of the query still shows a problem. DEFAULT NULL NULL is not standard SQL and will not be accepted by some databases. This is caused by the fact that in the column definition we didn’t specify a :default option, so we have the equivalent of :default => nil, which translates into DEFAULT NULL. This precedes the second NULL, which is there to indicate that the column accepts nulls. Again, even the second NULL is not part of the SQL standard. A field is nullable by default, therefore we only need to specify something if it must not accept NULLs (i.e. by appending NOT NULL).

The solution to this problem lies in changing the behavior of ColumnDefinition’s instance method to_sql in activerecord/lib/active_record/connection_adapters/abstract/schema_definitions.rb. This is the current definition:

def to_sql
  column_sql = "#{base.quote_column_name(name)} #{sql_type}"
  add_column_options!(column_sql, :null => null, :default => default) unless type.to_sym == :primary_key
  column_sql
end

Notice how the second argument of add_column_options! is a hash that sets the null and default values in every case. If these values are nil, the value NULL gets pushed into the final SQL query. We need to prevent this from happening by passing the two options only when they are not nil.

def to_sql
  column_sql = "#{base.quote_column_name(name)} #{sql_type}"
  column_options = {}
  column_options[:null] = null unless null.nil?
  column.options[:default] = default unless default.nil?
  add_column_options!(column_sql, column_options) unless type.to_sym == :primary_key
  column_sql
end

The migration file we saw above will now generate the following query (again, in the case of SQLite):

CREATE TABLE "posts" (
  "id" INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
  "title" varchar(255),
  "body" text,
  "created_at" datetime,
  "updated_at" datetime
)

This is standard SQL, if we exclude the primary key syntax which is specific to SQLite. More importantly, this would work and be accepted by other database systems like DB2 and Oracle. Version 0.9.5 of the IBM_DB gem monkey patches the issue in a similar fashion, in order to work around this problem and provide support for Rails 2.1. It is possible that the Oracle enhanced adapter 1.1.1 does the same thing. I didn’t submit a patch for this issue because several tickets regarding it are already open, and Nick Sieger already beat me to it. His patch was submitted more than a month ago but it hasn’t received any attention yet. I hope that this post will contribute towards changing that.

Conclusion

I think the last two bugs highlight a valid lesson: the importance of adhering to the SQL standard, particularly when implementing an Object-Relational mapper that supports so many databases. ActiveRecord takes the approach of generating SQL queries by mixing common SQL bits, defined outside of a particular adapter, with dialect-specific parts which are defined by each adapter. It is therefore crucial to ensure that the bits that end up in the SQL queries, independently from the adapter in use, are based on the SQL standard and not on the behavior of SQLite or MySQL.

The same principle applies when building a CMS system or a popular blog engine. If the project is aimed at being portable from one RDBMS to another with little hassle, attention should be payed to custom queries in order to keep them as close to the SQL standard as possible. Of course, it’s perfectly fine to create projects that are database specific and take advantage of that particular database’s strength. For example, DB2 Express-C offers the ability to handle XML in a native manner through a technology called pureXML. This is a fantastic feature, which can’t really be ported to a different RDBMS, but it’s worth its weight in gold, particularly when working in Ruby.

Lastly, don’t let this reflect negatively on your judgment of Rails’ code quality. A large and ambitious project like Rails is bound to have bugs here and there, no matter how scrupulous its developers are.

This Week in Ruby (July 11, 2008)

2008-07-23T19:34:16Z

This is the 12th episode of This Week in Ruby, please consider subscribing to my feed so as to not miss any weekly installments. Also, if you enjoy the series and this blog in general, please consider recommending me on Working With Rails.

I’d like to start this edition by apologizing to my readers for the delay in publishing this edition. Things got pretty hectic last week.

As far as I know, there are no updates regarding Ruby’s vulnerabilities, but if you’re aware of any, feel free to state so in the comment section. Meanwhile, BreakingPoint Systems published a couple of extra problems that were discovered while analyzing those pesky security issues. You can read about them here.

As you may have inferred, I’m quite interested in the optimization of Ruby code. Ilya Grigorik wrote 6 nice tips for optimizing Ruby MRI, which may come handy to you.

A new chapter was added to The Book of Ruby by Huw Collingbourne. Read more about it and download it here.

The Ruby community is big on TDD and BDD and there is no doubt that testing is fundamental for good quality software. RailSpike opens a can of worms with its thought-provoking article, Testing is overrated. Whether you agree or not with its findings, it is definitely worth a read.

Ethan Vizitei has had a couple of compelling entries lately. The first is about handling Gmail’s imap from Ruby and the second deals with refactoring Ruby code.

Sinatra is an ultralight Web framework, while Datamapper is considered by many to be a valid substitute for Active Record. Nick Plante shows us how to use them together to create a Pastie clone. If you are into Datamapper or would like to just get a feel for it, consider reading over this cheatsheet as well.

In conclusion, InfoQ published a Metaprogramming roundup and the second part of their RubyKaigi 2008 coverage. The most interesting bit is about the exciting prospective of standardizing Ruby. That would be a leap forward for the language and our community.

To keep the good times rolling, the third edition of This Week in Rails is available on the official Rails blog.

Ruby Shootout Status Update

2008-07-07T06:17:08Z

I want to provide those who are waiting for the Ruby shootout with a heads up. The benchmark suite needs some substantial changes in order to ensure accuracy and fairness for all the VMs involved.

This will delay the execution (and reporting) of the shootout further, but it will be worth it. I definitely prefer a shootout that’s published later in July (or heck even August) that is realistic, fair and provides interesting metrics (e.g. CPU time and memory) over an inaccurate one that was put together in a rush just for the sake of publishing it tomorrow.

For those interested in the technical details, we are trying to separate the parsing and “compilation” of definitions from the actual execution of the code (which needs to be timed). I accomplished this by creating a Proc for each benchmark, and then tested the time spent executing its call method. The problem with this approach is that it penalizes VMs that don’t JIT procs, like JRuby for example.

We also thought about defining a method instead of a Proc, but eval won’t accept class definitions or constants within methods. The workaround would be using MyClass.class_eval instead of class MyClass in the benchmarks and Module#const_set for the constants (or changing them to instance variables, for example). But we’re shooting for a cleaner solution in which we divide definitions from their actual execution in separate files, and only time the latter.

And of course, we also need to add cross-platform memory measurement into the picture. It may take a while, but stay tuned.

Ruby and Rails books, two years later (photo)

2008-06-27T22:39:22Z

Two years ago:

Today:

IBM’s crazy DB2 videos. What do you think?

2008-06-27T17:49:35Z

IBM is clearly synonymous with the term “Enterprise”, but I can testify that working here means interacting with many cool, down to earth people who are passionate about their profession. They are not the “corporate drones” that you’d expect. I sometimes think that there is a divide between the image of IBM and the reality of a company that’s mostly comprised of young people having fun while working hard.

Reading comments on sites like Slashdot or Reddit I see that IBM is often unfairly treated, despite being a champion of Open Source, with contributions in excess of those of Google and Amazon combined.

So I’m glad to see that IBM is not “taking itself too seriously” with a series of ads, campaigns and initiatives that portray a less rigid image. The silly “servers taking over” ads are examples of this.

More recently IBM has put out a few “unusual” videos about DB2 Express-C:

So I’d like to ask you to share your opinions about them. Are they “cool” and funny, or are they goofy and just trying too hard?