Austin.rb - home

ActiveRecord with NOT NULL columns

2007-07-28T03:45:00Z

Using ActiveRecord with an unchangeable legacy database schema poses some interesting challenges. Most of the work is a new style (for me) of composing adapters to make non-Rails stuff work within Rails' conventions.

ActiveRecord uses the database's native NULL to indicate an unset attribute [Ruby's nil]. This becomes a problem when the column definition enforces NOT NULL on columns that you may or may not be setting values.

The following Ruby code allows a table with NOT NULL columns to "just work" with ActiveRecord by setting them to reasonable default values.

class OldBacon < ActiveRecord::Base

  before_validation :prep_not_null_columns

  private
  
  def prep_not_null_columns
    self.class.columns.each do |c|
      unless c.null
        if read_attribute(c.name).nil?
          case c.type
          when :date
            new_value = Date.new
          else
            new_value = c.instance_values['original_default']
          end
          write_attribute(c.name, new_value)
        end
      end
    end
  end

end

Questionably defying good DBA practices? Perhaps. But sometimes you have to make the best with the hand your dealt.

TextMate's hidden "Find in Selection"

2007-07-25T21:51:00Z

[16:30] srogers:
So what's up with TextMate and replace string in selection?
i.e. why is it not there in the find dialog?

[16:37] marsipan:
srogers: hold down the option key while in the Find dialog
"Replace All" becomes "in Selection"

[16:38] srogers:
cool
thanks
(seems like checkbox would be better though)

[16:39] marsipan:
agreed.

Using Your Fixture Data in Development

2007-07-25T06:11:00Z

So you've spent hours putting together some sensible data for your tests in a bunch of tediously crafted YAML files, and you're staring at a blank development database. If only there were an easy way to slip that data from the test database back into development . . .

Your strategies for managing data will probably evolve as your project progresses, but I find that in the early stages when I'm trying to be "agile", I'm making frequent changes to the database and I don't really care about full-cycle management. In this stage, writing migrations for each change in DB is a drag, because I don't want to carry the kind of configuration management overhead during this rapid prototyping phase. Managing the data with migrations is even worse.

That's why I find it so convenient to prototype the DB grahically using OmniGraffle and just keep rolling back to migration zero, then recreating the database from the updated initial migration. You're thinking "Gee, doesn't that mean you're blowing your tables and data away every time you make a change"? And indeed, that's what brings us to this point.

Even in rapid-prototyping mode, I still have tests and fixture data. When I make a change to the DB, then I get the fixture data repaired, get the tests running, and then I want to "play" with the system in development. The data I want is in the test DB (because my tests loaded it from the fixtures). It's possible to trick a dump from the MySQL Administrator or command line to load in another database, but that's realy not convenient. To make this fast and easy, I created a script to copy the data from each table in the test DB to the development DB. It doesn't delete the table (which could change the options it was created with), it just moves the data.

#!/bin/csh
echo "Get tables . . ."
mysql -u person -ppass -h localhost -D my_development >table_list.tmp << EOF
show tables from my_test 
EOF
foreach tableName ( `cat table_list.tmp | sed -e '1d' | sed -e '/schema_info/d' `)
  echo "clean and load $tableName  . . ."
  mysql -u person -ppass -h localhost -D my_development << EOF
  delete from $tableName ;
EOF
  mysqldump -u person -ppass -c -n -t --skip-opt prototype_test $tableName | mysql -u person -ppass -h localhost -D my_development
end
rm table_list.tmp
echo "Done."

If you want to move the data from just one table, that's pretty easy too:

#!/bin/csh
# clean and reload the specified table in development from table in test DB
# USAGE:  reload_table table-name
echo "clean table $1  . . ."
mysql -u person -ppass -h localhost -D my_development << EOF
delete from $1 ;
EOF
echo load $1
mysqldump -u person -ppass -c -n -t --skip-opt my_test $1 | mysql -u person -ppass -h localhost -D my_development

These two scripts should allow you to quickly change your DB, blow it away and re-run your initial migration, get your fixture data into the test DB as your work through the tests, and then slide the data back into development. Once you get beyond the early stages of development, you'll want to increase change control and encapsulate your changes in migrations, and this technique may become less useful. But while things are flying fast and loose in the early stages, it really helps to keep things loose, and test driven to have the test data be the data in the system. If you get tired of writing YAML, once your prototype is running, you can use your system to help create test data and dump it to fixtures using the ar_fixtures plugin.

Diagrams to Migrations using OmniGraffle

2007-07-01T05:19:00Z

Once you go through the pain of setting up SQL Fairy (CPAN, yikes!), you can be designing your database visually with OmniGraffle in collaboration with other non-Railsy folks, and then generate your Ruby migration and stick it directly into your Rails project.

I'm in the process of embarking on a new Rails project with a moderate sized database. This project is based on an existing system, but with considerable redesign. I wanted to find a way to document the design so that other non-Railsy people can provide feedback on the design as it evolves, and do it in such a way that it would be natural to keep the documentation in sync with the Rails project.

Most DB tools are designed around an old-school method for database configuration management - the larger tools directly manage the database schema by pumping SQL through an ODBC connection. The smaller tools are focused on just visualizing an existing database. Neither of these options is well suited to developing in Rails. I'd really like to use the diagram as part of the creative process of developing the database rather than a post-hoc documentation step, and I don't want to short-circuit the Rails development process of managing the database through migrations.

After some searching I found graffle2sql.com - a site that takes database diagrams created with OmniGraffle and generates the SQL for the tables in the diagram. This is close, but what I really need is the Ruby for a migration, not SQL. I've been down this road enough to know that if the document requires a lot of tweaking to get it into the development process, it will quickly become stale and useless. A little more searching turned up SQL Fairy. This is a parser-based project that provides a wide variety of tools for manipulating SQL. This allowed me to set up a pipeline from an OmniGraffle DB diagram to a Ruby migration file using the following steps:

Make a DB diagram in OmniGraffle using the Graffle2SQL template
Convert it to SQL at graffle2sql.com
Use SQL Fairy to generate a YAML intermediate DB description
Use the template tool to rearrange the schema data into the general structure of a Ruby migration
Use string substitution to convert SQL type information into Ruby, yielding a Ruby migration file.

The last step is the only imperfect part of the process. The type information from SQL Fairy is somewhat limited, so complex types like decimal(x,y) don't translate properly. I'm not a Perl wizard (SQL Fairy is written in Perl) so there may be a better way to setup the template file to cover these cases.

In order to set this up, you need to install SQL Fairy and the template module. The template file I used is:

class InitialSchema < ActiveRecord::Migration
  # This file was auto-generated from SQL
  def self.up
[% FOREACH table IN schema.get_tables %]
    create_table "[% table.name %]" do |t|
  [% FOREACH field IN table.get_fields -%]
    t.column :[% field.name %],  :[% field.data_type %]  [% field.size %], :null => [% field.is_nullable %], :default => [% field.default_value %]
  [% END -%]
  end
[% END -%]
  end

  def self.down
[% FOREACH table IN schema.get_tables -%]
    drop_table "[% table.name %]"
[% END -%]
  end
end

and the script that drives the whole conversion process (OK, it's not Ruby, so shoot me) is:

#!/bin/csh
# syntax:  convert.csh file-root
# where the file to input is file-root.sql
# output will be in file-root.rb - the intermediate file file-root.yaml is removed

echo 'Generating YAML file . . .'
sqlt -f MySQL -t YAML $1.sql > $1.yaml
echo 'generating Ruby file . . .'
sqlt -f YAML -t TTSchema --template migration.tt $1.yaml > $1.txt
echo 'cleaning data type information . . .'
cat $1.txt | sed 's/:int  11/:integer/' | \
             sed 's/:varchar \([0-9]*\)/:string, :limit => \1/' | \
             sed 's/:char \([0-9]*\)/:string, :limit => \1/' | \
             sed 's/:tinytext  [0-9]*/:string/' | \
             sed 's/:timestamp  0/:timestamp/' | \
             sed 's/:date  0/:date/' | \
             sed 's/:float  ARRAY(0x1b06ac4)/:float/' | \
             sed 's/, :null => 1//' | \
             sed 's/, :default => $//' | \
             sed 's/, :null => 0/, :null => false/' | \
             sed 's/,  :string, :limit =>  255/,  :string/' | \
             sed -e '/, :id => false do |t|/N' -e 's/, :id => false do |t|\n      t.column :id,  :integer, :null => false/ do |t|/' > $1.rb
rm $1.yaml $1.txt
echo 'done.'

Once you go through the pain of setting up SQL Fairy (CPAN, yikes!), you can be designing your database visually in collaboration with other non-Railsy folks, and then generate your Ruby migration and stick it directly into your Rails project. This approach doesn't detect differences, so it's best suited for cases when you're working out the base schema by migrating back to VERSION=0, making changes to the migration, then migrating forward to VERSION=1 again. OmniGraffle is free for diagrams with 20 objects or less, so you can play around with the process to find out whether it works for you.

More Irb Tips

2006-10-20T19:23:00Z

I was talking to someone (can't remember who) at Rubyconf and they brought up #local_variable. This method shows you the names of all the local variables in scope.

Funny, I don't remember this function, and if I did know it, I had forgotten about it. So, I fired up irb and type it in to see what I get back:

local_variables   #=> ["_"]
x = 5             #=> 5
local_variables   #=> ["_", "x"]

Hmmm, this is interesting. I know there is a variable in irb that contains the value of the last expression, but it is not easy to remember. But, it looks like _ will do the same thing.

I'm always doing math expressions where I need the value of the previous expression. So now, instead of up-arrow and add to the equation, I can use _.

5/7.0    #=> 0.714285714285714
2*_      #=> 1.42857142857143

Of course, you can you this for any irb expression, and best of all, it is a variable name I can actually remember. :)

Quieting irb's Return Value

2006-10-06T16:38:00Z

It's easy to be overwhelmed by irb's automatic printing of return values, especially when dealing with collections of ActiveRecords in Rails' script/console.

This simple nugget will save you from an ocean of return values.

At the irb [or script/console] prompt, type:

conf.return_format = ""

The default (to print the return value) is:

conf.return_format = "=> %s\n"

To answer Jim's question (in the comments): you can easily limit the number of characters put into the result string by sprintf:

conf.return_format = "=> %.512s\n"

Replace 512 with your preferred return value character length limit.

Autovivification with Hashes

2006-10-05T15:47:00Z

For you Perl folk, you may be familiar with the term autovivification. In essence, that is where an empty data structure gets automatically created if it is referenced, but does not exist.

For example, consider a hash. Let's say we want a two dimensional hash such that we can write:

12	points = Hash.new points[1][2] = 5

The problem here is that points[1] must return a hash, and this hash is in turn accessed with the key '2'. It would be the same thing if we wrote:

123	points = Hash.new tmp = points[1] tmp[2] = 5

All this is fine and good except that it does not work. points[1] returns nil because the default return argument for a Hash when it is referenced with a non-existant key is nil. But, we can change that.

1234	points = Hash.new(Hash.new) tmp = points[1] #=> {} tmp[2] = 5 #=> 5 points #=> {}

Well, that wasn't such a good idea. points[1] returned a hash, but it never got applied to the original points hash. Even if we write this as points[1][2] = 5, we still get points to be an empty hash.

Blocks to the Rescue

What we need is to use the block form of Hash.new If a block is supplied to #new, it will be called with hash object and the key. So, to repeat the above, but with a block, we write:

12	points = Hash.new { \|h,k\| h[k] = {} } points[1][2] = 5 #=> {1=>{2=>5}}

If that was too quick to realize what just happend, we can break it down into two steps, like we did above, and see just what is happening:

    points = Hash.new{ |h,k| h[k] = {}}
    tmp = points[1]                         #=> {}
    points                                  #=> {1=>{}}
    tmp[2] = 5                              #=> 5
    points                                  #=> {1=>{2=>5}}

The expression points[1] actually creates a key-value pair and automatically sets the value to {}, just like we told it to inside the block that was passed to Hash.new.

Customizing Vivification

Now, that's all well and good, but what if we wanted to have our final value be an array. For example:

12	points[1][2] << 5 points[1][2] << 10

For this, we just need to make a small addition to our block that we pass to #new.

123	points = Hash.new { \|h,k\| h[k] = Hash.new { \|hh,kk\| hh[kk] = [] }} points[1][2] << 5 #=> {1={2=>[5]}} points[1][2] << 10 #=> {1={2=>[5, 10]}}

Well, now that's pretty nifty.

Endless Vivification

What if we need a hash that has an arbitrary number of arguments. For example, say we wanted to represent a directory tree with a hash. We could end up with something like:

12	tree["etc"]["samba"]["smb.conf"] #=> contents of "smb.conf" tree["home"]["me"][".ssh"]["authorized_keys"] #=> list of keys

Here is where we enter into the magic and mysterious land of recursive lambda's. In the example above, it would be equivalent to write the Hash constructors as:

12	blk = lambda { \|h,k\| h[k] = {}} points = Hash.new &blk

12	blk = lambda { \|h,k\| h[k] = Hash.new { \|hh,kk\| hh[kk] = [] }} points = Hash.new &blk

But, to have a 'never ending' constructor, we define the block using recursion:

    blk = lambda { |h,k| h[k] = Hash.new &blk }
    tree = Hash.new &blk
    tree = Hash.new &blk   #=> {}
    tree[:a][:b][:c] = 4   #=> 4
    tree[:x][:y] = 1       #=> 1
    tree[:fred] = "sally"  #=> "sally"
    tree                   #=> {:a=>{:b=>{:c=>4}}, :x=>{:y=>1}, :fred=>"sally"}

Now, isn't that cool!

It's not too bad once you get used to it, but it can be a bit confusing at first.

RubyConf 2006

2006-10-04T15:02:00Z

Well, RubyConf 2006 is nearly upon us and it looks like the state of Texas is going to be well represented by six Austinites.

Those attending are:

Jim F.
Hal F.
Matt
Wayne W.
David B.
Robert R.

Wayne must have made some kind of deal with a Leprechaun, because he was able to purchase a ticket just this week from zdennis. I bought mine ten minutes before the announcement that registration was open, and consider myself lucky since registration was only open for two hours.

Welcome

2006-10-04T14:42:00Z

Just wanted to welcome everyone to our Ruby website, and give it a little test drive.

Thanks to Mars for putting this together, with some help from Wayne, and thanks to Wayne for providing the hosting.

The Workings

2006-10-02T01:28:00Z

Our server set-up is as follows:

austinruby.com =

apache (port 80) <-> pound (load balancer) <-> mongrels (Rails servers)

We're starting out with three mongrels in the pound. Time will tell if more are required.

The original plan to serve static files from a separate virtual host has been nixed by my impatience.

See: The Workings r2 for updates since this article.

activate

2006-10-01T20:30:00Z

We're finally on-line! I love progress.

Final web server configuration is imminent. Then, I'll start adding authors.