Legacy Migrations made easy…

Legacy Migrations is a little gem that helps you migrate data from an old database structure to a new db and db structure. It keeps a status report for failed validations, and also lets you update data if all you need is a nightly data update. Here’s a quick sample that transfers data from the model Person to the model Animal (assuming the models are subclasses of either ActiveRecord::Base or CSV::Table):

require 'legacy_migrations'

#Assuming you've already created classes for source and destination tables
transfer_from Person, :to => Animal do
  match_same_name_attributes :only => [:sex, :age]
  from :name, :to => :pet_name
  from :from_record, :to => :species do |from_record|
    "Homo Sapien"
  end
end

Here’s the full list of options you can pass to ‘transfer_from’ (at the time of writing, of course):

  • :limit – Set a limit to the number of rows to transfer. This is useful when you’re trying to find faulty data in the source table, and don’t want to run the entire dataset.
  • :validate – Default = true. Use ActiveRecord validations when saving destination data. Reports errors in the StatusReport (a singleton object)
  • :source_type – Default = :active_record. Sets the source destination type. Options: :active_record: Assumes the From option is a class that inherits from ActiveRecord::Base, then iterates through each record of From table by using From.all.each… :other: Assumes the From option is an iterable ‘collection’ whose elements/items can respond to all columns speficied in the given block.
  • :store_as – Store the resulting generated record to be used in a future transfer_from or update_from operation. You can access this object like so:
    transfer_from Person, :to => Species, :store_as => 'new_species' do
      from :name, :to => :species_name
        "Homo Sapien"
      end
    end
    
    transfer_from Person, :to => Animal do
      from :pet_name, :to => > :name
      stored :new_species, :to => :species  # Uses the species that we
                                            # created in the previous
                                            #  transfer_from
    end
    

Aside from those helpers, you also get the ‘match_same_name_attributes’ helper that…wait for it…matches same-name columns between the source and destination tables. Here’s a quicky:

transfer_from Person, :to => Species, :store_as => 'new_species' do
  match_same_name_attributes :only => [:name, :age, :date_of_birth]
  # There's also an 'except' option, and if 
  # you don't use either, then the script
  # just transfers all same-name columns.
end

You can also use the gem to update data periodically using the ‘update_from’ operation. The
update_from operation uses the well-known squirrel plugin syntax to match records from the source database, to records in the destination database. If the source record doesn’t exist in the destination, then it gets automatically created. Think of this as ‘find_or_create_by’ on steroids. For example, let’s say you want to update first names in the Animal model, from the latest data in the People model, and the identifying and matching features between the source and destination tables are the date_of_birth and last_name columns:

update_from Person, :to => Animal do
  based_on do |from_record|
    date_of_birth == from_record.dob
    last_name ==  from_record.last_name
  end

  from :first_name, :to => :first_name
end

Here’s the git repo:

http://github.com/btelles/legacy_migrations

To install it just use:

gem install legacy_migrations

If you have any suggestions, please let me know!

Update:

By the way, this gem is considered beta software and to get a better idea for how to use it, you may want to look at the rspec files. They’re the most comprehensive documentation.

Here are the slides to a quick presentation I gave at our local Ruby Brigade, in case anyone’s interested.

2 Responses to “Legacy Migrations made easy…”

  1. Mario Olivio Flroes said:

    May 31, 10 at 4:07 pm

    This is really awesome. This is the legacy migration gem I have been waiting for! Though you should link back from your repo to this blog post in your readme or something. Also, why name make http://github.com/btelles/legacy_migrations an actual http link to the repo?

    :)

    I’ll probably be using this in the next two weeks. If there is anything I can contribute I will. Thanks again!

  2. Anonymous said:

    May 31, 10 at 4:49 pm

    Thanks for the suggestion Mario, and I’m glad you like the gem! I’ve made the changes you mentioned.
    Cheerio!


Leave a Reply