A Year of Commits

Why You Should Avoid Models in Rails Migrations

2021-04-10T08:00:00-04:00

Humble Beginnings

A simple Rails application exists with two models, Books and Authors.

class Book < ApplicationRecord
  belongs_to :author
end

class Author < ApplicationRecord
  has_many :books
end

After some domestic success, this simple Rails app goes international. A new column is required on the books table: country to denote which country published the Book.

A Reasonable Migration

To preserve data integrity, the country column should not allow null since null isn’t a country. Existing books also need to have a country set. This is all accomplished within a single migration in 3 steps:

Add a column
Update column for existing books with a reasonable value
Add a constraint that the column can not be null

class AddCountryToBooks < ActiveRecord::Migration[6.1]
  def up
    add_column :books, :country, :string, length: 2

    Book.update_all(country: 'US')

    change_column_null :books, :country, false
  end

  def down
    remove_column :books, :country
  end
end

This migration does the job but has hidden implications. A developer working in isolation may never run into the trap lurking in this code, but a team could.

Working with Others

Two developers work on this application, divvying up the wild world of books and authors but maintaining healthy work life balances. One developer goes on vacation and has a surprise waiting for them when they return.

	Developer 1 👩‍💻	Developer 2 👨‍💻
Day 1	Write Code	Write Code
Day 2	😎️ Time Off	🏗️ Create Migration
Day 3	😎️ Time Off	📛️ Rename Class
Day 4	😎️ Time Off	Write Code
Day 5	Run Migrations -> 🔥️ Error	Write Code

While Developer 1 was away, the Developer 2 was busy. They wrote the above migration, updated existing data and then a new feature request was completed: A model rename.

Developer 1 returns, updates their local environment, and runs rake db:migrate:

== 20210407191819 AddCountryToBooks: migrating ===================
-- add_column(:books, :country, :string, {:length=>2})
   -> 0.0015s
rake aborted!
StandardError: An error has occurred, this and all later migrations
canceled:

uninitialized constant AddCountryToBooks::Book

Book was renamed to Novel breaking the previous migration.

Avoiding Models

Using SQL

When used correctly, Rails migrations can count on the database being in an expected state. The application code moves from commit to commit but database structure (should) only change with migrations. Therefore, even if the Book class is now Novel, the database table books still exists at the time of the AddCountryToBooks migration.

Instead of Book.update_all, writing an update query in SQL and using the base ActiveRecord connection ensures this migration survives model changes like a rename. Because we can count on the state of the database structure, referring to the books table is safe while referring to the Book model may not be.

class AddCountryToBooks < ActiveRecord::Migration[6.1]
  def up
    add_column :books, :country, :string, length: 2

    ActiveRecord::Base.connection.execute(
      "UPDATE books SET country = 'US'"
    )

    change_column_null :books, :country, false
  end

  def down
    remove_column :books, :country
  end
end

Using Temporary Models

If writing SQL isn’t ideal, a temporary model can be used within the migration. Here, a new Book model is defined to manipulate the books table. Even if app/models/book.rb has been changed to app/models/novel.rb, this definition of Book will be valid within the scope of this migration.

class AddCountryToBooks < ActiveRecord::Migration[6.1]

  class Book < ApplicationRecord; end

  def up
    add_column :books, :country, :string, length: 2

    Book.update_all(country: 'US')

    change_column_null :books, :country, false
  end

  def down
    remove_column :books, :country
  end
end

Either of these migration will run correctly for Developer 1 no matter how many model renames happened while they were on vacation.

Problem at Scale

It might not seem like that big of a deal to have a single broken migration. But, within a larger team or with more complex migrations, this can easily become a painful hurdle a team needs to navigate.

Allowing migrations to rely on only the database as a source of truth enables all developers on the team to run migrations without issue. That is until the database or team inevitably get too large and people do database restores instead of running migrations, then this issue is far less common.

Making RSpec Tests More Robust

2020-07-19T08:00:00-04:00

RSpec is a popular framework for testing Ruby code. With an expect assertion, a developer can make sure their code calls the proper method or an acceptable result is returned. The expect().to receive matcher in a test overrides the default implementation and can cause some unintended side effects.

To demonstrate this potential problem, assume a very simple API client exists that can update models.

Testing an API Client

An example api_client.rb class defines a single put method that calls the underlying API with a Faraday connection.

# api_client.rb

class APIClient
  def put(url, body)
    client.put(url, body)
  end

  private

  def client
    Faraday.new('https://some-cool-api.com')
  end
end

Inheriting from api_client.rb is my_model.rb which defines an update method.

# my_model.rb

class MyModel < APIClient
  def update(payload)
    put('/my_models/1', payload)
  end
end

A typical test for the update method on MyModel in RSpec might look like:

# my_model_spec.rb

describe MyModel do
  describe 'update' do
    it 'updates the model' do
      expect(subject).to receive(:put)
      subject.update({ foo: :bar })
    end
  end
end

The subject in the above test is MyModel.new and is expected to receive the method put. Since MyModel#update calls the put method, this seems like a reasonable test.

$: rspec my_model_spec.rb
.

Finished in 0.0054 seconds (files took 0.07101 seconds to load)
1 example, 0 failures

Running the test produces a passing result, the MyModel class calls its parent method correctly and all is well. Until something changes that the test is unable to detect.

Breaking the Contract

If MyModel’s contract changes, the test should fail. If, for instance, the put method on the APIClient class is removed or commented out, the update method on MyModel would no longer work.

# api_client.rb

class APIClient
  # def put(url, body)
  #   client.put(url, body)
  # end

  private

  def client
    Faraday.new('https://some-cool-api.com')
  end
end

However, the test still passes despite this method being removed.

$: rspec my_model_spec.rb
.

Finished in 0.00523 seconds (files took 0.0699 seconds to load)
1 example, 0 failures

So what is going on here?

The culprit is this line: expect(subject).to receive(:put). The test happily accepts the call to APIClient#put despite it being removed. This is obviously problematic and, in a worst case scenario, could lead to an outage if a test suite is the only gateway to production code.

Starting an irb repl and calling the problematic method reveals that the code is no longer working despite the test’s result.

2.5.5 :001 > model = MyModel.new
 => #<MyModel:0x00005585f950ca10>
2.5.5 :002 > model.update(foo: :bar)
Traceback (most recent call last):
        3: from /home/yez/.rvm/rubies/ruby-2.5.5/bin/irb:11:in `<main>'
        2: from (irb):2
        1: from /tmp/foo.rb:8:in `update'
NoMethodError (undefined method `put' for #<MyModel:0x00005585f950ca10>)

Since this code communicates to an API, validating that the underlying put request is done successfully can be tricky. Unlike tests in a Rails application, MyModel#update can not be validated in a local database.

With a passing test and broken code, it is clear that wrong thing has been tested. As is, the test only asserts that a method calls another method and happily ignores everything else. One possible solution could be to add and_call_original to the end of the expect(). to receive line. This would make sure the put method is really there but would also make a real HTTP request in a test (which is generally bad practice).

An easy way to make this test more robust without going overboard is to stub the response of the HTTP client. Instead of mocking the put method in api_client.rb, tools like webmock can be utilized to stub the Faraday response. This enables the code to fake an HTTP request instead of of calling out to https://some-cool-api.com.

Testing with `Webmock`

The webmock gem allows developers to define custom responses for specific HTTP requests.

After installing the gem with gem install webmock, a stub can be written that matches the URL and request body to return a specific response.

WebMock.stub_request(:put, 'https://some-cool-api.com/my_models/1').
  with(body: /foo.*bar$/).
  to_return(body: '{ "success": true }')

In this case the stubbed response is { "success": true } and will be returned for any put requests to https://some-cool-api.com/my_models/1 with a request body matching foo followed by bar. This stub can be added in a shared file or at the beginning of my_model_spec.rb.

With the stub added, my_model_spec.rb can be updated to remove the old expect().to receive line and instead validate the response.

# my_model_spec.rb

describe MyModel do
  describe 'update' do
    it 'updates the model' do
      response = subject.update({ foo: :bar })
      expect(JSON.parse(response.body)['success']).to eq(true)
    end
  end
end

The stub works and running my_model_spec.rb shows the test passes.

$: rspec my_model_spec.rb
.

Finished in 0.00531 seconds (files took 0.33028 seconds to load)
1 example, 0 failures

Now, if APIClient is changed in the same way as before, the test will appropriately fail and alert the developer that the put method does not exist.

# api_client.rb

class APIClient
  # def put(url, body)
  #   client.put(url, body)
  # end

  private

  def client
    Faraday.new('https://some-cool-api.com')
  end
end

$: rspec my_model_spec.rb
F

Failures:

  1) MyModel update updates the model
     Failure/Error: put('/my_models/1', body)

     NoMethodError:
       undefined method `put' for #<MyModel:0x000055f148a80018>
       Did you mean?  puts
                      putc
     # ./my_model_spec.rb:17:in `update'
     # ./my_model_spec.rb:30:in `block (3 levels) in <top (required)>'

Finished in 0.00298 seconds (files took 0.4606 seconds to load)
1 example, 1 failure

Stricter Settings

RSpec allows developers to enable the flag verify_partial_doubles which will cause the original test to fail when the put method is removed. This setting is turned off by default.

RSpec.configure do |config|
  config.mock_with :rspec do |mocks|
    mocks.verify_partial_doubles = true
  end
end

With this setting in place, running the original code and test produces a failing result.

# api_client.rb

class APIClient
  # def put(url, body)
  #   client.put(url, body)
  # end

  private

  def client
    Faraday.new('https://some-cool-api.com')
  end
end

# my_model_spec.rb

describe MyModel do
  describe 'update' do
    it 'updates the model' do
      expect(subject).to receive(:put)
      subject.update({ foo: :bar })
    end
  end
end

$: rspec my_model_spec.rb
F

Failures:

  1) MyModel update updates the model
     Failure/Error: expect(subject).to receive(:put)
       #<MyModel:0x000055d4a4f740a8> does not implement: put
     # ./my_model_spec.rb:35:in `block (3 levels) in <top (required)>'

Finished in 0.00877 seconds (files took 0.46471 seconds to load)
1 example, 1 failure

This alternative does not require the use of a new gem and will fail correctly if the underlying contract is removed. However, if the put method or any layer between it and the API request changes, this test is not guaranteed to save a developer from that issue.

Determining what layer to mock and what to explicitly test is a very case by case basis.

Keep Tests Robust

Mocks and stubs in RSpec allow developers to make important assertions about their code. Unfortunately, mocking can also cause false positives when modifying real code.

Validating that the proper settings are in an application is a great first step towards more a more robust test suite.

Stubbing the response for the underlying API in this example enables a much more robust test that can alert developers when they make breaking changes. However, if the API response from https://some-cool-api.com changes, the webmock stub must be updated.

Stubbing and mocking tools should be used on a case by case basis. In the case of testing a third party API integration, providing canned responses in the test suite makes sense. In other “normal” Ruby or Rails tests, verifying doubles or other built in RSpec matchers could be all that’s needed.

Ruby Processes and Threads - Configuring a Web Server

2019-06-18T08:00:00-04:00

Multiple popular Ruby web servers exist. Each Ruby application is different and the ultimate tl;dr for configuring a web server is: it depends. This post will not prescribe one web server or configuration over another and will instead explain internal components most popular servers contain.

In order to facilitate more than one request at a time, a Ruby web server implements Threads, Processes, or both. These tools are used to enable concurrency and are beneficial in different ways.

`Threads`

Threads in Ruby are a solution for concurrent programming and can alleviate slow downs due to blocking code. This blocking is usually referred to as “I/O” or Input/Output blocking and occurs when a program must reach out for additional information. External API calls, reading from disk, and querying a database are all examples of blocking operations. When using multiple Threads, an application can continue to function while one Thread is waiting.

Most Ruby code in the wild is running on MRI (if you’re not sure what you’re using, there is a good chance this is what you use). Because of this, Ruby Threads are subject to the Global Interpreter Lock or GIL. The GIL prevents any two threads in the same process from running at exactly the same time making true parallelism not possible.

`Processes`

One way to allow for true parallelism in Ruby is to use multiple Processes. A Ruby Process is the instance of an application or a forked copy. In a traditional Rails application, each Process contains all the build up, initialization, and resource allocation the app will need.

Running multiple Proccesses can enable more efficient usage of some server resources like the CPU but is not without its downsides. Because each process must boot and provision an entire app, memory usage or database connection saturation can become a limiting factor.

Tempering the Metal

When configuring on a web server, an application’s request shape is the most important factor to consider. Web application requests can demand I/O operations, computationally taxing operations, both, or neither.

This example app has one controller named index_controller and a single route defined: index.

`index_controller.rb`

class IndexController < ApplicationController
  def index
    interval_sleep
    render json: { hello: :there }
  end

  def interval_sleep
    sleep(rand(2..10).to_f / 50)
  end
end

To emulate I/O blocking, this code uses a sleep call to randomly stall between 40ms and 200ms.

This application’s web server is Puma since it gives allows scaling of both Threads and Processes independently.

Most of the configuration is unchanged and the following two lines are all that will change during this testing:

`config/puma.rb`

threads_count = ENV.fetch("RAILS_MAX_THREADS") { 1 }
workers(ENV.fetch("WEB_CONCURRENCY") { 1 })

Here the configuration sets the number of Threads and Processes the application can use to fulfill requests.

With the configuration set and ready to go, the server is started with bundle exec rails server.

Under Siege

Now that the application code is written and the web server is configured, it can receive requests. To measure throughput, latency, memory usage, and CPU utilization, the tool Siege is used.

Siege is a benchmarking tool that has a very simple interface. Siege can be run via a small bash script with a few configurable options for each test.

`bin/siege.sh`

#! /bin/bash

CONCURRENCY=30
URL='http://localhost:3000/index'
CONTENT_TYPE='application/json'
REPS=20

siege -b --content-type $CONTENT_TYPE -c $CONCURRENCY -r $REPS $URL

# flags:
#   -b - benchmark means no gap between requests
#   -c - concurrency is number of requests to make at one time
#   -r - repititions is the number of times to run
#         the same amount of concurrent requests

This executable will request localhost:3000/index which routes to index_controller#index at a concurrency of 30 workers for 20 repetitions resulting in 600 total requests.

Keeping this script static allows for the web server’s configuration to change and each test to be congruent with others.

Metrics for Success

CPU Load Average

CPU load average is a measurement of how busy each processor is on a server. Basically, CPU load average can be used to determine how long a CPU is idling and waiting for something to do. These numbers are very dependent on the number of processors an application has. On a Linux or MacOS machine, you can cat the /proc/cpuinfo file to get a break down of how many processors are available to you.

On the machine used to test this application, 4 processors are available. Translating that to percentage of usage, we would see a value of 4.0 for our CPU load average if all processors were being completely used. Any value higher than that and we’d have work waiting to be done by a processor. Conversely, a value much lower than 4.0 indicates that some processors are not doing anything.

CPU load average is displayed with 3 numbers ex: 0.1 0.4 0.8 which translates to 1 min 5 min 10 min averages. If the 1 min average is higher than the 5 min and 10 min averages, server load is increasing. If the 1 min average is lower than the 5 min and 10 min averages, server load is decreasing. The only metric reported in this summary will be the 1 min load average since it is measured while the server is under siege.

Memory Usage

Memory Usage is the amount of memory (RAM) used by the application. When a server runs out of RAM, it creates virtual memory out of partitioned space on the disk to overflow into. This is called swapping and it can greatly reduce the efficiency of a server. Along with CPU load average, making sure a server’s memory usage is below capacity is an important way to keep a healthy and happy application.

The amount of memory an application uses in total should be balanced by any other processes running on that server while leaving some amount of room for utility applications. Using more than the memory available on a server will lead to performance issues.

I/O Intensive Threads v Processes and Throughput

With the metrics explained, let’s see the results of the siege on the I/O blocking application:

Processes	Threads	Duration(seconds)	Transactions/second	Memory(mb)	CPU Load
1	1	77.45	7.75	73	0.57
1	10	16.06	37.36	75	0.54
1	100	15.89	37.76	77	0.39
2	1	39.64	15.14	136	0.52
2	10	8.14	73.71	146	0.78
10	1	8.16	73.53	680	0.79
10	10	3.89	154.24	715	0.54

For the example I/O bound application, CPU load average never rises above 1.0 regardless of Process count. This means that the server is not using its processing power to its full potential because the requests do not demand it.

On the other hand, the number of Threads makes a big difference for I/O blocking operations. With a single Thread, it takes 77 seconds to complete all 600 requests. Just increasing that number of Threads to 10 makes the same work take 16 seconds! But look at what happens when Threads are increased to 100: the same jump in throughput doesn’t happen. This shows that the number of Threads an application can utilize has an upper limit .

Additionally, the memory usage quickly increases with the number of processes on the machine. For a very simple application like this with no database, the maximum memory usage of 715mb isn’t huge but on a traditional Rails application, each process could easily have a few hundred mb of memory usage. Along with hard caps for database connections, maintaining too many processes can quickly use more memory than a server has available.

Similar to Threads, infinitely scaling Processes is not a sure way to get the best results. In this example, a balance of 10 Processes and 10 Threads gives the best throughput at nearly a 700% memory increase.

CPU Intensive Application

To see how the number of Threads and Processes impact a more CPU intensive application the index_controller has been changed:

`index_controller.rb`

class IndexController < ApplicationController
  def index
    tax_cpu
    render json: { hello: :there }
  end

  def tax_cpu
    Array.new(100_000) { rand(100_000) }.sort!
  end
end

Now each request will create an array of 100,000 integers and sort them.

The same siege command is run against the same Thread and Process configuration counts.

Processes	Threads	Duration(seconds)	Transactions/second	Memory(mb)	CPU Load
1	1	18.34	32.72	114	0.64
1	10	18.16	33.04	134	0.79
1	100	18.45	32.52	184	1.05
2	1	10.26	58.48	230	1.12
2	10	11.73	51.15	270	0.89
10	1	8.73	68.73	1040	2.13
10	10	8.95	67.04	1180	2.65

CPU Intensive Threads v Processes and Throughput

For the CPU intensive code, only increasing Threads does nothing for overall execution time. The 600 requests require around 18 seconds no with 1 Thread and with 100 Threads.

On the other hand, doubling the number of Processes cuts execution time by almost half! However, increasing the number of Processes from 2 to 10 (a 500% increase) does not decrease the execution time by another 500%. Just like in the I/O bound application, scaling Processes will eventually experience diminishing returns, making each additional process less valuable.

During the CPU intensive code testing, the server never experienced a fully saturated load average of 4.0. This is because the example code was not all that taxing. In a real application serving end users, it is surprisingly easy to make full use of all CPU resources with enough Processes.

A Little Realism

In the real world, an application is not likely to be only CPU or I/O intensive but rather some mixture of both. To help add a little bit of variety to this test, the two operations are combined and run at a percentage:

`index_controller.rb`

class IndexController < ApplicationController
  def index
    if rand(3) == 1
      tax_cpu
    else
      interval_sleep
    end
    render json: { hello: :there }
  end

  def tax_cpu
    Array.new(100_000) { rand(100_000) }.sort!
  end

  def interval_sleep
    sleep(rand(2..10).to_f / 50)
  end
end

Now this controller will be I/O bound approximately 66% of the time and CPU bound the other 33%.

Note: While more realistic, this application is not a perfect representation of any real request structure out there and every individual app should be evaluated on its own.

Running the same siege command a final time on this application yields:

Processes	Threads	Duration(seconds)	Transactions/second	Memory(mb)	CPU Load
1	1	61.40	9.7	86	0.65
1	10	14.28	42.02	111	0.58
1	100	14.03	42.77	167	1.31
2	1	30.24	19.84	174	0.91
2	10	6.97	86.08	206	0.93
10	1	6.74	89.02	790	0.79
10	10	4.44	135.14	960	1.43

The number of Threads and Processes each have an impact in these results. Unlike the other versions of index_controller.rb, if too few Threads are allocated, the application will wait on I/O operations. Likewise, too few Processes tie up the CPU, blocking other requests from being served.

While the fastest possible results is 10 Processes and 10 Threads, it is important to look at the load average and memory usage. For nearly 1/5th the memory, 2 Processes and 10 Threads accomplishes the task in only 2 more seconds. A server that has memory issues or an application that is memory starved might adopt this more modest configuration to save memory. The 10 Processes, 10 Threads configuration also has the highest CPU load average. Just like memory concerns, a server with lower processing power might be better served with fewer Processes competing for CPU time.

Summary

The number of Threads and Processes can be the deciding factor on a web application’s performance. Additionally, understanding how the number of Procesess and Threads impact a server is useful for choosing and configuring a web server. Any code which uses concurrency can benefit from this knowledge. Balancing memory usage, external resource connection limits, and CPU load average is a great way to make sure a server is provisioned correctly and might even save some money.

When evaluating the best configuration for your web server, this simple list is a great starting point:

Use only the available memory for the server.
Use Threads for blocking code and Processes for CPU intensive code.
Use tools like siege for benchmarking and testing different configurations.

Things like thread starvation, semaphores, thread pooling and other concerns regarding concurrent programming were intentionally not mentioned in this post. Understanding how Threads and Processes work at a high level can enable the configuration and choice of web server, but if you want to write your own complex concurrent Ruby code, I’d suggest concurrent-ruby.

Improving Remote Work as a Software Engineer

2018-02-18T07:00:00-05:00

After working in the Bay Area for many years, I decided that being able to afford a house was something I’d like to do. That and family led me back to the East Coast and a new chapter of my career began: Remote Software Engineering.

Transitioning to working remotely in any profession has its challenges. After a few years as a remote software engineer, I have learned many things that don’t work and a few that do. Most of the things that didn’t work helped point me in the direction to those that do and hopefully these bits of insight can help others. These tips are most relevant to someone who is the only or one of few remote engineers on their team.

Presence

The largest and most obvious challenge one will face as a remote software engineer is a lack of presence. In an office, people can have small side conversations, ask about one another’s weekends, or simply say “hello” in the hallway. While seemingly inconsequential, these tiny interactions establish presence.

Maintaining presence is important if an engineer is a member of a team or has many stakeholders in their work. Presence can make the difference between being perceived as “a member of team” as opposed to “that one person who works on… What is it they do again?”. The former helps a person feel unified with the team they work for, the mission they are striving towards, and the product they are building; the latter is hollow and temporary.

So how does a remote software engineer maintain presence? With consistent yet relevant communication. Whether a company uses Hipchat, Slack, Google Hangouts, or another chat application, instant communication should already be baked into an organization’s workflow and can be utilized to establish presence. While over-communication is rarely valuable, sending short and well constructed messages about current progress or next steps helps keep team members informed and questions answered.

If instant messaging isn’t on the menu, status emails might do well in its place. There is nothing worse than an employer, manager, or team lead thinking that a remote software engineer isn’t delivering what they should. A single daily email can help provide visibility. The email doesn’t need to be very verbose. It can be as simple as a list of what’s been done today, any blockers that have been encountered, or an update to a nearing deadline. It might seem tedious or unnecessary but, to a manager or stakeholder, such an email can help reassure them of progress and collaboration.

Tone

Since communication is so important for a remote software engineer, the tone in which that communication is even more important. Tone is incredibly hard to convey over text communication. Because of this, it might take a few tries to accurately convey negative messages without sounding like an extremely irritated or otherwise nasty individual.

Code reviews are a great example of how the right tone can make all the difference. Think of code reviews in the recent past, unless a team goes out of their way to be super positive, most code review comments are a little negative. For example, a typical code review might have comments like: “This doesn’t look right”, _“Change to here"_, or _"I don't agree with this"_. The reason these comments are not an issue for a local team is because this is not the only comments they exchange. Perhaps a local team will have one-off side conversations where they explain their opinions in a more positive way. Or, completely separate to the code review, a local team shares some amount of small talk that is likely positive. Whatever the case, the local team has a mixture of positive interactions along with (most likely) negative code review comments. The remote software engineer does not have this luxury. All they have to offer is their code review comments which, most likely, are not filled with a very positive tone. This is what must be adjusted.

Utilizing a positive tone in code reviews and other communication help remind people that their remote team member is just that: a valuable member of the team, invested in their work and the happiness of their colleagues. To enable this perception, the remote software engineer can find opportunities for positivity that might be easily overlooked. Simple comments on how a method’s signature is nicely defined or a code comment is especially helpful are easy things to point out during a code review that express positivity.

Availability

For those remote software engineers in the same timezone as the rest of their team, availability shouldn’t be an issue. For everyone else, there may be times in which “after hours” communication can help boost one’s reputation as being dependable and invested despite being remote.

Responding in off hours is not the same as being always available. People need and benefit from downtime and the remote engineer is no different. Because of this, the availability of a remote employee should be flexible but not unreasonable. Responding to a few emails or chat messages after hours or during a morning coffee before their “normal work day” can go a long way. After all, if a company is willing to be flexible and work with a remote software engineer, shouldn’t that engineer also be flexible?

A Few RSpec Helpful Hints

2017-07-12T08:00:00-04:00

Two main frameworks dominate the Ruby testing world: Rspec and MiniTest. RSpec is a very expressive testing framework with many great features and helpers to make tests readable. When writing RSpec tests, here are just a few not so obvious hints that could make tests even easier to write, read, and maintain.

Assuming a system exists with Books and Authors, let’s utilize these hints to make testing easy.

class Book
  attr_reader :title, :genre

  def initialize(title, genre)
    @title = title
    @genre = genre
  end
end

class Author
  attr_reader :books

  def initialize(name, books)
    @name = name
    @books = Array(books)
  end

  def has_written_a_book?
    !books.empty?
  end
end

`subject` and `let` Variables

A great way to keep specs DRY and readable are via subject and let variable declarations.

For example, if we want to assert an Author has a name, a test without let and subject variables might look something like:

describe Author do
  before do
    @book_genre = 'Historical Fiction'
    @book_title = 'A Tale of Two Cities'
    @book = Book.new(@book_genre, @book_title)
    @author_name = 'Charles Dickens'
    @author = Author.new(@author_name, [@book])
  end

  describe '#name'do
    it 'has a name set' do
      expect(@author.name).to eq(@author_name)
    end
  end
end

While correct, additional tests asserting number of books, a different name, or other things about this Author could become very verbose.

Instead, we can introduce subject and let variables to keep things DRY and reusable:

describe Author do
  let(:book_genre) { 'Historical Fiction' }
  let(:book_title) { 'A Tale of Two Cities' }
  let(:book) { Book.new(book_genre, book_title) }
  let(:book_array) { [book] }
  let(:author_name) { 'Charles Dickens' }
  subject { Author.new(author_name, book_array) }

  describe '#name'do
    it 'has a name set' do
      expect(subject.name).to eq(author_name)
    end
  end

  describe '#books' do
    context 'with books' do
      it 'has books set' do
        expect(subject.books).to eq(book_array)
      end
    end

    context 'without books' do
      context 'books variable is nil' do
        let(:book_array) { nil }

        it 'sets books to an empty array' do
          expect(subject.books).to eq([])
        end
      end

      context 'books variable is an empty array' do
        let(:book_array) { [] }

        it 'sets books to an empty array' do
          expect(subject.books).to eq([])
        end
      end
    end
  end
end

Instead of needing multiple before blocks to set and reset instance variables, this code utilizing let variables is concise and easy to read. More specifically, the way these tests work is: each time an it block is run, the let variables within the nearest context are used to initialize the subject.

By setting important let variables, a context that tests if subject.books is an array based on an input of nil or [] is as simple as changing the let definition: let(:book_array) { nil }.

Loose Expectations

When a test does not care about specifics, RSpec allows the use of general expectations and placeholders. These placeholders can help mitigate a test’s complexity by only focusing on what is truly important.

1. `anything`

Just like it sounds, the anything argument matcher can be used when a method requires an argument but the specifics do not matter for the test.

If an Author test wants to assert that a Book has been written, but doesn’t care about the title or genre of the book, anything can be used:

describe Author do
  describe #has_written_a_book?' do
    context 'when books are passed in' do
      subject { Author.new(name, books) }
      let(:books) { [Book.new(anything, anything)] }
      it 'is true' do
        expect(subject.has_written_a_book?).to eq(true)
      end
    end
  end
end

2. `hash_including`

When testing a method that expects a Hash, some elements of the Hash may be more important than others. The hash_including matcher allows a developer to assert one or many key/value pairs within a hash without needing to specify the entire hash.

Assuming the Book class has a method which instantiates a new HTTP client (for fetching additional information), it might look something like:

class Book
  # ...
  def fetch_information
    HTTPClient.new({ title: title, genre: genre, time: Time.now })
              .get('/information')
  end
end

The test for this method should assert that the client is initialized with a few crucial pieces. The hash_including argument matcher works nicely here.

describe Book do
  describe '#fetch_information' do
    let(:book_genre) { 'Historical Fiction' }
    let(:book_title) { 'A Tale of Two Cities' }
    subject { Book.new(title, genre) }

    it 'instantiates the client correctly' do
      expect(HTTPClient).to receive(:new)
                        .with(hash_including(title: book_title,
                                             genre: book_genre))
      subject.fetch_information
    end
  end
end

The versatile hash_including argument matcher can specify key/value pairs of an expected hash or just keys. Here, the test only care that the Book’s title and genre are passed along.

3. `match_array`

In Ruby, two Arrays are equal if and only if they contain the same elements in the same order. In some tests, this strict equality criteria might not always be necessary. For those cases, RSpec provides a match_array matcher to make tests less brittle.

If the Author class retrieved its list of books from a database, the order of books might not be consistent due to default scopes or when records are updated.

Given a fetch_books method that looks something like:

class Author
  attr_reader :name

  def initialize(name)
    @name = name
  end

  def fetch_books
    BookDB.find_by(author_name: name)
  end
end

Utilizing match_array, a test can assert that the proper books are returned regardless of order:

describe Author do
  describe '#fetch_books' do
    let(:name) { 'Jane Austen' }
    let!(:books) do
      Array.new(2) do
        BookDB.create_book(author_name: name)
      end
    end

    subject { Author.new(name: name) }

    it 'fetches the books correctly' do
      expect(subject.fetch_books).to match_array(books)
    end
  end
end

Verifying Doubles

Mocking in RSpec is a simple way to ensure code that is expected to run, has indeed done so.

To fetch reviews for a Book, a third party API is used. The unit test surrounding that functionality should not actually call out to that API. Instead, a test should assert that the proper request is made.

class Book
  # ..
  def reviews
    Review::API.new(SUPER_SECRET_API_KEY)
               .get("reviews/?title=#{ title }&genre=#{ genre }")
  end
end

One way to test this code would be to write a stubbing method and return some test data:

describe Book do
  let(:book_genre) { 'Historical Fiction' }
  let(:book_title) { 'A Tale of Two Cities' }
  subject { Book.new(book_title, book_genre)  }

  describe '#reviews' do
    let(:fake_reviews) do
      [
        { critic: 'Pat M.', stars: 5, comments: 'Great Read!' },
        { critic: 'Sanjay R.', stars: 5, comments: 'Interesting!' },
        { critic: 'Rupa T.', stars: 4, comments: 'It was nice!' }
      ]
    end
    let(:test_api_client) { Review::API.new(TEST_API_KEY) }

    before do
      allow(Review::API).to receive(:new).and_return(test_api_client)
      allow(test_api_client).to receive(:get).and_return(fake_reviews)
    end

    it 'fetches them from the API' do
      expect(subject.reviews).to eq(fake_reviews)
    end
  end
end

This will work as long as the Review::API contract does not change how it fetches reviews. However, if the method does change for some reason, this code will pass and the application will fail in production.

Instead, an instance_double can be used to assert that a specific method is called while still returning test data:

describe Book do
  let(:book_genre) { 'Historical Fiction' }
  let(:book_title) { 'A Tale of Two Cities' }
  subject { Book.new(book_title, book_genre)  }

  describe '#reviews' do
    let(:fake_reviews) do
      [
        { critic: 'Pat M.', stars: 5, comments: 'Great Read!' },
        { critic: 'Sanjay R.', stars: 5, comments: 'Interesting!' },
        { critic: 'Rupa T.', stars: 4, comments: 'It was nice!' }
      ]
    end
    let(:test_api_client) do
      instance_double(Review::API, get: fake_reviews)
    end

    before do
      allow(Review::API).to receive(:new).and_return(test_api_client)
    end

    it 'fetches them from the API' do
      expect(subject.reviews).to eq(fake_reviews)
    end
  end
end

In one line, instance_double(Review::API, get: fake_reviews), the client class has been instantiated as an verifying double and the method get has been stubbed to return fake_reviews. The final important piece is: allow(Review::API).to receive(:new).and_return(test_api_client) which tells the class Review::API to use the double instead a new instance when calling new.

Now, if the Review::API instance method for fetching reviews changes the test will break, throwing a NoMethodError and hopefully saving a lot of debugging time.

A Successful Year of Commits

2016-04-26T08:00:00-04:00

At Rails Conf 2015, I met a lot of great people from the Ruby community. These people had a profound effect on me, instilling in me a new motivation to become part of something larger than myself. A year ago today, I embarked on a journey to better myself as a Software Engineer. I decided to commit to open source software every day for a year.

Even though open source software has been around for decades, up until a year ago I had thought it mostly magic and far beyond my reach. After Rails Conf 2015, it became clear to me that contributing to and becoming part of the open source community was not so far fetched.

The Plan

The word “plan” might be a bit generous for what I decided to do. I was going to “just keep doing commits for a year”. Surely doing something over and over again will yield good results…right? It has to have at least some effect on some aspect of my programming ability.

And so that is what I did. For one year, I made at least one commit to an open source repository (someone else’s or my own) every day:

I started this endeavor on April 26th 2015 and ended it today, April 26 2016.

Reality Check

At the beginning of the year, I was under the impression that I was going to be able to contribute to other people’s repositories every day without too much effort. That idea was soon shattered due to the short time constraints I had to work with. After all, I am happily employed and unable to devote more than a few hours a day to open source work. Exploring someone else’s code base, reproducing an issue and finding a solution can take more than a few hours.

It became clear that I needed to fill in gaps of time with my own projects. This ended up being much more fun than I had anticipated and I got to build some cool libraries that people would actually use:

However, this does not mean that my only contributions were to my own projects. Oh no, far from it, I forked repositories on Github like crazy and ended up contributing to a lot of great projects:

…and many more

Each of these projects taught me something new and introduced me to many different coding styles. My contributions ranged from one-liners to entire spec suite creations. If any maintainers of these libraries end up reading this post: Thank you for all the work you do. Open source development is done for free and I value the time you put into it.

Expect the Unexpected

A delightful surprise during this project was writing this blog. I had a feeling that I would enjoy writing every week and originally intended to only write about the Year of Commits project. However, a few weeks into the posts and I quickly found new inspiration. It turns out that people actually like reading my explanations of coding patterns, designs, and tutorials:

@jakeyesbeck Hey man! Your blog is awesome! Keep going! Also thanks for sharing your thoughts and learned things! 👍
— Gabriel Francisco (@fgbreel) February 19, 2016

Been really impressed with @jakeyesbeck's blog recently for great #Ruby and #RubyOnRails posts. Check it out! https://t.co/cIGGTdzqlL
— Brad Urani (@bradurani) February 21, 2016

Of course, not all feedback on my posts was positive, but all of it was useful. Having my opinion out there, my content distributed and consumed has been extremely motivating and completely unexpected. Finding topics I thought would be useful and interesting to others was always a hard challenge, but having a self-imposed obligation of one post per week kept me honest.

I now believe all software engineers should write. There are a lot of parallels between writing code and writing paragraphs; both are done with great care. Writing about something is a great way to express your ideas, share knowledge, and solidify concepts for yourself. Aside from the growth in my technical skills during this past year, having my writing appreciated made all my hard work worth it.

I was lucky enough to be featured in Ruby Weekly a few times and was even mentioned in a couple podcasts.

This support completely validated my commitment and helped make this project so productive.

What I Learned

The big question that most people I talk about this project ask is: “What have I learned while doing it?” Early on in the year, I would answer by explaining some new software pattern or Ruby library that I came across. But, after a few months, I realised something much more important: the value of perseverance.

Committing code every day for a week is not difficult. Writing software every day for a month is a little harder, and every day for a year is harder still. Within a year’s time, inconveniences are bound to appear. Be it an unexpected spell of food poisoning, an impromptu vacation to Las Vegas, or a simple release of a new Netflix show to binge watch, the coding must continue. All of these things did indeed happen to me over the course of a year and it was very tempting to say to myself: “What is the harm in missing one day?”

But one missed day meant more and more as time went on. At 100 consecutive days, missing one day would have been disappointing, at 200 days it would have been devastating and at 300 days, it was unthinkable. When people did not seem to care about libraries I worked on or blog posts I published, it was easier to think about skipping a day.

And here, a year later, I stand without missing a single day. Persevering through criticism and rejection, I kept going until my task was done. If a blog post was not well received, I had next week to try again. If a library was not found useful, I could tweak it until it was. Perseverance was my hammer and everything started to look like a nail.

And it worked. I learned a lot, met some great people, and hopefully helped a few people along the way. Believing in myself and my ability to persevere was my biggest lesson during this project and I will not soon forget it.

Would I do it Again?

In short, heck yes I would. This experience has been one of the most beneficial, positive, and growth inducing tasks I have ever done. Thinking back to a year ago, my attitude towards open source software and my own abilities as a software engineer have completely changed. Working with many different types of people and receiving a variety of feedback has been a very eye opening experience.

Too often in our daily lives are we subject to repetition. These patterns can be confining while not appearing as such. Breaking out of one’s routine and exposing yourself to a variety of new stimuli can help maintain an open mind about new ideas or technology.

This project is not for everyone. I was lucky enough to have the time and energy to complete a task of this scale and for that I am supremely thankful.

Looking Forward

I plan to keep writing and contributing to open source projects as frequently as time allows. I doubt I will ever obtain the same level of Github streak that this project allotted me, but I hope to continue to make meaningful contributions to open source projects.

The confidence and exposure to many different types of codebases is something that I will value for a long time. I am more than thankful for all the people who said kind things to me or gave me words of encouragement. Without those people, there is no way that I could have accomplished this.

Thank you all so much, now I think its time for me to take a little break. I plan to be back in a few weeks, refreshed and ready to write more.

How to Diagnose Ruby on Rails N + 1 Query Problems

2016-04-17T08:00:00-04:00

Diagnosing performance problems in a production Ruby on Rails application can be deceptively complex. When constructing a new application or expanding the features of an existing one, development environments that are not subject to typical production web traffic may not make performance issues evident. In those instances, the use of some simple (and mostly free) tools can help diagnose performance issues in production.

If an application is hosted on Heroku, the New Relic add-on can be added to the application for free. Despite it having a few limitations, the free version of New Relic can be extremely valuable for diagnosing some common performance issues.

New Relic will be the primary tool used in this post for analysis, but an application’s log files can prove to be just as valuable when analyzed correctly.

Setup

The example application has the following database structure and respective models:

create_table :users do |t|
  t.string :email
  t.string :first_name
  t.string :last_name
  t.timestamps(null: false)
end

create_table :posts do |t|
  t.integer :user_id
  t.integer :theme_id
  t.string :content
  t.timestamps(null: false)
end

create_table :themes do |t|
  t.string :name
  t.timestamps(null: false)
end

class User < ActiveRecord::Base
  has_many :posts
end

class Post < ActiveRecord::Base
  belongs_to :user
  belongs_to :theme
end

class Theme < ActiveRecord::Base
end

An example controller, home_controller.rb has a single action:

class HomeController < ApplicationController
  def show
    @user = User.find(params[:user_id])
  end
end

The above action renders a single simple view, show.html.haml:

%h3= "Posts from #{ @user.first_name } #{ @user.last_name }"

- @user.posts.each do |post|
  .theme= "Theme #{ post.theme.name }"
  .content= post.content

Additionally in the application’s Gemfile, the New Relic Ruby gem has been added:

source 'https://rubygems.org'
# ...
gem 'newrelic_rpm'

On a development machine under no contention, this action will quickly render a page consisting of a User's name information, a list of their posts and the theme each post belongs to.

However, there is an N + 1 query issue with this code. In a development environment, this issue might be overlooked or deemed satisfactory. It may not be until this code is released into the wild that a problem is seen.

The N + 1 Problem

An N + 1 query problem is responsible for an application’s execution of far more queries than needed. The 1 in N + 1 refers to first query executed, in this case when loading: @user.posts. The N then refers to number of queries that must be executed when traversing over the list of elements, in this case each post's theme.

As with most problems, the first step to solving them involves validating their existence. This is the step in which New Relic is utilized.

When using Heroku, after the New Relic add-on has been created for an application, it can be opened with:

heroku addons:open newrelic --app my-application-name

Otherwise, a user can simply Log-in at rpm.newrelic.com and navigate to their application after signing up and linking their application.

Once the desired application has been opened in New Relic, navigating to the the transactions tab and then the HomeController#show transaction will show our N + 1 issue.

At first glance, the graph of this simple endpoint is telling a alarming story. A large amount of time is spent in the database, fetching records necessary to render the page:

Below this graph, New Relic conveniently displays the call count breakdown for this transaction:

Immediately, we can see that the Postgres Theme find is called an average of 20 times per transaction. Compared to the single call count of the surrounding methods in this table, 20 seems excessive.

Investigation Station

With a direction to run in, it is time to revisit the code and find the offending line. Luckily, this application’s code line count totals to about 40, so the search is short.

Here is the line that causes the issue:

.theme= "Theme #{ post.theme.name }"

Each time post.theme.name is called the following query is executed:

SELECT  "themes".* FROM "themes"  WHERE "themes"."id" = $1 LIMIT 1  [["id", 3]]

While things like query caching, primary key indexes and other performance aiding features exist within production relational databases, issuing the same query 20 times per page load is still wasteful.

The Fix

Many solutions to this problem exist, each with their own pros and cons. The important part of this post is how to effectively use New Relic to diagnose a problem in a production Ruby on Rails application.

All that aside, what kind of effect does a solution to this problem have on our example application?

To rid this system of its N + 1 query, we can prepend some information to the simple find method, letting ActiveRecord do the work for us:

class HomeController < ApplicationController
  def show
    @user = User.eager_load(posts: :theme)
                .find(params[:user_id])
  end
end

Here the eager_load method will use the established foreign keys via the belongs_to relationships and pull in relevant posts and themes, resulting in two JOIN clauses in the query.

Note: This is just one solution for this particular problem with its own downsides. Any memory concerns dealing with a User that has far too many posts or similar problems are not addressed.

Opening up New Relic shows the the improvement:

A request previously rendering in 200ms is now as low as 40ms.

The breakdown of the transaction call count shows improvement as well:

Measure Twice, Measure Again, Just Keep Measuring

Understanding slow parts of a production Ruby on Rails application can mean the difference between an “OK” user experience and a great one. When developing these applications, it can be hard to keep track of every inefficiency that could result in sub-optimal N + 1 behaviour. Both new and seasoned Ruby on Rails applications can be vulnerable to these types of problems, knowing how to diagnose them is the first step towards their resolution.

New Relic was a crucial element in this example but it is not the only tool available. Many other analysis tools for Ruby on Rails applications exist and Rails itself gives developers insight into an application’s behaviour via its logs.

Things to Consider when Metaprogramming in Ruby

2016-04-10T08:00:00-04:00

Metaprogramming in Ruby is a polarizing topic. The most common purpose of Ruby metaprogramming is for code to alter itself at runtime. Metaprogramming can be used to achieve terse and more flexible code. However, it is not without its cost. As with most things, nothing of value is free, even metaprogramming.

Undoubtedly, there is a time and place for metaprogramming; but, awareness of concessions that need to be made to support a metaprogrammed solution is important.

Code Discovery and Readability

One problem with metaprogramming solutions are their obstruction of code discovery. When entering a new project or simply trying to re-familiarize oneself with an existing one, tracing code execution in a text editor can be quite difficult if method definitions do not exist.

For example, we can assume that a User class exists with a set of metaprogrammed methods:

class User
  [
    :password,
    :email,
    :first_name,
    :last_name
  ].each do |attribute|
    define_method(:"has_#{attribute}?") do
      self.send(attribute).nil?
    end
  end
end

Although a little contrived, this code is a list of simple convenience methods on a User class. This solution is easily extended to include additional attributes without a full method definition per attribute.

However, these methods can not be found using grep, silver searcher, or other “find all” tools. Since the method has_password? is never explicitly defined in the code, it is not discoverable.

A Work Around:

To combat this issue, some developers choose to write a comment listing the defined method names above the metaprogramming block. This simple solution can greatly help the readability of the code:

class User
  # has_password?, has_email?, has_first_name?,
  #  has_last_name? method definitions
  [
    :password,
    :email,
    :first_name,
    :last_name
  ].each do |attribute|
    define_method(:"has_#{attribute}?") do
      self.send(attribute).nil?
    end
  end
end

Performance

Depending on the amount of times a piece of code is executed, performance considerations can be extremely important. “Hot code” is a term used to describe code that is called frequently during an application’s request cycle. Since not all code is created equally, understanding the performance implications of different metaprogramming approaches is imperative when writing or modifying hot code.

The Setup

An example application needs to handle incoming data at scale. Upon receiving the data, it must make it accessible to the rest of the application. Myriad options exist to solve this problem, but we can assume that only a few are feasible for this Ruby codebase.

The incoming data looks like the following:

{
  "user": {
    "name": "Some User",
    "phones": [
      "818-555-5555",
      "415-555-5555"
    ],
    "email": "email@whatever.com",
    "birthday": "12-12-1900"
  }
}

Note: This data will be referred to in the following examples as incoming_data and we can assume it was been decoded from JSON into a Ruby Hash.

1. All Methods

One way to accept and integrate this incoming data would be create a class which maps all attributes under the 'user' key to methods:

class UserMetaMethods
  def initialize(hash)
    hash.each_pair do |key, value|
      self.class.send(:attr_accessor, key)
      self.send(:"#{key}=", value)
    end
  end
end

user = UserMetaMethods.new(incoming_data['user'])
user.email
# => email@whatever.com

This solution makes accessing the incoming data very consumer friendly. All attributes appear as methods that return positively to respond_to? and have corresponding instance variables per attribute.

2. `method_missing`

A group of metaprogramming solutions would not be complete without one utilizing method_missing. With method_missing, a non-existent method call can be intercepted on an object and evaluated with additional data unbeknownst to the original caller.

class UserMethodMissing
  def initialize(hash)
    @hash = hash
  end

  def method_missing(method_name, *arguments, &block)
    key = method_name.to_s
    if @hash.key?(key)
      @hash[key]
    else
      super
    end
  end

  def respond_to_missing?(method_name, include_private = false)
    @hash.key?(method_name.to_s) || super
  end
end

user = UserMethodMissing.new(incoming_data['user'])
user.email
# => email@whatever.com

The respond_to_missing? method is also defined to enable respond_to? and method calls to execute successfully. Read more information about respond_to_missing? here.

Note: Patterns equivalent to this are used in some popular libraries like OpenStruct and Hashie to achieve similar results.

3. “Regular” Object

As a control, a regular Ruby object can be created with specific attributes defined:

class UserRegular
  attr_reader :name,
              :phones,
              :email,
              :birthday

  def initialize(hash)
    @name = hash['name']
    @phones = hash['phones']
    @email = hash['email']
    @birthday = hash['birthday']
  end
end

user = UserRegular.new(incoming_data['user'])
user.email
# => email@whatever.com

An immediate downside to this approach is: if the contract of the external service changes this object may not be initialized with all pertinent data.

4. A `Hash`

No additional code is required for this approach, a consumer would simply use the resulting Ruby Hash after the received JSON is parsed:

incoming_data['user']['email']
# => email@whatever.com

Not a metaprogramming solution, but still a valid way of handling the passed in data. Using a simple Hash does not grant the flexibility of the other solutions but can be a great base-case for performance testing.

How They Compare

Finally, the exciting part: potentially relevant performance benchmarks.

The library we will use to test how each of these solutions does is benchmark/ips.

This library makes it simple to define different implementation sections and then compare them:

require 'benchmark/ips'

Benchmark.ips do |x|
  x.report('UserMetaMethods') do
    1000.times do
      u = UserMetaMethods.new(incoming_data['user'])
      u.email
    end
  end

  x.report('UserMethodMissing') do
    1000.times do
      u = UserMethodMissing.new(incoming_data['user'])
      u.email
    end
  end

  x.report('UserRegular') do
    1000.times do
      u = UserRegular.new(incoming_data['user'])
      u.email
    end
  end

  x.report('Hash') do
    1000.times do
      u = Hash(incoming_data['user'])
      u['email']
    end
  end

  x.compare!
end

Each report corresponds to a solution described above. The Hash report did not need to initialize a new object, but for consistency’s sake, a new Hash is initialized from everything under the 'user' key of the original Hash.

Running this code results in:

Calculating -------------------------------------
   UserMetaMethods     79.294  (± 2.5%) i/s -    399.000
 UserMethodMissing      1.531k (± 1.2%) i/s -      7.791k
       UserRegular    913.295  (± 1.4%) i/s -      4.628k
              Hash      3.141k (± 1.0%) i/s -     15.860k

Comparison:
              Hash:     3141.2 i/s
 UserMethodMissing:     1530.5 i/s - 2.05x slower
       UserRegular:      913.3 i/s - 3.44x slower
   UserMetaMethods:       79.3 i/s - 39.61x slower

Wow! Aside from simply using a Hash, the method_missing implementation is the fastest. Quick, everyone go change all the code to use method_missing! No. Stop. Do not do that.

While it might be faster than the UserRegular implementation, it is not without its drawbacks. A method_missing solution certainly has value in a variety of situations but a simple benchmark should not persuade anyone to simply switch their code around to gain the “speed up”.

What about existing libraries that have similar behaviour to method_missing (i.e. Hashie and OpenStruct)?

To add them to the existing benchmark, the corresponding classes must be made:

require 'ostruct'

class UserOpenStruct < OpenStruct
end

require 'hashie'

class UserMash < Hashie::Mash
end

Then two new report calls can add them to the existing benchmark:

Benchmark.ips do |x|
  # ...

  x.report('OpenStruct') do
    1000.times do
      u = UserOpenStruct.new(incoming_data['user'])
      u.email
    end
  end

  x.report('UserMash') do
    1000.times do
      u = UserMash.new(incoming_data['user'])
      u.email
    end
  end

  # ...
end

The results of these two additions is a bit of a surprise:

Calculating -------------------------------------
   UserMetaMethods     79.050  (± 2.5%) i/s -    399.000
 UserMethodMissing      1.537k (± 1.3%) i/s -      7.752k
       UserRegular    914.824  (± 1.4%) i/s -      4.576k
        OpenStruct     49.954  (± 6.0%) i/s -    250.000
          UserMash    194.411  (± 1.5%) i/s -    988.000
              Hash      3.140k (± 0.9%) i/s -     15.759k

Comparison:
              Hash:     3140.1 i/s
 UserMethodMissing:     1536.9 i/s - 2.04x slower
       UserRegular:      914.8 i/s - 3.43x slower
          UserMash:      194.4 i/s - 16.15x slower
   UserMetaMethods:       79.0 i/s - 39.72x slower
        OpenStruct:       50.0 i/s - 62.86x slower

Despite OpenStruct and Hashie seeming very similar to our homegrown method_missing solution, both yielded worse results. However, like other metaprogramming solutions, both OpenStruct and Hashie make up for this speed deficiency with flexibility.

If this were a real problem in a production application, OpenStruct and Hashie could certainly both be viable solutions. Unless the code path to utilize these libraries was scorching hot, their performance issues might not be a factor.

Why the Slowdown?

The reason that some metaprogramming solutions are slow has partially to do with the Ruby inline method cache. In Ruby, the inline method cache is responsible for storing methods that it knows about so as to avoid a costly look up operation every time. Metaprogramming interferes with this built in cache by invalidating its cache key.

Every time a class is reopened or a method is defined on a class, pieces of the inline method cache key change, resulting in a cache miss and method lookup. Metaprogrammed code (especially code that executes at every Object.new like UserMetaMethods) does not benefit from the inline method caching in the same ways as “normal” code does. For much more information, a man much smarter than I wrote a great article explaining Ruby inline method caching in detail.

Use the Right Tool for the Job

When iterating through a list of options, no single data point is sufficient enough to rule one option superior to all others. Benchmarks should be treated as a single data point and bring depth to a comparison, not rule it. After all, who cares how slow a piece of code is if it is never run?

Metaprogramming is a very powerful tool in the Ruby language that should be wielded with care. Like anything, using metaprogramming too much can cause unmaintainable code. This sort of code might be great for job security, but could be less performant, unreadable, and unmaintainable by coworkers.

When used correctly and in appropriate circumstances, metaprogramming can be a great asset. The trick is knowing when to use it and when to refrain. Sometimes just using a Hash might be the best solution.

How to Deal with Timezones the Active Support Way

2016-04-03T08:00:00-04:00

“Well the code is bad because we had to…” is a phrase one might hear when discussing timezone offsets or daylight savings time considerations.

Unless a developer is fortunate enough to work at a company whose userbase is entirely located in the UTC timezone, writing software aware of different timezones can be a daunting task. Luckily, Ruby on Rails’ ActiveSupport library has some very nice built in features that can prove invaluable when facing time related issues.

Built-in Timezones

ActiveSupport in Ruby on Rails 4.0+ has a built in list of all supported timezones on the TimeZone class:

ActiveSupport::TimeZone.all.map(&:name)
#=> ["American Samoa", "International Date Line West", "Midway Island", "Hawaii", "Alaska", "Pacific Time (US & Canada)", "Tijuana", "Arizona", "Chihuahua", "Mazatlan", "Mountain Time (US & Canada)", "Central America", "Central Time (US & Canada)", "Guadalajara", ...]

This list of timezones can be used when parsing strings into Time objects and converting an existing Time object from one timezone to another. For working with timezones in the United States, a handy us_zones method is also available on the same class.

One interesting detail about this list of timezones is the lack of daylight savings time qualifiers. Keeping the timezones agnostic of daylight savings helps simplify their use. A developer does not need to worry about using one timezone object over another due to the time of the year.

`Time.use_zone`

The ActiveSupport library adds some functionality to built in Ruby classes. One of those additions is the ability to set and retrieve the zone attribute on the Time class. After a zone is set, it can be used when parsing strings into Time objects:

Time.zone = 'Pacific Time (US & Canada)'
Time.zone.parse('2016-04-01 10:00:00')
#=> Fri, 01 Apr 2016 10:00:00 PDT -07:00

This code takes a timestamp string without a timezone specified and uses the value of Time.zone to correctly represent the time in the Pacific timezone.

However, an immediate red flag is present in this code. The zone attribute persists for the rest of the Ruby runtime, potentially causing unexpected behaviour at a later time.

Luckily, ActiveSupport solves this problem with the use_zone method. This method accepts the same set of strings as its single argument and expects a block.

Within the passed in block is the only place Time.zone is affected, eliminating the possibility of a zone sticking around longer than intended:

Time.zone.name
#=> "UTC"

Time.use_zone('Pacific Time (US & Canada)') do
  Time.zone.name
  # => "Pacific Time (US & Canada)"
  Time.zone.parse('2016-04-01 10:00:00')
  #=> Fri, 01 Apr 2016 10:00:00 PDT -07:00
end

Time.zone.name
#=> "UTC"

This temporary zone setting can be very helpful if a set of users are processed, each with their own respective timezone.

`TimeWithZone.in_time_zone`

When dealing with Time, Date and DateTime objects, converting each from one timezone to another can be tedious. The in_time_zone method removes some complexity from those operations.

Used alone, the in_time_zone can transform an existing object into an instance of TimeWithZone:

now = Time.now
# => 2016-04-04 03:55:24 +0000
now.in_time_zone('Hawaii')
# => Sun, 03 Apr 2016 17:55:24 HST -10:00

In any sane system, timestamps in a database are always stored in UTC, aka Zulu, time. When exposing these timestamps to users, the in_time_zone method can quickly switch the timestamp to a user’s local time.

To see a more creative use of in_time_zone, we can assume that an application must solve a scheduling problem. An example application wants to send its users an email at the exact same time relative to a user’s timezone. Regardless of where a user lives, they should receive an email at 10:30 AM local time.

A naive approach might not yield the intended result:

user.time_zone
# => Hawaii
send_time = Time.new(2016, 04, 01, 10, 30)
# => 2016-04-01 10:30:00 +0000
send_time.in_time_zone(user.time_zone)
# => Fri, 01 Apr 2016 00:30:00 HST -10:00

Whoops, that is not right at all. This code initialized a Time object for the “correct” send time but then in_time_zone not only moved the timezone of that object, it also modified the actual time. This would result in all users getting an email at the same exact moment but at an inconvenient time relative to where they live.

Potential solutions to this problem might include modifying the timezone part of an outputted string (the "+0000" at the end) and re-parsing it, but that solution can be prone to error and hard to understand.

Since the in_time_zone method works on the Date class, the simplest approach would be to initialize a new DateTime object in a user’s timezone and add 10.5 hours to it:

user.time_zone
# => Hawaii
beginning_of_day = Date.today.in_time_zone(user.time_zone)
# => Sun, 03 Apr 2016 00:00:00 HST -10:00
beginning_of_day += (10.5).hours
# => Sun, 03 Apr 2016 10:30:00 HST -10:00
sender = ImportantEmailSender.new(user)
sender.send_at!(beginning_of_day)

Great! An email will be sent at 10:30 AM Hawaii time and we successfully avoided any time string munging or other strange object casting. A user in London can also receive an email at exactly 10:30 AM their local time and this code need not grow in complexity.

Saving Time

Whether in code or everyday life, it seems that struggling with time is something we as humans are simply destined to do. Some good news is that ActiveSupport may not grant you any more wall clock time, but could save you some when wrangling different pieces of scheduling code together.

As usual, the API of the ActiveSupport::TimeZone class is much larger than just the two methods illustrated here. For more information, the offical ActiveSupport documentation is a great place to read further.

Five More Active Record Features You Should Be Using

2016-03-27T08:00:00-04:00

In a previous post, I illustrated a few helpful ActiveRecord features. The entire API of ActiveRecord cannot possibly be contained within a single or even a handful of digestible posts; but, here are at least five more pieces of that massive API that some might find useful.

Just like before, the example application used to demonstrate these features will be an imaginary Ruby on Rails application: “booksandreviews.com”:

class Book < ActiveRecord::Base
  belongs_to :author
  has_many :reviews
end

class Author < ActiveRecord::Base
  has_many :books
end

class Review < ActiveRecord::Base
  belongs_to :book
end

1. `pluck`

Introduced in Ruby on Rails 4.0, the pluck method helps keep memory allocation to a minimum when returning results from ActiveRecord queries. A great use case for the pluck method is when a database table backing an ActiveRecord object has a very large number of columns. In this situation, returning an object’s full dataset can be cause unnecessary memory allocation and potentially expensive deserialization (in the case of custom or JSON serialized columns).

To get a list of book_ids from the 'fantasy' genre, the pluck method is a simple to use:

Book.where(genre: 'fantasy').pluck(:id)
# SELECT "books"."id" FROM "books" WHERE "books"."genre" = 'fantasy'
=> [1, 3, 45, ...]

An important thing to recognize about the pluck method is its return value. Unlike its cousin, select, the pluck method does not return an ActiveRecord instance.

Multiple column names may be passed to pluck and the values of these columns will be returned in a nested Array. The returned Array will maintain the order of columns to how they were requested:

Book.where(genre: 'fantasy').pluck(:id, :title)
# SELECT "books"."id", "books"."title" FROM "books" WHERE "books"."genre" = 'fantasy'
=> [[1, 'A Title'], [3, 'Another One']]

While possible, it might not always be the best idea to request multiple columns in a single pluck call. Too many columns can produce an unruly result and nullify the performance gain pluck provided in the first place.

2. `transaction`

An important aspect of relational databases is its atomic behaviour. When creating ActiveRecord objects, maintaining this aspect is possible through the use of the transaction method.

For instance, if an exception occurs during the update of all of a Book's Reviews, a transaction can help mitigate harmful side-effects:

book = Book.find(1)
book.reviews.each do |review|
  review.meaningful_update!
end

If the meaningful_update! method throws an exception, all reviews before the erroring review will update appropriately and all after will not. If this method has a compounding side-effect (like incrementing a field), iterating through all reviews again would be harmful.

With a transaction, this problem does not exist:

ActiveRecord::Base.transaction do
  book = Book.find(1)
  book.reviews.each do |review|
    review.meaningful_update!
  end
end

After adding the transaction, this new code will revert all changes contained within the transaction if an exception is raised. It is equivalent to updating all records in a single query.

3. `after_commit`

Love them or hate them, callbacks in Ruby on Rails are a viable option to solve many problems. ActiveRecord callbacks are initiated at different times depending on the operation a model is about to undergo.

A fairly common use case for callbacks in Ruby on Rails revolves around what to do after a model is persisted. A model can be persisted by either save, create or update. Since the concept of “saving” is the common denominator for all these methods, it would be reasonable to use the after_save to trigger logic that should follow one of these persistence operations.

To add a new Book to a queue for a Review after it is saved:

class Book < ActiveRecord::Base
  after_save :enqueue_for_review

  def enqueue_for_review
    ReviewQueue.add(book_id: self.id)
    Logger.info("Added #{self.id} to ReviewQueue")
  end
end

We can assume that the ReviewQueue is a key/value storage (Redis or something similar) backed object whose purpose is to place new Books into a queue for critics to review. The Logger class simply outputs text to stdout.

When everything goes well, the callback works beautifully:

Book.create!(
  title: 'A New Book',
  author_id: 3,
  content: 'Blah blah...'
)
#=> Added 4 to ReviewQueue
#=> <Book id: 4, title: 'A New Book' ..>

ReviewQueue.size
#=> 1

However, if this code was wrapped in a transaction, and that transaction fails, the Book will not persist but the element on in the Redis-backed ReviewQueue remains.

ActiveRecord::Base.transaction do
  Book.create!(
    title: 'A New Book',
    author_id: 3,
    content: 'Blah blah...'
  )

  raise StandardError, 'Something Happened'
end

#=> Added 4 to ReviewQueue
#=> Error 'Something Happened'

ReviewQueue.size
#=> 1

This code has now generated an element on the ReviewQueue for a non-existent Book; however, if the after_commit callback is used, this problem will fade away.

Replacing after_save with after_commit:

class Book < ActiveRecord::Base
  after_commit :enqueue_for_review

  # ...
end

The same code results a much more desirable outcome:

ActiveRecord::Base.transaction do
  Book.create!(
    title: 'A New Book',
    author_id: 3,
    content: 'Blah blah...'
  )

  raise StandardError, 'Something Happened'
end

#=> Error 'Something Happened'

ReviewQueue.size
#=> 0

Awesome, the after_commit callback is only triggered after the record is persisted to the database, exactly what we wanted. Ruby on Rails provides many more callback hooks besides after_commit.

4. `touch`

To update a single timestamp column on an ActiveRecord object, touch is a great option. This method can accept the column name which should be updated in addition to an object’s :updated_at (if present).

A great time to utilize the touch method is when tracking when a record was last viewed. The good people at “booksandreviews.com” need to be sure their Reviews are being seen and want to know which Reviews have not been viewed in a while.

If the Review table has a last_viewed_at column which is of type timestamp, touch can easily update this column in a controller action:

class ReviewsController < ApplicationController
  def show
    @review = Review.find(params[:id])
    @review.touch(:last_viewed_at)
  end
end

A request which calls this controller action will result in the query:

UPDATE "reviews"
SET "updated_at" = '2016-03-28 00:43:43.616367',
"last_viewed_at" = '2016-03-28 00:43:43.616367'
WHERE "reviews"."id" = 1

Unlike setting other attributes, the touch method invokes an update query right away.

5. `changes`

ActiveRecord keeps a sizable amount of information about an object while it undergoes updates. A hash, accessible via the changes method, acts a central location for these updates. Additionally, ActiveRecord exposes a number of per-attribute helper methods to give even more visibility.

Each key in the changes hash maps to a record’s attribute. The value of each key is an Array with two elements: what the attribute value used to be, and what it is now:

review = Review.find(1)
review.book_id = 5
review.changes
# => {"book_id"=>[4, 5]}

a. `changed?`

A simple boolean method changed? is available to determine if anything on the object is different since its retrieval.

review = Review.find(1)
review.book_id = 5
review.changed?
# => true

However, if the same value is set on a model, regardless of the value’s object_id, changed? returns false:

book = Book.find(1)
book.title
# => 'A Book'
book.title.object_id
# => 2158279020
book.title = 'A Book'
book.changed?
# => false
book.title.object_id
# => 2158274600

Two “different” strings were set to book.title but since they resolve to the same characters, the object is not changed?.

When using PostgreSQL, the changed? method can save unnecessary begin and commit transaction calls for objects that have no changes:

class BooksController < ApplicationController
  def update
    @book = Book.find(sanitized_params[:id])
    @book.assign_attributes(sanitized_params)
    @book.save! if @book.changed?
  end
end

b. `<attribute>_was`

Accessing the same changes hash as the changed? method, <attribute>_was returns the value of an attribute before it was reassigned:

review = Review.find(1)
review.status = 'Approved'
review.status_was
# => 'Pending'

If a Review's status moving from "Pending" to "Approved" should remove it from an approval queue, the status_was method can be utilized in a callback:

class Review
  before_save :maybe_remove_from_review_queue

  def maybe_remove_from_review_queue
    if status == 'Approved' &&
        status_was == 'Pending'
      ReviewQueue.remove(id)
    end
  end
end

To verify the result, we can check that the ReviewQueue decrements its internal count of reviews pending review:

ReviewQueue.size
#=> 1
review = Review.find(1)
review.status
# => 'Pending'
review.status = 'Approved'
review.save!
# => true
ReviewQueue.size
#=> 0

Note: Additional considerations might need to be made since the save is not guaranteed to persist like described above in the after_commit example.

How to Build a Ruby on Rails Engine

2016-03-20T08:00:00-04:00

Ruby on Rails Engines are miniature applications whose purpose is to supplement a larger Ruby on Rails application. If functionality can exist independent from a main application, an Engine can provide a wonderful degree of encapsulation.

Recently, I created the and “gemified” the Passages Ruby on Rails Engine to help alleviate some routing frustration. This gem will be the be used as the example for demonstrating what goes into creating a Ruby on Rails Engine.

Revving Up

There are two ways to start building an Engine. One option is to use the built in generators to create directories and dummy classes. These generators behave in the same way as standard Rails generators.

To generate an Engine in this way, use the plugins built in generator:

$ rails plugin new passages --mountable

Note: Engines and Plugins are not exactly the same in the Ruby on Rails world but the --mountable flag tells the plugin generator to generate a full Engine.

The other approach is to simply create the needed directories and files by hand. This is was the way the Passages Engine was built, resulting in the following directory structure:

 |-app
 |--controllers
 |---passages
 |----<controller directories>
 |--views
 |---passages
 |----<views directories>
 |-config
 |--routes.rb
 |--initializers
 |---assets.rb
 |-lib
 |--passages
 |---engine.rb

Some directories that the rails generator would have added are missing (i.e. models, helpers, mailers). These directories might be necessary for some projects, but the Passages Engine did not have use for them.

The file at the heart of it all is engine.rb. This file is responsible for defining the engine and will also be utilized later to add optional enhancements an Engine can take advantage of:

module Passages
  class Engine < ::Rails::Engine
    isolate_namespace(Passages)
  end
end

An interesting line in this file is the isolate_namespace method call. This method helps ensure encapsulation for the Engine by isolating its controllers, helpers, views, routes, and any other shared resources between the main application and the Engine.

With isolation, an Engine does not need to worry about conflicting class or module names in the main application. Additionally, an isolated Engine will set its own name according to its namespace, accessible later via Passages::Engine.engine_name.

Since the Passages Engine is also a gem, it has a file named passages.rb in its lib directory:

module Passages
end

require 'passages/engine'
require 'controllers/passages/routes_controller'

This file is responsible for defining the Passages module and requiring the engine. It is the entry point for this gem’s logic.

Encapsulate Everywhere

The overarching theme of designing a good Ruby on Rails Engine is encapsulation. An application should not be negatively affected by its underlying Engines, they should simply support and bring new functionality the application.

To help reinforce this theme, Ruby on Rails requires that an Engine’s controllers, views, and assets all be nested in namespace modules and corresponding directory folders.

Controllers

In the Passages Engine, the RoutesController demonstrates this nesting.

 |-app
 |--controllers
 |---passages
 |----routes_controller.rb

module Passages
  class RoutesController < ActionController::Base
    # ...
  end
end

The same folder structure is used for views and assets.

Routes

Of course what good is a controller without a route to use it? A Ruby on Rails Engine also can define its own routes similarly to a stand-alone application.

Unlike the controllers, the routes.rb file is not contained in a passages folder, nor is it within the module Passages.

 |-config
 |--routes.rb

Passages::Engine.routes.draw do
  root to: 'routes#index'
end

In a standard Ruby on Rails application, the first line in a routes file is: Rails.application.routes.draw; however, within an Engine, the name of the Engine replaces Rails.application.

With a simple routes file in place, an application using the Passages Engine can run rake routes to see the new routes in action:

$ rake routes
     Prefix Verb   URI Pattern
   passages        /passages
      users GET    /users
            POST   /users
   new_user GET    /users/new
  edit_user GET    /users/:id/edit
       user GET    /users/:id
            PATCH  /users/:id
            PUT    /users/:id
            DELETE /users/:id
Routes for Passages::Engine:
  root GET  /      passages/routes#index

Note: This assumes the engine is mounted at /passages, more about mounting routes below.

Neat, the Passages routes are in a separate section to help differentiate them from normal application routes.

Remember the isolate_namespace method? One of the side-effects of not using an isolated namespace can be seen when asking for an applications routes.

If the namespace isolation is commented out:

module Passages
  class Engine < ::Rails::Engine
    # isolate_namespace(Passages)
  end
end

Then rake routes gives a different output:

$ rake routes
         Prefix Verb   URI Pattern
passages_engine        /passages
          users GET    /users
                POST   /users
       new_user GET    /users/new
      edit_user GET    /users/:id/edit
           user GET    /users/:id
                PATCH  /users/:id
                PUT    /users/:id
                DELETE /users/:id

Routes for Passages::Engine:
  root GET  /          routes#routes

Notice that now the root for the Passages Engine has had its prefix removed. This will cause Rails to look in the wrong place for the routes_controller:

ActionController::RoutingError
(uninitialized constant RoutesController):

While this might not be a huge deal for some applications, the fact that an Engine triggers a top level controller to be fetched is worrysome. What if the main application had its own RoutesController, the Passages Engine could incorrectly fetch that instead. Or if things are reversed, an Engine without an isolated namespace might incorrectly override an important controller in the main application.

Assets

Like controllers and views, an Engine’s assets are also nested under a folder bearing the Engine’s name.

 |-app
 |--assets
 |---javascripts
 |----passages
 |-----application.js
 |---stylesheets
 |----passages
 |-----application.css

This organization enables the layouts and other views in the Engine to only load the files it needs and not accidentally reference the main application’s application.js and application.css:

<%= stylesheet_link_tag 'passages/application', media: 'all' %>
<%= javascript_include_tag 'passages/application' %>

To enable precompiled assets, a few more lines need to be added in the same engine.rb file:

module Passages
  class Engine < ::Rails::Engine
    isolate_namespace(Passages)

    initializer("passages.assets.precompile") do |app|
      app.config.assets.precompile += [
          'application.css',
          'application.js'
        ]
    end
  end
end

This initializer line creates an initializer in the underlying railties to be evaluated when assets are precompiled via rake assets:precompile. With this, an application can successfully compile its own assets and those of the Passages Engine.

Mount Up

A Ruby on Rails Engine must be mounted by an application for it to be accessible.

The normal place for this to occur is in a main application’s routes.rb:

Rails.application.routes.draw do
  mount Passages::Engine, at: '/passages'
end

The '/passages' string can be replaced with any desired endpoint, the Engine does not care about the name.

Alternatively, an Engine can mount itself using the same initializer method in engine.rb:

module Passages
  class Engine < ::Rails::Engine
    isolate_namespace(Passages)

    initializer('passages',
                after: :load_config_initializers) do |app|
      Rails.application.routes.prepend do
        mount Passages::Engine, at: '/passages'
      end
    end
  end
end

In this case, the initializer has a specific placement: after the configuration initializers are loaded. This initializer then writes to the main application with Rails.application.routes.prepend and, as the name suggests, prepends the mount to the application’s routes.

Since the mount is added to the beginning of an application’s routes, it is possible that this mounted path (in this case '/passages') will be overridden by a route with the same name later on in the main application’s routes.rb file.

After an application mounts the Passages Engine, it can navigate to /passages. This request would be served by the Passages Engine like a normal Ruby on Rails application request.

Use With Caution

Auto-mounted Engines may sound like a great idea but it might be best to leave that decision to the consumer. A suggested approach to this is have an opt-in functionality with auto-mounting. Placing this logic behind a conditional (based on some kind of configuration variable) gives consumers of this Engine the power of auto-mounting without the worry of “magic” they did not ask for.

Next Steps

With the basic structure in place, a new Engine can be built like any other Ruby on Rails application. Controllers and their respective views can be created and placed under the appropriate namespaces and folders. Routes that utilize these controllers can be added as well. Assuming that both the main application and the Engine use the same ORM (or are at least compatible), even models can be created.

The Passages Engine was built as a standalone gem. Making it available was the same process as creating any other gem. A .gemspec file was created, the gem was cut to a version, and then finally hosted on Rubygems.org. An application that wishes to use the Passages Engine can install it by adding a new line to the Gemfile: gem 'passages'.

This same pattern can be used to make any stand alone Ruby on Rails Engine. However, it would be a wise decision to choose specific versions of Ruby on Rails to support. For example, the Passages Engine was built to support Ruby on Rails 4.X, it is not compatible with Ruby on Rails 3.X at all.

Built to Last

Going further, the creation of views, initializer files, migrations, and models can all be accomplished the same way one would in a regular Ruby on Rails application.

Not all of the Passages Engine was discussed in this guide, but feel free to read the source to find out more information. Also, I am always looking for eager contributors to either submit issues or pull requests with features they need.

The Power of Arel

2016-03-13T08:00:00-04:00

Many modern web applications have at least a few overlapping responsibilities. One of these responsibilities deals with maintaining state. A common choice for storing this state is by way of a relational database. Ruby on Rails applications assume the presence of a database by default and communicate with it via ActiveRecord.

ActiveRecord is an Object Relational Mapping (ORM). Under the hood, ActiveRecord uses a relational algebra library called Arel to compose queries for execution. Arel is a very powerful library readily available for when ActiveRecord falls short.

The Basics

Keeping up with tradition, assume a standard Ruby on Rails application exists with at least two models, User and Post:

create_table "posts" do |t|
  t.integer  "user_id"
  t.string   "category"
  t.datetime "created_at", null: false
  t.datetime "updated_at", null: false
end

create_table "users" do |t|
  t.string   "first_name"
  t.string   "last_name"
  t.string   "email"
  t.datetime "created_at", null: false
  t.datetime "updated_at", null: false
end

class User < ActiveRecord::Base
  has_many :posts
end

class Post < ActiveRecord::Base
  belongs_to :user
end

Ask: Find all posts within a category belonging to a user.

For such a simple query, ActiveRecord is capable of generating the appropriate SQL statement:

Post.where(user_id: 123,
           category: 'news')
    .to_sql

# => SELECT "posts".* FROM "posts"  WHERE "posts"."user_id" = 123 AND "posts"."category" = 'news'

However, if the query needed to be more complex, this simple where clause is not sufficient. For example, instead of all posts within a category for a user, only the posts which have a created_at in the current month are desired.

A naive ActiveRecord only solution might look like:

beginning_of_month = Date.today
                         .beginning_of_month

Post.where(user_id: 123,
           category: 'news')
    .where('created_at >= ?',
           beginning_of_month)
    .to_sql

# => SELECT "posts".* FROM "posts"  WHERE "posts"."user_id" = 123 AND "posts"."category" = 'news' AND (created_at >= '2016-03-01' )

This query will give the desired results but uses raw SQL. Using raw SQL is not always a bad practice, but in this case it seems unnecessary and could cause issue later.

`Model.arel_table`

Every ActiveRecord model can access its underlying Arel representation with the arel_table class method.

Post.arel_table

# => #<Arel::Table:0x00000103ae7f00 @name="posts", ...>

This Arel table exposes each column of the underlying table. These columns provide many available methods used for comparison with either another column or scalar value.

The same Post query from above using its arel_table looks like:

beginning_of_month = Date.today.beginning_of_month
arel = Post.arel_table

Post.where(
      arel[:user_id]
        .eq(1234)
        .and(arel[:category]
             .eq('news'))
        .and(arel[:created_at]
             .gteq(beginning_of_month)))
    .to_sql

# => SELECT "posts".* FROM "posts"  WHERE ("posts"."user_id" = 1234 AND "posts"."category" = 'news' AND "posts"."created_at" >= '2016-03-01')

After all the necessary conditions are applied to the arel variable, it is given to the Post class via the where method.

With this pure Ruby query generation, something looks a little different. Instead of:

AND (created_at >= '2016-03-01')

The query has changed to:

AND "posts"."created_at" >= '2016-03-01'

This change points to one of the issues of the raw SQL approach: its inflexibility when dealing with multiple tables.

Join Resistance

Iterating on the Post query, it now needs to return the relevant User. If the original raw SQL solution was used the user.created_at column and the posts.created_at column would clash, resulting in an error:

Post.where(user_id: 123,
           category: 'news')
    .where('created_at >= ? ',
           beginning_of_month)
    .joins(:user)
    .includes(:user)

# => ActiveRecord::StatementInvalid: PG::AmbiguousColumn: ERROR:  column reference "created_at" is ambiguous

The problem is clearly seen in the generated SQL:

SELECT "posts".* FROM "posts" INNER JOIN "users" ON "users"."id" = "posts"."user_id" WHERE "posts"."user_id" = 123 AND "posts"."category" = 'news' AND (created_at >= '2016-03-01' )

Since PosgreSQL does not know which created_at column to compare against this date, it can not fulfill the query.

This problem does not exist in the Arel solution:

arel = Post.arel_table
Post.where(
      arel[:user_id]
        .eq(1234)
        .and(arel[:category]
              .eq('news'))
        .and(arel[:created_at]
              .gteq(beginning_of_month)))
    .joins(:user)
    .includes(:user)
    .to_sql

# => SELECT "posts".*, "users.*" FROM "posts" INNER JOIN "users" ON "users"."id" = "posts"."user_id" WHERE ("posts"."user_id" = 1234 AND "posts"."category" = 'news' AND "posts"."created_at" >= '2016-03-01')

Note: The actual generated query has many column aliases in the select clause for the includes behaviour but is irrelevant for these examples.

With each element specifying both table and column names, this query is still pure Ruby and join friendly.

Advanced Selecting

Simple joins and comparison operators are useful but barely begin to explore the vast API contained within Arel. An interesting feature that Arel provides is how to create and utilize Common Table Expressions (CTEs).

A CTE is a temporary named result in an SQL database, derived from a SELECT query. This derived result remains present in the scope for additional SELECT, INSERT, UPDATE, or DELETE queries. CTEs can be thought of as alternatives to temporary tables, sub-queries, views, or other in-line user-defined functions.

Ask: Find all posts written within the last month, then return the email of each user who wrote the post.

Since no Post data is required, this ask is a great candidate for a CTE. A new Arel::Table can be created to house the CTE and then the join method on the User.arel_table makes it accessible:

post_arel = Post.arel_table
user_arel = User.arel_table

common_table_results = begin
  post_arel
    .project(post_arel[:user_id])
    .where(post_arel[:created_at]
            .gteq(beginning_of_month))
end

cte_table = Arel::Table.new(
              :posts_last_month)
composed_cte = Arel::Nodes::As.new(
                cte_table,
                common_table_results)

user_arel
  .project(users[:email])
  .join(cte_table)
    .on(users[:id]
         .eq(cte_table[:user_id]))
  .with(composed_cte)
  .to_sql

# => WITH "posts_last_month" AS (SELECT "posts"."user_id" FROM "posts" WHERE "posts"."created_at" >= '2016-03-01') SELECT "users"."email" FROM "users" INNER JOIN "posts_last_month" ON "users"."id" = "posts_last_month"."user_id"

At first glance, this code can be a bit intimidating but it is actually quite simple. Both the Post model and User model have had their underlying Arel representation referenced. Then, the CTE result is computed:

common_table_results = begin
  post_arel
    .project(post_arel[:user_id])
    .where(post_arel[:created_at]
            .gteq(beginning_of_month))
end

The most noticeable attribute here is the project method. This method tells the query what to SELECT. Since only user_ids are relevant in this CTE, that is all that is requested.

Next, the actual CTE is created (in Arel representation only, not in the database):

cte_table = Arel::Table.new(
              :posts_last_month)
composed_cte = Arel::Nodes::As.new(
                cte_table,
                common_table_results)

The As class under the Arel::Nodes namespace is created and then fed into the end of the main query via the with method:

user_arel
  .project(users[:email])
  .join(cte_table)
    .on(users[:id]
         .eq(cte_table[:user_id]))
  .with(composed_cte)
  .to_sql

Again, since only the email of the User is desired, the project method is used. The join method is similar to the ActiveRecord method joins but requires the on clause to tell the query how to join the tables. Finally, the with method receives the computed CTE representation and the query is successfully generated.

To actually use this query, another helpful method on ActiveRecord, find_by_sql can accept the full Arel representation, returning the desired User models with only the email column retrieved:

User.find_by_sql(
  user_arel
  .project(users[:email])
  .join(cte_table)
    .on(users[:id]
         .eq(cte_table[:user_id]))
  .with(composed_cte)
)

# => [#<User id: nil, email: "foo@bar.com">]

Why?

Like most tools, Arel may not be applicable in every situation. However, when used appropriately, it can make for some very well maintainable code that is resistant to unexpected failures. When dealing with multiple levels of an application stack, abstracting away complexities around their communication can help vastly simplify a codebase. ActiveRecord and Arel do a great job of insulating application developers from the nitty-gritty of database communication. Where ActiveRecord falls short, Arel is there to pick up the slack.

For extremely specific or complex database interactions, using Arel may not be the correct choice, but should at least be evaluated. Only interacting with the lowest level necessary for a resource has many benefits. An Arel solution to a problem has the potential to be more resilient to future changes.

The Real Cost of Technical Debt

2016-03-06T07:00:00-05:00

Writing software is an iterative process. Rarely is software written and then never revisited. When this iteration occurs, a software engineer is presented with multiple options. Usually, a single objectively correct option is present but may not be chosen due to time constraints or other outside pressures. When shortcuts are taken to alleviate these external pressures, technical debt is often accrued.

Permit Today, Pay Tomorrow

Technical debt is referred to as such due to its similarities with financial debt. A credit card enables a person to complete a purchase they may not otherwise be able to, resulting in a bill to be paid by that person in the future: a debt. Technical debt is the same concept; a current feature or product is completed faster than previously possible at the cost of future work: a technical debt.

If not apparent from name alone, technical debt is not a desired feature in a healthy software system. This debt can often appear benign, giving no cause for alarm or prompt resolution; however, I believe that thinking to be in error. Technical debt can severely damage a software application and may even cause its destruction.

Solution: Communicate the Importance of the Correct Approach

Open communication about why a solution might be slower to accomplish now, but result in a healthier system for the future can be very important. A conversation about the amount of technical debt a “fast” solution will generate might help reduce pressure.

Ongoing Overhead

Left unchecked, technical debt can become a constant complexity or time increasing abscess for an otherwise healthy software application. Suddenly, every task that must interact with or touch technical debt affected code is not as simple as it should be. The “simple change” that a product team might request can have its scope increased due to technical debt overages.

For example, we can assume that a typical web application decides that its new mobile friendly version should be a parallel implementation instead of investing time in making the existing experience responsive. This not DRY solution is immediately technical debt. When a new feature is to be added, it must be added in two distinct places instead of one. This decision might have been made due to the existing implementation having an amount of already unresolved technical debt, making it inflexible to change.

Solution: Plan for Overhead Reduction

To combat this problem, regular intervals of re-factoring and code cleanup should be inserted into a normal development flow. Making time to address technical debt can reduce a problem from an avalanche to a gentle dusting.

Re-factor Routines

To reduce technical debt, software must be re-factored to remove it and usually obtain the originally desired version of the code. These necessary changes, if not handled in a timely matter, can delay a future project or stop one in progress. Much like a bank charges interest for credit card bills, technical debt’s interest can result in application slowdowns, intermittent errors, or unusual edge cases until a re-factoring occurs.

Continuing the mobile implementation example, if the two experiences are to be combined into a single code path, this combination might incur a significant amount of work. Tack on application uptime concerns and maintaining pleasant experiences for active end users, the previous credit granted by this technical debt has less and less value.

On the other hand, if re-factoring is avoided for too long, the compounded technical debt interest can make feature development and iteration grind to a halt until addressed.

Solution: Embrace Re-Factoring and Learn Something New

Re-factoring code should not have the same negative connotation that technical debt has. Actually, re-factoring is a great time to try out new patterns and ideas. Since true re-factoring dictates that code adhere to the same contract it did before completion, it grants a level of safety and enables experimentation. Conducting these rewrites regularly can help reduce technical debt and grow a developer’s skill set.

Accessible Anti-Patterns

“I can resist anything except temptation.” - Oscar Wilde

The barrier for compromising standards when devising a solution weakens when it is routinely circumvented. As shortcuts become apparent, internal or external pressures to utilize these shortcuts might make them more attractive.

Admittedly, not all problems can be solved without the accrual of technical debt; however, when it becomes the Status Quo for development, alarms should start sounding. Maintaining vigilance and respecting proper development practices are the responsibilities for the professional software engineer. Allowing one’s self or one’s team to slide the quality meter too far towards the “quick and easy” side will result in more technical debt now, more code to re-work and repair later.

Solution: Keep Previous Technical Debt Payments in Memory

Human beings can be impulsive. To keep that inclination in check, remembering the cost of a previously dealt with bit of technical debt can help remind people the downsides of such solutions.

Inevitable Irregularity

Discipline is important when creating software applications able to stand the test of time. Having the discipline to push towards the reduction of technical debt inducing solutions can be difficult, but will result in a better system and brighter development future.

Even the most bullish proponents of correct solutions may find themselves in some amount of technical debt. It is not always avoidable. At times, it can even be the correct path. When external pressures carry a large enough weight (like the survival of a company) it might be correct to cut corners and take on the risk of technical debt.

Solution: Be Realistic

Technical debt will happen. When it does, leave notes for your future self to work with and remember that even the best in the business have cut a corner or two under pressure.

credit: xkcd

Ruby Spaceship <=> Operator

2016-02-28T07:00:00-05:00

Adhering to the law of trichotomy, the <=> operator (sometimes called the “Spaceship Operator”) works by comparing two elements and returning a -1, 0, or 1. While the original mathematical criteria applies to only real numbers, many programming languages implement the law of trichotomy as a general comparison between equivalent types.

Basics

At first glance, the return value of the <=> operator can be a bit confusing. A simple way to remember the significance of these return values is to break an expression down from left to right:

1. `<` –> `-1`

if a < b, then -1 is returned

2. `=` –> `0`

if a = b, then 0 is returned

3. `>` –> `1`

if a > b, then 1 is returned

Example:

> a = 3
> b = 5
> a <=> b
# => -1

> a = 3
> b = 3
> a <=> b
# => 0

> a = 10
> b = 3
> a <=> b
# => 1

Sorting

The <=> operator can be used alone for comparison or its contract honored within a block following the sort method.

By default, the <=> operator behaves as described above:

list = [8, 3, 1, 4, 0, 3]
list.sort { |a, b| a <=> b }
# => [0, 1, 3, 3, 4, 8]

Following this pattern, it is easy to sort a list in reverse by swapping operand positions:

list = [8, 3, 1, 4, 0, 3]
list.sort { |a, b| b <=> a }
# => [8, 4, 3, 3, 1, 0]

The sort method is extendable beyond explicit use of the <=> operator. A block passed to sort must only return either -1, 0, or 1 for sorting to work effectively.

If the same list were to be ordered in odds then evens:

list = [8, 3, 1, 4, 0, 3]
list.sort { |a, _| a.odd? ? -1 : 1 }
# => [3, 3, 1, 4, 8, 0]

Also, since -1, 0, and 1 are simple integers, creating compound <=> blocks is possible. If the same list were to be sorted odds then evens, with all odd and even numbers sorted in ascending order, it might look like this:

list = [8, 3, 1, 4, 0, 3]
list.sort do |a, b|
  if a.odd?
    if b.odd?
      # both are odd, default <=> behaviour is used
      a <=> b
    else
      -1 # a < b
    end
  else # a is even
    if b.even?
      # both are even, default <=> behaviour is used
      a <=> b
    else
      1 # a > b
    end
  end
end
# => [1, 3, 3, 0, 4, 8]

Far from elegant, this code effectively sorts the array of numbers first by odds to evens and then in ascending order.

Custom Methods

While thought of and generally referred to as an operator, <=> is actually a method. Defined originally on Object, any object has the opportunity to redefine this method to achieve custom sort behaviour.

An example class, Node, has a single attribute value:

class Node
  attr_accessor :value
end

By default, sorting an array of Node objects does not produce the desired result:

node1 = Node.new
node1.value = 20

node2 = Node.new
node2.value = 10

node3 = Node.new
node3.value = 30

list = [node1, node2, node3]
list.sort

# => ArgumentError: comparison of Node with Node failed

However, with a custom <=> method:

class Node
  attr_accessor :value

  def <=>(other_node)
    self.value <=> other_node.value
  end
end

list = [node1, node2, node3]
list.sort
# => [#<Node: @value=10>, #<Node: @value=20>, #<Node: @value=30>]

Boom! Custom sorting in the mix!

Admittedly, this same behaviour is possible with the built in sort_by method:

list = [node1, node2, node3]
list.sort_by(&:value)

# => [#<Node: @value=10>, #<Node: @value=20>, #<Node: @value=30>]

But, what happens when the Node class wants to expose a sorting mechanism while removing its value method? In that case, the sort_by method would no longer be sufficient.

Defining a custom <=> method also allows objects to include the Comparable mixin effectively. The Comparable mixin provides the conventional comparison operators (<, >, <=, etc.).

class Node
  include Comparable
  attr_accessor :value

  def <=>(other_node)
    self.value <=> other_node.value
  end
end

node1 = Node.new
node1.value = 200

node2 = Node.new
node2.value = 300

node1 > node2
# => false

node1 < node2
# => true

node3 = Node.new
node3.value = 250

node3.between?(node1, node2)
# => true

The Comparable mixin provides very useful functionality that can easily turn a custom class into a sortable list element. As with most patterns in software, using the right tool at the right time is very important. It may not always be appropriate to write a custom comparison mechanism for a class, but when it is, the collaboration of <=> and the Comparable mixin is worth evaluating.

Four PostgreSQL Tips

2016-02-21T07:00:00-05:00

PostgreSQL is an open source object-relational database used to power many production web applications. While many web applications interact with relational databases through Object Relational Mappers (ORM), direct SQL queries via a command line interface or graphical client are still common. When writing these queries, these four tips may come in handy.

All examples will assume the presence of a simple users table:

    Column     |            Type
---------------+-----------------------------
 id            | integer
 first_name    | character varying(255)
 last_name     | character varying(255)
 email         | character varying(255)
 created_at    | timestamp without time zone
 updated_at    | timestamp without time zone
 password      | character varying
 registered    | boolean
 registered_at | timestamp without time zone
Indexes:
    "users_pkey" PRIMARY KEY, btree (id)

1. Finding Duplicate Rows

A common mechanism for defending against duplicate rows in database tables are unique indexes. However, at the time of table creation, a unique index or two may have been forgotten. Duplicates in a table must be removed before a unique index may be added.

A great way to detect duplicates in PostgreSQL is by using window functions. Window functions are similar to aggregates; but, instead of grouping rows for the response, it maintains each row’s individuality.

Desired query: Find all duplicate users with the same first_name, last_name, and email, returning duplicate ids only (do not return the oldest id).

SELECT id from (
  SELECT id,
  ROW_NUMBER() OVER(
    PARTITION BY first_name,
                 last_name,
                 email
              ORDER BY id
    ) AS user_row_number
  FROM users
) duplicates
WHERE
duplicates.user_row_number > 1

This query will identify all the rows of the users table which share the same defined columns and return the primary key (id) of rows after the first via the duplicates.user_row_number > 1 condition. The result of this query can then be fed into a DELETE query to remove the duplicates.

The ROW_NUMBER() built-in function returns an incremented value assigned to each row that matches the criteria in the provided OVER clause. This OVER clause is followed by the window function: PARTITION, which divides matching rows into groups or partitions using the columns or functions which follow it. Learn more about window functions at the official PostgreSQL documentation page.

2. `interval`

Ever had to scope the results of a database query to a certain time range? So has every developer ever. Luckily, PostgreSQL provides a very readable convenience function: interval.

If a rolling count of user sign ups in the last 24 hours is desired, the following query makes it possible:

SELECT *
FROM users
WHERE created_at >= now() - interval '1 day'

The interval function accepts a variety of input types. Days, Hours, Minutes, Weeks, and more options are available.

interval subtraction can be used on any timestamp:

SELECT *
FROM users
WHERE created_at >=
  '2016-02-19 12:00:00'::timestamp - interval '3 weeks'

The :: notation casts the string 2016-02-19 12:00:00 to a timestamp, a requirement when manipulating time ranges with interval.

3. Remote Query Execution

A very powerful feature of PostreSQL, and more specifically the psql command line interface, is the ability to run remote queries. With the --command or -c flag, a query may be passed to any PostgreSQL server and executed.

A unique use for this feature is the ability to transfer a query’s results directly from one environment to another.

An extensive analysis of the users table for all users that signed up in January is requested. This analysis has the potential to be very taxing on a production database, resulting in adverse effects on end users. Additionally, no follower or read only database exists to alleviate this problem. Production data must be transfered to a local environment so that the analysis can run side effect free.

To accomplish this, the psql client has the ability to execute a query and return its result to stdout. Conversely, file contents or query results from stdin may also be read within the psql client.

Utilizing both aspects of this behaviour could look something like:

psql <REMOTE POSTGRESQL URL> \
-c "copy (
      SELECT *
      FROM users
      WHERE date_trunc('month', created_at) = '2016-01-01'
    ) to stdout" \
| psql <LOCAL POSTGRESQL DATABASE> \
-c "copy users from stdin"

As an alternative to pg_dump, this pattern is a quick solution for loading specific data from one database to another.

The copy command comes in two variants: copy, which will be run by the PostgreSQL backend (user “postgres”) and \copy which runs as the current user. Depending on permissions and what copy is attempting to do, one may be more appropriate than the other.

4. Referencing Columns by `SELECT` Position

If one were to issue a SELECT query with a GROUP BY or ORDER BY clause, the columns in these clauses can be referenced by their position in the SELECT section.

Desired query: Retrieve a count of all users which share the same first_name and were created_at on the same hour of the same day then order by the hour of creation and then first_name.

Without numeric referencing

SELECT first_name, date_trunc('hour', created_at), count(*)
FROM users
GROUP BY first_name, date_trunc('hour', created_at)
ORDER BY date_trunc('hour', created_at), first_name;

This query will satisfy the request, returning an ordered list users data grouped by their first_name and hour of the created_at.

With numeric referencing

Starting with 1, each value in the SELECT clause can be referenced by its numerical position. This can help reduce typos and copy pasting issues:

SELECT first_name, date_trunc('hour', created_at), count(*)
FROM users
GROUP BY 1, 2
ORDER BY 2, 1;

As queries become more complicated with references or additional aggregate functions, numerical referencing becomes even more valuable.

Use with Caution

While handy, referencing columns by their position is not always appropriate. Adding columns to the SELECT statement can invalidate previous referencing and cause unwanted behaviour. Additionally, with a significantly large number of columns in a SELECT statement, readability can suffer when referring to them solely by numbers.

As with anything, the use of numerical referencing should be deliberate and thoughtful.

Ruby Threads and ActiveRecord Connections

2016-02-14T07:00:00-05:00

Processing large data sets is a common problem faced by many production web applications. One solution is to divide the work amongst multiple processes and have each responsible for a single or significantly smaller batch of data. However, this solution is not without its problems. Machine provisioning limitations or financial barriers may invalidate this solution for a very large N.

Within the same vein as “divide and conquer” exists another solution, one which requires far fewer parallel processes: Threads. In the Ruby programming language, a Thread is a built-in object for concurrent programming.

Unlike independent processes, all Ruby Threads within the same process share memory, enabling each individual Thread to consume or process objects and elements from the same data store.

For this example, a database will be queried, results manipulated and finally returned to same database via ActiveRecord.

First Pass

Given a User model backed by a simple users table:

class User < ActiveRecord::Base
end

class CreateUsersTable < ActiveRecord::Migration
  def change
    create_table :users do |t|
      t.string :first_name
      t.string :last_name
      t.string :email
      t.boolean :validated, default: false
      t.timestamps null: false
    end
  end
end

The problem to solve is fairly straight forward:

All User records that are not already validated should be fetched and an external API hit with their email address for validation, then saved. If a User's email address is not valid, it should be removed.

Fast forwarding through time, it can be assumed that a completely serial solution has been written and deemed unsatisfactory. Then, during a second iteration, a bit of concurrent code was written:

class UserEmailValidator
  def self.run
    User.where(validated: false)
        .find_in_batches(batch_size: 30) do |user_batch|
          validate_emails(user_batch)
        end
  end

  def self.validate_emails(user_batch)
    threads = user_batch.map do |user|
      Thread.new do
        email_validator = EmailService.new(user.email)
        email_validator.validate

        user.email = nil if email_validator.invalid?

        user.validated = true

        user.save!
      end
    end

    threads.each(&:join)
  end
end

The important pieces of this code are:

find_in_batches(batch_size: 30)

To cut down on memory allocation, only 30 Users are fetched at one time. Each of these Users have their email address validated in a separate Thread.
threads.map(&:join)

The join method is used to make the program wait for all Threads to complete. If one Thread were to error out, some rescue and retry logic could be added to help.

Functionally adequate, this code should iterate over the entire set of User records, 30 at a time, and update them in parallel.

However, actually running this code will point out a very serious limitation.

Note: The GIL in MRI Ruby prevents this code from ever running truly parallel but since this code is input/output constrained, it does not make a difference.

Connection Accumulation

Moments after the above code runs for the first time, this error is seen:

ActiveRecord::ConnectionTimeoutError:
  could not obtain a database connection within 5.000 seconds

As it turns out, each Thread spawned had allocated its own connection to the database via ActiveRecord. Then, even after the Thread completed its work, the connection was not relinquished. Stale, idle connections began to accumulate within the Ruby process and before even 1000 User objects could be processed, the connection pool was empty.

Presumably, right after the self.validate_emails method completed for a batch of Users, the process should have released those connections to be used for either the next batch of Threads or other processes.

Since this liberation of database connections did not happen automatically, it seems that we must intervene.

Pool Full of Connections, then We Dive in it

To help ActiveRecord handle connection allocation and deallocation, the with_connection block is used. with_connection is a method that yields a connection to a block, then returns that same connection to the pool when the block has completed.

Basically, it ensures that the life cycle for a single database connection is only active within the block provided.

To make sure this solution will work, it is imperative that the database connection pool in either database.yml or in a connection initializer is set high enough.

An example database.yml:

production:
  adapter: postgresql
  database: production_database
  pool: 50

With a pool large enough, a with_connection block can be inserted around the pertinent code:

class UserEmailValidator
  def self.run
    User.where(validated: false)
        .find_in_batches(batch_size: 30) do |user_batch|
          validate_emails(user_batch)
        end
  end

  def self.validate_emails(user_batch)
    threads = user_batch.map do |user|
      Thread.new do
        ActiveRecord::Base.connection_pool.with_connection do
          email_validator = EmailService.new(user.email)
          email_validator.validate

          user.email = nil if email_validator.invalid?

          user.validated = true

          user.save!
        end
      end
    end

    threads.each(&:join)
  end
end

Running this very deliberate connection handling process now ensures that no more than 30 connections are claimed. Each Thread will connect to the database with one single connection and the with_connection block will return that connection to the predefined pool upon completion.

With the addition of this necessary connection pruning, the great User email validation effort can complete. Depending on the size of the data set and machines available, the 30 per batch limit could be easily increased, resulting in greater validation speed.

The database.yml configuration should have a pool size of equal to or greater than the number of Threads spawned. If 100 User objects needed to be processed at once, the pool should be >= 100.

Alternatives

Handling database connections and using raw Threads this explicitly can make some developers uncomfortable. Luckily, a few great libraries have been written to alleviate this problem.

In a previous post, EM Synchrony was utilized to make parallel HTTP requests. It also has the ability to parallelize ActiveRecord reading and writing without consuming excess database connections.

Another helpful library quickly becoming the most popular concurrency library is concurrent-ruby. The concurrent-ruby library provides convenient models for dealing with futures, promises and other parallelization data structures. However, the same ActiveRecord connections issues will occur with concurrent ruby objects as with plain old Threads.

Additionally, this code has been written for the sake of making a point. There are many ways to solve a problem and this is simply one of them.

How to Remove a Column with Zero Downtime in Ruby on Rails

2016-02-07T07:00:00-05:00

For a production Ruby on Rails application, uptime is paramount. Altering the structure of an application’s persistence layer is an operation that competes directly with uptime. Specifically, removing a table column within a relational database causes issues with the ActiveRecord ORM (the default relational mapping within a Ruby on Rails application).

However, this particular pain point has been removed as of Ruby on Rails 4.0.0, saving developers a lot of headache and greatly reducing the need for structural change coordination.

Old and Busted

To demonstrate the problem, a simple Ruby on Rails 3.2 application is created with a User model:

class User < ActiveRecord::Base
end

Supporting the User model is a PostgreSQL database table created with a migration:

class CreateUsers < ActiveRecord::Migration
  def change
    create_table :users do |t|
      t.string :first_name
      t.string :last_name
      t.string :email
      t.timestamps null: false
    end
  end
end

To ensure this application has as close to 100% uptime as possible, it can be assumed that it is behind some kind of pre-loader. As in, when code is deployed, existing requests are given time to complete before new requests are shepherded over to the new version of the code.

In a Ruby on Rails 3.2 application, a problem will arise when a column is removed from the database and the ORM does not have time to restart. In this case, even the pre-loader will not save an application from throwing an error about the missing column.

To emulate this problem, a rails console is run in the production environment and a User is created:

RACK_ENV=production rails console
> User.create(first_name: 'test', last_name: 'test')
# => #<User id: 1, first_name: "test", last_name: "test", email nil, created_at: "2016-02-07 21:03:26", updated_at: "2016-02-07 21:03:26">

In parallel, the psql command line client is used to connect to the same production database used by this server, all still running locally.

Within this psql prompt, the email column from the users table will be removed while the production console is still running:

psql app_production
psql=# ALTER TABLE users DROP COLUMN email;
ALTER TABLE

Jumping back to the RACK_ENV=production rails console, another User creation attempt results in an error:

> User.create(first_name: 'test2', last_name: 'test2')
# => PG::UndefinedColumn: ERROR:  column "email" of relation "users" does not exist

Kaboom! Rails attempts to save a new record to the users table and populate the email column even though it was never specified to do so. Seeing as this is not new behaviour, surely someone has come up with a solution to this problem that does not require downtime.

Yesterday’s Jam

As it turns out, a few different blog posts have been written about the solution to this very problem.

The TL;DR of these posts is that a multiple phase solution is required under the given circumstances:

1. Write code that `rejects` the column to remove
2. Deploy
3. Write migration to remove the column
4. Deploy + Migrate
5. Write code to remove the column `rejection`
6. Deploy

Phew! Three deploys and presumably three code reviews later, the column is finally removed. Assuming a system is built with continuous deployment guarded by a continuous integration suite, this whole process could take the better part of a work day in order to accomplish such a simple task.

Even worse, if an application has scheduled weekly deploys where only one deployment can happen per week, then this column will linger for three whole weeks. That is just absurd, what kind of barbarians can live this way?

New Hotness

Luckily, the incredible people who maintain and create features for the Ruby on Rails framework saw this behaviour and decided to correct it. As of a very influential commit during the beta release of Ruby on Rails 4.0.0, the framework does not aggressively attempt to populate attributes not specified during creation or update.

To prove this, a new Ruby on Rails 4.0.0 application is made with the exact same structure as above.

Then, the same production console is created along with a User:

RACK_ENV=production rails console
> User.create(first_name: 'test', last_name: 'test')
# => #<User id: 1, first_name: "test", last_name: "test", email nil, created_at: "2016-02-07 22:09:49", updated_at: "2016-02-07 22:09:49">

Again, in parallel, a psql client is initialized for the production environment database.

Within this prompt, the email column on the users table is removed:

psql app_production
psql=# ALTER TABLE users DROP COLUMN email;
ALTER TABLE

And finally, a new user is created in the still running RACK_ENV=production rails console:

> User.create(first_name: 'test2', last_name: 'test2')
# => #<User id: 2, first_name: "test2", last_name: "test2", email nil, created_at: "2016-02-07 22:13:18", updated_at: "2016-02-07 22:13:18">

Unlike before, the second insertion into the users table worked wonderfully, despite the email column’s removal.

As shown, the in memory representation of the User class still has the email attribute. The User object’s representation has not been reloaded, it simply just does not care about the email column when writing to the database.

The fact that the ORM does not specifically reload when writing to a table points out an interesting detail. It means that for this solution to fully work, the email column must not be referenced at all.

With this new found behaviour, the previous six step incantation comprised of three separate deployments can be reduced to three steps in total and just one deployment:

1. Remove references to column and write migration
2. Deploy
3. Once requests completely use new code, migrate

Wunderbar! The reject and deploy pattern is no longer necessary. Further more, development teams suffering within weekly scheduled deploys can achieve their desired results within a single iteration.

What Changed?

Unfortunately, the amazing change responsible for this vast improvement regarding Ruby on Rails column removal did not receive the attention is deserves. Many blog posts and guides were written about how to circumvent the older 3.2 behaviour, but when looking for information about this modern implementation, resources were scarce.

After reading through a very large amount of change logs and coming up empty, a bit of previous wisdom was utilized in order to track down the commit responsible.

The first step is to identify the offending code in 3.2. This way, the 4.0.0 version of the code can be analyzed, the discrepancy identified and the git revision responsible brought to light.

After some binding.pry insertions and utilization of the caller array, the offending code was traced to lib/active_record/attribute_methods.rb:244 (formatted for readability):

def arel_attributes_values(include_primary_key = true,
                           include_readonly_attributes = true,
                           attribute_names = @attributes.keys)
  attrs      = {}
  klass      = self.class
  arel_table = klass.arel_table

  attribute_names.each do |name|
    if (column = column_for_attribute(name)) &&
        (include_primary_key || !column.primary)

      if include_readonly_attributes ||
         !self.class.readonly_attributes.include?(name)
        value =
          if klass.serialized_attributes.include?(name)
            @attributes[name].serialized_value
          else
            # FIXME: we need @attributes to be used consistently.
            # If the values stored in @attributes were already type
            # casted, this code could be simplified
            read_attribute(name)
          end

        attrs[arel_table[name]] = value
      end
    end
  end

  attrs
end

While a little lengthy, the most obvious thing to note about this method is that its signature is comprised of entirely optional variables. Most notably, the final optional parameter, attribute_names, defaults to @attributes.keys which is the list of attributes for the given model.

In this case, the result of @attributes.keys is the complete list of keys for a User: ['id', 'first_name', 'last_name', 'email', 'updated_at', 'created_at']. Most notably, the email attribute is present, regardless of the fact that it was not given a value to insert. This means that by default, any create call will write to the email column without hesitation (just as previously observed).

While default behaviour can emit a code smell, it is only dangerous if it is used unknowingly. Since the parameter to determine attribute_names is available, chances are the caller of this method does indeed specify them, right?

Unfortunately, no:

def create
  attributes_values = arel_attributes_values(!id.nil?)
  new_id = self.class.unscoped.insert attributes_values

  self.id ||= new_id if self.class.primary_key

  IdentityMap.add(self) if IdentityMap.enabled?
  @new_record = false
  id
end

Here, in lib/active_record/persistence.rb:364 lies the method responsible for calling arel_attributes_values. Since it does not specify the list of attributes, the default behaviour prevails and all known attributes are returned.

The Big Fix

On a glorious Friday afternoon (probably) Jon Leighton made the commit which changed how people thought about column removal in Ruby on Rails.

The change enabled the concept of partial_writes. This pattern used the exiting changed hash present in every ActiveRecord model. The changed hash is manipulated as a model has attributes assigned to it.

In lib/active_record/attribute_methods/dirty.rb:77, partial_writes is used:

def create_record(*)
  partial_writes? ? super(keys_for_partial_write) : super
end

# Serialized attributes should always be written in case they've been
# changed in place.
def keys_for_partial_write
  changed | (attributes.keys & self.class.serialized_attributes.keys)
end

Before delegating to the same persistence.rb#create method, only the attributes present in the changed hash (along with any serialized attributes) will be used in subsequent insert queries.

Included in the February 2013 beta release of Ruby on Rails 4.0.0, this code enables a much more pleasant experience than the older work around. In fact, the author even pointed out this behaviour in his commit:

When inserting new records, only the fields which have been changed from the defaults will actually be included in the INSERT statement. The other fields will be populated by the database.

This is more efficient, and also means that it will be safe to remove database columns without getting subsequent errors in running app processes (so long as the code in those processes doesn’t contain any references to the removed column).

Unshackled

While this patch to the Ruby on Rails framework may not make every situation easier, it certainly reduces both cognitive and implementation overhead for many developers.

For more automated systems utilizing continuous deployment, two deployments may still be necessary since code running on old processes (within some sort of pre loader) will not be able to handle the column removal.

The deployment process for such a system may look more like:

1. Remove references to column
2. Deploy
3. Write migration to drop column
4. Deploy and migrate

Databases that power web applications are powerful and oddly fragile beasts. Manipulating these beasts while maintaining superb end user experiences requires one to constantly be learning. Hopefully with this behaviour, one more thing can be crossed off the check list of “things to remember when manipulating the database” and we can all just get back to work.

Nine Months of Commits

2016-01-30T07:00:00-05:00

It is hard to believe that already nine months have passed since the onset of my Year of Commits. In these past nine months, I have experienced a wide range of new aspects of open source development. This has truly been an eye opening experience, with many facets I doubt I would have experienced otherwise.

Confidence

An unexpected and fortuitous side effect of my Year of Commits has manifested as a significant boost in my overall coding confidence. Confidence is a very important aspect of anyone’s life. Without confidence, a person may place objectives out of reach or deem them too arduous to achieve. Believing in one’s self and having the confidence to stand up when knocked down is a trait that carries a lot of weight, especially when writing software.

Writing software can be a very subjective endeavor. Especially in languages like Ruby, there are many solutions to the same problem. These subtle differences can spark myriad conversations or criticisms of a person’s code. For those fortunate enough to write code for a living, think back to the times a pull request saw the most attention and accrued the most comments. Chances are, those comments pointed out style choice discrepancies or “we do not do it that way” assertions.

Maintaining a steady level of confidence in the midst of such feedback is paramount. I am lucky enough to have received feedback from many different people in many different situations. This feedback and perspective help inoculate a developer and encourage them to remember they are not their code.

Github Issues

Another exciting event since my last year of commits update pertains to Github issues. Github is a fantastic piece of software responsible for managing git repositories and enabling collaboration amongst developers. An issue may be opened by anyone with a Github account on any public repository of their choosing. These issues have been the primary avenue I have used to find which projects need help and what I can do to help them.

My excitement stems from the fact that, for the first time since I started Year of Commits, other developers have engaged with one of my projects (passages) and submitted some issues! Having consumers of software one builds is a great feeling. An even greater feeling is when those people care enough to submit issues they think can better the project.

Becoming part of the open source community was a goal of this endeavor and I am thrilled to have taken my first few steps towards accomplishing that goal.

Code Discovery

Practice makes perfect. When reading code you did not write, knowing where to start and identifying patterns in code organization helps a great deal. Since the goal of Year of Commits was to contribute to my own and other people’s code, I have been reading a lot of code that I did not write. This has been a fantastic learning experience for me. At a typical software engineering job, coworkers’ code starts to blend styles and eventually become fairly homogeneous.

By reading code written by many different people, new patterns and styles add a level of depth to code grokking that is hard to replicate in other ways. I came to appreciate this accidentally discovered benefit of Year of Commits so much, I wrote a whole post describing the process for foreign code traversal.

The Home Stretch

Year of Commits ends in April of 2016. Between now and then, I hope to learn even more about the open source software ecosystem and community. This project has had an incredibly positive affect on my life and I would prescribe something similar to anyone interested. My only goal between now and April is to have made a real contribution to a high visibility project.

The positive feedback I have received about this blog and the work that I have been doing helps make this easy. I would like to say thank you again to everyone on this journey with me.

Ruby Private Class Methods

2016-01-24T07:00:00-05:00

In the Ruby programming language, defined methods come in two variants: instance methods and class methods. Instance methods are available after an object has been initialized, creating an instance. Class methods, on the other hand, are available without creating an instance of the class they are defined upon.

Ruby methods can vary in visibility. Public methods are available in any context, while private methods’ availability is restricted within the instance of a class and its descendants. A third visibility scope, protected, behaves similarly to private methods, but protected methods can be called by other instances of the same class.

For a quick refresher, public and private instance methods look like this:

class Dog
  def do_trick
    bark
  end

  private

  def bark
    puts 'woof woof'
  end
end

When the public method is called:

> dog = Dog.new
> dog.do_trick
# => woof woof

And the private method:

> dog = Dog.new
> dog.bark
# => NoMethodError: private method `bark' called for <Dog>

Private class methods might not be as common as private instance methods, but they still have their place. For instance, a class method may require internal helper methods to complete its function. Whatever the reason, defining private class methods has value but is not always intuitive.

This example Dog class needs to maintain a list of tricks that will be used within the other public class methods. This list should not be accessible to any callers outside the Dog class.

The wrong way

A first pass at writing the private tricks method could look like:

class Dog
  private

  def self.tricks
    [:bark, :roll_over, :fetch]
  end
end

However, when testing the visibility of the tricks method:

> Dog.tricks
# => [:bark, :roll_over, :fetch]

Uh oh, no error was thrown indicating a that a private method was called, this method is completely public.

Why?

The reason that the above code did not produce a private method has to do with Ruby’s object hierarchy, interactions amongst internal classes, instances of those classes, and eigenclasses. A detailed write up about how Ruby’s objects work with one another can be found in a previous post.

When defining methods in a class, the default context and the context introduced by the self. method declaration are distinct.

The private method scope can not be used in the above way as it does not handle the context change between methods defined on the Dog class and defined within self context.

So what are the alternatives?

1. `class << self`

One alternative way to define class methods on an object is to use the class << self syntax. This syntax opens an eigenclass for the supplied argument. In this example, self within the Dog class will open the Dog's eigenclass.

class Dog
  class << self
    private

    def tricks
      [:bark, :roll_over, :fetch]
    end
  end
end

One thing to note is that when defining methods like this, the declaration has changed from def self.tricks to simply def tricks.

Now, the private scope is preserved and expected behaviour is achieved:

> Dog.tricks
# => NoMethodError: private method `tricks' called for Dog:Class

2. Modules

Modules in ruby are collections of methods and constants. These collections can be used as encapsulation tools or, in this case, alternatives to defining public and private class methods.

When a class extends a module, all the methods within that module become class methods on the subject class*. This pattern can be used to define the tricks method on Dog.

*Note: This only pertains to methods defined in a “typical sense” any def self. or def Dog. method definitions in a module do not automatically become class methods in the same way when extended.

class Dog
  module ClassMethods
    private

    def tricks
      [:bark, :roll_over, :fetch]
    end
  end

  extend ClassMethods
end

> Dog.tricks
# => NoMethodError: private method `tricks' called for Dog:Class

A benefit of this approach is readability. The module named ClassMethods, which is contained and thus encapsulated by the Dog class, is the clear home for all desired class methods.

Visibility modifiers like private behave accordingly and a simple extend at the bottom of the module definition brings it all together.

3. `private_class_method`

A third approach is to use the built in method Module#private_class_method. This method’s purpose is to change the visibility of existing class methods.

Unlike the other two solutions, this one does not require a different style of method definition from the incorrect solution:

class Dog
  def self.tricks
    [:bark, :roll_over, :fetch]
  end

  private_class_method :tricks
end

> Dog.tricks
# => NoMethodError: private method `tricks' called for Dog:Class

This method can also be in-lined during the class’ method definition:

class Dog
  private_class_method def self.tricks
    [:bark, :roll_over, :fetch]
  end
end

> Dog.tricks
# => NoMethodError: private method `tricks' called for Dog:Class

If a single class method must be private and saving lines is very important, this style might be applicable; but, it certainly sacrifices a level of readability.

A major strength of private_class_method is its explicitness. Unlike the other approaches, this one does not require a special module definition or method definition context switching.

Contributing to Open Source in 7 Steps

2016-01-17T07:00:00-05:00

Making contributions to open source repositories is a great way to give back to the community. Open source software has a rich history and is a crucial component in many successful software applications. Most open source projects are maintained by either one or many developers for free. Since “free” is not usually the price of food and shelter, these maintainers are not available to work on their projects at all hours, most of them have day jobs so they can eat and buy clothing.

To help keep open source projects up to date and bug free, members of the community regularly contribute a portion of their own free time. Since the start of the Year of Commits project, I have made quite a few contributes to open source software and distilled the process down to these seven basic steps.

1. Find a repository that has some issues

A great way to get a foot in the door in an open source repository is by solving one of its open issues. A project is much more likely to accept code that addresses an open issue as opposed to code that is added “just because”. Open issues are either bugs or feature requests and are tracked on Github.

Some really great tools exist for exploring Github’s vast index of repositories for relevant projects. CodeTriage is one of these awesome tools. CodeTriage enables a user to narrow down Github’s list of repositories by programming language. CodeTriage has a very nice UI, ordering a very large amount of Github projects in descending count of open issues.

Alternatively, Github has their own tooling around discovering projects with their explore feature. Trending repositories from this week or this month are available on a per programming language basis. The explore feature is great for popular projects but exposes far fewer projects than CodeTriage.

To demonstrate a typical contribution’s life cycle, the fetching-gem project has been chosen.

The fetching-gem library is a convenience Ruby library for accessing elements of nested hashes and arrays with dot notation and noisy feedback.

Recently, the fetching-gem's issues page had a feature request: “Make the library compatible with the Enumerable#first(n) behaviour”:

2. Read the contributing guidelines

Before starting work on a project, make sure that the contribution guidelines of the project have been met. Some more popular repositories have very explicit rules about the number of commits per patch, the logical grouping of those commits, and even commit message style concerns.

The fetching-gem's contribution guidelines are very simple, it merely instructs that a fork should be made and then a pull request from the forked repository should be made.

3. Fork it

Obtaining a copy of a project’s code is crucial when trying to make a contribution. In Github, the act of forking a repository creates a copy of the repository. This copy may be altered until the feature is built or bug is fixed without disrupting the original project. The Fork button can be found at the top of a repository’s Github page.

After the Fork has been made, the next step is to clone the forked copy locally:

$ git clone git@github.com:<your_username>/fetching-gem.git

Cloning the repository locally will automatically add the origin git remote URL to the code directory. Additionally, adding the source remote URL right away is a great way to make sure that local code and the master remote code can easily stay in sync.

$ cd fetching-gem/
$ git remote add upstream git@github.com:covermymeds/fetching-gem.git
$ git remote -v

origin git@github.com:yez/fetching-gem.git (fetch)
origin  git@github.com:yez/fetching-gem.git (push)
upstream  git@github.com:covermymeds/fetching-gem.git (fetch)
upstream  git@github.com:covermymeds/fetching-gem.git (push)

The above command adds a new remote, upstream, to the directory. With this remote in place, a git fetch upstream command will grab the latest updates from the original repository.

Exploring unfamiliar code is not the most intuitive process, but thankfully a number of blog posts have been written about the topic.

4. Reproduce the issue

With the fork cloned locally and remote pointers all set up, it is time to reproduce and address the issue. Since this particular issue was a feature enhancement ask, “reproducing” the issue consisted of calling a method with the wrong arguments and expecting it to fail:

> require 'fetching-gem'
> fetching_array = Fetching([1, 2])
> fetching_array.first(2)
ArgumentError: wrong number of arguments (1 for 0)

Success! The code definitely does not support the Enumerable#first(n) behaviour, so there is indeed work to be done.

5. Fix the issue

Both when adding code and altering existing code, it is important to make sure it is tested. In most cases, the project under revision will have chosen a particular testing library and have some amount of tests. Adding the appropriate tests at the same time as the patched code will boost the library’s maintainer(s) confidence in the patch.

Adhering to the style of the code under manipulation is also important. From test example descriptions to variable declaration styling, consistency is a complexity reducing concept that is extremely valuable later on.

For brevity’s sake, the finer details surrounding my example fetching-gem patch can be found on Github.

6. Submit a PR

With the contribution guidelines adhered to, the code altered, and the issue taken care of, it is time to submit the long anticipated patch.

A well crafted pull request (PR) message consists of a few key elements:

It is concise. Too much explination will be glossed over and ends up doing more harm than good.
A special “Resolved #" text exists, allowing Github to automatically reference and resolve the issue upon a successul merge.
If applicable, an example of the new behaviour is given.

The pull request that I recently added to fetching-gem looked like this:

Another possible check when submitting a PR involves continuous integrations. If a project automatically builds its test suite with a service like Travis, Github will supply some checks at the bottom of the PR:

If, for some reason, this status is not green and the tests did not pass, use the Details link to find out more information why.

7. Feel Awesome!

Pending continuous integration passing and the repository’s maintainer(s) signing off on the change, a patch will be accepted into the project. What a fantastic feeling that is! Awesome maintainers (like the fetching-gem maintainer mikegee) will show their appreciation directly on the PR, immortalizing your contribution for all days to come.

7b. Repeat 4 - 6

In the case that the submitted code is not exactly what the maintainer(s) is looking for, some additional work might be warranted. This back and forth could be thought of as frustrating, but it is also a great learning opportunity. When else is feedback so easily received about one’s code from people they would have probably never met otherwise?

Open source projects have tremendous value and a strong presence in the technology ecosystem. Contributing to these projects is an easy way to show appreciation and feel like a valued member of the open source community.

How To Parallelize Ruby HTTP Requests

2016-01-10T07:00:00-05:00

It turns out that managing web requests is quite important when doing web development. A web application backed by an external or internal API can issue a lot of requests when rendering a seemingly simple web page. How those requests are made and in what order is very important. With improper parallelization, an end user’s entire experience can go from delightful to horrific in a matter of seconds.

The Build Up

A basic Ruby on Rails application might have the following features:

A User can create a favorite of an item, creating a FavoriteItem.
A User can add an item to their wishlist, creating a WishlistItem.
A User can buy an item, creating a TransactionItem.

Each model this system uses is backed by an API. User, TransactionItem, WishlistItem, and FavoriteItem models all require a remote HTTP request for their information.

As a web application experiences growth, this structure is not uncommon. The same API might back a mobile app, website, and any other internal tooling to help this company with its day to day affairs.

The API that this application uses works in a two phase manner:

A user can be requested by their id.
- /users/:id returns a User corresponding to the given id.
Each supporting model is requested with the same user_id.
- /users/:id/favorite_items will return an array of FavoriteItems for the specified User.

The contract of this API is for all intents an purposes, non-negotiable. In-lined data or other request saving patterns are not available, the client must use the API as provided.

Sequential Approach

Within this example application, the most request intensive page is the User's history page. The history page consists of everything the user has done. Items a user has added to their favorites resulting in FavoriteItems, Items bought by the user resulting in TransactionItems, and Items added to a user’s wish list resulting in WishListItems.

To complement the (ex/in)ternal API, two helper methods exist on each model: remote_find which accepts an id or array of ids, returning the matching models, and remote_find_by_user_id which accepts a user_id and returns an array of ids for the corresponding model.

Leveraging the API and these two helper methods, a serial version of a User's pertinent data could look something like:

class UserHistoryController < ApplicationController
  def show
    @user = User.remote_find(params.require(:user_id]))

    @favorite_items =
      FavoriteItem.remote_find_by_user_id(@user.id)

    @wish_list_items =
      WishlistItem.remote_find_by_user_id(@user.id)

    @transaction_items =
      TransactionItem.remote_find_by_user_id(@user.id)
  end
end

However slow it may be, this code will indeed fulfill its duty. First the User is found, then all the supporting information about a User's history is retrieved one by one from the API.

In a worst case scenario, assume the API returns an individual request in about 150ms. With 4 requests, this means that only required page elements will take 600ms.

Tacking on view rendering, whatever data processing or formatting that needs to be done at the presentation layer, and finally some amount of Javascript, this page becomes objectively slow.

Enter `EM-Synchrony`

As with most problems, someone smart has had it before and probably done something about it. Luckily, this particular problem has been addressed by Ilya Grigorik’s EM-Synchrony library.

Grigorik’s library leverages the use of Ruby Fibers to parallelize* code execution.

*Note: Because of the Global Interpreter Lock (GIL) present in MRI, true parallelization is not possible.

The real benefit of using EM-Synchrony is the way it handles scheduling the underlying Fibers. A Ruby Fiber is basically a Thread that is not automatically scheduled by the Ruby VM. This means that it is up to the programmer to let the Fiber know when it should relinquish control to another Fiber.

Since this example is almost completely bound by Input/Output (I/O) operations, it is a perfect candidate for EM-Synchrony.

To use EM-Synchrony effectively, existing requests must be broken up into pieces that can accessed independently. A requests method can be created to extract details surrounding each request. The result of the requests method must be an enumerable.

Note: As explained below, EM-Synchrony can only be used with supported HTTP clients.

class UserHistoryController < ApplicationController
  # ...
  # ...

  private

  def requests
    {
      User => {
        method: :remote_find,
        arg: @user_id,
        instance_var: :@user
      },
      FavoriteItem => {
        method: :remote_find_by_user_id,
        arg: @user_id,
        instance_var: :@favorite_items
      },
      WishListItem => {
        method: :remote_find_by_user_id,
        arg: @user_id,
        instance_var: :@wish_list_items
      },
      TransactionItem => {
        method: :remote_find_by_user_id,
        arg: @user_id,
        instance_var: :@transaction_items
      }
    }
  end
end

For the sake of being explicit, this very redundant helper method will be used to iterate over requests and allow EM-Synchrony to process them in parallel:

class UserHistoryController < ApplicationController
  CONCURRENCY = 4

  def show
    @user_id = params.require(:user_id)

    EM.synchrony do
      EM::Synchrony::FiberIterator
        .new(requests, CONCURRENCY)
        .each do |key, request_hash|

        # Example of below with real values:
        #
        # key = Transaction
        # Transation.remote_find_by_user_id(@user.id)

        result = key.send(request_hash[:method], request_hash[:arg])
        instance_variable_set(args[:instance_var], result)
      end

      EM.stop
    end
  end
end

A few important lines of this solution are worth a closer look:

CONCURRENCY = 4

This line tells EM-Synchrony how many Fibers it is allowed to run at once. Since the example needed to request four remote resources, the concurrency amount is set to four.

The next line of interest is:

EM::Synchrony::FiberIterator
        .new(requests, CONCURRENCY)
        .each do |key, request_hash|

Here is where the actual work is done. After meticulously crafting the structure of the requests hash, the FiberIterator will pick the next element from the list and give it to a Fiber.

Finally the line stopping of Event Machine can not go unnoticed.

EM.stop

This method has literally no documentation but is in every example of using Event Machine, so it is probably very crucial.

And that is all there is to it! All four HTTP requests can now be run in parallel, resulting in a major speed up and a much happier user experience.

Caveats

A few considerations must be made in order to use EM-Synchrony effectively.

A compliant HTTP library must be used.
- As em-http-client’s the README a compliant HTTP library must be used when making the requests. Faraday is my favorite compliant library and it has worked great for me so far. Unfortunately, the very popular HTTParty library is not compliant and therefore no one may “Party Hard” when using EM-Synchrony and HTTParty.
One size does not fit all.
- Unlike the described example, not all code can be saved by EM-Synchrony. If a request depends on the result of another request, sharing those results between Fibers is neither simple, nor a good idea.

Passages Rails Engine

2016-01-03T07:00:00-05:00

Routing in the Ruby on Rails world can, at times, be a tad confusing. The official Rails guide is very helpful for the basics; but, as an application grows, it can become hard to remember specific details about every single route.

Existing Routing Tools

To remedy the issue of route complexity, a few helpful tools already exist. The most useful one is rake routes, which can be executed in the working directory of a Rails application. This tool requires that the application be on the developer’s local system, which is fine for applications that a developer owns, but what about services that the developer does not own?

While this problem might not exist for everyone, chances are at least one poor software engineer has been slapped in the face with a 404 page and shouted: “I know that route exists! Why doesn’t this work?”

Another helpful tool in development mode is this screen:

This page provides the ability to enter search terms that match words in the paths of specfic routes. However, similarly to rake routes, this screen is only accessible in development mode. While this tool is still extremely useful, there could be cases when route inspection would be helpful without running the server locally.

New Hotness

The Passages Rails Engine was created to fulfill two main purposes:

1. To expose routes of a Ruby on Rails application either during or outside development mode.

2. Enable searching on multiple pieces of a route’s information (HTTP verb, controller, path, etc).

Some might wonder by 1 would even be valuable. Imagine for a moment that a team of engineers decides that they need an internal API to power their various applications. Perhaps while developing this API, some basic documentation is written but never actually kept up to date. Maybe the API changes so fast that documentation just falls behind.

Whatever the reason, it becomes laborious for the consumers of that internal API to constantly ask which route does what and which parameters are expected in each URL.

Internal APIs are not the only candidate for Passages. An external API can also benefit from discoverable routing in lieu of or in addition to documentation. Since Passages has the option to use HTTP Basic Authentication, securing Passages is as simple as setting two ENV variables.

Purpose 2 is an attempt to reduce a large application’s route pile by expanding each route’s search surface area. A user of Passages might want to know of all the PUT routes or routes that map to a destroy action. Like its predecessor, Passages' route path search is also extremely valuable.

With (hopefully) the help of the open source community, the types of useful search features for Passages will expand from these very humble beginnings.

Demo

After following the installation guide from the project’s README, A very basic use of Passages would look like:

Contributing

Like most Open Source projects, Passages has many areas that welcome modification. Specifically, the design and search functionalities are suitable candidates for enhancement.

Please feel free to fork the repository or create an issue on Github if something seems wrong or out of place.

Currently, Passages only supports Ruby on Rails 4. It will most likely be upgraded when Ruby on Rails 5 is released but there are no plans to support Ruby on Rails 3 or older.

Things I Wish I Knew When I Started Programming

2015-12-27T07:00:00-05:00

If by some magical event I could go back in time to the day before I started my first job as a software engineer, this is what I would say.

Drive Slow

I mentioned a similar message in a previous post and I believe it is worth repeating. When starting a new job or a new career, it is easy to place external pressures on ourselves regarding time. This is a common issue that can arise when comparing one’s own work with those around them. As the new employee, noticing and measuring your own speed and efficiency against existing personnel can feel like the correct thing to do; however, chasing the “competition” or “proving that you are just as fast as them” will only result in high stress levels and bug riddled software.

The people who seem “fast” or come off as a “10x engineer” were not born that way. They took their time and learned their craft. And with that, they have slowly accelerated their process and patterns to make them appear extremely quick and efficient.

Deadlines should be thought of as real and valid; however, working at lightning speed to try and meet those deadlines will end up causing more work in the long run. What would be better, missing a deadline by a day or making a deadline and incurring so many bugs and tech debt that it takes two weeks to clean everything up?

ABL

Glengarry Glen Ross’s iconic scene has very clear message: Always Be Closing, meaning that one should constantly strive to be in the process completing a sale. When starting out writing software and even after years of doing it, a similar attitude can be extremely helpful: Always Be Learning.

From day one to day one thousand, learning and absorbing new information is very important. A key to learning new things is in asking questions. When starting a new job or a new career, there seems to be a stigma attached to asking questions. We are all under the impression that “they hired me to do this job, they must assume I know exactly what I’m doing.” This mentality can inhibit our natural inquisitive nature which is a great catalyst to learning.

Ridicule and impostor syndrome are the most likely the reasons stopping people from asking questions. The idea that questions might expose a person as not “good enough” or “smart enough” for their job is ridiculous and often unfounded. Getting over these fears early in one’s career is extremely important. Questions should be encouraged and celebrated. Ask questions every single day, ask a lot of them but make sure not to ask the same one too many times. No one enjoys repeating themselves.

It is Ok to be Wrong

Similar to not asking questions, most new engineers are terrified of being wrong. A golden rule is to be correct frequently, but no one is capable of being correct all the time. The trick to being wrong is how a person reacts when it happens. There is a huge difference between promoting a technique or solution and being closed-minded to alternatives.

If someone was never wrong, how could they learn? And if they did not learn, how could they Always Be Learning? Tackling instances of being wrong with an open mind helps refine a person’s skills, shaping them with feedback from others. While this mentality, like others already mentioned, is relevant throughout a person’s career, it is especially important in the beginning. In the beginning a person is likely wrong a lot of the time. This means that in the beginning a person has the most opportunity to grow; to be wrong and learn from those mistakes. Internalizing these mistakes can help bolster a person’s technical skills and confidence.

R.T.F.S.

At the onset of one’s career, it is easy to treat third party software or services as magical boxes of mystery. However, this is not necessary. It is actually very easy to open and read the source of many third party applications (especially in web development). Therefore, it becomes imperative to Read The Freaking Source.

Reading and having at least a basic understanding of how a piece of third party software works is paramount for a new developer. If foreign code is treated as magic, that is all it will ever be. This magic can become a crutch that an innocent new developer may find themselves leaning against more and more until it eventually breaks, leaving the developer confused and helpless.

Note: Not all third party software’s source can be read as not all of it is open source. If this is the case, do not try to read the source, it will not make much sense.

Who wants to be confused and helpless? No one wants that and no one needs to experience that. Reading the source code of a library or crucial bit of third party open source software can help bring light to an otherwise shrouded system, bringing a new level of depth to a developer’s understanding.

Introduction to Rails 5 Attributes

2015-12-20T07:00:00-05:00

Shortly after the tenth anniversary of Ruby on Rails 1.0, Rails 5.0 Beta has been announced. While the main character of this release was without a doubt ActionCable, other really great features have made their debut.

Types of Changes

One feature that particularly stood out is the introduction of ActiveRecord Attributes. This feature allows a developer to assert a specific type for a given attribute and an optional default value. It is not strict type validation (which I have a very strong affinity for), but it does define explicit type coercion that can be very useful.

Given an example application that deals with Transactions, a single table and model might exist:

class CreateTransactions < ActiveRecord::Migration
  def change
    create_table :transactions do |t|
      t.integer :user_id
      t.string :item_name
      t.integer :quantity
      t.string :success
      t.decimal :price
      t.timestamps(null: false)
    end
  end
end

class Transaction < ActiveRecord::Base
end

This structure, like most in the wild, could have been created before all edge cases were thought through. For some reason, success is a String instead of a Boolean. While this problem might seem trivial, imagine a system where hundreds of millions of transactions exist.

In such a system the entire column might not be able to change without incurring downtime. This is a perfect example problem that Attributes can help remedy.

Starting simple, a single line can be added to the Transaction class definition to coerce success to a boolean:

class Transaction < ActiveRecord::Base
  attribute :success, :boolean
end

Using the same logic that ActiveRecord uses for database column coercion, we can see attributes in action!

transaction = Transaction.new(success: 'yes')
transaction.success
# => true

transaction = Transaction.new(success: 'f')
transaction.success
# => false

transaction = Transaction.new(success: 0)
transaction.success
# => false

Just like that, the schema has been improved. Aside from raw SQL update statements, this code now prevents strings like "maybe" from littering the success column of the transactions table.

Before Attributes, a callback or custom setter method would have been the preferred way to address this problem. However, with this new built in solution for mismatched database column types, old one-off or ad-hoc solutions can be replaced with this new standard.

The Friendly Type

With such a clean DSL, it is obvious that a lot of time and thought went into the design of Attributes. Being able to specify how a model should interact with the persistence layer is a powerful tool.

The full list of supported types that Attributes provides can be found deep in the source code for ActiveRecord or right here:

#Supported Types

:big_integer
:binary
:boolean
:date
:date_time
:decimal
:float
:integer
:string
:text
:time

Each of these types has their own place and value in a system. Their purpose is simple: to make the experience between developer and framework fluid.

The Attribute feature is not restricted to database columns. If an attribute is only used during the life cycle of an object, it too can benefit from type coercion.

For instance, if a confirmed_at attribute had a useful purpose for a Transaction, but the format of it could vary, the Attribute module can step in and keep things clean.

class Transaction
  attribute :confirmed_at, :date_time
end

Without adding a new column or defining an attr_accessor for the confirmed_at attribute, the :date_time coercion works perfectly:

transaction = Transaction.new
transaction.confirmed_at = '2015-12-12 03:00'
transaction.confirmed_at
# => Sat, 12 Dec 2015 03:00:00 UTC +00:00

transaction = Transaction.new
transaction.confirmed_at = '2015/12/12'
transaction.confirmed_at
# => Sat, 12 Dec 2015 00:00:00 UTC +00:00

Since most Ruby on Rails applications deal with strings via form encoded data, it is easy to see just how valuable the Attribute module can be.

Out of the Box Typing

A fantastic detail about Attributes is that type coercion is not limited to only “supported” types. Any object that adheres to a proper contract may be used.

To help illustrate this feature, we can assume that all prices in the database have been changed to only deal with cents. This helps remove some complexities with floating point math oddities and prevents nefarious Office Space/Superman III bugs from creeping up.

Given a MoneyType object:

class MoneyType < ActiveRecord::Type::Integer
  def type_cast(value)
    if value.include?('$')
      price_in_dollars = value.gsub(/\$/, '').to_f
      price_in_dollars * 100
    else
      value.to_i
    end
  end
end

An important takeaway here is the inheritance and type_cast method definition. Both elements are necessary to create a custom type that the Attributes module can use effectively.

To use this new type, an initialized object is passed to the attribute method:

class Transaction < ActiveRecord::Base
  attribute :price, MoneyType.new
end

Then, if all keystrokes were done with style and grace, the effects should look something like:

Transaction.where(price: '$10.00')
# => SELECT * FROM transactions WHERE price = 1000

The observed behaviour here is very interesting. A string representing a money amount can be coerced to an Integer value representing the same data. Instead of the MoneyType’s type_cast method being extracted and used many times in controller or other model methods, it can be centralized to one specific spot.

Like the previous example, this solution is not the only way to achieve the desired result; however, it is unique enough to inspire new ways of thinking about these problems.

Type Validation

Avoiding hidden type coercion was a driving force when I created the validates_type gem. With Attributes, that coercion is still there but can be explicitly specified in a model’s definition. While this is a great step forward, I still find value in rejecting data of the wrong type. As a previous example can illustrate: why does the string "yes" map to the boolean value true? It does make some sense but "yes" is not a boolean.

The addition of Attributes will hopefully promote some new patterns and considerations around dealing with types in Ruby on Rails applications. Data consistency and reliability are important, it is exciting to see people making steps toward it in the Rails community.

Ruby Refinements

2015-12-13T07:00:00-05:00

The Ruby language provides many powerful tools for software engineers to utilize. For instance, classes that have been previously defined and evaluated can be reopened and changed. This is commonly referred to as “monkey patching”, a term which elicits almost universal disdain among Ruby developers.

The Problem

A reason that monkey patched code has issues lies in the scope of the changed code. If previously defined code is changed at an arbitrary time, all other parts of the application suffer from those changes.

Why would someone need to change an existing class? Perhaps an included gem needs to be altered in a small way to behave correctly in a very specific system. Or maybe, there exists a very dark area of a codebase that must not be touched directly for fear that the entire application will go under.

Whatever the case, patching code that has been already defined happens, and it usually happens poorly.

These problems can be especially nefarious if the patches are not in automatically loaded files. For example, say we have a Dog class:

dog.rb

class Dog
  attr_accessor :trained

  def bark
    "woof woof"
  end
end

Then in file loaded later, the bark method changes to be much more formal:

training.rb

class Training
  def train(dog)
    dog.trained = true
    dog.bark
  end
end

class Dog
  def bark
    "Woof woof, good sir."
  end
end

After the Traning class is loaded, any consumers of the Dog class will be in for a surprise whenever the bark method is called. Even worse, when a confused developer opens the dog.rb class to check bark’s functionality, they will not see the patched version.

dog = Dog.new
dog.bark
# => woof woof

require './training'
training = Training.new
training.train(dog)
# => Woof woof, good sir.

dog = Dog.new
dog.bark
# => Woof woof, good sir.

As shown, the second initialized Dog barks the same way as the trained Dog. Globally, the way a Dog barks has been changed after the training file has been included.

An alternative way to extend a class’ functionality is by using the built in ruby construct: Refinements. Refinements are context specific alterations to a class’ methods.

To refine the Dog class, a module needs to be written:

module SophisticatedDog
  refine Dog do
    def bark
      "Woof woof, good sir."
    end
  end
end

The interesting piece here is the refine method. This method returns an overlaid module specific to the class passed into it, allowing a very small scoped change of the Dog class.

To include this Refinement in the place where it is needed, the using method can be added to the Training class.

class Training
  using SophisticatedDog

  def train(dog)
    dog.trained = true
    dog.bark
  end
end

Now, the changes to the Dog method are contained within the Training class:

dog = Dog.new
dog.bark
# => woof woof

require './training'
training = Training.new
training.train(dog)
# => Woof woof, good sir.

Neither the passed in dog, nor newly created dogs’ bark method has been permanently changed:

dog.bark
# => woof woof

dog = Dog.new
dog.bark
# => woof woof

No more surprise behaviour! Each new Dog is created just as unrefined and unsophisticated as ever.

Super Cool

Another problem with traditional monkey patching is that the patched method is no longer accessible.

class Dog
  def bark
    'woof woof'
  end
end

# Then later:

class Dog
  def bark
    super + ', good sir.'
  end
end

Dog.new.bark
# => NoMethodError: super: no superclass method `bark' for Dog

The original implementation of Dog had a very specific string that might not want to be duplicated. When the class is monkey patched, the original method is replaced and super is not accessible.

In a less silly example, maybe the method being patched had some valuable code a patch could have used. Since Refinements are not strict code overwrites, they maintain the super functionality found in inheritance:

class Dog
  def bark
    'woof woof'
  end
end

module SubWoofer
  refine Dog do
    def bark
      super.split(' ').first
    end
  end
end

class DogTest
  using SubWoofer

  def call
    dog = Dog.new
    dog.bark
  end
end

DogTest.new.call
# => 'woof'

Dog.new.bark
# => 'woof woof'

The SubWoofer Refinement was able to call the existing Dog#bark method and change it how it saw fit. This keeps the code around barking DRY. Since Refinements are intended to extend or refine existing code, it makes sense that they could use the existing code to their benefit. When that existing code is changed, it would be arduous to make every single Refinement aware.

Using Refinements is a nice alternative to the sledgehammer that is monkey patching. Bending a class to fit a very specific need in an encapsulated manner can mean the difference between a stable system and one riddled with hard to track down bugs. While Refinements still have their own drawbacks, they are much less intrusive than the alternative.

How to use Rails Strong Parameters

2015-12-06T07:00:00-05:00

In the latest major version of Ruby on Rails, Strong Parameters were introduced. The intent of this addition was to enable consistent and reliable parameter checking. Using Strong Parameters is simple and intuitive. It provides a very clean method API to help keep controllers DRY.

However, knowing when to use Strong Parameters and how to use them correctly is very important. After all, what good is parameter validation if it is in the wrong place or conveys the wrong message?

Hit the Books

The trusty “booksandreviews.com” will serve as a good example application.

Given a controller that can create Authors, a default pattern without Strong Parameters could look like:

class AuthorsController < ApplicationController
  # POST /authors
  def create
    @author = Author.create!(author_params)

    redirect_to @author, notice: 'Author was successfully created.'
  end

  private

  def author_params
    params[:author]
  end
end

Assuming that there are no presence validations on the Author class itself (or in the form view creating this model), this code has no check to make sure that anything about an Author is passed in to create.

This means that any POST request is made to /authors will create a new Author. While this kind of problem would most likely not make it to production, let us assume that the poor people over at “booksandreviews.com” came from such humble code beginnings that this was the first iteration of the AuthorsController.

To make the AuthorsController more robust, the require method can be used on the params hash:

class AuthorsController < ApplicationController
  # ...

  private

  def author_params
    params.require(:author)
  end
end

Now if a POST to /authors does not contain a payload with an :author key in the body, the request will error:

ActionController::ParameterMissing

However, a request with the :author attribute will…

ActiveModel::ForbiddenAttributesError Also cause an error, apparently.

This is another feature of Strong Parameters. The reason an error occurs is that the author_params method did not specify which parameters could be mass assigned to the Author class. Passing the result of require directly to Author.create will trip this safety measure.

Shut ‘em Down

To correct the ForbiddenAttributesError and add some much needed structure to the AuthorsController, the permit method needs to be used in conjunction with require.

Adding a list of attributes as individual symbols will make sure that only those attributes can be passed through to Author.create!

class AuthorsController < ApplicationController
  # ...

  private

  def author_params
    params.require(:author).permit(:name, :country, :email)
  end
end

Great! Now a properly formatted POST containing only the correct attributes will create an Author appropriately.

But, why go through all the hassle? There should be a way to let all attributes pass, regardless of how crazy harmful they could be to our application we use to pay bills.

The permit! method is a completely permissive way to allow all attributes to be mass assigned to a model:

class AuthorsController < ApplicationController
  # ...

  private

  def author_params
    params.require(:author).permit!
  end
end

This will allow any attributes nested under :author to be set on an Author. A solution like this requires no declaration of attributes, but is this better?

The short answer is “no”.

The reason why being completely permissive with parameters is a bad idea is because not all attributes were created equally.

If “booksandreviews.com” had an admin or superuser column on their authors table, any POST with admin: true would create not a normal Author, but an all-mighty Author.

Specifying the attributes that can be mass assigned to an ActiveRecord model is not a new concept, with Strong Parameters this pattern is just too easy not to do.

Open Up Shop

Another great feature of using Strong Parameters is type assertion. Regardless of ActiveRecord’s type coercion, ensuring that data is as valid as early as possible is important.

Consider that an Author can have an array of genres which summarize their bodies of work. To ensure that this list of genres is an array, a hash syntax can be used:

class AuthorsController < ApplicationController
  # ...

  private

  def author_params
    params.require(:author)
          .permit(:name, :country, :email, { genres: [] })
  end
end

By adding { genres: [] } to the end of the permit call, it ensures that the genres value is always an array. This prevents random nonsense or a malformed request from compromising the Author class’ data integrity. If something other than an array is passed as the genres attribute, it will disregard it.

Alternatively, when a parameter is a hash (like preferences), all the keys in the hash need to be present in the permit call.

{
  "preferences": {
    "time_zone": "GMT",
    "subscribed": true
  }
}

class AuthorsController < ApplicationController
  # ...

  private

  def author_params
    params.require(:author)
          .permit(:name,
                  :country,
                  :email,
                  { preferences: [:time_zone, :subscribed] })
  end
end

If this is a tad confusing, check out more examples at the official guide for Strong Parameters.

Everything in Moderation

Strong Parameters is not appropriate for every situation. In the case that a certain number of parameters must be present, or if one parameter must only be present when not accompanied by another specific parameter, Strong Parameters will fall short. The time and place to use Strong Parameters is when creating or updating an ActiveRecord object.

Additionally, if the Ruby on Rails application in question was an API, Strong Parameters would only be effective in very specific circumstances. Since the feedback given by Strong Parameters is not as granular as an API would like to be, it could be hard to find many uses of it.

Validates Type 2.0

2015-11-29T07:00:00-05:00

One of the first projects that I worked on during this Year of Commits was validates_type. In case that name is too obscure, validates_type is a gem for validating that a specific value is exactly the type it is expected to be. The validates_type library is compatible to ActiveModel style validations.

Originally, the validates_type gem was very strict with what it validated against. It started with basic included Ruby types like Integer, Float, and String. This was a fine first step but proved too restrictive for basically any use case.

In a recent update, the validates_type gem has been extended to validate against any defined type in a system. With this update, the uses of this gem greatly increased. They increased so much that it is possible more than 2 people will use it, and that would be pretty awesome.

Still Valid

The validates_type gem can be useful for a system that cares about exact types at save time. For instance, a model that has an incorrect database column type (because of legacy or other reasons) can still control validation over its data just the same.

If a “junior developer who is now somehow the CTO” created the original system. And if that developer did something like create an is_published column on the authors table set as a varchar, validates_type makes that less of an issue.

class Author < ActiveRecord::Base
  validates_type :is_published, :boolean
end

If someone tries to assign a nonsense value to the is_published attribute, validates_type will ensure it does not save.

Note: callback skipping methods will still save bad values, just like any other ActiveRecord validators.

> author = Author.new
# => #<Author id: nil, name: nil, created_at: nil, updated_at: nil, is_published: nil>
> author.is_published = 'foo'
# => "foo"
> author.save!
# => ActiveRecord::RecordInvalid: Validation failed: is_published is expected to be a Boolean and is not.

This way, there is not a bunch of random data in the authors table because of a bad decision made a while ago. More examples on how this gem can be used are found at the prequel to this post and the project’s README.

The New Hotness

The 2.0 update brought flexibility to validates_type. This enables greater control over serialized attributes on ActiveRecord objects.

In a system with an intermediary class inserted between the column assignment and the column serialization, this extension can show its true value.

In an application that deals with Books, each book must store information regarding how it was published:

class Book < ActiveRecord::Base
  serialize :publishing_info, PublishingInformation
end

Perhaps each book can have such a variety of publishing information, or the data present can vary widely between each book. Whatever the reason, we can assume that the PublishingInformation class handles the input and extraction of the JSON data pertinent to a specific Book.

With that assumption, an issue could arise if another type of object was assigned to a Book’s publishing_information column.

> book = Book.new
> book.publishing_information = { foo: :bar }
> book.save!
# => UPDATE "books" SET "publishing_information" = $1, "updated_at" = $2 WHERE "books"."id" = $3  [["publishing_information", "{}"], ["updated_at", "2015-11-30 05:04:09.480707"], ["id", 3]]

If, like the above example, the PublishingInformation class is forgiving, this will silently fail. The book.publishing_information will not be set to the correct value.

On the other hand, if the PublishingInformation does fail, it might not do so in an obvious way.

However, with validates_type, this issue becomes explicit.

Adding the type validation to the book model:

class Book < ActiveRecord::Base
  serialize :publishing_info, PublishingInformation
  validates_type :publishing_info, PublishingInformation
end

Then the same save! call gives a different error:

> b = Book.new
> b.publishing_information = { a: :b }
> b.save!
# => ActiveRecord::RecordInvalid: Validation failed: publishing_information is expected to be a PublishingInformation and is not.

Now, a readable error is produced and nothing is left to the imagination.

A Reasonable Start

Was the used example extremely specific? Yes. But, I would wager that at least one application out there has had a problem similar to this and validates_type can ensure that it never happens again. Hopefully, this example and the problem it did end up solving with at least act as inspiration for this library’s other uses out in the wild.

I have found a use for this type of validation and hope others can as well. If a use case arises that this gem does not support but could easily, I invite everyone to contribute to it.

How to Write Future-proof Mocks in RSpec 3

2015-11-22T07:00:00-05:00

Tests are an important component in most software applications. Whether tests drive the development, or are strapped on after the fact, tests need to be reliable for future development to progress. An application’s complexity rises with time. During and after this increase, a rock solid test suite is crucial.

Let the Dogs Out

Given an application that deals with Dogs:

class Dog
end

Keeping track of grooming statistics about this application’s dogs is the obvious cash cow. But that sounds very tedious and complicated, so we can assume that a GroomingService exists that exposes a single Groomer object:

module GroomingService
  class Groomer
    def initialize(dog)
      @dog = dog
    end

    def groom
      # Some complex grooming logic
    end
  end
end

This service was included by the groomer gem and will become a core piece of the application. To keep everyone in the project sane and well rested, a robust spec suite is needed.

Spec It Up

Using RSpec and the GroomingService, a very basic test can be written to make sure the code works as expected.

Introducing the GroomingService into the Dog class could look like:

class Dog
  def groom
    GroomingService::Groomer.new(self).groom
  end
end

The first test should be to make sure that when the groom method on a Dog is called, it initializes and calls groom out to the GroomService::Groomer object.

When asserting that a certain set of methods is called on a particular object, the built in RSpec double method is very handy.

The double method returns a stand-in object to assert allow and expect messages against. Any method that is called on the test double which is not explicitly allowed or expected results in an error.

describe Dog do
  subject { described_class.new }

  describe '#groom' do
    let(:groomer) { double(GroomingService::Groomer) }

    it 'initializes and calls groom' do
      expect(GroomingService::Groomer).to receive(:new) { groomer }
      expect(groomer).to receive(:groom)

      subject.groom
    end
  end
end

As expected, this test passes beautifully:

$: rspec dog_spec.rb

Dog
  #groom
    calls new with self and then groom

Finished in 0.00773 seconds (files took 0.09367 seconds to load)
1 example, 0 failures

That can be it, right? Pack it up and put the dogs away, whoever let them out will have to do it all again later.

Think of the Future

After some time passes, the GroomingService::Groomer has changed its method signature. Instead of the groom method, it has changed to groom!. Chances are someone thought that either the method had too many side affects or that ! are just really awesome looking.

In any case, the included service changes and when the test runs:

$: rspec dog_spec.rb

Dog
  #groom
    calls new with self and then groom

Finished in 0.00735 seconds (files took 0.10532 seconds to load)
1 example, 0 failures

It passes? How can that be? The method changed from groom to groom!. The groom method does not even exist on GroomingService::Groomer. It seems that the specs do not care what does or does not exist on the double, they just happily pass.

Use `instance_double`

The answer here is to use instance_double. Like double, instance_double also returns a stand-in object that raises errors if methods not allowed or expected are called on the object.

The key difference is that instance_double checks the underlying object to make sure that it responds to methods before making assertions that they are called.

describe Dog do
  subject { described_class.new }

  describe '#groom' do
    let(:groomer) { instance_double(GroomingService::Groomer) }

    it 'initializes and calls groom' do
      expect(GroomingService::Groomer).to receive(:new) { groomer }
      expect(groomer).to receive(:groom)

      subject.groom
    end
  end
end

Now running this test, we see the failure we should expect:

$: rspec dog_spec.rb

Dog
  #groom
    calls new with self and then groom (FAILED - 1)

Failures:

  1) Dog#groom calls new with self and then groom
     Failure/Error: expect(groomer).to receive(:groom)
       the GroomingService::Groomer class does not
          implement the instance method: groom

Finished in 0.0054 seconds (files took 0.08288 seconds to load)
1 example, 1 failure

While no one likes failing specs, a failure here is always preferable to one in production.

Since the API that we do not own (Groomer) changed, it was a mistake to blindly mock it without asking the underlying object if that method existed. However, with this problem corrected, the code is more robust and development can continue.

Mocking with Confidence

Asserting that a method is called on an object is a very common pattern in Ruby tests. Additionally, Ruby does not have the benefit of compiled languages in terms of method invocation validation. We as Rubyists rely on automated testing to make sure that our applications work as expected, can be iterated upon, and handle third party library updates. While it may be attractive to use permissive mocking assertions, they can be dangerous and cause more problems than they are worth.

Five Active Record Features You Should Be Using

2015-11-15T07:00:00-05:00

Active Record is responsible for communicating with the persistence layer by default in Ruby on Rails applications. Using Active Record effectively and efficiently can greatly improve an application’s code.

In Ruby on Rails 4.0, some material changes have been made to Active Record. Understanding these changes, and how they are best utilized is important for any Rails developer.

To help explain these concepts, we can assume a Ruby on Rails application “booksandreviews.com” exists with three models:

class Book < ActiveRecord::Base
  belongs_to :author
  has_many :reviews
end

class Author < ActiveRecord::Base
  has_many :books
end

class Review < ActiveRecord::Base
  belongs_to :book
end

The smart people over at “booksandreviews.com” need to know what the state of their data to build metrics and make sales. Metrics need data and Active Record will be used to fetch that data.

1. Nested Queries

When doing database queries, the fewer the better. Since Active Record is responsible of crafting these queries, it is important to make sure it has all the help it needs. For simple queries this is rarely an issue, but more complex requirements can lead to sub-optimal results.

One day, Tim from sales comes rampaging through the office convinced that there must be a bug in the system. A recent sale for “booksandreviews.com” did not go well and he wants answers. Tim wants an analysis run. He wants all reviews published today, which are for books published in 2015.

Without too much thought, this approach seems reasonable:

book_ids = Book.where(publish_year: '2015').map(&:id)
# => SELECT "books".* FROM "books" WHERE (publish_year = '2015')

reviews = Review.where(publish_date: '2015-11-15',
                       book_ids: book_ids).to_a
# => SELECT "reviews".* FROM "reviews" WHERE "reviews"."publish_date" = '2015-11-15' AND "reviews"."book_ids" IN (1, 2, 3)

This will load the desired books, extract their id’s, and pass that result to the Review query. Not only does this generate two queries, it also wastes memory by creating an array of Book objects to map over and then another array of book_ids. With a large enough list of books, this could cause some serious problems.

Active Record’s where method returns an instance of ActiveRecord::Relation. These relations can be passed to other methods to aid in query construction. With the same request, we can save the map and array creation:

books = Book.where(publish_year: '2015')
# => ActiveRecord::Relation

reviews = Review.where(publish_date: '2015-11-15', book: books).to_a
# SELECT "reviews".* FROM "reviews" WHERE "reviews"."publish_date" = '2015-11-15' AND "reviews"."book_id" IN (SELECT "books"."id" FROM "books" WHERE "books"."publish_year" = '2015')

This still executes two SELECT statements but it nests them to let the database take care of memory allocation and optimization. The book_ids array is replaced with the books relation and is passed to the Review query.

Note: This can reduced to a single query with .joins, but for now we can assume that a nested query is desired.

2. DRY Scopes

Still fuming, Tim demands more information. Now he wants to know the list of all Books published in 2015 which have at least one approved Review. Since Reviews are subjective, they need to be approved in order to maintain the quality that “booksandreviews.com” is known for.

Luckily, a scope has been written on the Review class to accomplish this.

class Review < ActiveRecord::Base
  belongs_to :book

  scope :approved, ->{ where(approved: true) }
end

However, it is Books that we need to return, not Reviews. Repeating the scope definition, a join query can be used for this analysis:

books = Book.where(publish_year: '2015')
            .includes(:reviews)
            .references(:reviews)
            .where('reviews.approved = ?', true )
            .to_a
# => SELECT #long books and reviews column select# FROM "books" LEFT OUTER JOIN "reviews" ON "reviews"."book_id" = "books"."id" WHERE "books"."publish_year" = '2015' AND (reviews.approved = 't')

Books are returned at the cost of duplicating the approved scope. That means that the scope in Review changes, this code will not benefit from that change. The .includes and .references methods are used to ensure that we only return one Book (in the case of many Reviews belonging to the same Book).

The Don’t Repeat Yourself (DRY) principle was created for this exact reason. When identical code is not shared and instead repeated, changes to one version can have dangerous consequences on the other.

The good news is that Active Record provides precisely the medicine for this ailment: .merge.

With .merge, an existing scope can be used in another Active Record query.

books = Book.where(publish_year: '2015')
            .includes(:reviews)
            .references(:reviews)
            .merge(Review.approved)
            .to_a
# => SELECT #long books and reviews column select# FROM "books" LEFT OUTER JOIN "reviews" ON "reviews"."book_id" = "books"."id" WHERE "books"."publish_year" = '2015' AND (reviews.approved = 't')

Great! Now the results are the exact same and the code is DRY.

3. `where.not`

Typical insatiable Tim is back with yet another request to add to the brand new “totally not vanity metrics dashboard”. Now, he wants to know all the books not published in 2012.

Without even asking why such a silly request is necessary, some more code can be cranked out:

books = Book.where('publish_year != 2012').to_a
# => SELECT "books".* FROM "books" WHERE (publish_year != '2012')

Like before, this code works but could be better. There is some raw SQL in there that the next developer might not understand well enough to manipulate. Whatever the reason, it is best to rely on abstraction instead of explicit SQL.

To help solve this dilemma, the .not modifier has been introduced in Active Record 4.0.

books = Book.where.not(publish_year: 2012).to_a
# => SELECT "books".* FROM "books" WHERE (publish_year != '2012')

The result is the same but look how much nicer that is. Not only is the raw SQL gone, the code is more positive too.

4. `first` and `take`

Since “booksandreviews.com” has been around since 2012, chances are it upgraded from Ruby on Rails 3.0 to 4.0. One notable change from Active Record 3 to 4 is the behaviour of .first.

In Ruby on Rails 4.0+, the .first method returns the first row after the table has been ordered by its id.

Author.where(first_name: 'Bill').first
# => SELECT  "authors".* FROM "authors" WHERE "authors"."first_name" = "Bill" ORDER BY "authors"."id" ASC LIMIT 1

This will work fine for every table that has an id column. However, if a table does not need an id column, this method causes a problem.

Despite each Author having an id, complex joins might cause an issue with an implicit ORDER BY on queries.

To alleviate that problem, the take method can be used instead of first:

Author.where(first_name: 'Bill').take
# => SELECT  "authors".* FROM "authors" WHERE "authors"."first_name" = "Bill" LIMIT 1

This behaves in a much more explicit way, returning the same information without a default ordering.

5. `.unscoped`

During the development life of “booksandreviews.com”, countless modules have been built and gems included. Amidst this chaos, someone must have typed gem install hairball and horribly altered the Author class. This has led to the new guy Mike’s complaint that: “Authors are missing data”.

Mike knows that authors have a first_name but for some reason it is not being returned:

authors = Author.where(last_name: 'Smith').take(5)
authors.map(&:first_name)
# => [nil, nil, nil, nil, nil]

What Mike does not know is that one of those hairball gems added a default scope to all Active Record objects that begin with the letter “A”. However impossible this bug is, it exists and it is ruining Mike’s day.

What Mike needs is the .unscoped method. This method removes all existing scopes on an Active Record relation.

authors = Author.unscoped.where(last_name: 'Smith').take(5)
authors.map(&:first_name)
# => ['Frank', 'Frank', 'Jim', 'Frank', 'Frank']

(Is anyone else concerned that there are four Frank Smith authors?)

With the .unscoped method, all harmful default scopes are removed and the Franks are free.

Queries on Queries

With these five techniques (and probably a lot more), naiive Active Record queries can stay DRY and intuitive. The exhaustive list of what Active Record can provide can be found at Rails Guides.

Ruby DelegateClass

2015-11-08T07:00:00-05:00

Objects are a big deal in Ruby. A previous post about Ruby Objects can corroborate: Ruby Objects are pretty cool.

There are many ways to work with Ruby Objects. Standard inheritance, module inheritance, decorators, and composition are all commonly found in a codebase. One class in particular seems almost universal across applications: the User class.

From social networks to online retails stores, web applications most always have a user. Additionally, the User class might interact with nearly all other classes. This can make the relationship and responsibilities of that User awkward or unwieldy over time.

As an example, assume that an application “booksandreviews.com” exists. This application serves both authors and critics, each representing a User. These User types have different responsibilities and privileges.

Assuming a Rails application, the basic user model will look most likely like:

class User < ActiveRecord::Base
end

One approach to extend this class and make Author and Critic classes is to use the Single Table Inheritance pattern. STI is a common pattern in Rails applications to share responsibilities between classes and store all records in the same place. This approach can work and might fit current needs just fine, but it might be beneficial to think outside the single table for a second.

After all, no one said a User had to be either an Author or a Critic, what if they were both?

Delegators, Mount Up!

An alternative way to architect a system like the one described is by using delegation. Object delegation is a way of composing objects to achieve flexibility and maintain encapsulation.

Ruby provides a few ways to delegate objects. An interesting option that will nicely serve this application’s needs is DelegateClass. The DelegateClass method accepts a class and returns a new class. The returned class takes the passed in class’s instance methods and defines delegating methods.

Defining Author and Critic classes:

class Author < DelegateClass(User)
end

class Critic < DelegateClass(User)
end

These classes both expect to be initialized with an instance of User, then will delegate methods to the passed in user by default. This allows each class to define their own logic for determining what an Author or a Critic has access to.

Initializing these classes is as simple as loading the user then passing it in:

user = User.find(8)
critic = Critic.new(user)
critic.id
# => 8

Business Logic

Assuming that Book and Review records exist in this same application, related to a user by a user_id, both new classes can handle those resources independently.

class Author < DelegateClass(User)
  def written_books
    @written_books ||= Book.where(user_id: id)
  end

  def has_written_books?
    written_books.present?
  end

  def top_selling_book
    written_books.max_by(&:revenue)
  end
end

class Critic < DelegateClass(User)
  def written_reviews
    @written_reviews ||= Review.where(user_id: id)
  end

  def has_reviewed_book?(book_id)
    written_reviews.where(book_id: book_id).present?
  end

  def top_viewed_review
    written_reviews.max_by(&:views)
  end
end

See how nice it is with the User class responsible only for standard User information? That leaves the Author and Critic classes available to house business logic the User should not have to concern itself with.

The User class can keep caring about things like updated_at timestamps, important booleans like is_a_confirmed_user?, and any other methods that indicate how a general user interacts with the software. Also, it remains the sole responsibility of the user to interact with the persistence layer. The Author and Critic classes will delegate instance methods like save and update to the underlying user.

Adding Yet Another Type

Imagine some time passes and “booksandreviews.com”’s membership is popping off like gangbusters. Then one day, Tim from sales comes down and says: “We are literally making no money, we need to sell these books or something”.

Tim is asking for a new use case for the product, a new type of User will be needed: the Consumer. As before, a Consumer could be an Author, Critic, both, or neither.

Had this application been built with STI, I would wager some spaghetti code would form trying, to shoehorn the same user into three roles.

However, with the DelegateClass pattern we have used, it becomes trivial:

class Consumer < DelegateClass(User)
  def transactions
    @purchased_books ||= Transaction.where(user_id: id)
  end

  def purchaed_books
    transactions.map(&:book)
  end

  def needs_marketing_email?
    # Tim says we need to email people to "encourage"
    #  them into buying things.
    transactions.length == 0
  end
end

Now this new class can be utilized to spam the users of “booksandreviews.com” until Tim is content.

An important note is that instances of Author, Critic, and Consumer are not instances of User.

user = User.find(9)
author = Author.new(user)
author.is_a?(User)
# => false

Keeping it Closed

The DelegateClass approach keeps the User class open to extension but closed for modification. Adhering to this principle becomes very beneficial if the backing of the User model changes.

Maybe in the future, “booksandreviews.com” experiences astronomical growth, resulting in a huge service oriented architecture overhaul. Suddenly, a UserService exists which communicates with an internal API. If that happens, our code is safe.

The Author, Critic, and Consumer classes need not change:

user = UserService.find_user_by_id(10)
consumer = Consumer.new(user)
consumer.id
# => 10

As previously stated, there are many ways to solve this particular problem. Arguments for module inheritance or single table inheritance could be made and might result in just as valid solutions.

Six Months of Commits

2015-11-01T07:00:00-05:00

Six months ago I started a challenge for myself. I wanted to contribute to open source software, a task that I had previously thought out of reach and intimidating. A plan that I had devised to overcome this mindset was to achieve consistency. For a full year, I made a commitment to make at least one commit to an open source repository (someone else’s or my own) every single day. Additionally, I decided to write an article every Sunday about either my projects, my journey, or something interesting I encountered along the way.

To be honest, I did not give my original goal all that much thought. I primarily was making a hypothesis: “If I just keep doing something, chances are that I will improve”.

Looking back, I believe that I have improved. How much I have improved has been hard to measure, but the journey itself has helped me understand the real value I was gaining. However, only after reflecting on my progress could I really understand that there was an at all.

Re-visit

Routines and patterns enable unintentional autopilot. The drive to work, the morning coffee run, even the same weekly meeting are examples of every day life patterns. Can one really distinguish between their commute last Wednesday versus the previous? When our brain does the same thing over and over, it automatically optimizes itself. We forget the details about our days and what distinguishes them from one another. This year of commits project has been no different. Since I have been writing software every single day, I find myself with another daily pattern.

I am lucky that this project exists in a world where history is king. From Github to Google, I have the ability to rewind time and watch my journey step by step. While some days have been lighter than others, the overall picture is clear: This has been one of the best things I have done for my professional development and my state of mind. I might have not noticed this development unless I took my own advice: to stop to take a moment.

Persevere

There have been a number of surprises during the last six months. A few commits were done right at the wire, racing against midnight. Others felt like they took the entire day to complete. Additionally, I have experienced a few very hard times. Days that sapped all motivation for writing code. But so far, I have not missed a single day. I think that if I had missed one day, it would be simple to miss multiple. For now, I am still in a streak which is helpfully demonstrated on Github. Internalizing this perseverance has been crucial.

Repeat actions build confidence. I remember at the onset of this project, the first pull request that I did to a new repository had me literally shaking. I must have read each commit message text twenty times before pressing the “Open Pull Request” button. Now, with many pull requests successfully merged, the previous anxiety has vanished. Confidence in the correct quantities can be extremely powerful. This confidence has helped me develop my technical skills and has been instrumental for receiving feedback and criticism.

Thanks

A person who I hold in high regard recently said they found my effort inspiring. I never intended for this endeavor to yield glory or admiration; but, few things are more motivating than hearing kind words of encouragement. To know that I have not been shouting in a dark room and that my effort has meant something feels amazing.

To all the people who have been supportive and hopefully will continue to do so, you are the reason that I am gaining so much out of this experience. You are the reason I will continue this path forward. You are the reason I will complete a Year of Commits.

How to Traverse Foreign Ruby Code

2015-10-25T08:00:00-04:00

A Ruby on Rails project will most likely contain large amounts of third party software. Software written by other people can fluctuate greatly in terms of documentation. Even very well documented software might have pieces that are more shrouded than others.

When these opaque pieces of code start to cause issue or incite curiosity, spelunking through these libraries is easier with simple patterns and the right tools.

0. Get the Right Tools

An important piece of technology when reading code is a text editor. The right editor will make searching for methods fast and opening files painless.

Another vital component when traversing Ruby code is a runtime debugger. For this, the pry gem is a personal favorite. Using pry is as simple as requiring it, then adding binding.pry at the desired stopping point. Alternatively, some people like to debug with output statements in code execution; but, this is 2015 and I like things that I can interact with.

1. Use `source` and `source_location`

This example will use ActiveRecord’s store_accessor method as the subject of investigation.

The store_accessor method is useful for storing data with a volatile structure. If an application wants to prototype a feature or is unsure about how useful a certain data structure might be, using store_accessor is reasonable. In this case we can assume that the store_accessor column is the json type.

Given a User class, with a column named settings, we can define two store_accessors: is_registered and contact_method.

class User < ActiveRecord::Base
  store_accessor :settings, :is_registered, :contact_method
end

This model responds to contact_method= and serializes the result into the json column.

If we wanted to see how this was defined, we need to start a console and look at where the contact_method= method exists.

To see the definition of a method, method and source are helpful.

user = User.new
puts user.method(:contact_method=).source
#=> define_method("#{key}=") do |value|
#     write_store_attribute(store_attribute, key, value)
#   end

It seems that the contact_method= method is meta-programmed. This does not say much but gives a good starting point. Now, to find which file this method is defined in, source_location is used.

user.method(:contact_method=).source_location
#=> ["$RVM_PATH/.rvm/gems/ruby-2.2.1@global/gems/activerecord-4.2.4/lib/active_record/store.rb", 85]

As assumed, the method which defines the store_accessor methods is inside of ActiveRecord, specifically on line 85 of store.rb.

2. Place a Reasonable `binding.pry`

With the location of the method found, binding.pry has a logical place to go. Since gems are not magic, the ActiveRecord gem can be opened and its source easily read. Opening gems is fairly simple, set a desired EDITOR and open via the bundle command. For this example, subl is mapped to Sublime Text.

EDITOR=subl bundle open activerecord

Then, navigating to line 85 of store.rb, a binding.pry can be added to stop code execution when the contact_method= method is used.

# In activerecord-4.2.4/lib/active_record/store.rb

define_method("#{key}=") do |value|
  require 'pry'
  if key == :contact_method
    binding.pry
  end
  write_store_attribute(store_attribute, key, value)
end

Note: adding require 'pry' might not be necessary, depending on the bundle this code is running in.

With the binding.pry in place, launching a new rails console will use the modified code. When a user’s contact_method= method is called, pry will take over.

user = User.new
user.contact_method = { phone: true }

    85: define_method("#{key}=") do |value|
    86:   require 'pry'
    87:   if key == :contact_method
 => 88:     binding.pry
    89:   end
    90:   write_store_attribute(store_attribute, key, value)
    91: end

We have liftoff! The code is halted in the meta programmed definition of this setter. The local variables here can be accessed to see what is actually going on:

[1] pry> store_attribute
# => :settings
[2] pry> value
# => {:phone=>true}
[3] pry> key
# => :contact_method

This tells us some, but it seems that the real logic is in the write_store_attribute method. While the code execution is stopped, we are able to use the same method().source and method().source_location calls from before to gain even more insight.

[4] pry> puts method(:write_store_attribute).source
# => def write_store_attribute(store_attribute, key, value)
# =>   accessor = store_accessor_for(store_attribute)
# =>   accessor.write(self, store_attribute, key, value)
# => end

[5] pry> method(:write_store_attribute).source_location
#=> ["$RVM_PATH/.rvm/gems/ruby-2.2.1@global/gems/activerecord-4.2.4/lib/active_record/store.rb",
 129]

3. Make an Exit

Pry comes with a number of helpful commands for code navigation. An important command is exit, which will stop the at a specified break point and let the code continue to either the next break point or until completion.

Multiple binding.pry lines may be added to different files in order to jump from break point to break point. If we wanted to dig deeper, to see what the accessor variable is in write_store_attribute, we could place a second binding.pry.

# In activerecord-4.2.4/lib/active_record/store.rb

def write_store_attribute(store_attribute, key, value)
  accessor = store_accessor_for(store_attribute)
  binding.pry
  accessor.write(self, store_attribute, key, value)
end

Now, exit and re-running rails console shows both break points in action.

> user = User.new
> user.contact_method = { phone: true }

    85: define_method("#{key}=") do |value|
    86:   require 'pry'
    87:   if key == :contact_method
 => 88:     binding.pry
    89:   end
    90:   write_store_attribute(store_attribute, key, value)
    91: end

[1] pry> exit

    129: def write_store_attribute(store_attribute, key, value)
    130:   accessor = store_accessor_for(store_attribute)
 => 131:   binding.pry
    132:   accessor.write(self, store_attribute, key, value)
    133: end

pry> accessor
#=> ActiveRecord::Store::StringKeyedHashAccessor
pry> puts accessor.method(:write).source
# => def self.write(object, attribute, key, value)
# =>   super object, attribute, key.to_s, value
# => end

More break points, more knowledge. Each new binding.pry gives a new context which subsequently opens more avenues of exploration. This is obviously not the end of the store_accessor logic, but a great first step has been made.

Note: The @ command can be used to show the current bound context. This is especially helpful if many different debug or output statements have been used in one bind point.

4. Rinse and Repeat

Following the same pattern, any depth of code can be reached by placing a binding.pry, observing results and repeating. Traveling through code that is foreign will also help build confidence. When developers stop assuming that things are black boxes of magic, everyone benefits.

Using these simple techniques, code previously hidden or otherwise out of reach becomes accessible and easy to traverse. Finally, I would advise to keep trips down the code rabbit hole short, or you might shave one too many yaks.

Building a Simple Web Server with Ruby 2.0+ (Part 2)

2015-10-18T08:00:00-04:00

In a previous post, a very simple Ruby server was created to listen to HTTP requests. While great for a first step, this example server does nothing more than respond with “Hello World”. Greetings are nice and polite, but I think we can do better.

Pro-filing

A reasonable feature for this simple server is the ability to serve files. When retrieving files, the server must remain secure, only serving files that should be readable by clients. Additionally, if a requested file does not exist, the server should make the client aware.

Since a request can have multiple parts, the server will need to parse out the noise from the desired file. For instance, if the request looks like /path/to/my_file.html?query=params&are=cool, the server should remove all query parameters and search for my_file.html nested within the /path/to/ directory.

With an incoming request:

GET /path/to/my_file.html?query=params&are=cool HTTP/1.1

A simple file fetching method might look like:

require 'uri'

SEVER_ROOT_DIR = '/var/www'

def fetch_file(request_string)
  request_parts = request_string.split(' ')

  # Remove query params and HTTP verb, version
  path = URI(request_parts[1]).path

  full_path = File.join(SERVER_ROOT_DIR, path)

  File.open(full_path) if File.file?(full_path)
end

Given a request string input (from request.gets in the existing code), this method returns an instance of File if it can find the requested file or nil if it does not exist. The SERVER_ROOT_DIR is used to ensure the file lookup is centralized to where the server expects the files to be.

Putting it all together, the server can now fetch and return files that exist.

require 'socket'
require 'uri'

SERVER_ROOT_DIR = '/var/www'

def fetch_file(request_string)
  request_parts = request_string.split(' ')
  path = URI(request_parts[1]).path
  full_path = File.join(SERVER_ROOT_DIR, path)
  File.open(full_path) if File.file?(full_path)
end

server = TCPServer.new(8080)

loop do
  Thread.new(socket.accept) do |request|
    request_string = request.gets
    file_to_return = fetch_file(request_string)

    if file_to_return.nil?
      header = "HTTP/1.1 404 Not Found\r\n"
      response = 'File not found'
    else
      header = "HTTP/1.1 200 OK\r\n"
      response = file_to_return.read
    end

    header += "Content-Type: text/plain\r\n"
    header += "Content-Length: #{ response.bytesize }\r\n"
    header += "Connection: close\r\n"

    request.puts header
    request.puts "\r\n"
    request.puts response

    request.close
  end
end

Now, a curl request for an invalid file produces a 404:

curl localhost:8080/bad_file.html -I

HTTP/1.1 404 Not Found
Content-Type: text/plain
Content-Length: 14
Connection: close

More Logic, More Problems

A few problems are immediately evident with this code. One is that any file that exists on the system can be requested. For instance, the file /etc/passwd is a common target for immature web servers to accidentally expose.

To combat this, the fetch_file method can throw out all directory movement inducing path parts:

def fetch_file(request_string)
  request_parts = request_string.split(' ')
  path = begin
    insecure_path = URI(request_parts[1]).path

    secure_request_parts = insecure_path.split('/').reject do |part|
      part == '..' || part == '.' || part == ''
    end

    secure_request_parts.join('/')
  end

  full_path = File.join(SERVER_ROOT_DIR, path)

  File.open(full_path) if File.file?(full_path)
end

This will change a path from /../../../my/hidden/file to /my/hidden/file, nullifying the attempt to expose private files.

Speaking the Same Language

A second issue with our new and improved tiny web server is in the response type. Currently, all responses indicate that the type of the file returned is text/plain. Instead, the response type can be extracted from the file returned. To determine a file’s type, a good place to start is by examining the extension.

For the request /path/to/my_file.html?query=params&are=cool, the server must be able to identify that the .html extension maps to the content type text/html.

With the addition of a simple mapping method, the server can respond more intelligently:

def content_type(file_extension)
  {
    html: 'text/html',
    txt: 'text/plain',
    json: 'application/json'
  }[file_extension] || 'text/plain'
end

This method accepts an extension (like .html) and maps it to a Content-Type. If no mapping can be found, it assumes that the file is in plain text.

Using some additional File.extname extraction, using this content_type method is simple:

require 'socket'
require 'uri'

SERVER_ROOT_DIR = '/var/www'

def fetch_file(request_string)
  request_parts = request_string.split(' ')
  path = begin
    insecure_path = URI(request_parts[1]).path

    secure_request_parts = insecure_path.split('/').reject do |part|
      part == '..' || part == '.' || part == ''
    end

    secure_request_parts.join('/')
  end

  full_path = File.join(SERVER_ROOT_DIR, path)

  File.open(full_path) if File.file?(full_path)
end

def content_type(file_extension)
  {
    html: 'text/html',
    txt: 'text/plain',
    json: 'application/json'
  }[file_extension] || 'text/plain'
end

socket = TCPServer.new(8080)

loop do
  Thread.new(socket.accept) do |request|
    request_string = request.gets

    file_to_return = fetch_file(request_string)

    if file_to_return.nil?
      header = "HTTP/1.1 404 Not Found\r\n"
      response = 'File not found'
      extension = nil
    else
      header = "HTTP/1.1 200 OK\r\n"
      response = file_to_return.read
      extension = File.extname(file_to_return.to_path)
                      .split('.').last.to_sym
    end

    header += "Content-Type: #{ content_type(extension) }\r\n"
    header += "Content-Length: #{ response.bytesize }\r\n"
    header += "Connection: close\r\n"

    request.puts header
    request.puts "\r\n"
    request.puts response

    request.close
  end
end

The important line is:

extension = File.extname(file_to_return.to_path)
                .split('.').last.to_sym

This code results in a symbol to pass to content_type that returns a meaningful content type for the client.

The results of a request using this new content type parsing are just as expected:

curl localhost:8080/path/to/my_file.html -I

HTTP/1.1 200 OK
Content-Type: text/html
Content-Length: 79
Connection: close

There it is! The text/html content type returned correctly by the server when an html file was requested.

Good Progress but Far from Perfect

This new iteration has added some good depth to the server; however, a plethora of issues remain. This server still has no concept of thread pooling for memory management, no authentication for restricted access files, and can only fetch basic files.

While this might be sufficient for a pet project or educational purposes, I must reiterate that using a more mature and maintained web server is preferable.

It sure has been fun building it though, right?

Building a Simple Web Server with Ruby 2.0+

2015-10-11T08:00:00-04:00

The Ruby language can be utilized for a variety of different purposes. The most popular of these is as a web scripting language. The entry point to these applications, the web server, is not something that is usually built from the ground up. Most applications have the lines gem 'puma' or gem 'unicorn' in their Gemfile. Those are high quality web servers that were built and are maintained by very talented people, but how do they work?

The aforementioned web servers are just Ruby gems written by Rubyists. While they are awesome gems, they are not magic. A basic web server can be written to satisfy some very simple requests.

`GET` Basic

The job of a web server is to take an incoming request, decide what it means and respond accordingly. Essentially, we want to read some text, process it, and send some other text as response. To create a very simple server, the built in Socket library will be utilized.

The goal of this tiny server will be to listen to localhost requests on port 8080.

require 'socket'

server = TCPServer.new(8080)

loop do
  request = server.accept
  request.puts 'hello world'
  request.close
end

And there it is! A ridiculously tiny web server written in Ruby. Dissecting this code, we see how simple the Socket library makes this task.

server = TCPServer.new(8080)

This line makes a new TCPServer instance that will accept requests to port 8080 and defaults to localhost as the host. The important method on the TCPServer instance is accept.

loop do

Since this server is meant to respond to any request on port 8080, an infinite loop is created so that more than one request can be fulfilled before the process exits.

request = server.accept
request.puts 'hello world'
request.close

Here is the real logic of the server. The accept method returns a new instance of TCPSocket which can be used to communicate with a client. A response string, 'hello world', is then printed to the client and the connection is closed.

Saving this code to example_server.rb, the server can be started with ruby example_server.rb in a console.

Using curl to test this server results in:

$: curl localhost:8080
hello world
curl: (56) Recv failure: Connection reset by peer

Alright, alright, the 'hello world' has been printed to the console but it looks like there might be an issue. Connection reset by peer, what the heck does that mean? It means that the server did not send enough information to the client, particularly about when to close the connection. The client is expecting more data than this very naive server is providing. The client expects header data.

Look Ma, no Head(er)!

An HTTP header consists of many colon-separated key-value pairs deliminated by a return and newline combination. Everything from a status code to response size can be returned in an HTTP header.

For this simple example, a status code, Content-Type, Content-Length, and Connection will be added to the response header.

require 'socket'

server = TCPServer.new(8080)

loop do
  request = server.accept

  response = 'hello world'

  header = "HTTP/1.1 200 OK\r\n"
  header += "Content-Type: text/plain\r\n"
  header += "Content-Length: #{ response.bytesize }\r\n"
  header += "Connection: close\r\n"

  request.puts header
  request.puts "\r\n"
  request.puts response

  request.close
end

With this additional information, a client using this server will know the type, size, and connection status of its request. This will enable the client to respect the content type, parse the result intelligently and close the connection.

An interesting detail about HTTP headers is the way they end. A single line consisting of a return newline, request.puts "\r\n", is how the client knows that the HTTP header is finished.

Note: If a server like this is created for any real requests, hard-coded response headers is not the best idea.

Head’s up

Now, the same curl localhost:8080 test to the new and improved server returns no error:

$: curl localhost:8080
hello world

The new headers are visible via curl localhost:8080 -I:

$: curl localhost:8080 -I
HTTP/1.1 200 OK
Content-Type: text/plain
Content-Length: 11
Connection: close

The server is now responding in a sensible way, but a problem still exists. If more than one request is made, the first request will be processed and all others will wait.

This implementation is not adequate for concurrent request handling, but another built in solution is readily available.

Threads the Needle

An easy way to achieve basic concurrency with this server is via Threads. Out of the box Ruby uses the Matz’s Ruby Interpreter or MRI. MRI is not capable of true concurrency (due to the Global Interpreter Lock) but built in threads will be just fine for this small server.

This example can be extended to use threads by wrapping the responsibility of the request in a block:

require 'socket'

server = TCPServer.new(8080)

loop do
  Thread.new(server.accept) do |request|
    response = 'hello world'

    header = "HTTP/1.1 200 OK\r\n"
    header += "Content-Type: text/plain\r\n"
    header += "Content-Length: #{ response.bytesize }\r\n"
    header += "Connection: close\r\n"

    request.puts header
    request.puts "\r\n"
    request.puts response

    request.close
  end
end

Voila! A simple threaded Ruby server!

Note: As with the previous solution, this might not be fit for a production environment since every single request spawns a new thread within a single process.

What now?

Now, this simple server will fulfill concurrent requests in a way that will make sense to its consumers. However, it always responds with 'hello world', regardless of the request data. To fetch and parse the incoming data, request.gets is available:

require 'socket'

server = TCPServer.new(8080)

loop do
  Thread.new(server.accept) do |request|
    response = 'hello world'

    request_input = request.gets
    # handle request information

    header = "HTTP/1.1 200 OK\r\n"
    header += "Content-Type: text/plain\r\n"
    header += "Content-Length: #{ response.bytesize }\r\n"
    header += "Connection: close\r\n"

    request.puts header
    request.puts "\r\n"
    request.puts response

    request.close
  end
end

Other improvements would be header configuration, incoming request parsing, and thread pooling.

Spending a thousand hours iterating on this code will certainly improve its quality, but it still might not allow this simple server stack up to the leaders in the space. I would recommend using a more battle tested web server for an actual production application. At least some of those awesome servers’ mysteries have been explained.

To see even more improvements, check out part 2!

Someone Wrote that Code on Purpose

2015-10-04T08:00:00-04:00

One of the great misconceptions of software engineering is the belief that “any code written in the past is probably wrong”. For some reason, an accepted mindset (usually among junior developers) is that older code was written with haste or by someone who did not fully understand the problem at hand, invalidating its existence.

A multitude of rationalizations may arise when the thought of refactoring software from the ground up is an option. After all, the person who wrote that code may not even be around anymore. They could have just been learning the language, surely that is a valid enough reason to completely decimate their work. What possible justification could there be for the opposite? The code just feels wrong, and I, the new domain expert, will rectify this situation!

The Devilish Details

A person who stumbles on older code can not possibly have the same level of understanding as its original author. Do not misunderstand, it is completely possible that the original author had an insufficient amount of knowledge when writing the code; however, if details have been lost between code inception and re-discovery, removing that code could have dire consequences. How dire could they be? Well according to a recent Bloomberg article, it could be millions of dollars.

According to the article:

The Core Engineering developer did not understand that the changes he made to the code sequences would affect the directed ISOs generated by Latour’s trading algorithms.

The key words to take away here is that the developer literally did not understand the code. The developer that made this mistake needed to simply understand the code that he or she was editing. However, due to time constraints, inexperience, or just plain arrogance, that crucial understanding was never reached. The small yet significant details of how the code worked within itself and with external services were never discovered, leading to a catastrophe.

Assume Smart People Wrote It

A simple rule that may help avoid multi-million dollar code rewrites is to make the assumption that the person who wrote that older piece of software is smart. With some exception, software currently running a business or application was written with care and planning. Time may have passed and the form or shape of the code might not be modern, but in those misplaced curves and edges might lurk the solution to some rarely seen and deadly problems.

In very lucky situations, the older software might be written by a current employee. This makes that person the obvious best candidate to ask about why the code might “look weird” or “does this obviously wonky thing”. Asking this person about the code or tagging them in the inevitable pull request might just save a few production meltdowns.

Take a Break and Re-visit

If no resources are present and a full code obliteration seems like the right approach, it would be wise to walk away and think for a short amount of time. Thinking about a rewrite in a relaxed environment could help make sure no bit is left unturned. Remember, the code in question was most likely reviewed, iterated on, and deliberately turned into its current state.

If solo breaks do not help get the cognitive juices flowing, try explaining the software design and purpose to an ever attentive Rubber Duck. Quackers might not offer the best repartee, but speaking about the code aloud might shine some light onto the darker bytes.

Learn and Burn

Finally, if the code has been understood and its existence is not longer required, feel free to unleash napalm and build elegant skyscrapers on top of the ruins. Because sometimes, and hopefully only sometimes, that old software hiding under a mountain of neglect really is in need of a phoenix-style rebirth. It is possible that it was written by that intern who only got the job because they were the son or daughter of the CEO’s neighbor. Maybe it was code written by an independent contractor who thought it would be funny to name their variables after famous superheros. Whatever the case may be, stopping to make very deliberate changes will pay dividends.

Taking time to understand old or opaque code can mean the difference between a healthy face lift to an application and accidentally removing its liver. How are you supposed to have a celebratory drink after a grand refactoring without a liver?

How to Create a Custom Enumerable

2015-09-27T08:00:00-04:00

Ruby is a wonderfully flexible language. An example of this flexibility is in the ability to define a custom collection class that acts as an Enumerable object. In Ruby, a collection that acts as an Enumerable is basically a class which holds a list of objects and exposes helpful methods for iteration and collection. An example of this pattern built into Ruby is the Array class.

In keeping with a theme, let us assume that an application about dogs exists. In this example, a DogKennel class exists that will hold information about each dog in the kennel and detailed information about said kennel. We can also assume that this class is meant to be used as a collection of dogs, exposing helper methods for information about the kennel. Why would we use this collection class over a typical Array? One reason might be that the consumer of this class expects a list of dogs and some additional metadata. That metadata can be easily exposed in this class without having to wrap the class or extract the metadata from an included Hash.

class DogKennel

  attr_reader :dogs, :location, :operating_hours

  def initialize(dogs, location, operating_hours)
    @dogs = dogs
    @location = location
    @operating_hours = operating_hours
  end
end

1. Add `include Enumerable`

To make our DogKennel into something that can iterate over its dogs easily, we can add the Enumerable module. The desired functionality will have iterative functions as instance methods, so include will be used:

class DogKennel
  include Enumerable

  attr_reader :dogs, :location, :operating_hours

  def initialize(dogs, location, operating_hours)
    @dogs = dogs
    @location = location
    @operating_hours = operating_hours
  end
end

This include will allow the DogKennel class to inherit a large number of useful methods; however, none of these methods are able to be used until a basic iterative each method is defined. If a method such as map is called in this state, the DogKennel class will throw a NoMethodError until the each method is defined.

2. Define an `each` Method

The each method is the building block for all other iterative methods the Enumerable module includes. For this example, we can assume that the consumer of DogKennel will want to iterate over the dogs in the kennel. The each method must accept a block and either yield or pass the block along to another method:

class DogKennel
  include Enumerable

  # ...

  def each(&block)
    @dogs.each(&block)
  end
end

Hooray! Now, the DogKennel class can respond to each, map, select and all the other documented Enumerable iterative methods.

Some examples of how this class can now be used:

dog_kennel = DogKennel.new(['Fido', 'Spot', 'Bandit'], 'Tallahassee', '09:00 - 18:00')

dog_kennel.each { |dog| puts "#{dog} is here!"}
# => Fido is here!
# => Spot is here!
# => Bandit is here!

dog_kennel.select { |dog| dog.length == 4 }
#=> ['Fido', 'Spot']

As always, this is not the only way to achieve this solution in Ruby. Ruby-ists are blessed with a plethora of tools to do any task. I am of the opinion that for this specific task, making a custom collection class is clean and concise. Delegation might be an attractive option but could prove to be overly verbose for this use case.

3. Consider Comparisons

In this new collection class, the sort method will default using the <=> (sometimes called spaceship) operator. This combined comparison operator returns 0 if first operand equals the second, 1 if first operand is greater than the second and -1 if first operand is less than the second.

This will work fine for strings and other basic object types; however, if a more complex object held the information about each dog, the <=> method must be defined on that object.

A Dog class is created to represent a dog’s attributes:

class Dog
  attr_reader :breed, :name, :color

  def initialize(breed, name, color)
    @breed = breed
    @name = name
    @color = color
  end
end

Then, instances of Dog are then given to DogKennel on initialization and sorted:

fido = Dog.new('Daschund', 'Fido', 'brown')

bandit = Dog.new('Labrador', 'Bandit', 'black')

dog_kennel = DogKennel.new([fido, bandit], 'Tallahassee', '09:00 - 18:00')

dog_kennel.sort
#=> ArgumentError: comparison of Dog with Dog failed

This error is to be expected since the Dog class has no way to compare one instance of itself against another. To maintain feature parity with the first example, we can define the <=> method that introspects on each Dog’s name:

class Dog
  # ...

  def <=>(other_dog)
    self.name <=> other_dog.name
  end
end

Now, we can sort those puppies!

dog_kennel.sort
#=> [#<Dog:0x007fcb4a01c800 @name="Bandit", @breed="Labrador", @color="black">,
     #<Dog:0x007fcb4914b880 @name="Fido", @breed="Daschund", @color="brown">]

It’s Raining Iterators and Dogs

Having the Enumerable module included and the <=> comparison operator defined, the DogKennel class can now be used in the same convenient way as other collection classes.

The strength behind this approach is expressed through its maintainability. Since no long list of methods was defined, and no duplicative code was written, this code can exist alongside active development of the Enumerable module. For the example in question, this could be an overkill solution. But, in a more complex and sophistiacted system, using the Enumerable could be the best option.

Just learn Rails (Part 3) HTTP status codes

2015-09-20T08:00:00-04:00

So you want to be a Rails superstar? To live large, big servers, requesting tars. Writing code all over the world, gotta make commits constantly. Great, I feel the same way.

Assuming you have already had that “I should just learn Rails” moment, the idea of writing an API for a super awesome application might be on the horizon. And, of course, the framework that makes the most sense is Ruby on Rails. Given that is the direction things might naturally evolve, creating an API with Rails can gloss over a few crucial design concepts and considerations.

HTTP status codes are important

When the Hypertext Transfer Protocol (or HTTP) was conceived, the idea of a server responding with multiple distinct pieces of data came into existence. Officially, HTTP is a stateless application-level protocol for distributed, collaborative, hypertext information system. In other words, it is a standard way that one server can talk to another in a predefined fashion. This is especially useful when writing a system that will facilitate requests from someone other than the person which built it; e.g. an HTTP API.

The aspect of HTTP that is most relevant in this post are status codes. These codes explicitly tell consumers of an application how they should react to a request. From authorization issues to an incorrect arity of parameters, a large amount of information is exposed via HTTP status codes.

Rails status code support

Let us assume that an application exists to look up information about books. To get a specific book, a RESTful route is provided by the application at https://mycoolbookapp.com/books/12. This route might execute the following controller action:

class BooksController < ApplicationController
  def show
    render json: Book.find(params[:id])
  end
end

This simple controller action can respond with three HTTP status codes. A 200 will be in the response of any Book which matches the :id supplied. Likewise, a 404 will automatically be returned if the Book requested does not exist. Finally, if some “code in need of improvement” is accessed and fails, a 500 is returned to let the user know that the server has almost certainly caught fire (surely no code pushed to production has bugs in it).

For a very simple API, this response structure is adequate. However, achieving more complicated responses might not be so straight forward. For instance, if a user must be logged in to access a Book, the correct response might be 401, to explicitly let the user know that they are not authorized to see that resource.

If HTTP status codes are unknown or underutilized, an inexperienced developer might write that response that attempts to only use the text body of a response.

class BooksController < ApplicationController
  before_filter :authenticate_user

  def show
    if @user.nil?
      response = { error: 'User is not logged in' }
    else
      response = Book.find(params[:id])
    end

    render json: response
  end
end

This code will respond with a 200 and a message denoting an error. This is not the correct way to return a result. This is the equivalent of someone allowing you to leave a store with an item and not pay, then arresting you for it. It just does not make sense and should be avoided.

Instead, Ruby on Rails gives us very helpful symbols that can be used to accurately convey the appropriate response to end users.

To only return the correct HTTP status code, the books controller can be rewritten to:

class BooksController < ApplicationController
  before_filter :authenticate_user

  def show
    if @user.nil?
      return head :unauthorized
    end

    render json: Book.find(params[:id])
  end
end

Additionally, you can provide even more information to the consumer of your API with a combination of HTTP status codes and a response body:

class BooksController < ApplicationController
  before_filter :authenticate_user

  def show
    if @user.nil?
      response = { error: 'User is not logged in' }
      status = :unauthorized
    else
      response = Book.find(params[:id])
      status = :ok
    end

    render json: response, status: status
  end
end

Communicate effectively

This example has exposed the difference between a decent API and one that understands how HTTP should work. While the specific problem could be solved with CanCanCan, it is important to understand how and why those libraries work the way that they do.

If one were to continue down the naive path, returning hashes or strings with incorrect HTTP status codes, things would become messy quickly. That response structure unjustly handcuffs the API clients to be unnecessarily tolerant of ad-hoc text responses. But, if the API conforms to HTTP standards, a client knows exactly what each response means. A 400 status code in the response will give context to what a supplied error means. The client can respond correctly and know that their request resulted in an error. However, if a response body is the only way to denote that something went wrong, what stops a new developer on the API from changing the error key to Error and ruin everyone’s day?

Instead of capturing in text every single detail about why a request did not result in an expected response, use HTTP status codes to supplement and communicate effectively.

Rescue Groups Ruby Wrapper

2015-09-13T08:00:00-04:00

People need companions. We are a social species that enjoys surrounding itself with family and friends. But sometimes, and maybe more often than just sometimes, people crave companionship from something that cannot talk back. This is probably one of the reasons so many people like pets. A pet cannot disagree with you or say judgmental remarks as you fill up that wine glass for a fourth time. No, a good pet will simply exist near you, occasionally asking for pets and food. Pets can be obtained from breeders for huge stacks of cash, or they can be adopted for significantly cheaper.

The Adoption Option

About 2.7 million animals are euthanized each year in animal shelters across the United States. That is some sad stuff right there. These animals could be hanging out with you at your house, clawing up the couch or barking at nothing. Does that not sound like just the best time? If only there was a Ruby library that some super awesome Software Artisans could use to make discoverable pet data a reality. Well now, there is! Over the past few months, as part of the Year of Commits initiative, I have built a wrapper for the RescueGroups.org API.

The RescueGroups API is free to use, but requires signing up for an API key. To do that, fill out this form and they will email you an API key for you to use.

1. Install the `rescue_groups` Gem

To install the gem, simply add rescue_groups to a ruby project’s Gemfile:

Gemfile

source 'rubygems'

gem 'rescue_groups'

Then in an initializer or at the beginning of the execution flow for using the library, add your new fancy API key to the configuration.

RescueGroups.configuration do |config|
  config.apikey = 'your api key'
end

2. Use the Gem!

After the gem is installed, some handy dandy search functionality is right at your fingertips. Call me biased but I rather like the ActiveRecord find and where API. In fact, I like it so much that the rescue_groups gem implemented its own version of both.

Searching

If an ID is known for an animal, simply pass it to find and it will return an Animal object. If an Array is given to find, it will return an Array of Animals.

Animal.find(1)
#=> <Animal id: 1, name: 'Fluffy' ...>

Animal.find([20, 30, 40])
#=> [<Animal id: 20, name: 'Mittens' ...>, <Animal id: 30, name: 'Mr. Doom' ...>, <Animal id: 40, name: 'CatDog' ...>]

Where is very similar, any of the supported fields for Animals, Organizations, and Events can be used to find the matching records.

Organization.where(name: 'Pets-R-Us')
# => [<Organization id: 1, name: 'Pets-R-Us' ...>]

Animal.where(color: ['black', 'brown'])
# => [<Animal id: 1, color: 'black' ..>, <Animal id: 5, color: 'brown' ..>]

Animal.where(location_postal_code: 94117)

The default behavior asserts that the fields are equal, i.e. An animal that is brown will be returned when Animal.where(color: 'brown') is used. Other comparisons are possible for more complex queries.

Animal.where(general_age: { less_than: 5})
# => [<Animal id: 1, age: 2 ..>, <Animal id: 3, age: 1 ..>]

Organization.where(name: { contains: 'shelter'}, location: 90210)
# => [<Organization id: 1, name: 'Big Animal Shelter', location: 90210 ...>, <Organization id: 2, name: 'Small Animal Shelter', location: 90210 ...>,]

Relationships

While I was in the spirit of borrowing, I did so with Ruby on Rails style relationships. In the rescue_groups gem Animals and Events belongs_to Organizations and, inversely, Organizations has_many Animals and Events. When one of these models’ relationship is called, the respective relationship is automagically fetched from the RescueGroups.org API.

organization = Organization.find(1)
# => <Organization id: 1, name: 'Pets', city: 'Dallas' ...>

# This will issue a remote call that is equivalent to Animal.where(organization_id: 1)
organization.animals
# => [<Animal id: 1, organization_id: 1 ...>, <Animal id: 2, organization_id: 1 ...>, ...]

3. Make a Pet Project

Now with all this pet data available, put it to work. I have really enjoyed making this library but am always looking for feedback. If anyone would like to know more or help contribute to this project please feel free! Github PRs and issues are always welcome.

Stop and Take a Moment

2015-09-06T08:00:00-04:00

Someone in all our lives has offered up the advice: “Slow down”. It might have happened once or, if we are lucky, many people have said it. Regardless, the words did not sink in. At least not at first. And even to this day, maybe the words are yet to truly be absorbed and understood. But those words do not mean anything, it is just something people say, right? Wrong. These little words carry more weight with them than most conversations in a day.

“I am just too busy”, “There is no time to waste, how could I slow down?”, we say in response. Because in the moment, these phrases make sense. Things that are right in front of our face are the most important. We may plan slightly in the future or be cognizant of a few days in the past, but we stay busy. After all, being busy has become our natural and most comfortable state.

Busyness is mindlessness

In an earlier post, I said to keep moving, to keep working. The message of this article is not in contradiction to that one, but should be thought of as complementary. Staying still will cause fatigue, muscle atrophy, and many other problems; but, not enjoying the moments we are in can be just as detrimental. Within this world full of stimuli, it is extremely easy to preoccupy ourselves with just about anything. Think about yesterday or the day before, how many tasks or activities flowed into each other? How long did the entire day feel? For me, days do not feel nearly as long as they used to. Moments flow from one to another without even being noticed. It almost feels like a trance, a mindless sea of moments washing over each other.

People like to stay busy, to feel valuable and like they have a place in the world. However, this mentality can lead a person to miss out on those special moments that deserve a bit more attention.

One moment can mean a lifetime

“You don’t know what you’ve got until its gone” is a phrase said a great deal, but not often heard. This concept has recently rang very true for me. Memories that seemed so small and insignificant suddenly have a great deal of meaning. It might have been a 5 minute conversation or a multiple day excursion. Either way, the memories that we have and the moments that we use to define ourselves are happening to us every single day. It is important that they are noticed and appreciated.

In times of sadness, reflection is effortless. It is unfortunate that these feelings are most often surfaced due to despondency, but it can serve as a valuable lesson. A lesson to stop and take a moment to appreciate where we are and who is among us.

Gone but never forgotten

I have lost a friend. Truly, I have a lost a brother. While the same blood did not run through our veins, our experiences and connection made us family. I am lucky to have such fond memories and character defining interactions, but I wish that I had taken more time. More time to understand, to appreciate, and to be in the moment when times were at their best. I believe that we are who we are not only by our own volition, but also the influence and teachings of those around us. Some of those teachers we recognize right away, others are taken for granted.

I wish that I had said how much I valued our time together, how much I have learned and what you really meant to me. I would not be who I am today without you. I regret never telling you in person, but feel blessed that I have as many fond memories as I do.

Dedicated to Christopher Adam Bell

Active Support and Time

2015-08-30T08:00:00-04:00

If a list was compiled of useful Ruby libraries, Active Support would be close to the top. Based on the documentation alone, it is apparent that a very large amount of time went into thinking about the roles Active Support should play and the functionality it should support. From string manipulation to internationalization, Active Support seems to simply do it all. However, even a profoundly useful library like Active Support still has a few inconsistencies with everyone’s favorite subject: Time Calculation.

Time wounds all heals

Personally, the thing that keeps me up at night, the monster in my closet, the beast waiting to grab my foot when I accidentally let it hang over the bed while sleeping, is time calculation in programming. Regardless of the task, however simple it may seem, calculating time offsets or time zone correction continues to be a tremendous pain point. Now, to demonstrate just how crazy things can be, let’s pick on Active Support.

Active Support grants a myriad of methods for even the most impatient Software Artisan. For example, if one were to ask for this instant in time, a day ago, Active Support makes it possible.

1.day.ago
# => 2015-08-29 18:44:55 -0700

Likewise, basic date addition and subtraction is just as easy.

three_days_from_now = Time.now + 3.days
#=> 2015-09-01 18:46:34 -0700

three_days_from_now + 12.hours
# => 2015-09-03 06:48:28 -0700

Great! Very simple, seemingly arbitrary time calculation has never been easier. So what could be the catch? Obviously there isn’t one, if there were then this article would have to continue and I’m sure everyone has had quite enough.

`to_i` is not always easy

Had enough? Too bad. The topic of to_i still has to be discussed with relation to the Time class. Ruby’s Time class exposes a method to_i which can be used to convert a Time to an Integer. This Integeris the sum of seconds passed since Epoch. Using Epoch time can be useful for some systems that want to avoid doing timezone conversion. In fact, it might be so attractive an option that a developer’s first instinct might be to save and calculate everything in Epoch time. However, this might not always be the best idea.

Active Support follows suit with allowing to_i to be called on its convenience methods. For example, 1.day.to_i calculates the number of seconds in the day.

1.day.to_i
# => 86400

And 1.week.to_i calculates the number of seconds in a week.

1.week.to_i
# => 604800

Additionally, 1.month.to_i calculates the number of seconds in a month.

1.month.to_i
# => 2592000

Wait a second, that only accounts for 30 days. If I learned anything in kindergarten, it was that the teacher really hates it when you teach yourself to whistle during nap-time. Also, I am pretty sure that not all months consist of exactly 30 days.

Bad times

Introducing this inconsistency in time calculation can lead to unintended results. For example, note the difference between subtracting time objects and subtracting to_i calculated times:

Time.now - 3.months
# => 2015-05-30 19:07:44 -0700

Time.now - 3.months.to_i
# => 2015-06-01 19:07:59 -0700

So which is it Ruby? Was 3 months ago the 1st of June or the 30th of May? The problem persists even if everything is converted to integers first:

epoch_now = Time.now.to_i
# => 1440986921

epoch_three_months_ago = epoc_now - 3.months
# => 1433210921

Time.at(epoch_three_months_ago)
# => 2015-06-01 19:08:41 -0700

Ok, but why?

So why does this happen? Why does 1.month return the same integer regardless of which month it is. Because, how could it know? A month is not meant to exist as an abstract concept, there is no canonical “month”, they are either 30 days, 31 days, 28 days, or 29 days long (I didn’t forget about you, leap year!). This is just a small demonstration of why time calculation is never as simple as it may seem. There is no silver bullet to avoid it all. Even Epoch time conversions can lead to some real bad times.

I, Object!

2015-08-23T08:00:00-04:00

I am lucky enough to work with wonderfully talented people every single day. This is a guest post by one of my magnificent coworkers, Kristján Pétursson. Thank you for allowing me to share this knowledge with everyone!

This post is adapted from my answer to this Stack Overflow question. If you want to start from the beginning of the universe and build out, go read that one. If you prefer to start with something you can touch and work backwards, here we go.

Ruby likes ducks. Which is to say that when we’re coding, and we have an object, we don’t particularly care what kind of object it is, so long as it responds to the messages we send it. It might be a Duck or a Child or a Doctor, and as long as when we call #quack we hear a noise, all is well. That’s called Duck Typing, and Ruby digs it.

So if we have some arbitrary object and we ask it to #quack, the Ruby interpreter needs to figure out where the object’s #quack method is. Nothing’s been compiled, and Ruby lets you define methods pretty much any place or time you like, so #quack needs to be looked up at runtime. That’s called Dynamic dispatch, and it’s how Ruby handles ducks.

Now for the thing we can touch; let’s make a `Duck`

class Duck
  def quack
    puts "Quack, I say!"
  end
end

duck = Duck.new
duck.quack
#=> Quack, I say!

Surely, this is no surprise. You’ve done this in the past, or quieter things like it, and you understand just fine that a method called on duck will be found in Duck. But we less frequently do things like:

def duck.quack
  puts "I'm tired of quacking."
end
duck.quack
#=> I'm tired of quacking.

other_duck = Duck.new
other_duck.quack
#=> Quack, I say!

Hm, so if we can just redefine duck.quack without messing up other_duck, where is the second #quack method? It turns out every object has a Singleton Class where it can stash all its personal possessions. Other words for singleton class include metaclass, eigenclass, and virtual class, but Ruby implements a method called #singleton_class, so we’ll use that one. You can see #quack on duck.singleton_class, but not on other_duck.singleton_class:

puts duck.singleton_class.instance_methods(false).inspect
#=> [:quack]
puts other_duck.singleton_class.instance_methods(false).inspect
#=> []

And then you can see the original #quack on Duck where we left it:

puts Duck.instance_methods(false).inspect
#=> [:quack]

Drawing it out

Now we can start drawing diagrams, which is great, because people like diagrams almost as much as Ruby likes ducks. When we ask duck to #quack, it starts looking for the method on duck.singleton_class and then works its way up until it finds it.

             +----------+
             | Duck     |
             |   #quack |
             +----------+
                  ^
                  |
                  +-------------------------------------+
                  |                                     |
           +---------------+                    +---------------+
duck ~~~~> | #<Class:Duck> |   other_duck ~~~~> | #<Class:Duck> |
           |   #quack      |                    +---------------+
           +---------------+

I’m using ~~~> to move from objects to their singleton class, and then ---> to move from classes to their superclass.

And indeed, duck.singleton_class.superclass == Duck. The #<Class:Duck> singleton class is an anonymous class brought into existence just for duck. other_duck has its own singleton class that doesn’t have #quack defined on it, so it traverses upwards and finds #quack on Duck instead.

We can see the whole lookup path with #ancestors, and can check exactly where a method is defined with #method. #ancestors includes the singleton class as its first entry because that’s the first place we look for a method.

puts duck.singleton_class.ancestors.inspect
#=> [#<Class:#<Duck:0x007fe793031dd0>>, Duck, Object, Kernel, BasicObject]
puts duck.method(:quack)
#=> #<Method: #<Duck:0x007fe793031dd0>.quack>
puts other_duck.method(:quack)
#=> #<Method: Duck#quack>

There are more things in #ancestors than we’ve drawn yet, though I bet you saw them coming. Moving up from Duck, we get to Object. Everything in Ruby is an Object. No, everything. Yes, everything, even BasicObject, which is an ancestor of Object - just go with that for a second. Maybe pretend there was time travel involved.

When you class Duck, there’s an implicit class Duck < Object so your class can inherit everything Object has and be a good citizen. The falses we were using earlier to look at instance_methods lets us look only at the methods that class is defining, rather than everything it has inherited, but in reality:

duck.public_methods
#=> [:quack, :nil?, :===, :=~, :!~, :eql?, :hash, :<=>,
#    :class, :singleton_class, :clone, :dup, :itself, :taint,
#    :tainted?, :untaint, :untrust, :untrusted?, :trust, :freeze,
#    :frozen?, :to_s, :inspect, :methods, :singleton_methods,
#    :protected_methods, :private_methods, :public_methods,
#    :instance_variables, :instance_variable_get,
#    :instance_variable_set, :instance_variable_defined?,
#    :remove_instance_variable, :instance_of?, :kind_of?, :is_a?,
#    :tap, :send, :public_send, :respond_to?, :extend, :display,
#    :method, :public_method, :singleton_method,
#    :define_singleton_method, :object_id, :to_enum, :enum_for,
#    :==, :equal?, :!, :!=, :instance_eval, :instance_exec,
#    :__send__, :__id__]

And actually, Object didn’t really define any of those itself. It inherited some from BasicObject and then included Kernel to get the rest. When you include a module, it’s inserted into the list immediately after the singleton class, which explains the end of the #ancestors list. Our whole object setup looks like this:

       +-------------+
       | BasicObject |
       |   #==       |
       |   #!        |
       |   ...       |
       +-------------+
            ^
            |
            |      +----------+
            |      | Kernel   |
            |      |   #nil?  |
            |      |   #===   |
            |      |   ...    |
            |      +----------+
            |           ^
            |           |
            +-----+-----+
                  |
             +----------+
             | Object   |
             +----------+
                  ^
                  |
                  |
                  |
             +----------+
             | Duck     |
             |   #quack |
             +----------+
                  ^
                  |
                  |
                  |
           +---------------+
duck ~~~~> | #<Class:Duck> |
           |   #quack      |
           +---------------+

This is now everything we can look at to decide how duck responds to a message. But what about class methods on Duck? Well, remember I said everything in Ruby is an Object - that means everything in our diagram is like duck, and has a singleton class and ancestry chain. In fact, everything here except for duck and Kernel are instances of Class, so we can build them out the same way we built duck. Kernel is an instance of Module, and has the appropriate singleton class with Module as its superclass, but drawing that makes the diagram pretty messy.

                                      +--------+
                                      | Module |
                                      +--------+
                                          ^
                                          |
                                      +-------+
                                      | Class |
                                      +-------+
                                          ^
                                          |
       +-------------+         +----------------------+
       | BasicObject | ~~~~~~> | #<Class:BasicObject> |
       +-------------+         +----------------------+
            ^                             ^
            |                             |
            |      +----------+           |
            |      | Kernel   |           |
            |      +----------+           |
            |           ^                 |
            |           |                 |
            +-----+-----+                 |
                  |                       |
             +----------+        +-----------------+
             | Object   | ~~~~~> | #<Class:Object> |
             +----------+        +-----------------+
                  ^                       |
                  |                       |
                  |                       |
                  |                       |
               +------+           +---------------+
               | Duck | ~~~~~~~~> | #<Class:Duck> |
               +------+           +---------------+
                  ^
                  |
                  |
                  |
           +---------------+
duck ~~~~> | #<Class:Duck> |
           +---------------+

When you define a class method on Duck, you’ve probably noticed, but maybe sort of glossed over, that you declare it on self.

class Duck
  def self.in(lake)
    # Return all the Ducks in lake
  end
end

This is no different than when we def duck.quacked to put a new method on duck (but not other_duck). In this context, self is Duck, so we’re stashing .in on Duck.singleton_class in exactly the same way.

All together now

Ok, one more iteration. Once again, everything is an Object - even Module. You can discard that time traveling ancestry paradox from before, and we’ll just add lines for the actual paradox.

                                                                +----------+
                                                                |          |
                                      +--------+       +-----------------+ |
   +----------------------------------| Module | ~~~~> | #<Class:Module> | |
   |                                  +--------+       +-----------------+ |
   |                                      ^                     ^          |
   |                                      |                     |          |
   |                                  +-------+         +----------------+ |
   |                                  | Class | ~~~~~~> | #<Class:Class> | |
   |                                  +-------+         +----------------+ |
   |                                      ^                                |
   |                                      |                                |
   |   +-------------+         +----------------------+                    |
   |   | BasicObject | ~~~~~~> | #<Class:BasicObject> |                    |
   |   +-------------+         +----------------------+                    |
   |        ^                             ^                                |
   |        |                             |                                |
   |        |      +----------+           |                                |
   |        |      | Kernel   |           |                                |
   |        |      +----------+           |                                |
   |        |           ^                 |                                |
   |        |           |                 |                                |
   |        +-----+-----+                 |                                |
   |              |                       |                                |
   |         +----------+        +-----------------+                       |
   +-------> | Object   | ~~~~~> | #<Class:Object> |                       |
             +----------+        +-----------------+                       |
                  ^                       |   ^                            |
                  |                       |   |                            |
                  |                       |   +----------------------------+
                  |                       |
               +------+           +---------------+
               | Duck | ~~~~~~~~> | #<Class:Duck> |
               +------+           +---------------+
                  ^
                  |
                  |
                  |
           +---------------+
duck ~~~~> | #<Class:Duck> |
           +---------------+

Module is an Object, which inherits from BasicObject, which is a Class, which inherits from Module, which is an Object, and so on until you hit the turtles. #ancestors and other methods that would have trouble with this loop have special cases in the Ruby source for when they find BasicObject, and just pretend that’s the end of the line.

If you start from any of these objects and traverse up, right-to-left, depth-first, you can build the ancestry chain showing in what order methods will be found. Everything has a singleton class to handle the things we declare on it, and there’s only a little bit of cheating to make the whole thing work. But what did you expect from a system filled with ducks?

Author: Kristján Pétursson

Gems != Magic

2015-08-16T08:00:00-04:00

Ruby gems can seem magical. Usually, one (or more accurately: thirty) solution to a problem exists in the form of a gem. Using gems is easy and intuitive. Simply add a line in the project root’s Gemfile and run bundle install. Then, either require the name of the gem or, if using Ruby on Rails, start using the gem’s code immediately anywhere in the project. What could possibly go wrong?

Gems on gems on gems

Unchecked and unmanaged, a project’s Gemfile might turn into something that looks like:

source 'https://rubygems.org'

gem 'linkbuilder'
gem 'http_helper'
gem 'the-color-red'
gem 'devise'
gem 'cancancancancan'
gem 'nokogiri'
gem 'crack'
# ...
# ....
# .....
gem 'cucumber'
gem 'avacado'
gem 'you-get-the-idea'
gem 'these-are-too-many-gems'
gem 'really'

It is very possible that all these gems do not have any adverse side effects. They might not even have any horrible monkey patches to String or Object. This post has more information on “Why do most Rails apps use so much memory per process?”.

Without being able to see inside of them, intrusive gems might be mixed into this mass and we would have no idea. At a glance, it is not immediately obvious how to differentiate between benign and malignant gems. It becomes infinitely easier to understand how gems interact if we remember that gems are not magic.

Gems are just Libraries

There it is! The rabbit is out of the hat. Somewhere, some amount of cats have been released from their respective bags. Gems are not magic silver bullets injected into a codebase. Instead they are nothing more than libraries of code. Gems are (hopefully) well encapsulated groups of files that amount to a (again hopefully) few pieces of helpful functionality. With this new understanding of what gems really are, it means that parsing, modifying, and creating gems is within reach. All that is left is to figure out how to pry them open.

Opening gems

Understanding gems is a great way to learn different design patterns and idiomatic Ruby conventions. Good gems have in-line documentation, great gems have comprehensive Github documentation. Github is the most common place to find the source code for gems and a large number of them have really great method and API documentation. This level of detail might be all that is required; however, for the more opaque gems, the bundle open command is extremely helpful.

bundle open <gemname> will use the system default text editor to open the contents of a gem. Since all gems are installed locally on the system, they should be available to open from a project’s root. If a default editor is not set, or the default is not desired, one can be added to the beginning of the command: EDITOR=vim bundle open <gemname>.

A common starting place for a gem’s logic is in a file with the same name as the gem. Since most gems need to encapsulate their logic, a namespace or wrapper class is usually created with the same name as the gem. In this file it is equally common to find a list of required files or the entry point for the underlying logic. This file is usually in the root of the gem’s source or in a lib directory.

In the validates_type root file:

Great! Now what?

Now, go out there and open up all the gems in the entire world! Read them, understand them, be one with the gems. Get a flavor for the different types of Ruby code that can exist. Then, after complete code nirvana has been achieved, submit a pull request to the gem’s source with some great additions. Or, place a few bindings in a few obscure methods to see how the code flows in and out of the gem. A master craftsman understands all of the tools in his toolbox, the same should be true for software.

How to Process Large Data Sets with Ruby

2015-08-09T08:00:00-04:00

The need for data migrations in mature systems is real. At times, requests for these migrations can appear at random. One minute, a system is behaving as specified, happily fulfilling requests, and then bam! All the user objects suddenly need an extremely crucial attribute. Well that seems relatively simple, right? All that is needed is a simple ruby script to iterate over all users and update every user with this essential piece of data.

To demonstrate such a problem, we can assume the following:

A Ruby on Rails application exists with a User class, each with a phone_number attribute.

class User < ActiveRecord::Base

  validates_presence_of :phone_number
end

The application is relatively popular, which results in 2,000,000 users:

User.count
#=> 2000000

Finally, the requested task:

Add a “+1” to the beginning of all user phone numbers (we are assuming that all phone numbers belong to users in the USA or Canada).

Disclaimer: Ruby is probably not the best tool for this kind of data migration but for argument’s sake we can assume it is the only one available.

Serial scripts at scale are slow

Without giving it too much thought, an approach to solve this problem might look something like:

User.find_each do |user|
  user.phone_number = "+1#{ user.phone_number }"

  user.save!
end

This approach will work. The find_each method will make sure the memory footprint of the script stays low (it will not load every user into memory at once) and the phone numbers will be updated.

However, this will be painfully slow. Even if the system is able to update 20 users per second, it will take approximately 27 hours to complete.

2,000,000 users / 20 users per second
= 100,000 seconds to process all users

100,000 seconds / 60 seconds
~= 1,666 minutes

1,666 minutes / 60 minutes
~= 27 hours

Resque to the Rescue

Resque is a very useful Ruby library for creating background jobs. Redis is utilized as the storage for these jobs and individual Resque workers pick one job off each queue at a time.

With a small amount of code reorganization, Resque enables processing multiple users at the same time. The basic idea behind this approach is divide and conquer.

0. Install Resque and Redis

1. Extract phone number updates into a new class

This new class will accept a single user and update all associated phone_numbers.

class PhoneUpdater
  def initialize(user)
    @user = user
  end

  def update_phone!
    @user.phone_number = "+1#{ @user.phone_number }"

    @user.save!
  end
end

2. Create a Resque Job to use this logic

A Resque Job class needs to conform to a simple API. A queue must be defined and the class must have a class method named perform. The parameter given to perform is the same that is passed to Resque when a job is enqueued.

class PhoneUpdaterJob
  @queue = :phone_updates

  def self.perform(user_id)
    user = User.find(user_id)
    updater = PhoneUpdater.new(user)
    updater.update_phone!
  end
end

3. Create script to enqueue ResqueJobs

Iterate over the user set and enqueue a job per user. The second argument is the id of the user to process.

User.find_each do |user|
  Resque.enqueue(
    PhoneUpdaterJob,
    user.id)
end

4. Run many parallel Resque workers

This is where the fun happens. By initializing many different Resque worker processes, they all will read from the phone_updates queue and process users in parallel. The QUEUE specified matches what was defined in PhoneUpdaterJob.

# Open a terminal in the root of your project...
QUEUE=phone_updates rake resque:work

# Open another terminal in the root of your project...
QUEUE=phone_updates rake resque:work

QUEUE=phone_updates rake resque:work
# Open another terminal in the root of your project...

# etc.

5. Try not to overload the system

Now with 2,000,000 queued Resque jobs, each Resque worker will drastically decrease the execution time of the overall task. However, system constraints should not be ignored. Database connection saturation, CPU usage and memory usage should be used in calculation of how many Resque workers to run at once.

Parallelism is awesome

Although the parallel solution is not as simple as its serial counterpart, the benefits are extremely apparent. The parallel solution is not without its weaknesses, but they do not make it invalid. It iterates over the entire user set twice, but parallelizing the slow phone_number update queries makes up for that inefficiency ten-fold.

Faster Ruby Testing: Only Test What Matters

2015-08-02T08:00:00-04:00

Automated testing is important. Fast, exhaustive automated testing is even more important. Tests are responsible for ensuring the code you spend hours creating actually works. A great test suite can be a safeguard against bugs, a directional guide towards extending the code, and an accurate measurement of the codebase’s health. The key to writing good tests is understanding where pieces of responsibility begin and end. Maintaining small concise automated tests can make all the difference.

Speed is King

Aside from the general importance of short feedback cycles, a fast test suite will greatly improve your work-flow and mood. Conversely, a slow test suite can make your development cycle a living nightmare. How many unfortunate software artisans have dealt with something like:

Bleh! What a horrible thing to deal with. What, am I supposed to just wait 5 whole minutes every time I want to test my system? A test suite this slow is simply unacceptable.

So what could be causing this slowness? More than likely, a number of tests in this application are doing too much. To try and replicate and solve this problem, let’s assume we have a Ruby on Rails application we test using RSpec.

Testing Database Models is Slow

We can assume that there exists a very important class called MyAwesomeClass:

class MyAwesomeClass < ActiveRecord::Base
  def assign_and_save!
    saved_on_the_weekend = weekend?
    self.save!
  end

  def weekend?
    Time.now.saturday? || Time.now.sunday?
  end
end

If we wanted to test that the assign_and_save! method works correctly, we might write a test like:

describe '#assign_and_save!' do
  it 'assigns and saves' do
    awesome_test = MyAwesomeClass.new

    expect do
      awesome_test.assign_and_save!
    end.to change(MyAwesomeClass, :count).by(1)

    expect(awesome_test.saved_on_the_weekend).to_not be_nil
  end
end

These tests make sure that the model is saved correctly to the database and a value is assigned to saved_on_the_weekend. However, the test code is overstepping the boundaries of the method it is testing. The assign_and_save! method’s job is to simply assign a value to an object and then save it. It does not care about how the actual saving works, that is the job of other validations on the model and ActiveRecord.

To avoid this, we can assert that the save! method is called, which will not actually write to our database:

describe '#assign_and_save!' do
  it 'assigns and saves' do
    awesome_test = MyAwesomeClass.new
    expect(awesome_test).to receive(:save!)
    awesome_test.assign_and_save!
    expect(awesome_test.saved_on_the_weekend).to_not be_nil
  end
end

Voila! We have made the same assertions about our code and did not do any slow input/output operations.

Now I know what some of the more detail oriented Software Artisans reading this will initially think: “What about validation concerns with the model?” and “If you stub out save! like that you can’t be sure it worked”. While these are totally valid points, there exists a simple solution for dealing with the uncertainty of the save:

The `valid?` method

describe '#assign_and_save!' do
  it 'assigns and saves' do
    awesome_test = MyAwesomeClass.new
    expect(awesome_test).to receive(:save!)
    awesome_test.assign_and_save!
    expect(awesome_test.saved_on_the_weekend).to_not be_nil
  end

  it 'is save-able' do
    awesome_test = MyAwesomeClass.new
    allow(awesome_test).to receive(:save!)

    awesome_test.assign_and_save!
    expect(awesome_test).to be_valid
  end
end

With the new test assertion, we make sure that the model’s valid? method returns true, ensuring the model will be saved properly.

The valid? method is what ActiveRecord uses before saving the model to the database. This method returns either true or false and writes the data to the database if true was returned. Now the code is tested and ActiveRecord’s’ validation errors are considered.

Not Perfect, but Good Enough

However, there is one more thing that this test will not catch: database specific uniqueness constraints. If your application uses a database to enforce uniqueness and not in ActiveRecord validations, this testing method will fail. For that you will need a more robust integration tests which guard against duplicate data.

All that aside, not actually saving data will be fine for the 90% case which is most codebases.

Only Test What Matters

In this example, removing the testing of save! resulted in a speed increase. However, the idea of only testing the crucial parts of a method is not only about speed, it is about encapsulation. If a method’s only responsibility is to call helper methods, that is the only thing that should be tested. The test should simply assert that the helper methods are called, not the logic within them. Following this pattern will help keep test code small, concise, readable, and input/output operations to a minimum.

Ninety Days of Commits

2015-07-26T08:00:00-04:00

It has been ninety days since I started the Year of Commits initiative. The criteria behind this challenge was to make a commit to a public Github repository every day. Technically, a commit like the following satisfies that criteria:

However, if I allowed this commit to be my sole contribution for a day, I would be cheating no one except myself. In reality, I have been averaging 3 to 4 decent size commits a day. Spending around an hour or two total per night. Github is an awesome piece of software that tells a visual tale of when I am most productive:

Over this quarter of a year, I have accrued some interesting learnings about developing open source software.

1. Writing software every day just feels good.

Not missing a single day of writing software has made a significant improvement in my confidence and attitude. In the same way exercising daily makes your body feel more sturdy, developing software daily keeps my mind focused and clear. Like warming up a car before a drive, writing software every day makes it very easy for me to jump into a new project or build a new feature. I think most of my fellow software artisans have been in this situation: You go on vacation or maybe just a very relaxing weekend, then the next day at work or when you open that side project again, you end up spending a lot of time getting familiar with the software. Even if you just wrote it a few days ago, the break has made some tiny details less sharp. Writing software every single day helps eliminate this.

2. Immature software is expected

While writing software, there are a total of 0 people that look at what they just wrote and say: “This software is already bad and I have only just written it”. However, this statement can be more true than we would like. The potential in a new project is always the best part, right? Developers around the world usually jump at a change to do green field development. After all, with no legacy software around we can make what ever our heart desires. It can be clean, DRY, clever software with all the best design patterns. But, at least in my experience, all software starts off painfully immature. This is just fine. The shape and structure of good software grows over time. This is particularly true for my Year of Commits projects. Since I must write software every single day, I revisit my software extremely frequently. This re-visitation greatly increases the iterations my software goes through and I end up with much more polished work.

3. Software development is not just making it work

If you take away business requirements, deadlines, conversion rates, etc., you are left with something pure: software development. Year of Commits has enabled me to experience developing software at a slower pace without these extra degrees of “purpose”. Because of this, I have learned the difference between software that just works and carefully architect-ed software. I am not boasting that the software I write is now amazingly designed and executed, I have just noticed that when I invest more time, it makes a difference. We, as Software Aristans need time to stop and smell the ones and zeros. Given a decent amount of time and attention, all software can mature to something really extraordinary.

I am optimistic about completing my goal of a Year of Commits. With 3/4 the distance ahead of me, let’s hope I can keep that graph solid green.

Just learn Rails (Part 2)

2015-07-19T08:00:00-04:00

In a previous post I explained how “just learning Rails” is not as straight forward as the phrase portrays. Even the novice programmer who learns Ruby on Rails in a methodical progression may still run into hardship. However, this is not entirely the fault of young programmer, he or she was simply enabled into bad habits. Ruby on Rails enables the complete disregard of encapsulation. Notice how I did not say that Ruby on Rails prescribes less encapsulation, it simply enables it.

This idea, much like any idea posted to the Internet, is not new. Many intelligent people have come to this same realization. One such example is the Trailblazer framework, which aims to help inject some encapsulation and abstraction mechanisms Rails is lacking. (For those interested, a book has been written to more fully explain this framework).

Alright, so what am I really talking about? Ruby on Rails has certain features that make it easy to forget about encapsulation.

The Autoloader

A keyword that most Ruby on Rails developers rarely see on a daily basis is require. A piece of Ruby on Rails magic called autoloading greatly decreased the frequency of the require keyword in Ruby on Rails applications. This has many positive aspects for new and seasoned developers alike. But, it is not without drawbacks. One such drawback is that it makes ActiveRecord objects ubiquitous. Whenever a developer has the urge to reach into the database, regardless of the context, they are able to do so.

This enables horrific code.

Imagine we have a view template that we want to display a user’s username and their pictures. Without any guidance, this following code could come into existence:

#
# Reminder, this is terrible code. Never do this.
#

<div>
  <!--
    Let's just go through all the users in the database.
    This is a new application so we don't have that many users,
     it will never run into scalability issues.
     -->
  <% User.all.each do |user| %>
    <div class='username'>
      <%= user.username %>
    </div>

    <div class='pictures'>
      <!--
       Make sure we have the only the picture url, just go ahead
        directly to the database and let it handle
        the downcasing that we, for some reason, need.
       Also, Jim in Business Ops thinks it will be
        super great to order the pictures randomly each time!
      -->
      <% Pictures.select('downcase(url)')
                .where(user_id: user.id)
                .order('random()').each do |url| %>
        <img src="<%= url %>"/>
      <% end %>
    </div>
  <% end %>
</div>

#
# Never replicate the above code or you will be very sad
#

Notice how deep in the presentation layer (the view) we have direct database calls. This completely violates the encapsulations that an MVC framework should provide. Since all the models exist everywhere, nothing stops a programmer from writing it in this way. The code above should make every ruby programmer cringe.

Separating concerns and responsibilities helps prevent repetition and boosts comprehension. The above code should be a composition of multiple objects with entirely different responsibilities. That concept is not always intuitive for some developers, especially those just starting out. One might ask: “Why would I make a whole mess of objects to do the same thing that these 14 lines can do?”. The simple answer is because these problems are not new. Ruby on Rails did not expose these abstraction and encapsulation issues for the first time in the history of software engineering. They have been around for a very long time. Smarter software artisans than I saw these problems and created guidelines to solve them. Following their lead will save all of us in the long run.

So what can we do about our misguided code above? Let us try moving some responsibilities around and see how it changes.

Add a dash of encapsulation

# The user class just needs to know about the database connection.
# ActiveRecord takes care of that detail for us, adding a scope
#   is a supplementary addition to that logic.
class User < ActiveRecord::Base
  scope :with_pictures, ->{ includes(:pictures) }
end

# We can create a PORO (Plain Old Ruby Object) to house the user
#  object and extract, downcase, and randomize its picture urls.
class UserPresenter
  def initialize(user)
    @user = user
    @pictures = user.pictures
  end

  def username
    @user.username
  end

  def picture_urls
    @pictures.map do |photo|
      photo.url.downcase
    end.shuffle
  end
end

# The controller handles linking up each User with a UserPresenter
#  then saves the list to a variable to be used by the view.
class UsersController < ActionController
  def show
    @user_presenters = []

    User.all.with_pictures.each do |user|
      @user_presenters << UserPresenter.new(user)
    end
  end
end

Finally, the view logic only knows about a presenter for each user. From the views perspective, the presenter object only has two methods: username and picture_urls. It does not need to know about database structure, schema, syntax or business logic. It has one job, to display data:

<div>
  <% @user_presenters.each do |presenter| %>
    <div class='username'>
      <%= presenter.username %>
    </div>
    <div class='pictures'>
      <% presenter.picture_urls.each do |url| %>
        <img src="<%= url %>"/>
      <% end %>
    </div>
  <% end %>
</div>

Conclusion

The code that was rewritten is far from perfect. However, it is a noticeable improvement over its original version. I am in no way claiming to be the creator of these design patters, simply a spokesperson for good encapsulation and object oriented design. When respected and correctly utilized, Ruby on Rails is a fantastic tool for web development. When learning Ruby on Rails, it is imperative to stand by software engineering paradigms. Just because a framework allows it, doesn’t mean that it is necessarily a good thing to do.

Create a Free Tech Blog

2015-07-12T08:00:00-04:00

The Internet is wondrous technology. If anyone remembers Geocities and Personal Home Pages, you can appreciate just how much the Internet has evolved. This evolution has recently enabled creating a tech blog completely for free. When I say free, do I mean completely free? Yes, completely free! Even this very blog, my Year of Commits, is hosted for free using the same technologies outlined in this post. You too can have a free tech blog after following these 2 simple steps.

Step 1. Use Jekyll

Jekyll is a free open source static site generator. The Jekyll documentation page has some great tutorials that make getting started extremely painless. Jekyll is also a ruby tool. To contribute to Jekyll, check out their github page.

Jekyll makes it easy to start creating content without worrying about CSS, page templates, or link architecture. It even generates an RSS feed of your content. All pages can be written in Markdown and automatically converted to HTML upon building or serving the site. The jekyll build command is what this post will focus on, but jekyll serve is worth checking out too.

To get started with Jekyll:

$: gem install jekyll
$: jekyll new mytechblog
$: cd mytechblog/
$: touch _posts/2015-07-12-my-first-post.markdown

This will create a new directory with some default configuration files and subdirectories. Inside this new directory, the example created a 2015-07-12-my-first-post.markdown file within the _posts/ subdirectory. This file will contain the content for the first blog post.

Jekyll Markdown pages require a specific format at the top of the file. This format is called front matter. It is imperative that all Jekyll posts have proper front matter in order to be converted to HTML.

Step 2. Use Github Pages

Github has been a great tool for managing version controlled software for years. Now, Github has released an equally awesome tool called Github Pages. To get started with Github Pages, simply create a public repository with the pattern: <yourusername>.github.io (Remember to replace the <yourusename> with your actual username). Now, every time a commit is pushed to this repository’s master branch, the blog will be automatically updated. Since this repository is required to be public, make sure that no sensitive information is accidentally committed within posts.

To set up a directory and point it to the github.io repository:

$: mkdir username.github.io
$: cd username.github.io/
$: git init .
$: git remote add origin git@github.com:username/username.github.io.git

The files uploaded to the Github Pages repository should be static HTML. This is where Step 1 comes in. Since this blog will use jekyll build, the <yourusername>.github.io repository should consist of HTML files generated by Jekyll.

To generate the HTML files:

$: cd mytechblog
$: jekyll build --destination=~/username.github.io
$: cd ~/username.github.io
$: git add .
$: git commit -m "First blog post"
$: git push origin HEAD

And there you have it! A successful blog created entirely for free. Now, the example blog is accessible at https://username.github.io.

(Optional) Step 3. Use a custom URL

To serve a blog at a custom URL, simply add a single file to the root of the <yourusername>.github.io repository. The filename must be CNAME and it should include a single line: the custom URL.

An example CNAME file:

mytechblog.com

Additionally, the DNS provider that mytechblog.com’s DNS is served from must include an ALIAS, ANAME, or A record to the Github IP. More information on setting a custom domain can be found at the Github help page

Final Step: Blog for free about cool technology

Now that a completely free blog is newly created, you have the ability to update it as frequently as you choose. Share your knowledge at expeditious paces by simply making commits to a public repository. Never worry about hosting costs, server configuration or deployment again. Simply write Markdown, build static HTML, and commit!

Modules in Ruby

2015-07-05T08:00:00-04:00

Currently, one hundred percent of Year of Commits’ commits have been to Ruby repositories. Ruby has many strengths and is a very malleable language. Ruby can be written functionally or object oriented. I tend to lean toward the latter and find myself using a few different patterns regularly. One powerful tool Ruby provides is the use of Modules.

Modules

When code is to be shared between classes, a Module is created to encapsulate the shared functionality. A very simple module looks something like this:

module Ripe
  def ripe?
    puts "this is ripe!"
  end
end

`extend`

With the Ripe module, there are two ways to include its methods in a class. We can extend the module, which makes the method ripe? a class method on the Banana class:

class Banana
  extend Ripe
end

Banana.ripe?
# => this is ripe!

`include`

Alternatively, we can include the module, making ripe? an instance method:

class Banana
  include Ripe
end

banana = Banana.new
banana.ripe?
# => this is ripe!

That’s great right? Right! We have methods on methods on methods. As long as we can make modules for our instance and class methods separately, we will be golden. But, what about all the times that we need to define both class and instance methods in one module? Surely it would be crazy to have RipeClassMethods and RipeInstanceMethods right? That looks like amateur hour. There must be a better way.

`def self.included`

A very helpful method for dealing with included modules is the included class method. This method is built in Ruby and any class has access to it. By using the included method, it is possible to mix class and instance methods within a single module.

When using self.included, we are able to determine which methods are accessible on the instance and on the class. We will use a new inner module named ClassMethods to encapsulate our desired class methods. In our example below, the single parameter base is the class in which the Ripe module is included.

module Ripe
  def self.included(base)
    # Just like in normal module extension,
    #  the extend method is used to make all methods in
    #  ClassMethods into proper class methods on base
    base.extend(ClassMethods)
  end

  module ClassMethods
    def ripe?
      puts 'this is ripe!'
    end
  end
end

Now, including the functionality from Ripe is the same as the include example above:

class Banana
  include Ripe
end

Banana.ripe?
# => this is ripe!

With this new structure, adding instance methods is as simple as adding them outside of the ClassMethods module.

module Ripe
  def self.included(base)
    base.extend(ClassMethods)
  end

  def color
    'yellow'
  end

  module ClassMethods
    def ripe?
      puts 'this is ripe!'
    end
  end
end

class Banana
  include Ripe
end

Banana.ripe?
# => this is ripe!

banana = Banana.new
banana.color
# => yellow

Boom, boom, pow, we have some nicely organized shared code. We are now able to easily define instance and class methods for our various important fruit related modules.

Staying Motivated

2015-06-28T08:00:00-04:00

Motivation is the most important thing you can control. Only with proper motivation can one reach their full potential. A Year of Commits is a project that requires a decent amount of motivation. I, like nearly every human ever, am not always the defintion of a “driven individual”. It takes a large amount of effort to persevere through low points and come out the other side with something to show for it. It has been 63 days of my Year of Commits initiative and I have found a few helpful ways to keep my motivation up and morale high.

1. Make small deliverables for yourself

If you embark on grandiose endeavors frequently, consider breaking pieces of them apart into small, digestible morsels of motivation goodness. Every project does not need to go at break neck speed instantly. This is especially applicable in software development. You know that project that you’ve been planning? The revolutionary one that you’re so excited about? That project has a greater chance of completion if you don’t expect finish the whole thing in an afternoon. Setting yourself up with unreasonable expectations can deflate even the most cocksure individuals. If you take some time to prioritize small deliverable pieces, you will be much more likely to achieve something. Speaking from experience, there comes a time when you have to sit back and wonder: “Why do I have 10-15 projects around 30% completion?”. It might just be that you were biting off a bit more than you can chew. Take smaller bites, you will get there if you just keep going.

2. Keep moving

A complementary behavior to breaking things into smaller pieces is to make every day count. As the person committed to working every single day for year, I might be a tad biased. However, I believe that maintaining at least some kind of constant pace of effort will keep motivation from dropping off a cliff. Procrastination is the greatest enemy of motivation. If you let yourself delay your tasks once, it becomes increasingly easy to delay them again. Subsequent delays will mostly likely be in the not so distant future. If you make your tasks part of your routine, it can have the opposite effect. I bet you don’t even think about all the time you spend doing things that are just “part of your routine”. Imagine if the task you want to accomplish, the world changing, earth shattering, unbelievable awesome project that you have queued up was as easy to fit into your schedule as brushing your teeth. There exists no magic that some of the greatest achievers have found and harnessed. They simply kept working until the job was done. To them, working on what mattered to them was as routine as brushing their teeth or putting on their shoes.

3. Believe that you are good at what you do

As it so happens, your kindergarten teacher was right about a thing or two. Attitude is important in order to help you stay motivated. A self debilitating mindset can vacuum the motivation out of nearly anything. Having the right attitude will help you soar, help keep that fire of motivation stoked and ready. Impostor syndrome, doubts, or anything in that general mindset will be nothing but a hindrance. Understanding your skill sets, knowing where you want to improve and where you have strengths, will help keep you motivated to achieve your goals. This might be the most obvious piece of advice but that does not mean it is the easiest to achieve. In software development, our confidence in our abilities is constantly tested. I try and combat this by remaining calm, remembering “I am not my code”, and taking a few deep breaths now and again. Confidence takes time to build. If you are among the lucky few that simply have it in abundance, great news, you just need to keep it strong!

So now with all this new information, ask yourself: Are you good? Yeah you are good. You are better than good, you’re motivated! Keep moving, keep building, keep creating and you will get to where you want to go. You are the one at the helm, the one in control of your journey. Motivation is the fuel.

Rails style subset validation

2015-06-21T08:00:00-04:00

Do you suddenly wake up in a cold sweat, wondering if there is finally a way to validate that your data is a proper subset that you desire? I too have experienced this horror, and that is why I made a rails style subset validator.

Validations über alles

Data validation is important. Having data you can depend on is the difference between a great application and one that might be classified as: “amateur hour”. Rails provides a nice amount of validators via ActiveModel::Validations, but not an exhaustive set (get it?). One of those missing pieces involves validating sets of data. Just in case you haven’t had your coffee yet, we define a set as an abstract data type that stores distinct values in no particular order.

Ruby does indeed have the Set class but most code I’ve come across simply uses the Array class. So, for simplicity’s sake, all examples will use Array.

Now that we have all been convinced that subset validation is the greatest thing since sliced bread, how about some examples?

Example usage

To include validates_subset, simply add gem 'validates_subset' in your Gemfile of your projects. Or type gem install validates_subset in your terminal.

For rails applications, the gem is automagically available to you. For other ruby frameworks, just add the line: require 'validates_subset' wherever you need the library. For this, we can assume that our application is using Rails 4.0 and we have a class that looks like:

class HasASubset < ActiveRecord::Base
  attr_accessor :foo

  validates_subset :foo, [1, 2, 3]
end

Using our handy dandy test class, we can see that:

A valid subset is valid:

test = HasASubset.new
test.foo = [1]
test.valid?
# => true

An empty set is also very valid:

test = HasASubset.new
test.foo = []
test.valid?
# => true

An invalid set is… invalid:

test = HasASubset.new
test.foo = ['banana']
test.valid?
# => false

And a non-set is invalid:

test = HasASubset.new
test.foo = 99_999
test.valid?
# => false

Holy guacamole that is some sweet validation. But wait, there’s more! Since this validator is built on top of rock solid ActiveModel::Validations logic, all modifiers are supported.

Modification nation

If your particular dataset needs something to be a subset or nil, it is a simple as:

class HasASubset < ActiveRecord::Base
  attr_accessor :foo

  validates_subset :foo, [1, 2, 3], allow_nil: true
end

If you needed the data to allow_nil and only validate on create, you could easily define the validation as:

class HasASubset < ActiveRecord::Base
  attr_accessor :foo

  validates_subset :foo, [1, 2, 3], allow_nil: true, on: :create
end

Any combination of modifiers is supported by validates_subset. For a complete list of modifiers, have a look at the rails documentationn.

As always, please contribute to and use this software for the low low price of free.

You are not your code

2015-06-14T08:00:00-04:00

It may have been a mentor, a colleague, or even a random vagrant on your way to work. One of these people has been wise enough in to let you in on the well known fact that: “You are not your code”. The words themselves seem so simple, so obvious and straightforward. But simple as it may be, this concept is paramount if you are to survive as a Software Artisan and as a functioning member of society.

When writing software, having a sizable ego is not always appropriate. In fact, it could become a hindrance that will prevent you from being productive and happy. Software contribution is about coordinating with others to make the best possible software. It is very hard to coordinate if a person is unable to relinquish control of their projects because they feel defined by them.

The first time someone said that I was not my code, I took offense. I didn’t understand what they really meant. All I could think was: “What do you mean? I wrote the code. I put my heart, soul, blood, sweat, and tears into the code. It is the epitome of me. The code I write defines me! “. It was that last statement that demonstrated how naive I really was. Your code was crafted by you, maintained by you, and evolved with you. But, like a carpenter is not a chair, you are not your code.

During my Year of Commits, it has been very important to internalize the difference between my code and myself. If I have too much ownership and pride dealing with the code I write, it makes it impossible for me to contribute as part of a community. The difference between writing software and other creative forms of work lies in the longevity of what is created. For a more traditional artisan, a painting or a house will last much longer than the average piece of code.

You, me, even that guy at work who seems to be the fabled “10x engineer”, write transient code. Our code will not outlive us. Heck, the average lifespan of code written is between 6 months and 2 years. That is nothing compared to other professions. What if civil engineers only had to make a bridge that lasted for 6 months? That would be one poorly built bridge that no one would want to drive over. But when writing software, that is often the expected life of the code. With that, it becomes quite clear that code written to not last a century should never be used to define a person’s life. If you had the desire to really define yourself by your code, you would probably spend 10 years writing and perfecting every single component. This is not feasible nor realistic. We are not building bridges to last decades upon decades, neither are we creating pieces of art to hang in museums for generations. We are building software to last “long enough”. We build it because we love to and because someone is most likely paying us to do it (I still smile sometimes when I realize I get paid to write code).

At the end of the day, everyone wants to be proud of the job they have done and the things they have achieved. People who write software are very lucky. We get to go home every day having created something new or made something better. However, we must remain humble. Humility will be our saving grace. It will be the thing that prevents us from going gray at the age of 35. Letting go of the code that we write will help us keep that bottle of Glenlivet from diminishing at an expeditious pace. Our families will not have to wonder when we will return home because we let our code go. We can rest easy knowing “that new guy who uses tabs instead of spaces” will be taking over the code we wrote a month ago, and that is ok. Because in the end, the code we write will most likely be deleted within a year of its conception.

You are not your code. You love to write code but are not defined by it. The code we write enriches our lives. We learn a great deal from the problems that we solve, but we are not defined by the solutions. I am a Software Artisan who’s code will be forgotten along with everyone else’s. I am happy I got to write it at all.

Short Feedback Cycles

2015-06-07T08:00:00-04:00

Forty-two days into my year of commits initiative has come and gone. With it, I have found some great processes for making my contributions efficient and effective. Feedback is extremely important when writing software. After all, the code you write is probably designed to work, right? Right. A good way to verify correctness in your software is through automated tests. These can either be unit tests for specific methods and functions or integration tests; those that make sure a complete code path behaves as expected. Regardless, we write these tests to ensure our code works now and in the future. “In the future” is where I have started to contribute to projects.

Good projects have test files that the original author(s) provide. Great projects have fast test files that the original author(s) provide. Fast tests provide short feedback cycles. Short feedback cycles prevent bugs and help make new feature development and refactoring much more painless. Feedback governs our behaviors in and outside the development world. Without it, we would not be as efficient or effective at the tasks that are important to us. From writing software to training for a half marathon, feedback matters a great deal.

In a recent contribution of mine, I relied heavily on automated testing and short feedback cycles. It was important to me that I was able to identify how the specs were changing as I upgraded them to RSpec 3. I used Guard to help run my tests automatically when any of the code changed. Guard is a tool to automatically run your automated test files as you are making changes to the code.

Guard has been around for a long time. I chose to use it due to how simple it is to set up. All you need to do is include the guard-rspec gem in your repository create a Guardfile.

An example Guardfile (which exists in the root of your repository) would look something like this:

guard :rspec, cmd: 'rspec', notification: false do
  watch(%r{^spec/.+_spec\.rb$})
  watch(%r{^lib/(.+)\.rb$})    { |m| "spec/lib/#{m[1]}_spec.rb" }
  watch(%r{^models/(.+)\.rb$}) { |m| "spec/models/#{m[1]}_spec.rb" }
  watch('spec/spec_helper.rb') { "spec" }
end

Let’s break this file down and see exactly what is going on.

watch(%r{^spec/.+_spec\.rb$})

This line tells Guard to watch each spec file and run the contained tests if the file has been saved. The file does not need to be changed, only saved for guard to run the specs.

watch(%r{^lib/(.+)\.rb$})    { |m| "spec/lib/#{m[1]}_spec.rb" }
watch(%r{^models/(.+)\.rb$}) { |m| "spec/models/#{m[1]}_spec.rb" }

These two lines do the same thing in different directories. The first looks in the lib directory for anything ending in .rb and runs the associated spec files in spec/lib/. For example, if we have changed a file, lib/foo.rb, then the spec, spec/lib/foo_spec.rb, is run. The second line does the exact same thing for the models directory.

watch('spec/spec_helper.rb') { "spec" }

This last line tells guard to run a specific directory (in this case, spec) whenever our spec/spec_helper.rb file is saved.

With our Guardfile set up, seeing the successes and failures of our test suite while we do development is as easy as running guard in the root of our directory structure. Personally, I use guard --clear to make the terminal clean up after itself between runs. This creates a nice blank slate to run each spec on and requires no additional scrolling.

Guard has been a very valuable asset to me during my Year of Commits and it, or technologies in the same space, would be worth looking into for anyone searching for short feedback development. Go out there, keep your Guard up and make some great contributions to other people’s code!

Writing good API wrappers

2015-05-31T08:00:00-04:00

Both in my day job and in my year of commits, I spend a lot of time thinking about APIs.

For the uninitiated, an Application Program Interface (API) is an avenue for one piece of software to speak to another. This could be a remote, web-based, HTTP API. Or, the API might be an internal interface for one portion of a software system to talk to another. Designing an API that is simple and maintainable is crucial if it is intended to be used and quickly adopted by colleagues or 3rd party developers.

Being one of those 3rd party developers, I find myself using and writing wrappers to these APIs. Here, I will explain what makes a good API wrapper. Let’s postulate about some example wrapper designs.

The Barely Abstractor

The mantra of this design is: “I will take away that annoying HTTP element and you do literally all the rest.”

We can assume that a library called api_requester exists to wrap our very important 3rd party remote API (Which we will also assume is a RESTful API).

If we wanted to retrieve an object our code might look something like:

require 'api_requester'

params = { object_id: 1 }
headers = { 'Content-Type' => 'application/json' }

object_wanted = APIWrapper.get(
                  'relative/path/to/object/',
                  params,
                  headers)

# => { big: :hash, of: :attributes}

To post an object via this wrapper:

require 'api_requester'

params = {
  object_id: 1,
  object_name: :foo,
  object_type: :bar,
  object_description: 'I am an object'
}
headers = { 'Content-Type' => 'application/json' }

APIWrapper.post(
  'relative/path/to/object/',
  params,
  headers)
# => { big: :hash, of: :attributes}

So, you get the idea. That idea is verbosity. However, this approach is not all negative.

Pros:

Resilient to API changes

A big problem with wrapping APIs outside of your own control is endpoint churn, change, and deprecation. With such a verbose wrapper, the consumer is in complete control of the request. From URL changes to parameter addition and deletion, a consumer of `api_requester` is able to adapt without updating their library (just their own source code).

Transparent

Since `api_requester` does attempt to abstract out the intricacies of our 3rd party API, the consumer of `api_requester` knows exactly how the underlying API works. Understanding the nuances of the API being "wrapped" might influence the consumer's system architecture positively.

Cons:

No abstraction

Without abstraction, a consumer of `api_requester` must be one with the 3rd party API's documentation. To ensure competent use, the consumer must become familiar with all possible endpoints and usage patterns. This nearly completely defeats the purpose of providing a wrapper.

Does not minimize 3rd party surface area

One of the main purposes of wrapping a 3rd party API is to minimize its overall surface area. Making small, distinct interaction points is important for the consumer of `api_requester`. Fewer moving pieces means fewer points of failure. It is probably not necessary for every single end point and function to be exposed by `api_requester`.

The Over-abstraction contraption

Unlike our api_requester, the mindset behind this pattern is: “Make sure no one can actually understand what is going on behind the scenes”.

Wrapping the same RESTful 3rd party API, api_contraption, will be our next library. Its code might be used in the following way:

require 'api_contraption'

object_wanted = APIWrapper.fetch_an_object
# => <Object @variable=:thing, @other_variable=:other_thing>

To post an object via this wrapper:

require 'api_contraption'

object_wanted = APIWrapper.fetch_an_object

object_wanted.update_object_attribute!
# => <Object @variable=:updated, @other_variable=:also_magically_updated>

As demonstrated, this library is nearly 100% magic. APIWrapper exposes arbitrary methods like fetch_an_object and returns a magical object with instance variables set.

Pros:

Actual abstraction

Consumers of `api_contraption` do not need to understand the underlying API's full functionality. The wrapper has provided (hopefully) a small handful of useful methods and classes to expose the heart of the API it wraps.

Cons:

Inflexible

Whenever the underlying API needs to change a response or request contract, the `api_contraption` has to change. This can be a very tiring exercise for its consumers. However, if the API being wrapped is very stable, this inflexibility might not be noticed as greatly.

Makes the underlying API a black box

While the `api_requester` above did not do enough abstraction, this `api_contraption` does too much. It prevents developers from discovering useful features of the 3rd party API that might be helpful to them. Granted, a consumer could refer to the service's own documentation for feature discovery but who says that they should be forced to?

So which is better?

The answer is neither. A desirable solution exists somewhere between these two examples. An API wrapper should be terse yet flexible, simple yet sophisticated. Making a consumer of your wrapper upgrade with every API change is not scalable and will drive people away from it. At the same time, if a consumer cannot see the value your library gives them, why would they bother to use it?

A good abstraction, a positive value add piece of software can be defined by a few key features:

Flexibility
Usefulness
Readability

Each of our example libraries had one or two of these features but not all three. To make up for what was lacking let’s try and make a hybrid of the two: an api_wrapper.

The middle ground

Requesting an Object

require 'api_wrapper'

object_wanted = APIWrapper::DesiredObject.find(123)
#=> <DesiredObject @id=123, @name='The One', @author='The Architect'>

Updating an object

require 'api_wrapper'

object_wanted = APIWrapper::DesiredObject.find(123)

object_wanted.update!(author: 'Mr. Smith')
#=> <DesiredObject @id=123, @name='The One', @author='Mr. Smith'>

At first glance it might not seem like all too much is different between this solution and the previous two. However, a few key differences are present.

We no longer have a generic superclass APIWrapper to interact with, it has become a namespace. With this namespace, clients can freely inherit their own objects from our DesiredObject class and make modifications as they see fit.
Abstraction is still very much in play. URL structures, HTTP payloads and other small minutia about the request is abstracted away from our consumers and they are given very logical methods find and update. These methods enable our consumer to interact with their resources in a familiar and pleasant way.

I am not saying this solution is perfect, but it is objectively better than the other two. A lot of factors go into making a great API wrapper. Adhering to the three ideal attributes: Flexibility, Usefulness and Readability will at least point your projects in the right direction.

Contributing

2015-05-24T08:00:00-04:00

Since starting this 365 day adventure, I have done a mixture of contributing to my own projects and to others. At the on-set, I made an assumption that it would be an even distribution of commits between my own projects and existing ones. However, this proved much harder than I originally thought. After a sizable amount of effort, I did find a few projects to contribute to. 28 days into this endeavor and seems like I might actually be able to accomplish my goal.

One of the existing projects, pluck to hash, was a very young project when I found it. The project was released without any test coverage so I attempted to rectify that. Thankfully my pull request was accepted! It felt great to finally contribute to a project that wasn’t my own. I also learned a few things by doing so.

The first important learning from making a contribution was understanding that the caliber of contributors to open source software was not vastly superior to “normal” software engineers. I had previously thought of people who wrote open source libraries to be some sort of battle-tested, hardened, bug-resistant demi-gods sent here from Krypton in order to write code for us mortals. This was a very debilitating thought. For years it was a wall between me and open source software development. I didn’t even entertain the idea of surpassing that wall and inevitably gave up on making meaningful contributions. Alright, that is probably a little dramatic but I think it makes my point for me. It was intimidating to contribute! Then, like a breath of fresh air, my pull request was accepted and these preconceptions vanished. It was awesome. It was super awesome (and objectively I didn’t even do that much).

A second learning from finding projects to contribute to was how to find them. It seems like an important thing to know, right? There are a few tools one can use to find projects that are ripe for the commit-in. Using the explore function on Github, you can find the top trending projects on a per language basis. This process is how I found the puck to hash gem and decided to contribute. A big difference between the pluck to hash repository and others is the fact that pluck to hash did not have any open issues. Contributing to a repository with no open issues could be considered risky. After all, if a project has open issues then at least some work is welcome and expected. A project with no open issues can not be assumed to want, need, or accept contributions. Luckily for me, the pluck to hash maintainer was open to my additions.

A great resource for finding projects ordered by issue count is CodeTriage. CodeTriage does a great job of indexing projects hosted on Github in descending order of their open issues. Finding the right projects to contribute to is hard, but with CodeTriage and the built in issue trackers in Github, contributing to repositories that matter to you and to others is achievable.

Use these resources and start making some contributions of your own.

Just learn Rails

2015-05-17T08:00:00-04:00

Obligatory YoC update: Still going strong, 21 days in a row!

The only real city I have lived in is San Francisco. I assume that in other cities, like New York and Los Angeles, people bring up topics relevant to industries popular there. So I would imagine that in New York, a person could overhear something about: banking, stock trading, fashion, and expensive brunches. While in Los Angeles, you might wander by people having conversations pertaining to: making movies and working at the Cheesecake Factory. I have come to this conclusion because everyone in San Francisco talks about writing software.

I personally love hearing conversations about software as often as I do (I did move to San Francisco for the purpose of writing it). One theme that seems to pop up often is how people can get into writing software for themselves. This is great, more people in the space means more ideas and more innovation. However, some people seem to be under the impression that breaking into software development (specifically web development) is as simple as “just learning Rails”. I hope to explain why that mentality is a little naive.

So what’s the problem with just learning Ruby on Rails? You might be able to pick up software development through learning a framework and even get a job that will pay you real money. However, I sincerely doubt that it will be a pleasant experience for long. Frameworks are a single tool a person can use to correctly and efficiently do their job.

I would wager that everyone is familiar with the type of work that a carpenter might do, but just for laughs, I’ll give you an example. A carpenter could be contracted to build a wooden table and chairs for a client. He would shape the wood, sand, treat, stain and polish it until the dining set was made. I highly doubt he would be able to do all that with just a single tool. Can you imagine the quality of his product if he only used a chisel? This is a bit of a stretch but I feel it portrays the point I am trying to make. Artisans need multiple tools to do their jobs effectively and efficiently (fyi: if you don’t call yourself a software artisan then you are missing a great branding opportunity).

In the same way that a chisel is not enough for the carpenter, a framework is not enough for our software artisan. The beauty of frameworks like Ruby on Rails is that they can get you into making a product quickly. But, as soon as you hit any kind of interesting problems, you’ll have to rely on something else. It might even be something that your framework has abstracted from you and discouraged you from learning for yourself.

For instance, Rails ships with a custom ORM: ActiveRecord. This reasonably straight forward library makes interacting with your database simple enough to start persisting objects. Great! What happens when you start persisting large amount of objects? What happens when the relationships that you have created between these objects start to cause performance losses? Your users get mad. Not great. They get so mad they don’t even know why they are using your product and they bounce. To remedy this problem, you would have to tweak your database indexes in a way that speeds up your queries for you. You’d have to know how a relational database works, how indexes work and when to use the proper ones. If you would be using postgres, it would be imperative that you understand what explain analyze can tell you. As you traverse down this path of improving your product you will be outside of the framework and thus without your only tool.

While this is just one example, I think it makes my point relatively well. We could spend page after page of other examples not given to you by your framework: JavaScript, Redis, Memcache, CSS, Git, a variety of Linux commands dealing with debugging and server maintenance, etc. And even after all of that, after you learn everything about those technologies and can make web software that scales, there are other aspects of software development. Like being on a team that either does agile, waterfall or quick stream methodology. Or how to deal with merge conflicts on a complex system, architectural design decisions, 2 spaces vs 4 (spoiler: its always 2).

The point of this post is not to discourage, I want to simply educate. Learning new tools and technologies is important and a ton of a fun! Just make sure that when you have to get our your toolbox you have more than just a chisel in there. Challenge yourselves to look outside the framework. Figure out what your frameworks are doing for you and learn why someone built it to do those things. Understanding the underlying technologies a framework utilizes is key to your success and to the community’s.

I look forward to seeing what all you software artisans build.

validates_type

2015-05-10T08:00:00-04:00

In keeping with my apparent obsession with types, I created another helpful library: validates_type. I wanted a nice, lightweight way to assert that the types of my attributes are exactly what I need them to be once I save them to the database.

No one is particularly fond of littering their code with trys and is_a?s. To remedy this, I make my models assert confidence that the data in my database is what I expect. I’m not talking about type coercion, type casting or any other munging of types that might happen on a typical read/write from your handy dandy ORM of choice. No no, this is the real deal, the bees knees, the elbows of a gazelle: actual type validation.

Data quality is important to me, and should be important to you too. This library is an attempt at injecting some structure into your typical Rails/Sinatra app in order to keep your data clean and your system happy.

Alright so when would this kind of thing be necessary? Let’s see an example of how automagic type coercion can bite us.

Let’s say you’re using Rails, and let’s pretend for a moment that for some reason, you have some code that looks like this:

A very important class:

class ImportantResource < ActiveRecord::Base
  # Attribute named :settings with what we want to be an array
  # of important things encoded and stored in either
  # a nice postres json column or a text column
  store_accessor :settings, :array_of_things
end

And a matching very important controller:

class ImportantResourceController < ApplicationController
  def create
    resource = ImportantResource.new
    # let's set the array_of_things to a parameter that is passed in
    #  via a form or something
    resource.array_of_things = params[:array_of_things]
    # great, lets save that resource
    resource.save!
    # cool everything worked, this could not have gone better
  end
end

Oh no, george! What about if something other than array is passed to that parameter??

Well let’s see what might happen. If params[:array_of_things] = 'a random string', any other method or object that needs to interact with ImportantResource would have a bad time unless they explictly validated array_of_things is an array. We don’t have any ORM coercion to fall back on here since this is a store_accessor.

Wouldn’t it be nicer to have ImportantResource validate itself so no other object needs to care about array_of_things’s array-ness?

This sounds awesome to me, and I agree. Let’s check it out and how you can harness the power for yourself. What if instead of letting ImportantResource behave in any which way, we locked that thing down? It might look something like this:

class ImportantResource < ActiveRecord::Base
  # Attribute named :settings with what we want to be an array
  # of important things encoded and stored in either
  # a nice postres json column or a text column
  store_accessor :settings, :array_of_things

  validates_type :array_of_things, :array
end

Now if we try to save any other value to array_of_things that isn’t an array, we receive validation errors added to important_resource.errors in the same way other ActiveRecord/ActiveModel classes behave! This will keep us honest with what we set our data to.

So why use this pattern at all? Some of my data might be the wrong type but who cares?

I think you should care, I sure as heck do. For instance, what if the software we’re writing is an API that people need to use? Wouldn’t you rather them be confident that your documentation (that you painstakingly wrote for hours) is correct?

For example, lets say you build a medical API: SuperImportantMedicalAPI. And in a super important response about a patient, a boolean value is returned for patient_is_allergic_to_nuts.

This totally legitimate example shows how imperative it is that SuperImportantMedicalAPI’s types be locked down. If people are depending on your awesome API to save them from nut allergies, don’t let a string like 't' be your downfall. The people need true and can handle true, give them what they want.

Great? Great! Here’s some FAQs:

Q: Do the validations behave exactly the same as ActiveModel::Validations?

A: Exactly the same.

Q: So does that mean I can chain other validations on?

A: Heck yes, check it out:

# custom messaging for errors
validates_type :array_of_things,
               :array,
               message: 'This needs to be an array!!!'

# chaining on other validation like numericality
validates :other_thing,
          type: { type: :integer,
                  numericality: { greater_than: 1 } }

# making sure object is a string and included in a specific list
validates :yet_another_attribute,
          type: { type: :string,
                  inclusion: { in: ['foo', 'bar'] } }

Q: Can I use this with ActiveRecord or ActiveModel?

A: Y to the es! Yes! Isn’t that cool? Any ruby code that uses either ActiveModel or ActiveRecord can use all the power of this gem on day

Q: Does this cost at least $1,000 every time I need to use it?

A: Close, you can use this library right now for the low low price of $0. I tried to make it lower but just couldn’t do it.

Q: Where can I read more?

A: Right freakin’ here

Ruby Boolean

2015-05-03T08:00:00-04:00

Week one of my Year of Commits initiative is complete, 51 more to go. I did not plan on telling people about this until a few weeks into the task (in case history repeated itself and I gave up 20% in). But that didn’t happen; people found out about it almost immediately, demonstrating yet again just how easy it is to find anything on the Internet. Thanks, Obama.

So what happened in the first week? Did I change the world of open source forever? Did I contribute a monumental change to one of the most well renown ruby libraries out there? Nope, I made a gem that at its core is 3 lines long. I made a boolean gem.

I can see that most people would wonder: “Why?”, “Why did he make this useless thing? Doesn’t he know that ruby does not have the boolean type by design?”. The answer is simple. I found myself over and over again typing some kind of nonsense like this:

# For some reason we need to know if this value
# is a boolean or not
#
# @boolean?
#   param: value - what we want to know is a boolean or not
#   return: true/false
def boolean?(value)
  value.is_a?(TrueClass) || value.is_a?(FalseClass)
end

This, however, could also be questioned. “Who cares if something is a boolean or not? This is ruby, duck-type that shiz”. Sure, that makes sense, I can do that. But do I want to do that? No. Why not? Because sometimes knowing the type of something can be important. You might not always have a background process like ActiveRecord’s type coercion to clean up your types for you. Or maybe you actually want something to be a boolean, not 't' or 'f'.

This gem is not alone. Other gems that do similar and much more functionality exist as well (i.e. bool and boolean)

With the ruby-boolean gem, the same kind of code can be re-written:

# Super important business logic method
def some_method_where_booleans_matter
  if thing.is_a?(Boolean)
    # Do boolean related stuff
  else
    # Throw a fit about it not being a boolean
  end
end

Great! Now I only have to check one class instead of two. My life is now perfect, what could possibly make this any better? Nothing.

This project was a fun introduction to the 365 commit-a-thon I am embarking on. Maybe someone will even use it.

A Year of Commits

2015-04-23T09:15:09-04:00

Some people have no problem being motivated to be excellent every single day. I am not one of those people. While I love what I do, I think I can do better. It is very easy to make excuses for why I have countless side projects around 20% completion. Maybe people believe me when I say “I just don’t have time right now.” or “I’ve just been slammed at work.”. Those excuses are bullshit. There are plenty of people with way more on their plate than I. That is why I want to embark on a new adventure – “A Year of Commits”.

For the next year, I will make at least 1 commit per day to a public repository on github. I am counting the commits by the day they are made, not they day they are merged in (in cases of public PRs).

Additionally, I will make 1 blog post per week. That’s 52 blog posts total counting this one (in case for some reason you don’t know how many weeks are in a year or whatever). It’s my first blog so who knows what will happen. Do people put pictures in blogs?

So who cares? Why would I go through the trouble of making this blog to exist alongside the gazillion other ones out there? Because I want to. I think it will be not only a great motivator for me but potentially provide some comic relief to any wandering interneters that happen to stumble upon this.

A Year of Commits

Why You Should Avoid Models in Rails Migrations

Humble Beginnings

A Reasonable Migration

Working with Others

Avoiding Models

Using SQL

Using Temporary Models

Problem at Scale

Making RSpec Tests More Robust

Testing an API Client

Breaking the Contract

Testing with Webmock

Stricter Settings

Keep Tests Robust

Ruby Processes and Threads - Configuring a Web Server

Threads

Processes

Tempering the Metal

index_controller.rb

config/puma.rb

Under Siege

bin/siege.sh

Metrics for Success

I/O Intensive Threads v Processes and Throughput

CPU Intensive Application

index_controller.rb

CPU Intensive Threads v Processes and Throughput

A Little Realism

index_controller.rb

Summary

Improving Remote Work as a Software Engineer

Presence

Tone

Availability

A Few RSpec Helpful Hints

subject and let Variables

Loose Expectations

1. anything

2. hash_including

3. match_array

Verifying Doubles

A Successful Year of Commits

The Plan

Reality Check

Expect the Unexpected

What I Learned

Would I do it Again?

Looking Forward

How to Diagnose Ruby on Rails N + 1 Query Problems

Setup

The N + 1 Problem

Investigation Station

The Fix

Measure Twice, Measure Again, Just Keep Measuring

Things to Consider when Metaprogramming in Ruby

Code Discovery and Readability

A Work Around:

Performance

The Setup

1. All Methods

2. method_missing

3. “Regular” Object

4. A Hash

How They Compare

Why the Slowdown?

Use the Right Tool for the Job

How to Deal with Timezones the Active Support Way

Built-in Timezones

Time.use_zone

TimeWithZone.in_time_zone

Saving Time

Five More Active Record Features You Should Be Using

1. pluck

2. transaction

3. after_commit

4. touch

5. changes

a. changed?

b. <attribute>_was

Testing with `Webmock`

`Threads`

`Processes`

`index_controller.rb`

`config/puma.rb`

`bin/siege.sh`

`index_controller.rb`

`index_controller.rb`

`subject` and `let` Variables

1. `anything`

2. `hash_including`

3. `match_array`

2. `method_missing`

4. A `Hash`

`Time.use_zone`

`TimeWithZone.in_time_zone`

1. `pluck`

2. `transaction`

3. `after_commit`

4. `touch`

5. `changes`

a. `changed?`

b. `<attribute>_was`

`Model.arel_table`

1. `<` –> `-1`

2. `=` –> `0`

3. `>` –> `1`

2. `interval`

4. Referencing Columns by `SELECT` Position

1. `class << self`

3. `private_class_method`