Categories
Programming

Common Core JS Quick Setup

Common Core JS am pleased to announce a rapid application development tool for Rails: It’s called “Common Core JS.”

The source code is on Github. When you install it, the gem is pulled from RubyGems.

The finished result of the example app we will build today can be seen here. (I recommend you type out each step and only refer to this if you need to.)

It’s based on the idea that you will embrace a common naming convention for your Rails app, will use AJAX rendered-everything (that means all the forms are remote: true), and want a common set of tools for rolling out a quick, dashboard-like app in Rails.

It comes with poor-man’s authentication built-in. Poor man’s auth is safe and works fine, but it isn’t designed for you to grow your entire app on and you should graduate out of if you have any granularity to your access control.

Common Core is a fantastic tool for rapidly building prototypes. It can also be used to create API back-ends for Javascript-heavy apps.

It is a blunt instrument and you should wield it carefully. In particular, you’ll want to remove specific code that it creates because it can create situations where users may have the access they shouldn’t.

Quick Setup

Let’s build the fastest app you can possibly build in Rails.

I’ll call it MyGreatApp, but yours can be named anything of coruse.

rails new MyGreatApp

Once you have the app generated (and even on faster processors Rails new still takes a hot minute or two), go to modify the Gemfile

Add the Gem

gem 'common_core_js'

To your Gemfile.

Then run: bundle install

Setup The Common Core

Then run the Common core install generator (which is implemented as a generator, not a rake task)

bundle exec rails generate common_core:install

Now you’ll create a User objeect, with a migration and model.

do this using. Note that you should not add any fields that will conflict with Devise fields, like email (I’ll add those in the next step).

bundle exec rails generate model User name:string joined_date:date timezone:integer

Add Devise

Add devise to your Gemfile

gem 'devise'

run bundle install, then install devise on the User model:

rails generate devise:install
rails generate devise User

Take a look at the fields on your User database table now. (Here I’m using DB Browser for SQLite)

Notice the fields added name, joined_date, and timezone I’ve put a red box around and the devise fields, including email and encrypted_password, I’ve put a blue box around.

Now you have the most bare-bones Rails app with Devise gem installed. This is great because you get free sign up, login and forgot password fundionality immediately.

Go to

http://127.0.0.1:3000/users/sign_up

Add jQuery

Add jQuery to your package.json via yarn

yarn add jquery

Go to config/webpack/environment.js and add this code in between the existing two lines

const webpack = require('webpack')
environment.plugins.prepend('Provide',
  new webpack.ProvidePlugin({
    $: 'jquery/src/jquery',
    jQuery: 'jquery/src/jquery'
  })
)

The complete environment.js file looks like so (the part you are adding is shown in red.)

const { environment } = require('@rails/webpacker')

const webpack = require('webpack')
environment.plugins.prepend('Provide',
  new webpack.ProvidePlugin({
    $: 'jquery/src/jquery',
    jQuery: 'jquery/src/jquery'
  })
)

module.exports = environment

Add require("jquery") to your app/javascript/packs/application.js file so it looks like so.

While we are here, let’s also add the common_core javascript too using require("common_core")

The change code is shown in red.

require("@rails/ujs").start()
require("turbolinks").start()
require("@rails/activestorage").start()
require("channels")
require("jquery")
require("common_core")

Adding Bootstrap

Next let’s add Boostrap to the Gemfile

gem 'bootstrap', '~> 4'
gem 'font-awesome-rails'

Next delete app/assets/stylesheets/application.css

And replace it completely with a new file

app/assets/stylesheets/application.scss

(Do not save the old application.css file or rename and append to it; do not include the contents from the old file in the new .scss file)

@import 'bootstrap';
@import 'font-awesome';
@import 'common_core';

Test it

Now go ahead and start your rails server with bundle exec rails server

Go to /users/sign_up and create a new user account (be sure to enter the same password twice)

You will then be redirected to the Home page, which is now the Ruby on Rails “Yay you’re on Rails.” That’s fine— go ahead and customize your root URL.

Today’s Example App

I’ll make a super-simple system today where Users have many Events An Event belongs to a Format, which in our fictional world is only two choices: Zoom or Outdoor.

Events have a name, a starting datetime, and an ending datetime, and a “publicize” date on which should be before the starting time.

The two datetime fields (starting_at and ending_at) and the date field (publicize_on) can be empty (nil), but if set they are enforced to be: starting_at must be before ending_at. publicize_on must be before starting_at.

An Event will belong_to a Format (class & table). Formats will have a name field and only two records: Zoom and Outdoor. (We can assume they will be id 1 and id 2 in the formats table.)

All Events must belong_to a format, and when we create or edit an Event we can switch its format, but the format cannot be blank (null).

We already have the User object, and we also already have all the login, logout forgot password, and more provided by Devise.

We want the users to log-in and go to a dashboard of their own events.

They should be able to create, edit, & delete their own events with only the validations discussed above.

They should not be able to edit or create events belonging to other users, even by hacking the query parameters.

Finally, the user should be able to edit their own name and timezone, but not any other user’s name or timezone.

Because we want to name our routes from the perspective of the context-interaction, we’ll namespace the controller to Dashboard:: in our Ruby code and /dashboard in the URL. The controller will be at controllers/dashboard and the views will be at views/dashboard

Make the models

Since you already made the User model in the devise setup, let’s go ahead and create the Events and Formats tables.

run

bundle exec rails generate model Event user_id:integer name:string start_at:datetime end_at:datetime promote_on:date description:string format_id:integer

Then open up the migration file and ed the description line, adding a larger than 256 limit, like 400

Next run bundle exec rake db:migrate to create the table.

Before I go further, let’s edit our models just a bit.

Open models/user.rb and add has_many :events

also add validates_presence_of :name

class User < ApplicationRecord
  # Include default devise modules. Others available are:
  # :confirmable, :lockable, :timeoutable, :trackable and :omniauthable
  devise :database_authenticatable, :registerable,
         :recoverable, :rememberable, :validatable

  has_many :events
  validates_presence_of :name
end

Likewise, on the Event object, defined models/event.rb, you’ll need to add the reflexive relationship for belongs_to :user and belongs_to :format

class Event < ApplicationRecord
  belongs_to :user
  belongs_to :format
end

Now make a formats table

bundle exec rails generate model Format name:string

      invoke  active_record
      create    db/migrate/20200808233939_create_formats.rb
      create    app/models/format.rb
      invoke    test_unit
      create      test/models/format_test.rb
      create      test/fixtures/formats.yml

Modify the migration file to create to dummy Formats, adding this to after end of the create_table block

Because the COVID quarantine prohibits indoor events during 2020, we want to make only two format records: “Zoom” and “Outdoor” events.

Format.create(name: "Outdoor")
Format.create(name: "Zoom")

Your edited migration looks like

class CreateFormats < ActiveRecord::Migration[6.0]
  def change
    create_table :formats do |t|
      t.string :name
      t.timestamps
    end

    Format.create(name: "Outdoor")
    Format.create(name: "Zoom")
  end
end

Then run the migration itself, which will now make the table & the two format records.

bundle exec rails db:migrate

====================================
-- create_table(:formats)
   -> 0.0025s
== 20200808233939 CreateFormats: migrated (0.0027s) ===========================

Now we have two Formats in our database.

Localized Timezone Support

In order to show dates, we use a localized date display: the date & time is always shown to the user in their own date. To do this you have a few choices: (1) You can save the timezone to the user’s table, and let them set it for themselves, (2) You can show everybody the server’s date

Option #1 – Store timezone On the User object

We already took care of this by adding timezone to our User object.

Option #2 – Use the Server’s Timezone

If the auth object (current_user) does not respond to timezone, the Rails “system clock” will be used. The system clock’s timezone as set by the Rails app is used. This is often the timezone of the headquarter’s of the company that owns the application. (That is, if you do not know the user’s context, you simply use your own company’s context instead.)

Make the Controller, Views & Specs

Next we’re going to do the thing. We’ll make two controllers: Dashboard::EventsController for editing events and Dashboard::UsersController for the user editing their own name.

First, let’s create the Events Controller

rails generate common_core:scaffold Event namespace=dashboard --with-index

A few things to note

  1. Use the ‘generate’ command, not a rake task.
  2. When passing the model name, pass it in the singular form
  3. Here I’ve provided the namespace= with a value of dashboard. You will this is important to how our code comes out.

Here is the heart & soul of the common core: 5 .js.erb files, 5 .haml files, a controller, and a controller spec. Also along for the ride came controllers/dashboard/base_controller.rb as well as views/dashboard/_errors.haml, and layouts/_flash_notices.haml

Take a peak through all the generated code now.

Pay particular attention to _line.haml and _form.haml. You will note _form.haml conveniently is used for both the new/create actions and also for the update action, unifying the layout of your record across CRUD.

You can use both to customize your app quickly and easily.

One more quick step, add this to your routes.rb file.

Make sure to nest the :events route within the :dashboard namespace, as shown here. If you aren’t familiar with namespacing in Rails, check out this blog post on my other blog, The Rails Coach.

Rails.application.routes.draw do
  devise_for :users

  namespace :dashboard do
    resources :events
  end
end

Start your server with

bundle exec rails server

If the Common Core finds null for timezone on your User object, it will default to either (1) whatever is set for your Rails app in either application.rb or an environment file, or, if don’t have this set (2) the clock timezone of the server that is running your Ruby application.

It’s generally a good idea to set your Rails app timezone to the same timezone of your company’s headquarters, and then don’t change it, because that way if your server happens to move from one timezone to another (for example, you migrate from a server on the East coast to the West coast), your app will be unaffected. If your company changes timezones, you can either leave the Rails app as-is or change it, but be sure to note any place where your default timezone comes through.

config.time_zone = 'Eastern Time (US & Canada)'

Done!

We can now do all of these fancy thigns.

Create an event. Leave name or format blank, get an error.

When you create a new event, you must give it a name and Format. Notice how if you don’t, the Rails-side logic will return the form shown with the erroneous fields marked in red.

Edit an Event

Your model-level validations — name and format as required — are enforced in the update action as well.

Deleting An Event

Adding Validation

Add this to your Event class in app/models/event.rb

validate :start_at_before_end_at, if: -> {!start_at.nil? && !end_at.nil?}

def start_at_before_end_at
  if end_at < start_at
    errors.add(:start_at, "can't be after end at")
    errors.add(:end_at, "can't be before start at")
  end
end

*Validation magic*

Finally, to add validation on all the date fields, here’s our completed Event model

class Event < ApplicationRecord

  belongs_to :user
  belongs_to :format


  validates_presence_of :name

  validate :start_at_before_end_at, if: -> {!start_at.nil? && !end_at.nil?}
  validate :promote_on_before_start_at, if: -> {!promote_on.nil? && !start_at.nil?}

  def start_at_before_end_at
    if end_at < start_at
      errors.add(:start_at, "can't be after end at")
      errors.add(:end_at, "can't be before start at")
    end
  end

  def promote_on_before_start_at
    if start_at < promote_on
      errors.add(:promote_on, "can't be after start at")
    end
  end
end

Account Dashboard

Next we’re going to create the very simplest of Account Dashboards. Remember that Devise already handles log in, log out, and forgot password, meaning most of the heavy lifting of user authentication has been taken care of.

In this simple app, we want an Account dashboard that let’s us edit only two fields: name and timezone.

First let’s add the route to routes.rb

Rails.application.routes.draw do
  devise_for :users
  # For details on the DSL available within this file, see https://guides.rubyonrails.org/routing.html

  namespace :dashboard do
    resources :events
    resources :users
  end
end

Next let’s generate some scaffold

rails generate common_core:scaffold User namespace=dashboard

We now instantaly have a very basic dashboard for the User to edit their own details

The final finishing touch here will be to make the Timezone into a drop-down.

To do this, we’ll create a non-AR model :

class UsTimezone
@@_US_TIMEZONES = {
-5 => 'Eastern',
-6 => 'Central',
-7 => 'Mountain',
-8 => 'Pacific',
-10 =>'Hawaii–Aleutian'
}
def self.all
@@_US_TIMEZONES.collect{|k,v| OpenStruct.new({label: v, value: k})}
end


def self.utc_to_name(input) # in hours
utc = input[0...-2].to_i
return @@_US_TIMEZONES[utc]
end
end

Next go into the views/dashboard/users/_form.haml and we’re going to make our first cutomization

We’re going to add this:

= f.collection_select(:timezone, UsTimezone.all, :value, :label,  {:prompt => true, value: @user.try(:timezone) }, class: 'form-control')

The full file looks like this

.row
  %div{class: "form-group col-md-4 #{'alert-danger' if user.errors.details.keys.include?(:name)}"}
    = f.text_field :name, value: @user.name, size: 256, class: 'form-control', type: ''
    %label.form-text
      Name

.row
  %div{class: "form-group col-md-4 #{'alert-danger' if user.errors.details.keys.include?(:joined_date)}"}
    = date_field_localized(f, :joined_date, @user.joined_date, 'Joined date', current_user.timezone)
.row
  %div{class: "form-group col-md-4 #{'alert-danger' if user.errors.details.keys.include?(:timezone)}"}
    = f.text_field :timezone, value: @user.timezone, size: 256, class: 'form-control', type: ''
    %label.form-text
      Timezone
.row
  %div{class: "form-group col-md-4 #{'alert-danger' if user.errors.details.keys.include?(:email)}"}
    = f.text_field :email, value: @user.email, size: 256, class: 'form-control', type: ''
    %label.form-text
      Email

First, take away the strikethrough text above. This is the text field for the timezone that we don’t want.

In its place, add the new collection_select

.row
  %div{class: "form-group col-md-4 #{'alert-danger' if user.errors.details.keys.include?(:timezone)}"}
    = f.collection_select(:timezone, UsTimezone.all, :value, :label,  {:prompt => true, value: @user.try(:timezone) }, class: 'form-control')
    %label.form-text
      Timezone

We now have a nice drop-down for our Timezone field. You can replicate this pattern for any field that you want to turn into a drop-down.

Conclusion

Common Core JS harnesses the power of many great things about Rails:

• Database migrations

• ActiveRecord assocations (has_many, belongs_to, etc)

• Scope chains for access control

• Devise for athentication

Remember, make your models first: Add limits and defaults to your database fields by modifying your migrations. Then add the relasionships between the tables using standard ActiveRercord has_many, belongs_to, and has_one.

Then build the common core scaffolding & customize the views and controllers it produces.

With these powerful tools, you can build a dashboard-like app in minutes, complete with simple interface buttons that let your users accomplish most of what they’ll need. The philosophy is that you will want this dashboard as you initially introduce people to your product. The main logic of your application will likely live more in the models, service objects, and domain layer (business logic) parts of your Rails app. For this reason, you are encouraged to customize the files only lightly. (Add some verbiage or change the CSS to customize the look & feel.)

The code you build with common core is cheap and disposable. It is not very modern, but it gets the job done. It is just “good enough” to launch a sophisticated app on, but it isn’t good enough to impress your users with a really good UI.

For that, you’ll want to throw away the front-end code and replace it with a modern JS UI like React, Vue, Ember, or Angular.

By that time, the Common core will have already helped you build (1) a prototype, (2) your business logic, and (3) possibly even some controllers you can re-use (for example, remove the format.js responses and replace them with format.json to change the controllers into a JSON-responding API controller.)

Enjoy your rapid prototyping!

Categories
Programming Tools

Remember how strftime works in Ruby? Neither do I.

For a Good Strftime

https://www.foragoodstrftime.com/

Fancy little website to help you create the stftime syntax for Ruby.

Categories
Coaching Series

FAT MODELS AND PREFERRING COMPOSITION OVER INHERITANCE

Today I’m gonna talk about Fat Models.

Fat Models are unavoidable when you are learning Rails. That’s because when they teach you MVC— that is model, view, controller— they tell you: Don’t put your business logic into your view. And then they tell you don’t put your business logic into your controllers. So where do you put your business logic?

Unfortunately, you’re sometimes taught or encouraged to put them into the models. Thus, the models get very large. And when I say large I mean hundreds and hundreds of lines of code.

This is not something you should do. Instead, you should understand your models as the bare-bones operations that read or write to your database only. Then you should learn and use the other patterns I’m going to talk about today to build what we call a business domain.

This lesson can be considered an Introduction to Domain Driven Design, which will discuss explicitly later on.

Today we’ll look is meant when we say “prefer composition over inheritance.”

Perhaps you have heard the term: “prefer composition over inheritance.” But what does it really mean understand what that means?

Well, give me a few minutes of your time and I’m going to take you in this lesson into a broad overview of 9 different patterns you can use to break out of the MVC cycle.

Fat Models Skinny Controllers

In the early days of Rails people said “fat models, skinny controllers.”

In the really early days when the apps themselves were much smaller. People only actually said “fat models skinny controllers” for a short time because then the models got unmanageably large.

Rails was pioneering in the basic concept of splitting Models (persistency logic), Views (view logic) and Controllers (display and response logic).

Notice that the descriptions I used were very intentional and specific. Most people teach the “M” in MVC as “business logic” and the “V” as “view logic or display logic.” Then they try to describe what a controller is. What’s a controller, exactly?

Hmm. Well, for Rails, in the context of a GET request, it:

1) typically prepares a query in anticipation of view being rendered (like setting up a query, may be based on search criteria)

2) Handles authentication, authorization and access control if appropriate. (Arguably, the implemented class Contrtoller doesn’t typically handle this, but the Controller object, as a concept, is where this is responsible)

3) Interacts with the Rails request layer for params, headers, etc.

That’s pretty much it. In other languages, Controllers are sometimes called ViewControllers. You shouldn’t think a Controller as really anything more than a set of code that deals with how to present stuff and respond to things.

That’s why I call it “display logic and response logic”

Breaking out of the MVC Antipattern

If you’re new to Rails or programming, the first thing you learn is MVC (model, view, controller). Then, you should unlearn it.

In domain-driven design, which really I won’t even get to until pattern #6 below, we separate out Rails models from the business domain. That is, we think the ‘M’ in the traditional Rails sense as only what is necessary to read to and write from the database. This way, we still get to use all of the good parts of ActiveRecord while working towards what is fundamentally a domain layer that doesn’t live in any of the models, views, or controllers.

That, in a nutshell, is the bird’s eye view of what I’m going to talk about today.

We might do this by creating simple methods that will instantiate these service objects when needed.

Service objects have the unique property of coming into memory when needed and then disappearing when your code finishes running on each request.

The name of the game here is to compose the other objects of multiple smaller objects that will keep the business logic in the business domain layer.

Today is a broad overview of some answers to the question “How do I deal with fat models?” We’re gonna look at

1. Classic Delegation

2. Inheritance (bad)

Then I’ll talk about the classic composition patterns, including:

3. Composition with Modules (& Using Helpers in Rails View) and Composition with Rails Concerns

(And I’ll cover what a Rails cocern is and how it adds to normal Ruby modules.)

Then I’ll cover patterns found in larger Rails apps:

4. Service Objects

Then we have the aforementioned Introduction to domain-driven design, and other “out-of-the-box” architecture concepts, like:

5. Domain Context Interaction as per James Coplien in Lean Architecture

A look the Trailblazer gem – a complete domain driven design system that separates business logic from persistence logic.

At this point, you will understand what I mean when I say we are “separating business logic from persistence logic.”

Then we’ll take a quick look at 3 more patterns:

6. The “pub-sub” pattern: Publish-Subscribe and we’ll have a quick look at a Gem called Whisper

7. The “Interactor” Pattern: Interactor and ActiveInteraction, and U Case gems – to perform complex business operations together

8. The “mutation” pattern: Mutations using a gem called (unsurprisingly) Mutations

It’s a big lesson so be sure to take your time with it. As well, I will return to some of the higher-level concepts in future courses to get more hands-on with these ideas.

Again, these nine types of abstraction are presented here without strong bias (except #2 which is generally considered bad). You should learn them all and learn how to reach for the right abstraction at the right time. I’ll have another lesson in this course about too early abstractions. For now, I’m presenting this as a very high-level overview so that new and intermediate developers understand the landscape of possibilities.

Well, in truth the sky is the limit! But in the real world, these patterns as outlined here are based decades of industry -wide working and re-working of what are commonly known as “design patterns.”

Although the classic “design patterns” in programming languages that predate Ruby are more numerous than these 9, I’ve chosen these 9 to focus on because they are a great way to teach someone who has only learned about MVC what the alternatives are.

1. Classic Delegation

Ok so the first topic is called delegation.

Delegation is simply we move logic out of a class and delegate it to another class. It is a common pattern and one of the first ones you learn. Consider for example a Thing object that can export itself to XML, JSON, or CSV

class Thing
  # no delegation — all export methods are here
  def as_csv
    #...
  end

  def as_xml
    #...
  end

  def as_json
    #...
  end
end

As our model gets “fat,” we’ll want to move those specialized methods out of it. Delegation is our first strategy.

Examine our new Thing object, and another object called Converter::Thing

class Thing
  # delegate to a converter, passing self as the object
  def converter
    Converter::Thing.new(self)
  end
end
class Converter::Thing
  attr_reader :thing
  def initialize(thing)
    @thing = thing
  end

  def as_xml
    # ...
  end

  def as_json
    # ...
  end

  def as_csv
    # ...
  end
end

Here we’re simply moving the methods out of the original object and into another object. It is important to note it is a “simple” move.

In other words, we’re just fundamentally moving code around and splitting it out into new objects. We aren’t actually changing anything fundamental about how we think about functionality and objects — we’re just changing where we think about functionality and objects.

As the complexity of your app grows, the more basic solutions (like this one, “delegation”) will only be building blocks. This is fundamentally abstraction.

Let’s move on to pattern #2.

2. Inheritance

Classical inheritance is what they teach you when learning computer science as, well, classical inheritance. Its name offically means “inherticance using classes,” but as a tounge-in-cheek joke the double entdre is now that it is “classic” as in outdated. The easiest way to describe classical inheritance is this way:

models/animal.rb

class Animal
def blood_temperture
raise "superclass must implement"
end
end

models/mammal.rb

class Mammal < Animal
  def blood_temperature
    "warm"
  end
end

models/bear.rb

class Bear < Mammal

end

Ok so what have we achieved? We can ask questions about the animal and, for example, if we want to implement a different species or genus, we would know where to implement things like: Does the animal have hair? Does it have skin or fur? How does it reproduce?

Species of animals lend themselves particularly well to the teaching about classical inheritance. It’s a great use case for teaching, but unfortunately, classical inheritance isn’t often as useful or practical in the real world.

Some may think that categorization and graphing of this complex hierarchy is the stuff of OO developers. In some ways it is and in some ways, it isn’t. In some ways, an obsession with over-categorization is what gets OO a bad rap.

Think about a developer who learns a pattern— like inheritance— and then everything they implement is done with inheritance. It’s like they keep repeating the same solution for every problem. Why? Because our brains operate in the mechanism that our brains were just operating.

That is— once you start doing something one way, you are cognitively biased to repeat the same solution to every new problem that same way. This doesn’t actually make sense and you shouldn’t be that developer.

Because more often than categorization and graphing of a complex hierarchy you as the developer are considering how and why external users— that is, an end user— come into play with the data.

Experienced OO developers say “prefer composition over inheritance” so let’s a take a look at composition.

The rest of this lesson, along with the other 29 lessons in this course is available on Teachable + a special bonus: the Rails Coach cheat sheet mailed directly to you.

Categories
Coaching Series

UNINITIALIZED CONSTANT (INSIDE OF A RAKE TASK!)

This post is part of my Stepping Up Rails: Go From Good to Great series. Get the complete series on Teachable!

So you’re inside of a Rake task and you invoke an object in your app. In a classic blunder, your Rails app tells you

Uninitialized Constant: Thingy

What?! “Where’s Thingy!” you proclaim. It can’t be. You check the spelling. You double-check Rails’ object-file name to make sure it’s the same name as the object. (This way, Rails will autoload your file — there’s a post dedicated to that coming up in this series.) In other words, it checks out.

The Rake task: the Rails’ developer’s best friend. A utility knife of utility knives— the Rake task is where you get stuff done. But unfortunately not when it can’t find your Rails objects.

You can do all kinds of things in Rake tasks.

Background jobs are the primary function of Rake tasks.

For instance, queues are run in Rake tasks.

In other words, lots of Rails operates on Rake tasks. They are the underpinning of most SAAS (Software-As-A-Service) back-ends that are written in Ruby on Rails.

Remember you must always specify to the task the symbol :environment . That’s the key, like so:

task my_task: :environment do
  # do some stuff here
end

If you fail to do this, Rails will load your task, but won’t load your environment, so it won’t have access to your models or application objects. You’ll get funny errors, like Uninitialized Constant and the like.

Always remember, load your environment in Rake tasks.

Categories
Coaching Series

Who is N+1?

This post is part of my Stepping Up Rails: Go From Good to Great series. Get the complete series on Teachable!

Ahh, the elusive N+1 query. Perhaps you’ve heard of him from your engineering pit, cursed under the breath of engineers or beshrewed out load when such an engineer encounters an endpoint (or web request) that’s slow, slow, slow. The N+1 query is very often the culprit.

See, in web development, we like things fast, fast, fast. That is, we want our webpages to load really fast. (And typically that means 200ms or faster).

Let’s say you have a database of teachers and students. Your teachers have many students.

Teachers have many students

In Rails, this might be expressed as a Rails class Teacher that has_many :students.

Let’s load the students in the HAML via the association.

This teacher teacher.name has teacher.students.count students

When viewed in the browser, it might look something like so:

Example output showing count of studetns

Finally, if you examine the Rails console, you’ll see Rails makes 3 queries: 1) to load up the teacher, 2) to count the number of students (this is because I explicitly asked for the count of students) , and 3) it the loads all of the associated students in the final query.

Example SQL output showing count query

Let me propose making this slightly more complex to demonstrate how to create an N+1 query. Remember, N+1 queries are not something we want. I’m going to show you how to create an N+1 query and then how to fix it.

Teacher Student ReportCard ERD Entity Relationship Diagram

Students have many ReportCards. In this simple example, we’re going to be querying for the last report card for each student.

We have some HAML that looks like this:

ruby source code HAML showing teacher and students output

Here I want to show an unordered list of the name of the students and the last report card for each student.

When we run this, we’ll see this in our browser:

Students list example output showing report card with N+1 query

In our console, we will see:

Example console Rails log output showing N+1 queries

Take a look at that nasty N+1 query. See how repetitive it is? For these seven records, it might seem fast now. But as your app grows, your database server will strain and struggle to keep up.

Won’t Fly With N+1 Queries

While it might look trivial now, those “ReportCard Load” lines are telling you something important: Your app won’t scale with N+1 queries.

As a result of the object relational mapping (ORM) we are doing with Active Record, we will do something called pre-fetching. That’s when we tell Rails to fetch all of the ReportCards we will need to display quickly in one single query.

As you can see from the query, without telling Rails to side-load the related data, Rails must loop through the Student results as it fetches each of the “last” report cards for the student, causing it to query 8 times, 7 times for the report cards and once for the student (thus, “n + 1”). In fact, it does it the other way around, but we call the phenomenon “n+1” and not “1+n”.

As the developer you must know N+1 Queries

Using an ActiveRecord helper method called :includes(:____) we will chain this instruction to load the associated objects onto our ActiveRecord query.

(Do not confuse this with the ruby command include [singular], which is to load a Ruby file other than the one you are executing—they are unrelated: include is Ruby and looks like this

include ‘byebug’

.includes, on the other hand (notice the dot ‘.’) comes after an Active-Relation (also known as an “AREL”), like so:

@teacher.students.includes(:report_cards)

This directive tells Active Record to smartly fetch the student data in “as few queries as possible.” You will note I’ve put this in quotation marks because Rails actually does some advanced stuff under the hood to choose one of two loading strategies: JOINing the table in the original query or pre-fetching when the data is needed in a single, composite query for all of the associated Students.

using includes(:___) to prefetch

Notice in this example that Rails makes precisely two queries to the database (and remember, each query has overhead, or a time that it takes for the webserver to communicate with the database server and for the database server to process the request): once for the students and then again for all of the report cards.

SELECT `students`.* FROM `students` WHERE `students`.`teacher_id` = 4  [["teacher_id", 4]]

  ReportCard Load (0.6ms)  SELECT `report_cards`.* FROM `report_cards` WHERE `report_cards`.`student_id` IN (22, 23, 24, 25, 26, 27, 28)

Boom! Your N+1 query is gone. Take special note that Rails has queried for the students associated with teacher_id 4 and found that they all have ids of 22, 23, 24, 25, 26, 27, and 28. Since we only need those report cards records for the seven associated students, Rails has concatenated the ids of the students in the second query, to form the WHERE clause. As in: WHEREIN (22, 23, 24, 25, 26, 27, 28)

This technique is called pre-fetching or eager loading. We’re telling Rails that we’ll need the associated report_cards on each student when we are looping through the students.

Prefetching fixes N+1 queries

The key to fixing your N+1 queries is to realize that Active Record relations are instantiated in one place and then invoked “just-in-time,” that is, typically when the view is rendered.

N+1 query is fixed in the Rails log output

Now, finally, here there one more thing to note: AR has three different methods for loading your associations: includes, eager_load, and preload. A fourth method, joins, allows you to join a table in your SQL without necessarily loading the associated records.

Includes will load all of related the records referenced as arguments in the includes method. Typically includes will load the records like you see above (in a separate query using WHERE … IN (…) syntax)

However, if you have a where or order clause that references a relationship, includes will instead us a left outer join.

Rails has two ways of dealing with N+1s: 1) “big join” and 2) “separate queries.”

.includes — uses either strategy #1 or #2 depending on whether or not there is a where or order clause that references a relationship. .eager_load always uses strategy #1 and .preload uses #2.

I recommend that you always start with .includes. When your app gets large and your queries are slow, then you will be forced to optimize, which means measure and tweak your queries to make them faster. During this process is when you will experiment with .eager_load vs. .preload and determine on a case-by-case basis which one is most optimized for you.

I hope you’ve enjoyed this brief introduction to N+1 queries.

THE EXAMPLE APP FOR THIS POST CAN BE FOUND AT

https://github.com/the-rails-coach/who-is-en-plus-one

Categories
Coaching Series

Identity Crisis

This post is part of my Stepping Up Rails: Go From Good to Great series. Get the complete series on Teachable!

Javascript developers talk about the hoist, where a reference to a variable inside of a given scope in a section of code below affects code above it, effectively hoisting the variable up into the reserved namespace. Did you know that Rails has its own hoist effect?

This little puzzle comes in an oft-confused part of Rails & Ruby. The attribute getter (attr_getter) that is made available to all Rails ActiveRecord objects automatically.

Remember that a Ruby object has ApplicationRecord as its base object (or ActiveRecord::Base before Rails 5), and ApplicationRecord makes certain things available to the object-relational mapping (ORM), which is essentially what ActiveRecord is. This ORM is what marries the Ruby object to the database object. Think of ORM like a two-way tunnel— the Ruby object needs data so it pulls it from the database, the database takes data that is updated from the Ruby object.

But POROs (Plain Old Ruby Objects) don’t actually have a database behind them. Rails objects do. That’s where the ApplicationRecord magic comes in.

The Rails Magic Loading Getter

One quirk to remember for Rails developers is that inside of Ruby ApplicationRecord object, there’s a secret (and magic) getter for any attribute in your database. Let’s say in a Person class you might have

class Person < ApplicationRecord
  def name
    first_name + " " + last_name
  end
end

This works because Rails looks to the Ruby object first for a first_name and last_name. Assuming you haven’t overridden them, Rails doesn’t find them on your Ruby object and falls through to method_missing (that’s the secret method that is called when a Ruby object doesn’t have a method object— it’s how so much Ruby magic works.)

The method_missing implementation looks for these fields on the Rails class which now comes from the ORM (or the database). So basically it’s like having an attr_reader — or a ‘getter’ — automatically for any of your database fields.

But keep in mind this doesn’t work in reverse! There is no setter field that allows you to do first_name = "John"

That’s why when you’re setting a Rails object, you must always use the self keyword followed by dot (.)

self.first_name = “John”

Getter Not Setter For the Hoist

Now, you’ve told Rails to update the ORM-backed object. Why does this distinction matter? Well, first of all first_name = "John" doesn’t actually do what you intended it to do (you probably intended it to update the object’s first name). What it does is it creates a local variable in the local scope with the name first_name, which, confusingly, has no effect on the Rails object’s field first_name.

Consider this method on an object that checks if someone can make a request. The API rate limits their requests to 10 requests in a rolling, 10-second timeframe.

def can_make_request?
  unless last_request_at.nil?
    secs_since_last_request = (Time.now - last_request_at).floor

    requests_available += secs_since_last_request
    requests_available = [10, self.requests_available].min
  end
end

The intention of this code is to update the requests available to either 10 (if the last request was more than 10 seconds ago), or to add the number of seconds since the last request to the current count, but never allow the count to exceed 10. Unfortunately, we aren’t actually saving the data back to the database.

Dropping a byebug into this method reveals a curious thing:

def can_make_request?
  unless last_request_at.nil?
    secs_since_last_request = (Time.now - last_request_at).floor
   byebug
    requests_available += secs_since_last_request
    requests_available = [10, requests_available].min
  end
end

Here, if we ask for either id or name, we get it. But if we ask for requests_available, Rails tells us the result is nil.

Notice that we haven’t even modified requests_available, and unlike id and name, this field, even when read (with the getter) comes up as nil:

4:

5:   def can_make_request?

6:     unless last_request_at.nil?

7:       secs_since_last_request = (Time.now - last_request_at).floor

8:       byebug
=>  9:       requests_available += secs_since_last_request

   10:       requests_available = [10, self.requests_available].min

   11:     end

   12:   end

   13:   

(byebug) id

1

(byebug) name

"Cruickshank Inc"

(byebug) requests_available

nil

Ruby Hoist

How does that make sense?
Well, it’s the Ruby hoist we were talking about before, and it demonstrates why Rails provides only a getter and not a setter. You see, inside of the scope of can_make_request? Ruby has already seen on line 8 that we are going to use a local variable called requests_available. When we call it on line 8, we actually get nil because Ruby has already hoisted the variable’s definition and namespace to the top of the local scope, just like in Javascript.

Here, the local variable — which hasn’t even been set yet— is returned to us instead of the Rails getter that calls through to the database. (which you can see when we call id and name).

This is easily fixed! Just remember to use self. whenever we want to save information back to the object. (You will also need to actually save the object too!)

The basic rule of thumb is:

You can get a Rails database field using the magic getter that is its own name, but if you want to save back to the database, use self.____ = instead.

Categories
Coaching Series

MATZ IS NICE AND SO WE ARE NICE

This post is part of my Stepping Up Rails: Go From Good to Great series. Get the complete series on Teachable!

It is said of Ruby’s creator Yukihero Matsumoto, ‘Matz is Nice and So Are We.’ It is this mantra about the Ruby community and programming language, the vision of Rubyists with an ethical calling towards treating each other nicely— an interdependence, of a kind, on the transitive property of niceness (that is, that nice can be passed on from one individual to another).

When we code we are nice. We treat one another with dignity and respect; we are good to ourselves; and we are good to the codebase itself. We don’t try to fit ugly code into structures. We are nice to the codebase and we are nice to us. Matz is nice.

When we make something work well, we proclaim, ‘NICE!‘ as an expression of triumph. We are nice.

Nice is about being considerate. It requires human empathy to understand the other person. Matz is nice.

Nice means we put-up other people up instead of putting them down. We are nice.

Nice is how we design a system— with “just enough” knowledge to guide our choices now, and nicely extensible for the future. Matz is nice.

Nice, to many people, means courteous and pleasant. To Rubyists, it means that but so much more. It’s a way of thinking, it’s a way of being, it’s a core value. What we put in we get out and what we get back and give back.

Before it was a Linux operating system, “Ubuntu” was a South African spiritual philosophy:

I am because you are because I am

In our inter-disciplined practice of Ruby, We are Nice Because Matz is Nice and We are Nice Because We are Nice— it is the core of our modern approach to Ruby and the Rails ecosystem of developers.

So be nice, because it will come back to you 1000 times.

Categories
Programming

Halfway to One Point Oh: UTM Version 0.5

Today I’ve finished version 0.5 of my new Gem, Universal Track Manager.

It’s a plug-and-play Rails engine that you install into your Ruby on Rails application with just three simple steps (see the README). You can then immediately pick up your visitors’:

IP address
Ad campaign where they came from
the browser they are using

In my next version, I’ll add support for http referrer and more too. Give it a try today.

If you like the Gem, please ‘star’ it on Github or download it from RubyGems (you do that just by running bundle install). Also, consider supporting it today with a small contribution today through the Github sponsors program. Sponsors levels start at just $1/month.

Categories
Programming

A First Look : Universal Track Manager

Today I’m announcing ‘a first look’ at my new Gem: Universal Track Manager. It’s an ambitious project that’s going to have nearly universal appeal and utility.

Visitors come to your site every day. Along with their visits, 4 key pieces of information come along for the ride:

— IP address
— browser name (which lets you infer operating system)
— UTMs showing if they clicked from another site, or if they came from online advertising (typically you can “auto-tag” your ad campaigns and your UTMs will be magically populated)
— Http referrer, which shows if they clicked directly from another site to your site (Even when no UTMs are set)

Universal Track Manager, a play-on-words that shares an acronym with “UTM Parameters,” is your one-stop shop to automatically pick up this information and stash it into your database. You can think of it like a built-in Google Analytics (without the fancy dashboard).

As if that weren’t ambitious enough, with a tiny bit of trickery I’m planning support for optional Viewport size (width X height), which can let you determine if the user is on a desktop or mobile browser. (coming soon)

I’m pleased to announce Version 0.0.3, the first version I’m dubbing as ‘public beta.’ Although this is production-quality code, it should be used with caution until it is no longer in BETA status. You are welcome to give it a whirl on your Rails projects today. With an easy 3-step installation into any Rails 4+ app and you can sit back and sweep up tracking info on your visitors.

*MOST* of the core functionality now works! This version 0.0.3 implements fully support for timestamping your visits, the user’s IP address and browser. Support for UTMs & HTTP referrer and more coming soon! If you are curious now’s a great time to try it out, please submit feedback via Github.

Links:

Github Repo

Rubygems page

Categories
Programming

Jason FB’s 10 Magical Ruby Developer Tools

1. deivid-rodriguez/byebug
Byebug is a fantastic debugger available for Ruby 2 (and presumably above). Drop

gem ‘byebug’

into your Rails app Gemfile and bundle install. In either your test run or your development run, write

byebug

on a single line of your app and voila. When you hit that line, you will drop into the debugger.

If you’re not developing a Rails app, you can include ‘byebug’ at the top of your Ruby file.

Full docs here.

2. pretty print (pp)

Pretty print is one of my favorite introspection weapons to help see variables more clearly.

pretty print, which you write as pp, prints out your object with each attribute on its own line. Take for example this Spree::Country object, shown here on the Rails console without pretty print

2.4.6 :010 > x
=> #<Spree::Country id: 232, iso_name: “UNITED STATES”, iso: “US”, iso3: “USA”, name: “United States”, numcode: 840, states_required: true, updated_at: “2019-05-19 17:16:07”, zipcode_required: true>

Now, with pretty print, the same object is conveniently displayed with each attribute as its own line. This is invaluably helpful when you have deep nesting of objects.

2.4.6 :009 > pp x
#<Spree::Country:0x00007fd8507ea358
id: 232,
iso_name: “UNITED STATES”,
iso: “US”,
iso3: “USA”,
name: “United States”,
numcode: 840,
states_required: true,
updated_at: Sun, 19 May 2019 17:16:07 UTC +00:00,
zipcode_required: true>

3. puts, .to_s, and inspect

OK, so we get a 3-in-1 here: When you call puts on an object, .to_s will be called and then output to your screen. So you should make your objects have a .to_s that is human readable, possibly even for use in, say, a drop-down menu or label. 

def class Person
 attr_accessor :first_name, :last_name

 def to_s
   “#{first_name} #{last_name}”
 end
end

inspect, on the other hand, is specifically for developers. In this method, you would write out as much information as you the developer (or next developer) want to see, including the keys (ids) of your objects if those will be helpful:

def class Person
 attr_accessor :first_name, :last_name

 def inspect
   “Person id: #{id} – first: #{first_name}; last: #{last_name}”
 end
end

Your objects should have both .to_s and .inspect on them, and you can try these universally named Ruby methods on other people’s objects to examine them. A well-formed codebase implements them or has helpful output for both of these.

4. .to_yaml
Pretty print’s cousin is the .to_yaml method, which will take your object and convert it into yaml. Take for example this arbitrary object, which you will notice contains a :ghi key that has a nested object as its value:

2.4.6 :023 > x= {abc: 1, def: 4, ghi: {ye: 6, nm: 3}}
=> {:abc=>1, :def=>4, :ghi=>{:ye=>6, :nm=>3}}
2.4.6 :024 > x
=> {:abc=>1, :def=>4, :ghi=>{:ye=>6, :nm=>3}}

.to_yaml on its own will produce a string that will output with newline characters, like so:

2.4.6 :025 > x.to_yaml
=> “-\n:abc: 1\n:def: 4\n:ghi:\n :ye: 6\n :nm: 3\n”

To make this more useful, try puts along with .to_yaml

2.4.6 :026 > puts x.to_yaml

:abc: 1
:def: 4
:ghi:
 :ye: 6
 :nm: 3

5. x.method(:_____).source_location

(where :_____ is name of the method — as a symbol — you are trying to search for)

OK, so the ultimate secret weapon of Ruby debugging is this little-known method that will magically — and I mean magically — tell you where a method was defined. That’s right, I mean the actual line number itself.

2.2.5 :002 > a.method(:hello)
=> #<Method: Apple(Thingy)#hello>
2.2.5 :003 > a.method(:hello).source_location
=> [“/Users/jason/Projects/nokogiri-playground/app/models/thingy.rb”, 2]

Look, ma, take a peek into my hard drive and you would find that the hello method is actually defined on the file at the full path /Users/jason/Projects/nokogiri-playground/app/models/thingy.rb on line 2.

Like magic it works for Rails and Gem code too, and is invaluable when you are ready to dive into the APIs you are working with.

6. x.methods
By default this method will return a list of all of the methods on an object. Watch out because you’ll get all the methods on the ancestor chain too.

In older versions of Ruby, you could use this method to examine the instance methods that were defined on this class only (excluding the ancestors), but unfortunately this no longer works.

If you pass this method false, like so:

x.methods(false)

…things get more interesting: then you get only the class methods defined on this object’s class itself. (Remember in Ruby class methods are defined with self.)

7. brunofacca/active-record-query-trace

An excellent gem that’s still a non-optional workhorse in my development practice – especially when debugging a legacy codebase. This gem will display for ALL of your SQL queries where in your Ruby or Gem code the active record update commands are coming from.

Follow the instructions in the Gem to create an initialize file and set it up. My only tip here that adds to the docs is that you’ll want to set the number of lines:

ActiveRecordQueryTrace.lines = 10

I find that when debugging a problem in my own Rails app I want this set to a lower number (like 5 or 10) and when debugging a problem in Gem code or in Rails I need this at a much higher number (like 50 or 100).

8. flyerhzm/bullet

Understanding N+1 queries is a significant litmus test that sets amateurs from the professionals. Bullet is like a magic bullet – literally named so – for finding your N+1 queries.

Bullet is a great gem that you should install in either development or test, not in production. Often because it does add overhead to your speed, I install it but leave it configured so that it is turned off by default and any developer on the team who needs it can turn it on.

You CAN and SHOULD turn it on periodically too, to examine where your app is producing N+1 queries.

Here’s the catch with Bullet: Remember that Active Relation objects are created as chains of conditions before they get translated and executed as SQL. That’s why when you do this you must carefully consider

query = Country.where(name: “United States”)

When you do this in your Rails console, you will see it run the SQL right away.

2.4.6 :039 > Spree::Country.where(name: “United States”)
 Spree::Country Load (0.7ms) SELECT `spree_countries`.* FROM `spree_countries` WHERE `spree_countries`.`name` = ‘United States’ LIMIT 11

It only does this because of the ‘print’ effect the Rails console has on your objects. If you string another .where onto this object, when translated into SQL, ActiveRecord will combine the queries:

query = Spree::Country.where(name: “United States”); query.where(iso: “US”);
 Spree::Country Load (1.0ms) SELECT `spree_countries`.* FROM `spree_countries` WHERE `spree_countries`.`name` = ‘United States’ AND `spree_countries`.`iso` = ‘US’ LIMIT 11

The reason this is important is that to understand where your N+1 queries are coming from you need to understand when you are creating your Active Relation objects and when they are invoked. They are not the same place, although on the Rails console it makes you think it is one and the same. When you grok this, you will see why Active Record’s side loading (which loads a related set of objects in a single optimized query, taking the number of queries from N+1 to 1+1=2) is both efficient and can be tricky to work with, especially with objects that have many relationships.

Don’t be fooled: Side-loading is nearly always more efficient than N+1 queries. 

Bullet tells you where those pesky N+1 queries are invoked, but not where you are creating them. What you then need to do is trace your code (manually) to figure out where the queries are being created, which hopefully should be near in the code to where they are being invoked ( but in the case of complex filtering logic might not be).

Here you need to add the appropriate .joins(:____) to your code anywhere between when the objects are set up and when they are invoked by Active Record. Note that you’ll only want to join in those additional tables if they are actually used. If not, you don’t need the over head.

You’ll know you’ve solved your N+1 queries because you won’t see them output in your Rails log, and they’ll disappear from the Bullet code.

9. better_errors

Since Rails 4 adopted a near identical default, this used to be more interesting. For a Rails 3 app it can bring your error crash console up to Rails 4 standards.

10. Introspect, Introspect, Introspect but remember Ruby’s last-line quirk

Always look at what you’re doing. Drop into your debugger, look at your variables, clone & freeze them, look for race conditions, look for flip-floppers, watch out for your own confirmation bias. Remember that when in the debugger or on the Rails console and you type a SINGLE VARIABLE and HIT RETURN, the console will interpret that action as-if you had called .inspect

2.2.5 :007 > a
=> #<Apple id: nil, created_at: nil, updated_at: nil>
2.2.5 :008 > a.inspect
=> “#<Apple id: nil, created_at: nil, updated_at: nil>”

In some cases, the act of inspection actually changes the object (like in the case of an Active Relation, in which case it invokes the query), so keep that in mind (we might call this the “observer effect” in software development.)