Categories
Programming

Announcing Nonschema Migrations — NOW FOR RAILS 6!

Nonschema migrations version 5.0.0: RubyGems Page, GitHub page

Now compatible with Rails 6, nonschema_migrations is the best way to separate your schema changes from your data changes.

Want to run no-downtime deploys with data-only migrations? No problem.

Install it in your Rails app today to see what a difference data migrations can make.

Shout out to Mikls Fazekas who hails all the way from Gyenesdis, Hungary for the pull request for this release!

Categories
Programming

Heroku with a SSL/TLS purchased from SSLRenewals – BYO Instructions

Buy yourself a “RapidSSL Certificate” at SSL Renewals. After you place your order, they may not send you an email, so log in and go to “Orders” and look for the “Generate” button

You must have access to admin@ your domain to proceed. They will send an email to verify your domain.

Generate a private key and CSR on your computer. (see instructions for mac OS or Windows)

Submit the .crt file, or certificate signing request, to SSL Nenewals. select for Web Server “Apache + MOD SSL”. Copy & paste the crt file into text box.

Use DNS authentication. It is easiest. You will need to set a TXT record on your domain name.

For “Please select the Secure Hashing Algorithm (SHA) you would like to use.” choose “SHA2-FULL-CHAIN”

Follow the instructions to create a TXT file. For the apex (root) domain, it should have no subdomain (leave blank). For a subdomain, it will instruct you to create a “domain” for a special content string.

The string is provided in what SSLRenewals calls “Alias Name”. Copy & paste this into into the “content” setting of your new TXT record (do this in your DNS manager)

At the end of the process, go to the SSL Renewals tab, find your order, and find the “Download Certificates” button. You will receive a folder called Certs-12345678

In it there will be several files:

CACertificate-INTERMEDIATE.cer
CACertificate-ROOT.cer
PKCS7.p7b
ServerCertificate.cer

You want to open in a text editor these three files: CACertificate-INTERMEDIATE.cer, CACertificate-ROOT.cer, ServerCertificate.cer

create a new file that contains the concatenated version of these three files, in this order:

ServerCertificate.cer
CACertificate-INTERMEDIATE.cer
CACertificate-ROOT.cer

Name your new file “-CHAINED” to distinguish it from the others.

Log into the Heroku web interface. Make sure you have deployed at least once.

On the command line, remove the automatically added Heroku SSL using

heroku certs:remove -a your-app-name

Verify with
heroku certs -a your-app-name

Back in the Heroku web interface, go to Settings > Configure SSL.

On the first screen, the “public certificate” is the new concatenated file you created. On the second screen the “private key” is the key you generated when you created the key & CSR.

(Reminder: never share your private key except with your web server provider.)

Categories
Programming

Amazon RDS on Heroku Still Works

Although Heroku has put a lot of attention into their own Postgres-based datbase architecture, Heroku Data, this article still works for rebels who want to use Amazon RDS on Heroku:

https://devcenter.heroku.com/articles/amazon-rds

Note that I can’t seem to get “sslca” to work as a config option in my database.yml file (in Rails), I get this I get an error after deploying, upon connection

SSL_CTX_set_default_verify_paths failed

This happens when configuring by database.yml, which I shouldn’t be doing anyway.

Nonetheless, the configuration by DATABASE_URL (which is preferred for security anyway), does actually work.

Although it continues to work in 2018, this may not be good long-term strategy as my last conversations with Heroku support led me to conclude that their failover policies for catastrophic failures aren’t set up for an RDS- backed solution unless you go into Heroku Private Spaces, which you need to be an enterprise client for.

Categories
Programming

The Five Secret Heroku Slug Cleaning Commands

This is to fix build, slug, and caching problems related to asset compilation during slug compilation.

Everyone needs a little spring cleaning, right? No, really. Sometimes when I switch around buildpacks or change up assets I run into a strange asset cache problem. Here’s the five magic commands to purge your slug compile process, from less-invasive to more-invasive. When you have a nasty asset cache problem, usually #4 is the one you need.

To use the Heroku repo plugin, I think you’ll want to install it following these instructions.
Note that the Heroku plugins are installed LOCALLY on your machine not on your environment. They are simply shell scripts that perform a series of actions on a remote Heroku environment.

Use with caution. After each one, re-push your branch to Heroku.

1.

heroku run rake tmp:cache:clear -a appname

(then push your branch to Heroku)

2.

heroku run rake assets:clobber -a appname

(Rails 4+, for Rails < 3, use heroku run rake assets:clean instead)
(then push your branch to Heroku)

3.

heroku repo:gc -a appname

This will run a git gc -agressive against the applications repo.

(then push your branch to Heroku)

4.

heroku repo:purge_cache -a appname

This will delete the contents of the build cache stored in the repository. This is done inside a run process on the application.
(then push your branch to Heroku)

5.

heroku repo:reset -a appname

This will fully empty the remote repository.
(then push your branch to Heroku)

Categories
Programming

How to get your Heroku app to optimize your images for you automagically using Paperclip, image_optim and special buildpacks

Site speed can be said to be the number one issue facing web developers today.

Whether it’s this KISS Metrics block post, another KISS Metrics block post, study after study show that delivering your content fast, fast, fast is make-or-break factor in today’s web economony. That’s why it’s so important that your images are optimized for the web.

Photoshop and other tools export notoriously large files — well over 1 MB. This is unacceptable in today’s world, where 33% of mobile users in the US are on 3G connections.

If you’re on Rails using Paperlcip, I’ve got a great solution to explore for you today: Image-Optim. You can automagically compress all your images, inside the Rails pipeline and also the ones you upload with Paperclip. On Heroku, you’ll need to use two special buildpacks to make this work. As well, because Heroku uses an ephemeral file system, Paperclip needs to be configured to use an AWS bucket as its storage.

Before continuing, figure out which Heroku stack you are on. For most Rails apps, that will either be the older Cedar 14 (released in 2014) or the newer Heroku 16 (released in 2016) stack. In all subsequent examples “your-app-name” means the actual name of your Heroku app.

To find out, use heroku stack -a your-app-name

If you’re on Heroku 16, for example, you’ll see this, with “heroku-16” appearing with an asterisks and in green:

=== your-app-name Available Stacks
* heroku-16
 cedar-14
 container

There are slightly different buildpacks for the different stacks so be sure to confirm which one your app is on before continuing.

Cedar 14

First, refer to my blog post from last year, about how to add the ImageMagick buildpack to your Cedar-14 herokubuikd.

The instructions above will direct you to do add this buildpack first:

heroku buildpacks:add -i 1 https://github.com/jasonfb/heroku-buildpack-cedar14-imagemagick704

Then add another buildpack to your Heroku environment

heroku buildpacks:add -i 2 https://github.com/bobbus/image-optim-buildpack

(you’ll note here you are using the index flag to put this buildpack into position 2 because you already should have the Imaegmagick buildpack at position 1)

You should now have 3 buildpacks, which can be check with heroku buildpacks like so:

$ heroku buildpacks -a your-heroku-app
=== your-heroku-app Buildpack URLs
1. https://github.com/jasonfb/heroku-buildpack-cedar14-imagemagick704
2. https://github.com/bobbus/image-optim-buildpack
3. heroku/ruby

Heroku 16 (Updated 4/22/2018)

For Heroku 16 the current working solution is slightly different.

Please note here you are counter-intuitively pushing both buildpacks into “position #1” of your buildpacks. Since they are ordered (and the order matters), Heroku will bump the first one you push into position #2 when you push the second one. Needless to say, run them in exactly this order.

heroku buildpacks:add -i 1 https://github.com/ello/heroku-buildpack-imagemagick -a your-appname

and

heroku buildpacks:add -i 1 https://github.com/weibeld/heroku-buildpack-graphviz -a your-app-name

You should see a message like this:

Buildpack added. Next release on <your app name> will use:
1. https://github.com/weibeld/heroku-buildpack-graphviz
2. https://github.com/ello/heroku-buildpack-imagemagick
3. heroku/ruby

If you mess it up, use heroku buildpacks:remove -i X (where X is the position # you want to remove to remove the errant ones. The heroku/ruby is there by default.)

Gemfile setup
Then add to your Gemfile these 4 gems, (for the sake of this post I will assume you already have gem ‘paperclip’ in your Gemfile).

gem ‘paperclip-optimizer’
gem ‘image_optim’
gem ‘image_optim_rails’
gem ‘image_optim_pack’

To get this working on Heroku, you’ll actually need to work through a few more steps: database setup, AWS. For the lazy, check out the example which you can find at the end of the blog post.

Here’s my has_attached_file. In this example, I’m creating only two styles: a thumbnail, and an optimized version.

Notice that I’ve turned off lossless compression, in other words, allow_lossy: true

With this safeguard on (allow_lossy: false, which is default), I’m usually able to only get an image down to about 75% of its original size.

A large 909KB file was only reduced down to 730 KB; whereas Optimizilla was able to get it down to a whopping 189 KB.

With the safety guard switched off allow_lossy: true, I get much better results but much worse quality.

1st Example
Here, I define a thumb and a optimized.

has_attached_file :attachment, {
 styles: {
  :thumb => ‘125×100>’,
  :optimized => ‘%’
 },
 processors: [:thumbnail, :paperclip_optimizer],
 paperclip_optimizer: {
  nice: 19,
  jpegoptim: { strip: :all, max_quality: 10, allow_lossy: true },
  jpegrecompress: {quality: 1},
  jpegtran: {progressive: true},
  optipng: { level: 2 },
  pngout: { strategy: 1}
 },
 convert_options: { :all => ‘-auto-orient +profile “exif”‘ },
 s3_headers: { ‘Cache-Control’ => ‘max-age=31536000’}
}

2nd Example
Here, I define a thumb and a large.

Remember, when configured together the whole thing looks like this, see the “Per style setting” on this paperlclip-optimizer doc:

(this is an example that mimics the paperclip-optimizer docs)

 has_attached_file :avatar,
          processors: [:thumbnail, :paperclip_optimizer],
          paperclip_optimizer: {

          },
          styles: {
           thumb: { geometry: “100×100>” },
           large: {
            geometry: “%”,
            paperclip_optimizer: {
             paperclip_optimizer:
{
              jpegrecompress: { allow_lossy: true, quality: 4}},
              jpegoptim: { allow_lossy: true, strip: :all, max_quality: 75 }
            }
           }
          }

The Magic Sauce

The docs say you should have allow_lossy set to its default, which is is false. Using this setting this way means your images come out with no quality loss. In my tests, I’ve found that this setting should be turned on, overriding the default.

I recommend paying attention to two important settings
jpegoptim max_quality – 0 through 4, with 4 being best quality
jpegrecompress quality – 0 through 100%, with 100% being best quality

In my tests, I’ve found that the following are acceptable for production websites with high-quality images.

Option A
jpegoptim max_quality quality: 4; jpegrecompress quality: 80
this yields 20-40% compress images of the uncompressed JPGS

Option B
jpegoptim max_quality quality: 3; jpegrecompress quality: 60
this yields 10-20% compress images of the uncompressed JPGS

As far as I can tell, jpegoptim max_quality setting appears to have very little effect on the file size, where as the jpegrecompress quality setting has the most dramatic effect, especially on larger files. The values for jpegrecompress quality are 0-4, with 0 being the least quality (most savings) and 4 being the best quality. With a settle of 4, you can’t perceive any quality loss, but you don’t get the benefit of extremely optimized files. I recommend a setting of 3, which is barely noticeable in terms of quality loss but a significant boost in file size.

Test App

I threw together a test demo here. You can read the source of this demo app on Github.

Please note this Heroku (production) app is configured with a few extra goodies:

AWS setup for a basic Amazon S3 bucket
Postgres setup for Heroku

The jpegoptim max_quality and the jpegrecompress max_quality

You’ll notice my example app here creates 5 different versions, using the same jpegoptim setting (jpegoptim: { allow_lossy: true, strip: :all, max_quality: 75 }, but 5 different quality settings on the jpegrecompress setting (be sure to note the jpegrecompress takes a quality parameter of 0-4; the jpegoptim setting takes a max_quality setting of 0-100)

In my example app I’ve split the settings for jpegrecompress and jpegoptim into a global setting and a per-style setting. Its setup differs from the examples above.

In my sample app, I’ve set the jpegoptim max_quality setting to 75 and created five different jpegrecompress settings: 0, 1, 2, 3, and 4, named:

optimized_compress_0
optimized_compress_1
optimized_compress_2
optimized_compress_3
optimized_compress_4

(you’ll see these in the has_attached_file in app/models/asset.rb)

So go ahead, upload a color-rich un-optimized image. In my experiments, I found that quality settings 4, 3, 2, and 1 yield approximately the same file size, with only a small dip in file size when you went down to 0.

However, the noticeable loss in quality begins to happen even at quality setting 3, so it seems to me why not use quality setting 4. You will be baking in an automatic guard against very large un-optimized images coming into your app. You’ll need to play around with these two settings.

Important Addendum (2017-03-09)

I am adding an important addendum to this post. After switching around my buildpacks on Heroku, I ran into a strange Sprockets error:

undefined method `dependency_digest’ for #<Sprockets::StaticAsset:0x007fefb93d0d28>

The only way I found to fix this was to purge my assets in slug compilation. This will mean your 1st push after purging will take an extra long time to slug compile.

If you run into that error, do this before you push to your environment:

heroku repo:purge_cache -a appname

Also see this Stack overflow post. I corresponded with the maintainer of Sprockets regarding this issue, and he suggested later versions of Sprockets may have addressed this issue (we are on Rails 4.1 with Sprockets 2.12.4).

Update 08/12/2019

From the PaperclipOptimizer installation:

PaperclipOptimizer uses image_optim to do the heavy lifting. image_optim automatically inserts itself
into the asset pipeline and tries to compress your /app/assets/images as well. By default, it enables
all the optimization libraries it supports, and it will fail if you do not have all of them installed.

Either add

  config.assets.image_optim = false
  
to your config/application.rb to disable this, or check https://github.com/toy/image_optim#from-rails
for how to configure this properly.

Categories
Programming

How to trim down your Heroku Slug

First you need to know how to connect to your Heroku application using bash. You can then use the du -h command to get a read on how big the files inside your slug are.

heroku run bash -a yourappname

Once you’ve done that, here’s the magic command that will show you how big each folder in your slug is:

du -h --max-depth=1

Max-depth of 1 will give you 1 level deep output of how big each of those root folders are. You can change the max-depth to see more granular introspection of each folder. (However, realistically, it is probably more useful to cd down to some subfolders and run du -h --max-depth=1 again.)

Some observations I have made:

1. Obviously you basically want to get as much out of your assets/ folder as you possibly can. Almost everything I do these days is tied to some kind of managed content, so I can safely put almost all the images and videos in my app into a Paperclip-backed data model. Using S3, Paperclip stores all the different sized versions it will render remotely, so all that stuff stays out of the slug.

2. Beware of Gem bloat. If you don’t use it, you probably want to get it out of there.

3. Generally speaking just keep large files out of your app. If you’re needing a lot of larger data files re-think your architecture to work in a more distributed, background job kind of way that doesn’t require the files to be in the slug. (Remember, worker dynos can always download from or upload to S3 and storage on S3 is much cheaper than keeping things in your slug.)

4. Use .slugignore (well documented so I won’t go into it here.)

5. I found that using gems directly from Rubygems seems to save a lot of space compared to using gems which point to a Github repo.

You can recognize a gem pulled from its Rubygems version because it usually has few or no parameters, or only a version parameter:

(pulling from Rubygems)

gem ‘spree’, ‘2.1.12’

It turns out that because Spree can be a little lazy about building the Gems, they recommend you just point your Gemfile to the github repo. (To be fair, they maintain a lot of bugfixes across several branches, and pointing to a branch on Github makes much more sense if you have software that is retroactively fixed like Spree)

(Pointing to a github repo)

gem ‘spree’, git: ‘https://github.com/spree/spree.git’, branch: ‘2-1-stable’

The difference here is that Bundler can just pull a compiled gemfile in the first example, whereas it has to download the repo and turn it into a Gem in the second example. In my app, the latter (pulling from Rubygems) saves me a whopping 70 MB compressed in my slug. Although I haven’t confirmed, I suspect this is because Bundler is actually very efficient at packaging up a Gem and excludes all the support files you don’t want when you create the gem, whereas the ‘lazy’ way of just pointing to a Git repo has the side-effect of including all the files in the repo in your finished slug.

Categories
Programming

Heroku Why Oh Why?!?

Heroku you now have SIX different versions of top-level navigation across your website(s)

Who in the world told you that was a good idea?

Categories
Programming

How can I create an inline WYSIWYG editor where I can attach Images to the content I’m editing?

[This is an excerpt from an email I answered on the Rails-Talk list today]

That’s actually pretty hard, but with a bit of work you will be able to pull that off.

You have two basic strategies:

1) Make the user upload the image file somewhere else (like amazon S3 bucket) using their own client (like an FTP client), then copy & paste the URL into your WYSIWYG editor (wrapped inside an image tag) – most editors let you do that easily.

2) provide an interface for the user to upload the image from their computer to your website (I see now why you were going down the paperclip route). Then make some kind of interface that allows them to drop a string-style reference to the image into the WYSIWYG editor. In the app I’m currently working on, we have a special macro inside the editor so you use something like this:

[IMAGE:123]

When output, this macro actually replaces the block of text with the image with id 123. (While editing, the editor doesn’t actually see the image, they only see the macro)

If you’re on Heroku, or designing for scale, you have some special considerations when creating a web app that accepts large file uploads. Check out this article here which explains how it is done.

In particular, see the note on this page that says:

Large files uploads in single-threaded, non-evented environments (such as Rails) block your applications web dynos and can cause request timeouts and H11, H12 errors. For files larger than 4mb the direct upload method should be used instead.

If your images files are large (they say larger than 4 MB, but I would even say larger than 500K), you need to do direct upload to S3. This is documented here.

As you can see, this is actually a complicated can of worms (which is why you should strongly consider if option #1 above is better for you since it is much easier and quicker to implement)

You could probably write an uploader using method #2 described above that works with TinyMCE and inserts some kind of high-level macro or the actual image tag using javascript. But you definitely would have to get your hands dirty with javascript.

If you want to go with Method #2, I strongly recommend that you DO NOT do pass-through uploading on Heroku. Although it will work for very small files, at scale you will create long running-request bottlenecks that will affect other users of your app – people who aren’t even using the upload tool will see slow performance. The s3_direct_upload gem (below) is one solution to this problem (it is an implementation of what the Heroku article discusses when it says “Direct upload”)

see also:
Direct Uploads to S3 with Rails & Paperclip
s3_direct_upload gem

As an alternative do “building this yourself,” you might consider this Rails engine-style gem calls Comfortable Mexican Sofa, which serve as a fully-fledged content management system (or “CMS,” you will take notice its cheeky name)
Comfortable Mexican Sofa

Categories
Programming

Pushing to Heroku

What you may miss off-the-bat is that Heroku’s copy of your app is actually a git repository. It’s like github’s repository of your code, but it is used for the special purpose of deploying the app.

When you do a deploy to Heroku, you are pushing only your latest commits to the master branch of the Heroku repository for your app.

Type more .git/config at the command line of one of your apps, you will see a git remote for the heroku repository

[remote “heroku_abc_app”]
  url = git@heroku.com:abc-app.git
  fetch = +refs/heads/*:refs/remotes/heroku/*

In my case, the name of my heroku app is abc-app (made up) but I’ve attached it to a git identifier (just something I type at the command line) called heroku_abc_app

To deploy this app, I would use:

git push heroku_abc_app master:master

This is saying I want to put my local master branch (the first “master”) to the master branch of the remote repository (the second “master” after the colon) to the repository which is identified by heroku_abc_app (defined in .git/config), in my case that repository happens to be at git@heroku.com:abc-app.git

You can also push a different branch to Heroku, like this

git push heroku_abc_app some_branch:master

This is saying you want to push some_branch onto the master branch of your your app. Be careful – if you push a branch and then try to push another branch (or master) onto a non-downstream git timeline, you will get rejected. You can easily fix this with --force at the end of your command.

Although Heroku’s git repository acts just like a real git repository, most of us use github as the authoritative source for our app’s code and the Heroku git setup only for deploying.

Categories
Programming

Heroku requires Postgresql, gem ‘pg’

Ran into an annoying chicken/egg involving deploying to Heroku with a shared database. It seems that Heroku isn’t happy unless you have gem ‘pg’ in your Gemfile. If you are using the shared database and fail to have gem ‘pg’ in your Gemfile, you app actually deploys OK on Heroku but then generates an Application Error when you call a page. If you check the heroku logs you see something like this: