Animoto

The other day, one of our servers started failing. After examining the damage for a minute, I clicked “Relaunch” on our RightScale console. I paused to imagine how this would be handled before the cloud. I couldn’t remember, but I’m sure it was painful.

(Techie details:  Amazon had sent us a warning saying “your instance is degraded”, but by the time I got to it, /mnt was completely gone and the kernel log was full of bad block errors on /dev/sdb. It was an ephereral instance, so everything important was already in S3. RighScale’s “Relaunch” feature terminated the current server while launching a new one. Precicesly 3.2 minutes later, everything was back to normal.)

We have recently rolled out support for oEmbed on all of our video player pages.

For those unfamiliar with oEmbed, it is a standard for representing embeddable media contained on a web page. Clients can retrieve details of this content without having to parse the webpage and so forth. This means that you can easily access Animoto video’s with just a simple URL.

To see this in action I would recommend you take a look at 280Slides. Create a new presentation and select “Movies” from the toolbar, the Media Inspector should open. If you click the “Enter URL” button and enter any Animoto video’s URL , you should see it appear in your document, no clunky embed code required.

You can query our oEmbed service using both JSON and XML, and the URL’s look like this:

http://animoto.com/services/oembed?url=http://animoto.com/play/q6W7MjkrqYuiI1mZwAtaSg&format=json
http://animoto.com/services/oembed?url=http://animoto.com/play/q6W7MjkrqYuiI1mZwAtaSg&format=xml

In the response, we provide details such as the name and description of the video, and the HTML for embedding the video player.

For more information, check out our developer guide.

HAProxy is a wonderful load balancer, but it requires restarting to change the configuration.  Even the latest version doesn’t seem to allow adding and removing servers dynamically.  Unfortunately, restarting HAProxy has several bad side-effects:

• The old HAProxy process must stop listening for TCP connections, then another HAProxy process must start listening. No mater how fast your server is, there will be a short period where neither process is listening, and your OS will return a “Connection refused” error.  If your server is under heavy load, the “short period” can stretch out many 10s of milliseconds. If you have a web server like Nginx or Apache in front of HAProxy, you will see the 503s in your web server logs. If you have HAProxy listening directly on port 80, then you’ll never notice the lost connections (but your users will).
• When  HAProxy first (re)starts, it doesn’t know which backend servers are down. If some of them turn out to actually be down, you will get a few dropped or delayed requests while it figures that out.

This caused us to restart HAProxy as infrequently as we could get away with.  On the other hand, restarting our application servers was also causing bad things to happen:

• When stopping, sometimes the application server doesn’t serve the remaining requests correctly. (It’s not like normal tests cover this case.)
• HAProxy sometimes drops or delays connections while detecting that a backend is down.
• Most application servers have “start-up” anomalies: Some don’t load your code until the first request. Some (like mongrel_cluster) will try to start serving requests while still starting up (and generating high CPU load). Some will page-cache your home page or generated assets, so you want them ‘warmed up’ before letting them take requests from users.

We can fix the first problem with the magic of Linux IPTables to temporarily “remap” our TCP ports. This lets HAProxy restarts be seamless. In turn, that lets us fix the second problem: We now always “take a server out of rotation” (remove it from HAProxy configuration) before working on a it. Read the rest of this entry »

We decided to A/B test some of our emails, and it turned out this is simple in rails.  The one missing piece I had to dig a bit for was specifying which template ActionMailer uses.

(The example code is all on github)

First, you need to bucket the emails into A or B buckets.  A simple way to do this is based on the email recipient, based on their id.  Something like:

class User < ActiveRecord::Base
  # even id is A, odd is B
  def a_b
    # Note: doesn't deal with new records.
    id.even? ? :a : :b
  end
end

Then you can change the email settings or body based on the user’s A/B bucket. If you are making large changes to the email bodies, you may want to use different templates for the A/B buckets. To do this, call the template function:

class WelcomeMailer < ActionMailer::Base
  def welcome(user)
    from       'Welcome Bot '
    recipients user.email
    subject    "Welcome, my friend"
    body       :user => user
    if user.a_b == :b
      template "welcome_experimental"
    end
  end
end

Extra credit: add Google Analytics tracking to measure the performance of each mail

I found an incredibly handy trick that I’ve wanted to know for ages through Clifton Griffin’s blog here.

I oftentimes have to grep through huge log files on multiple boxes when trying to track down an issue. 99% of the time, the event will have happened in the last few days, so I don’t want to search through an entire month’s worth of information and through gigabytes of data, which takes forever and puts a heavy load on the box. Until now, I’ve just used less and then reverse searched, but this doesn’t work when I’m trying to search via capistrano on multiple boxes. I could of course just use tail -n and set some reasonable number, but that seems hacky when I just want to search backwards until I hit something.

Enter tac, the reverse cat. Now it’s as simple as:
tac production.log | grep -m1 WhatImLookingFor

I’m using -m1 to stop after the first result is found and to prevent it continuing through the file. Here’s the difference on a search I just ran.

[root@domU] time grep -m1 5tg3ifaOGMzhLCh5i0A95g production.log
real	7m4.896s

[root@domU] time tac production.log | grep -m1 5tg3ifaOGMzhLCh5i0A95g
real	0m0.107s

A 3,970-fold increase? Not bad. Of course ideally we’d be using syslog with something like Splunk, but until then this will dramatically reduce search times.

I’m proud to present FilmStrip, an experimental AS3.0 library that lets you process code-based animations to video with natural-looking motion blur. Take a look at the sample video I showed at Flash on Tap here (MP4, 800k).

Yes, some sad little red dice — quite a demo huh? But pause it and step through the frames and you’ll see some very realistic motion-blur, produced by FilmStrip. This looks quite different from a standard directional box blur.

It also takes time to render, so what you’re seeing isn’t realtime. It’s a series of Flash-generated frames that have been saved out using AIR and processed into an MP4 after the fact using a utility called FFmpeg. So if it’s not realtime, is it useful? Well… possibly. :-) A spaceship game could feature cinematic explosions that are really pre-rendered video clips. These could be quite seamlessly integrated since they’re made from the real game models, and done easily without jumping into another 3D or video program. A website or kiosk could similarly feature fast, smooth navigational swoops. And so on.

In general FilmStrip is a small, snap-on utility that is designed to be used with your existing projects. It can render Papervision3D scenes or Sprite-based scenes, and even stack various 2D and 3D scenes in a single render. FilmStrip is extensible to other 3D environments and works with any tween engine – even that one your grandpa wrote last week. ;-) Capture modes include WHOLE_SCENE  for a fast snapshot series, EACH_OBJECT to generate unique blurs based on actual motion, and blur modes include NONE, MATTE_SUBFRAMES and SPLIT_SUBFRAMES — plus a bunch more blur settings for fine tuning. Toss in some animated Pixel Bender and BitmapFilters and you might be doing some fairly decent post-processing, right out of ActionScript!

Filmstrip and accompanying project filmstripAIR (frame dumper & playback utilities) are available here:
http://github.com/animoto/
http://github.com/animoto/filmstrip/
http://github.com/animoto/filmstripAIR/

This is by no means a finished product! It’s Open Source MIT licensed, and you’re fully encouraged to play with the code freely. Please spread the word about FilmStrip and post your feedback.

Many heartfelt thanks to everyone who’s supported this project along the way!

The Flash on Tap conference postponed last year to the end of this month, resulting in a tizzy of topic updates. Sliding through the closing gates with just moments to spare before they went to press, my topic ended up getting an exciting upgrade. This was due to the inclusion of a surprise guest, and Animoto’s approval to show off a little of the R&D I’ve been doing for them. Here it is:

Proof of Concepts: Flash Touch Tables + Compositing

This session will cover two thrillingly uncharted directions for developers: DIY Flash touch tables, and a new system for exporting an animated Papervision3D scene as video.

Todd Greco and Robert Lewis will explain how the Portland studio Fashionbuddha has begun building their own multi-touch table units and then programming exciting new tools for them in Flash, and how you can do the same. Their Aquarium app generates creatures, then lets you synthesize their DNA to produce new hybrid monsters.

Moses Gunesch of Animoto.com has devised a way to render a Papervision3D scene to frames of video, including natural-looking motion-blur! This new technique could be used to pre-render stunning seamless transitions within an interactive experience, or to composite and render Flash-generated videos from AIR apps.

Hope to see you all at Flash on Tap — the first Flash conference that officially sanctions beer as its ulterior agenda! ;-)

We’ve been pretty dogmatically RESTful since our API was first written–not that it was too difficult considering Rails’ built-in proclivity to it. However, I recently confronted a issue with REST that has split some devs out there, and I’d love an opinion:

What is the best resource URI for multiple but discrete resources?

For instance, I am working with our Render objects (incidentally, for the Rails people out there, that naming has surprising _not_ come back to haunt us), and the RESTful endpoints for this resource are as follows:

/renders [GET, the collection method "index"]
/renders [POST, the collection method "create"]
/renders/1 [GET, the member method "show"]
/renders/1 [PUT, the member method "update"]
/renders/1 [DELETE, the member method "destroy"]

Any developer familiar with REST paradigms will immediately recognize those. However, what if I want to load more than one render object in a single query, but I want to use individual IDs and instead of getting a complete index listing?

Two ideas sprung (sprang?) to mind:

/renders?ids=1,2
/renders/1,2

Apparently, Highrise uses the latter protocol (or at least, this post from 2007 says so), but then I got to thinking, and suddenly that didn’t seem right. Here’s why:

In my mind, in spite of my general aversion to query params (and the fact that using IDs in query params is inconsistent with the more typical case of IDs being used in the URL routing), the one that uses query params is actually more correct because the root endpoint, /renders, returns what is expected, what it normally does: a list of renders within a “renders” block (in XML, which we use). Moreover, it’s pretty common for index/listing endpoints to accept query parameters to filter (or paginate) the results, so an “ids” parameter (which, incidentally, I prefer to “id” for its conspicuousness) can just be seen as yet another filtering param.

If you go with the latter protocol, it’s true that you can get Rails to parse the “1,2″ into params[:id], which seems clever and convenient (and plus you get to learn about ActionController::Routing::SEPARATORS), but it then becomes an endpoint (a controller and action) that returns inconsistent data formats: sometimes it returns multiple renders in a “renders” block, and sometimes it returns a single render with no “renders” block.

The one advantage I can see to the latter method is access control: we have some consumers that are allowed to use the /renders endpoint for index/listing of renders, and others that are not–and I would like all consumers to be able to list multiple renders with multiple IDs, not least because it probably means they’re cutting down on further API requests. With the latter protocol, this is handily handled for us; with the former, it’s not. However, this is pure coincidence and, like the auto-parsing into params[:id], shouldn’t be taken as a guiding point of evidence: an easy work around is just to pass the request off from the “index” to “show” action after all the access control before-filtering has taken place.

My suspicion as to why this isn’t more clear out there is that REST was never meant to load multiple objects: you should just make multiple requests to the singular resources. However, in the real world we have real reasons to want to do this, and I don’t think it behooves us to slalom down the mountain of Roy dogma.