Rubyists

This is a fairly minor bugfix (thanks diegoviela) to fix the Playback app. The 0.4.4 should be available via gem.

Mostly i'm sorry this blog has been so idle, we've been working hard on FSR, FXC, and related components, but much of it on commercial or proprietary projects. In the coming months we'll be using this knowledge to release applications which enhance the functionality of open source telecom.

Thanks to all the new contributors, and welcome to the core team Harry Vandberg. He's pushed development along while we played end user for a few months, we're happy to have him.

As performance boosts are about speed, we’ll start with the benchmarks. Here is a run of spec/bench.rb, from the url_escape source tree.

Escape

Unescape

URLEscape provides these two methods as a C extension, suitable for use on ruby 1.8.6-8 and 1.9.1+; tested on linux, XP, and Vista.
The jruby version uses the java stdlib’s java.net.URLEncoder and URLDecoder. We only see a 200-700% increase with this change, and would like to improve on those numbers.

Why?

Josh Susser initially noticed the ability to overload #escape and #unescape while testing a client application. At the same time, we had just come across a bottleneck when regression testing FXC (a web app which serves configuration information to the FreeSWITCH softswitch) where requests were being delayed in our rack middleware, which parses the POST data sent by FreeSWITCH and routes requests to the ramaze application for processing. The delay was noticeable under loads of only 50 req/second; where rack became the bottleneck, not ramaze, the db, or any other factor. Adding the above library (on linux, with ruby 1.9.1) removed the delay in rack, pushing the work back to to the web app (or database) where it’s free to be as slow as it must. Optimally we’d like to perform at a speed equal to the database, making it the final bottleneck in a dynamic application.

Installation and Usage

To use URLEscape standalone

Install with one of the following methods:

• gem install url_escape
• get the tarball from RubyForge
• get the source from GitHub and rake install in the source top-level.

Then simply require “url_escape” and you have access to URLEscape.escape(string) and URLEscape.unescape(string)

To use URLEscape’s escape/unescape in place of CGI or Rack::Utils versions

gem install rackfastescape

or

gem install cgifastescape

This will install url_escape if it’s not already installed, as well.

You can optionally install rackfastescape or cgifastescape from rubyforge’s tarball or the github source (rake install as with urlescape). If you use tarball or source rake install, *you will have to manually install urlescape first.*

Once installed, simply use

require "rack_fast_escape"

to replace Rack::Utils version, or

require "cgi_fast_escape"

to replace the CGI version.

What else?

The ability of large posts to slow down a web application cannot be removed by just speeding up the POST parser. In order to alleviate the risk of such large POSTs being used to deny a service, firewall or web server throttling or limiting is a more reliable protection to enable. Here are a few examples:

• Lighttpd: http://lighttpd.net

  • Offers mod_evasive which limits connections per ip, as well as the ability to limit the data rate per connection.

• Nginx: http://nginx.org

  • Flexible limiting system, per vhost, per user, per connection.

• Netfilter/QoS (linux): http://l7-filter.sourceforge.net/

  • Allow classifiation of HTTP packets so iptables/tc or whatever utility you’d like can have the info it needs about the HTTP protocol to make limiting/dropping/queueing decisions

  • Others: Apache, Squid, Litespeed, many others will have various methods of limiting size and frequency of requests.

Side Note

When speccing these libraries, a few implementation differences came to light which we’ll highlight here.

1. Rack::Utils and CGI both throw errors on a mixed ASCII and Unicode string in ruby 1.9.1 and above
2. Java’s URLDecoder and URLEncoder do not escape or unescape mixed ASCII/Unicode properly.

URLEscape (the C version) handles these cases properly, though we don’t expect you’d see them much in proper requests.

Thanks

To Evan Phoenix, Josh Susser, Trey Dempsey, Jayson Vaughn, Michael Fellinger, Kevin Berry, and all the other contributors of ideas and support who made this product a reality.

License

Nothing to fear, it’s MIT

With FSR well under way and being used both internally and in the wild for FreeSWITCH application development, the next logical step will be a Configurator for FreeSWITCH itself. FreeSWITCH has long lacked the standard Administrator/User configuration available via web or GUI, FXC is intended to allow web interfaces for configuration to be built easily with any Rack application utilizing the FreeSWITCH xml_curl interface. The first pass of FXC will include some Rack middleware which turns FreeSWITCH requests (to /) into easy to organize routes such as /directory/register/internal/1000, /dialplan/public/8885551212, /configuration/acl.conf. The goal is to eliminate the gruntwork of routing by POST variables, exposing clean Rack-app routes up the stack (for ramaze, sinatra, etc). The following is an example with ramaze.

middleware.rb

module FXC
  module Rack
    class Middleware
      def initialize(app)
        @app = app
      end

      def call(env)
        r = ::Rack::Request.new(env)
        return @app.call(env) unless r.params["section"]
        path = r.params["section"] + "/"
        path << case path
        when "dialplan/"
          dp_req(env, r)
        when "directory/"
          dir_req(env, r)
        when "configuration/"
          conf_req(env, r)
        end
        env["PATH_INFO"] << (env["PATH_INFO"].match(%r{/$}) ? path : "/#{path}")
        @app.call(env)
      end

      private
      def dp_req(env, r)
        s = [r.params["Caller-Context"]]
        s << r.params["Caller-Destination-Number"]
        s.join("/")
      end

      def dir_req(env, r)
        s = []
        if r.params["purpose"]
          s << r.params["purpose"].gsub("-","_")
          s << r.params["sip_profile"]
        elsif r.params["action"] and r.params["action"] == "sip_auth"
          s << "register"
          s << r.params["sip_profile"]
          s << r.params["sip_auth_username"]
        elsif r.params["user"]
          s << "voicemail"
          s << r.params["sip_profile"]
          s << r.params["user"]
        end
        s.join("/")
      end

      def conf_req(env, r)
        s = []
        if r.params["key_name"] == "name"
          s << r.params["key_value"]
        end
        s.join("/")
      end

    end
  end
end

controller/dialplan.rb

# Copyright (c) 2008-2009 The Rubyists, LLC (effortless systems) <rubyists@rubyists.com>
# Distributed under the terms of the MIT license.
# The full text can be found in the LICENSE file included with this software
#
module FXC
  class Dialplan < Controller
    map '/dialplan'
    layout :dialplan

    def index(*args)
      Ramaze::Log.info("Got unhandled dialplan request: " + request.inspect)
      not_found
    end

    def default(number)
      Ramaze::Log.info("got default dialplan request for #{number}")
      not_found
    end

    def public(number)
      @did = FXC::Did.first(:number.like /#{number.sub(/^1/,'1?')}/)
      if @did 
        @user = @did.user
        @targets = @did.targets
        Ramaze::Log.info("Routing #{@did.number} to #{@user.dialstring}")
        render_view(:index)
      else
        Ramaze::Log.info("Got public dialplan request for #{number}, but no DID matches: ")
        not_found
      end
    end
  end
end

In the FXC::Dialplan controller, each method represents a FreeSWICH dialplan context. Undefined contexts (in this ramaze controller) will fallthrough to the index method and be logged.

Finally, the FreeSWITCH configuration to send requests to above app becomes a single line/single url

conf/autoloadconfigs/xmlcurl.conf.xml

<configuration name="xml_curl.conf" description="cURL XML Gateway">
  <bindings>
    <binding name="fxc">
      <param name="gateway-url" value="http://127.0.0.1:9292/" bindings="configuration|directory|dialplan"/>
    </binding>
  </bindings>
</configuration>

Next step is completing all the methods available for each binding type (configuration, dialplan, directory). Once complete FreeSWITCH web config on Rack should follow rapidly. That will be the “Look Ma, No XML” FXC release (TBA).

The Problem In the past with BDD and writing specifications for ruby applications which used postgresql, I’d resorted to environment variables specifying some environment, which I’d then place a conditional somewhere so if the environment was “test” it would use that already-existing database server and database. The permissions problems with this were numerous, so we eventually moved to keeping an instance of postgres up that was just a test server for these test databases. Maintenance was not easy.

Other tactics we’ve used, especially when starting a project or on one which does not use postgres-specific code, we’d use Sqlite3 databases in-memory for our specifications. This led, at times, to inconsistencies with how postgres would handle data or functions, and again became a maintenance nightmare as soon as we wanted to put the power of postgres to work and Sqlite3 could not support the specs anymore.

On a small project using Ramaze, Sequel, and Bacon for specs, manveru and I came across a solution which, while not as simple as Sqlite in memory, accomplishes the same encapsulation with none of the security risks of the postgres test server or test-database on a multi-purpose server scenario.

The Solution Create a database in a ram filesystem (/dev/shm mounted as tmpfs on linux), initialize a postgres cluster there, start the db server, and the user who does so has God rights to create/drop/manipulate the db without chance of affecting anyone else. The db_helper.rb we use to accomplish this follows

begin
  require "sequel"
rescue LoadError
  require "rubygems"
  require "sequel"
end
require "logger"
ENV["PGHOST"] = PGHOST = "/tmp"
ENV["PGPORT"] = PGPORT = "5433"
SHM = "/dev/shm"
ENV['PGDATA'] = PGDATA = "#{SHM}/fxc"
DB_LOG = Logger.new("/tmp/fxc_spec.log")

def runcmd(command)
  IO.popen(command) do |sout|
    out = sout.read.strip
    out.each_line { |l| DB_LOG.info(l) }
  end
  $? == 0 ? true : false
end

def startdb
  return true if runcmd %{pg_ctl status -o "-k /tmp"}
  DB_LOG.info "Starting DB"
  runcmd %{pg_ctl start -w -o "-k /tmp" -l /tmp/fxcdb.log}
end

def stopdb
  DB_LOG.info "Stopping DB"
  if runcmd %{pg_ctl status -o "-k /tmp"}
    runcmd %{pg_ctl stop -w -o "-k /tmp"}
  else
    true
  end
end

def initdb
  raise "#{SHM} not found!" unless File.directory?(SHM)
  return true if File.directory?(PGDATA)
  runcmd %{initdb}
end

def createdb
  runcmd %{dropdb fxc}
  runcmd %{createdb fxc}
end

raise "initdb failed" unless initdb
raise "startdb failed" unless startdb
raise "createdb failed" unless createdb

DB = Sequel.postgres("fxc", :user => ENV["USER"], :host => PGHOST, :port => PGPORT)
require 'sequel/extensions/migration'
require File.expand_path('../../lib/fxc', __FILE__)

# go to latest migration
Sequel::Migrator.apply(DB, FXC::MIGRATION_ROOT)

require FXC::SPEC_HELPER_PATH/:helper

The Result Running our specs is now much faster, especially after the first run which initializes the cluster.

## Before we run, stop the test db and wipe it out of memory
bougyman@zero:~$ pg_ctl -o "-k /tmp/" stop
waiting for server to shut down.... done
server stopped
bougyman@zero:~$ rm -rf /dev/shm/fxc/
## Run the specs (it's initting a whole new db in ram on this run)
bougyman@zero:~$ time rake
(in /home/bougyman/git_checkouts/fxc)
   1/4: spec/model/target.rb                  1 passed
   2/4: spec/model/user.rb                    1 passed
   3/4: spec/view/dialplan.rb                 1 passed
   4/4: spec/view/directory.rb                3 passed
6 specifications (29 requirements), 0 failures, 0 errors

real    0m8.176s
user    0m0.200s
sys     0m0.056s
## Run the specs again (db is initialized already here, just run specs)
bougyman@zero:~$ time rake
(in /home/bougyman/git_checkouts/fxc)
   1/4: spec/model/target.rb                  1 passed
   2/4: spec/model/user.rb                    1 passed
   3/4: spec/view/dialplan.rb                 1 passed
   4/4: spec/view/directory.rb                3 passed
6 specifications (29 requirements), 0 failures, 0 errors

real    0m2.417s
user    0m0.216s
sys     0m0.044s
## Much faster on runs after the db is initialized