Tech Notes From The Trenches

I ran into this issue today: Dealing with conflictive files/directories in projects using Subversion. For example: Log directories and configuration files. But I’ve seen this problem before in my projects. I really like working in teams. But I also like having my own work environment.

I don’t really like sharing the database I use for development with other people. I guess it’s because of my experience and many past headaches due to teammates introducing conflicting changes (the system starts failing, ‘magically’)

This is a good practice suggested by Subversion: Use .tmpl files for conflictive resources in your projects. In my Rails projects, the conflictive file is usually database.yml -

My teams like to work with a development database in the Aycron office. I like to work with a local database. I believe it’s faster and safer. I save time by using my local MySQL database.

Here is what you should do, in order not to share the same database.yml:

1. Create your Rails project structure

[etagwerker@benteveo trunk]$ rails test-local -d mysql

2. Rename database.yml to database.yml.tmpl

[etagwerker@benteveo test-local]$ cd test-local

[etagwerker@benteveo test-local]$ mv config/database.yml config/database.yml.tmpl

3. Commit all your code to the SVN repository

4. Create your local copy from your template

[etagwerker@benteveo test-local]$ cp config/database.yml.tmpl config/database.yml

5. Ignore database.yml for the rest of your project

[etagwerker@benteveo test-local]$ cd config

[etagwerker@benteveo config]$ svn propset svn:ignore database.yml .

6. Commit the config directory

[etagwerker@benteveo config]$ svn ci

You’re done! Your teammates can do whatever they want with the database.yml, you can do whatever you want and there will be no conflicts over that file. Because it will never get committed to the SVN repository.

I ran into this weird issue with attachment_fu and had no idea what was wrong with my code. I’m using ActiveScaffold and attachment_fu to manage my entities and images. The exception was not that useful and I only found the solution thanks to Diego, our Rails guru @ Aycron.

Here is the exception:NoMethodError (undefined method `content_type’ for “10.jpg”:String):
/vendor/plugins/attachment_fu/lib/technoweenie/attachment_fu.rb:294:in `uploaded_data=’
/vendor/plugins/active_scaffold/lib/attribute_params.rb:106:in `send’
/vendor/plugins/active_scaffold/lib/attribute_params.rb:106:in `update_record_from_params’
/vendor/plugins/active_scaffold/lib/data_structures/action_columns.rb:68:in `each’
/vendor/plugins/active_scaffold/lib/data_structures/action_columns.rb:55:in `each’
/vendor/plugins/active_scaffold/lib/attribute_params.rb:49:in `update_record_from_params’
/vendor/plugins/active_scaffold/lib/attribute_params.rb:71:in `update_record_from_params’
/vendor/plugins/active_scaffold/lib/data_structures/action_columns.rb:68:in `each’
/vendor/plugins/active_scaffold/lib/data_structures/action_columns.rb:55:in `each’
/vendor/plugins/active_scaffold/lib/attribute_params.rb:49:in `update_record_from_params’
/vendor/plugins/active_scaffold/lib/actions/create.rb:73:in `do_create’
/Users/etagwerker/.gem/ruby/1.8/gems/activerecord-2.0.2/lib/active_record/connection_adapters/abstract/database_statements.rb:66:in `transaction’
/Users/etagwerker/.gem/ruby/1.8/gems/activerecord-2.0.2/lib/active_record/transactions.rb:80:in `transaction’
/vendor/plugins/active_scaffold/lib/actions/create.rb:72:in `do_create’
/vendor/plugins/active_scaffold/lib/actions/create.rb:28:in `create’
/Users/etagwerker/.gem/ruby/1.8/gems/actionpack-2.0.2/lib/action_controller/base.rb:1158:in `send’
/Users/etagwerker/.gem/ruby/1.8/gems/actionpack-2.0.2/lib/action_controller/base.rb:1158:in `perform_action_without_filters’
/Users/etagwerker/.gem/ruby/1.8/gems/actionpack-2.0.2/lib/action_controller/filters.rb:697:in `call_filters’
/Users/etagwerker/.gem/ruby/1.8/gems/actionpack-2.0.2/lib/action_controller/filters.rb:689:in `perform_action_without_benchmark’
/Users/etagwerker/.gem/ruby/1.8/gems/actionpack-2.0.2/lib/action_controller/benchmarking.rb:68:in `perform_action_without_rescue’
/usr/local/lib/ruby/1.8/benchmark.rb:293:in `measure’
/Users/etagwerker/.gem/ruby/1.8/gems/actionpack-2.0.2/lib/action_controller/benchmarking.rb:68:in `perform_action_without_rescue’
/Users/etagwerker/.gem/ruby/1.8/gems/actionpack-2.0.2/lib/action_controller/rescue.rb:199:in `perform_action_without_caching’
/Users/etagwerker/.gem/ruby/1.8/gems/actionpack-2.0.2/lib/action_controller/caching.rb:678:in `perform_action’
/Users/etagwerker/.gem/ruby/1.8/gems/activerecord-2.0.2/lib/active_record/connection_adapters/abstract/query_cache.rb:33:in `cache’
/Users/etagwerker/.gem/ruby/1.8/gems/activerecord-2.0.2/lib/active_record/query_cache.rb:8:in `cache’
/Users/etagwerker/.gem/ruby/1.8/gems/actionpack-2.0.2/lib/action_controller/caching.rb:677:in `perform_action’
/Users/etagwerker/.gem/ruby/1.8/gems/actionpack-2.0.2/lib/action_controller/base.rb:524:in `send’
/Users/etagwerker/.gem/ruby/1.8/gems/actionpack-2.0.2/lib/action_controller/base.rb:524:in `process_without_filters’
/Users/etagwerker/.gem/ruby/1.8/gems/actionpack-2.0.2/lib/action_controller/filters.rb:685:in `process_without_session_management_support’
/Users/etagwerker/.gem/ruby/1.8/gems/actionpack-2.0.2/lib/action_controller/session_management.rb:123:in `process’
/Users/etagwerker/.gem/ruby/1.8/gems/actionpack-2.0.2/lib/action_controller/base.rb:388:in `process’
/Users/etagwerker/.gem/ruby/1.8/gems/actionpack-2.0.2/lib/action_controller/dispatcher.rb:171:in `handle_request’
/Users/etagwerker/.gem/ruby/1.8/gems/actionpack-2.0.2/lib/action_controller/dispatcher.rb:115:in `dispatch’
/Users/etagwerker/.gem/ruby/1.8/gems/actionpack-2.0.2/lib/action_controller/dispatcher.rb:126:in `dispatch_cgi’
/Users/etagwerker/.gem/ruby/1.8/gems/actionpack-2.0.2/lib/action_controller/dispatcher.rb:9:in `dispatch’
/usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel/rails.rb:76:in `process’
/usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel/rails.rb:74:in `synchronize’
/usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel/rails.rb:74:in `process’
/usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.5/lib/mongrel.rb:159:in `process_client’
/usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.5/lib/mongrel.rb:158:in `each’
/usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.5/lib/mongrel.rb:158:in `process_client’
/usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.5/lib/mongrel.rb:285:in `run’
/usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.5/lib/mongrel.rb:285:in `initialize’
/usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.5/lib/mongrel.rb:285:in `new’
/usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.5/lib/mongrel.rb:285:in `run’
/usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.5/lib/mongrel.rb:268:in `initialize’
/usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.5/lib/mongrel.rb:268:in `new’
/usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.5/lib/mongrel.rb:268:in `run’
/usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.5/lib/mongrel/configurator.rb:282:in `run’
/usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.5/lib/mongrel/configurator.rb:281:in `each’
/usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.5/lib/mongrel/configurator.rb:281:in `run’
/usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.5/bin/mongrel_rails:128:in `run’
/usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.5/lib/mongrel/command.rb:212:in `run’
/usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.5/bin/mongrel_rails:281
/Users/etagwerker/.gem/ruby/1.8/gems/activesupport-2.0.2/lib/active_support/dependencies.rb:489:in `load’
/Users/etagwerker/.gem/ruby/1.8/gems/activesupport-2.0.2/lib/active_support/dependencies.rb:489:in `load’
/Users/etagwerker/.gem/ruby/1.8/gems/activesupport-2.0.2/lib/active_support/dependencies.rb:342:in `new_constants_in’
/Users/etagwerker/.gem/ruby/1.8/gems/activesupport-2.0.2/lib/active_support/dependencies.rb:489:in `load’
/usr/local/lib/ruby/gems/1.8/gems/rails-2.0.2/lib/commands/servers/mongrel.rb:64
/usr/local/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `gem_original_require’
/usr/local/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `require’
/Users/etagwerker/.gem/ruby/1.8/gems/activesupport-2.0.2/lib/active_support/dependencies.rb:496:in `require’
/Users/etagwerker/.gem/ruby/1.8/gems/activesupport-2.0.2/lib/active_support/dependencies.rb:342:in `new_constants_in’
/Users/etagwerker/.gem/ruby/1.8/gems/activesupport-2.0.2/lib/active_support/dependencies.rb:496:in `require’
/usr/local/lib/ruby/gems/1.8/gems/rails-2.0.2/lib/commands/server.rb:39
/usr/local/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `gem_original_require’
/usr/local/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `require’
/script/server:3
-e:2:in `load’
-e:2

My code was organized so that a Page entity had one Image entity. The data structure was correct. The problem was in my controllers. images_controller was fine. I could check that by uploading images using that particular controller. The problem was with pages_controller. pages_controller uses images_controller to create/edit pages and their associated images.

pages_controller was missing a few lines of ActiveScaffold configuration. Particularly the lines associated with multipart forms. See ActiveScaffold:: create API.

The solution is to add the following lines to the pages_controller ActiveScaffold configuration:

config.create.multipart = true
config.update.multipart = true

That did the trick. I’m writing this down here so that I don’t forget the solution. Oh and maybe someone else will find this useful, if they ever run into such a useless exception message.

Here are a few notes about executing the stress test:

• If you are using only one computer, remember that you are using only one CPU, one memory, one internet connection and one operating system. So I would advise not to do many things at once in your computer while you test (for example, don’t download that torrent!)

• Also, I would not to do any CPU-intensive activity while stress testing my web application

• Each single test may fail. I didn’t have that problem often, 1 out of 12 tests. But one of them did not finish correctly. You can see this by comparing the total requests per test (compared to other tests you performed) – For example, your internet connection might unexpectedly go down and make your test useless

• Have a good connection. You should properly measure the delay you have between your computer and the server.  The delay I had between my client and server was minimal

I ran a set of tests to have a base set of results. The initial set of results was not good (though I was expecting this). I noticed there was a high load on the database. Mongrel was not taking most of the server resources, MySQL was.

With our team, we decided to implement a mongrel-level cache to reduce database access and improve performance/throughput.

The second set of tests showed a big improvement. Performance improved a lot! The cache reduced the load on the database. This was very noticeable. We applied the cache to the most visited sections. To know which were the most visited sections, we used historical Google Analytics data from the project.

The set of tests didn’t vary on the client side. I simulated a load of 100, 200, 500, 750, 900, 950 and 1100 concurrent users.

The result improvements were 3 times better (the lowest), 15 times better (the highest) and  11 times better (the average)

To sum things up, JMeter is a very useful tool for stress testing web applications. You will have to read their documentation to understand the graphic results (numbers are shown but units are not always shown).

If you haven’t developed any performance-related code (eg. cache) for your application, run a first set of stress tests before you do. This will give you a base for improvement.

Develop your performance improvements, test them, make sure they are working as expected (eg. cache invalidation working properly) and then run your second set of tests. You will definitely make a difference on the throughput of your application. If not, you will have to keep making improvements until you see the results.

Have fun!