Random Musings of an Insane Mind

After working on a Pylons project for a week or so, there was a minor part of it that I felt didn’t need the complexity of a framework. Some quick benchmarking of the most minimal Pylons/SQLAlchemy project I could muster came in around 200 requests per second which put me at roughly 12 million requests per day based on the typical curve.

Within 15 minutes of installing Tornado and using their simple hello world example, I imported SQLAlchemy and ended up boosting this to 280 requests per second. As I really didn’t need any of the features from the ORM, I decided to use tornado.database which isn’t much more than a bare wrapper to python-mysql. Even with a single worker process, I was able to get 870 requests per second. 56 million requests per day, without any tuning?

I’m reasonably impressed. Once I put it on production hardware, I’m thinking I’ll easily be able to count on double those numbers if not more.

Next weekend, Traffic Server.

Not that it impacts this site, but, here’s another fine example of a company that doesn’t quite understand distributed computing.

67.218.116.171 - - [22/Jun/2010:12:22:40 -0400] "GET /robots.txt HTTP/1.1" 200 102 "-" "Mozilla/5.0 (Twiceler-0.9 http://www.cuil.com/twiceler/robot.html)"
67.218.116.168 - - [22/Jun/2010:12:30:12 -0400] "GET /robots.txt HTTP/1.1" 200 102 "-" "Mozilla/5.0 (Twiceler-0.9 http://www.cuil.com/twiceler/robot.html)"
67.218.116.167 - - [22/Jun/2010:12:31:57 -0400] "GET /robots.txt HTTP/1.1" 200 102 "-" "Mozilla/5.0 (Twiceler-0.9 http://www.cuil.com/twiceler/robot.html)"
67.218.116.169 - - [22/Jun/2010:12:32:55 -0400] "GET /robots.txt HTTP/1.1" 200 102 "-" "Mozilla/5.0 (Twiceler-0.9 http://www.cuil.com/twiceler/robot.html)"
216.129.119.45 - - [22/Jun/2010:12:33:52 -0400] "GET /robots.txt HTTP/1.1" 200 102 "-" "Mozilla/5.0 (Twiceler-0.9 http://www.cuil.com/twiceler/robot.html)"
216.129.119.49 - - [22/Jun/2010:12:37:30 -0400] "GET /robots.txt HTTP/1.1" 200 102 "-" "Mozilla/5.0 (Twiceler-0.9 http://www.cuil.com/twiceler/robot.html)"
216.129.119.42 - - [22/Jun/2010:12:37:51 -0400] "GET /robots.txt HTTP/1.1" 200 102 "-" "Mozilla/5.0 (Twiceler-0.9 http://www.cuil.com/twiceler/robot.html)"
67.218.116.165 - - [22/Jun/2010:12:40:05 -0400] "GET /robots.txt HTTP/1.1" 200 102 "-" "Mozilla/5.0 (Twiceler-0.9 http://www.cuil.com/twiceler/robot.html)"
67.218.116.170 - - [22/Jun/2010:12:40:27 -0400] "GET /robots.txt HTTP/1.1" 200 102 "-" "Mozilla/5.0 (Twiceler-0.9 http://www.cuil.com/twiceler/robot.html)"
216.129.119.47 - - [22/Jun/2010:12:41:25 -0400] "GET /robots.txt HTTP/1.1" 200 102 "-" "Mozilla/5.0 (Twiceler-0.9 http://www.cuil.com/twiceler/robot.html)"
216.129.119.40 - - [22/Jun/2010:12:42:52 -0400] "GET /robots.txt HTTP/1.1" 200 102 "-" "Mozilla/5.0 (Twiceler-0.9 http://www.cuil.com/twiceler/robot.html)"
216.129.119.43 - - [22/Jun/2010:12:43:01 -0400] "GET /robots.txt HTTP/1.1" 200 102 "-" "Mozilla/5.0 (Twiceler-0.9 http://www.cuil.com/twiceler/robot.html)"
67.218.116.164 - - [22/Jun/2010:12:43:37 -0400] "GET /robots.txt HTTP/1.1" 200 102 "-" "Mozilla/5.0 (Twiceler-0.9 http://www.cuil.com/twiceler/robot.html)"
216.129.119.45 - - [22/Jun/2010:12:52:25 -0400] "GET /%7Emcd/crossovernext.html HTTP/1.1" 404 354 "-" "Mozilla/5.0 (Twiceler-0.9 http://www.cuil.com/twiceler/robot.html)"
216.129.119.41 - - [22/Jun/2010:12:55:37 -0400] "GET /robots.txt HTTP/1.1" 200 102 "-" "Mozilla/5.0 (Twiceler-0.9 http://www.cuil.com/twiceler/robot.html)"
67.218.116.166 - - [22/Jun/2010:12:56:53 -0400] "GET /robots.txt HTTP/1.1" 200 102 "-" "Mozilla/5.0 (Twiceler-0.9 http://www.cuil.com/twiceler/robot.html)"


All that for a 404 on a page that hasn’t existed in 10 years.

One of our client systems has a Raid 1 setup using two 1 Terabyte drives. Last night, Debian’s consistency check launched, but, his system was doing some heavy disk IO due to some scripts that were being processed and the system was estimating close to 1000 hours to complete the check.

md3 : active raid1 sdb8[1] sda8[0]
      962108608 blocks [2/2] [UU]
      [===>.................]  check = 15.1% (145325952/962108608) finish=60402.6min speed=224K/sec

To abandon the check, we issued:

echo idle > /sys/block/md3/md/sync_action

Which allowed the machine to skip the rest of the test. While I don’t like disabling the checks, we’ll reschedule this one to do the check after they are done doing their work.