Random Musings of an Insane Mind

I heard a comment from a developer the other day:

You don’t need indexes on small tables.

So I asked what the definition of a small table was. He said, anything with a few hundred rows. So I said, 2300 rows? Well….. 24000 rows? Well….. 292000 rows? That’s large. I showed him unindexed queries in his application dealing with tables that had 2300, 24000 and 292000 rows.

Avoid tablescans

When MySQL deals with a query that is unindexed, it does a full tablescan to see if each record in the table meets the criteria specified. On a small table, if the query is executed frequently, the MySQL query cache might be able to serve the query. However, on a larger table, or a table with large rows, it must read every row, check the fields, possibly create a temporary table in ram or disk, and return the results. On a small site, you might not notice it, but, on a large system, forcing tablescans on tables with even a few thousand rows will slow things down considerably:

Uptime: 60016 Threads: 11 Questions: 105460332 Slow queries: 197769 Opens: 5819 Flush tables: 1 Open tables: 1320 Queries per second avg: 1757.204

Slow queries are sometimes unavoidable, but, often, slow queries are missing an index.

Use the slow-query log to find potential issues

When analyzing a system to find problems, putting:

log-queries-not-using-indexes

in the my.cnf file and restarting mysql will log the unindexed queries to the slowquery log.

What can be indexed?

The rule of thumb when writing indexes is to write your query in such a way that you reduce the result set as quickly as possible, with the highest cardinality possible. What does this mean?

If you are collecting data of the IP address and the Date, your query against date,ip will actually be worse than ip,date. Imagine receiving 40000 hits to your site on the same date. If you were looking for the number of hits that a particular IP had, you would search the 41 hits they have made over time, and then the 8 that they had today. If you queried by date,ip, you would search 40000 rows then would receive the 8 they had today. Each index you have, adds extra overhead and an index file should be as small as possible. IP addresses can be represented in an unsigned int which takes much less space than the varchar(15) usually used. Remember when you index a varchar field, indexing will spacepad the key to the full length. If you have a variable length field you want indexed, you might be able to figure out the significant portion of that field by finding the average length and adding a few characters for good measure and indexing fieldname(15) rather than the entire field. If a query is longer than the 15 characters, you have still created a significant reduction in the number of rows that it must check.

Cardinality refers to the uniqueness of the data. The more unique the data, the lower the chance that you’ll have thousands of records that match the first criteria. When the data is very similar, the index as built on disk will become imbalanced resulting in slower queries. Since MyISAM and InnoDB use a B-Tree index (or R-Tree if you use a spatial index), data that is similar when inserted, can create a very imbalanced tree which leads to slower lookups. An optimize table can resort and reindex the table to eliminate this, but, you can’t do that on an extremely large, active table without impacting response times.

# Query_time: 0 Lock_time: 0 Rows_sent: 1 Rows_examined: 3323
SELECT * FROM websites_geo where (zoneid = ’5135′) LIMIT 1;

In this case, zoneid is not indexed on the table websites_geo. Adding an index on zoneid eliminates the tablescan on this query.

Check for equality, not inequality.

An index can only check equality. A query checking to see if values are not equal, cannot be indexed.

# Query_time: 0 Lock_time: 0 Rows_sent: 5 Rows_examined: 2548
SELECT * FROM websites where (id = ’1056692′ && status != ‘d’ && status != ‘n’) order by rand() LIMIT 5;

# Query_time: 0 Lock_time: 0 Rows_sent: 10 Rows_examined: 2544
SELECT * FROM websites where (status != ‘n’ && status != ‘d’ && traffic > 3000) order by added desc LIMIT 10;

These two queries show two different issues, but, deal with the same fundamental issue. First, id is not indexed which would have at least limited the result set to 9 records rather than 2548. The status check isn’t able to use an index. On the second query, status is checked followed by traffic. There are other queries issued that check status,traffic,clicks_high. When we look at status (which should be an enum or char(1) rather than varchar(1)), we find that there are only 4 values used. By indexing on id,status and status,traffic,clicks_high, we could alter the queries as such:

SELECT * FROM websites where (id = ’1056692′ && status in (‘g’,’ ‘)) order by rand() LIMIT 5;

SELECT * FROM websites where (status in (‘g’,’ ‘) && traffic > 3000) order by added desc LIMIT 10;

which would result in both queries using an index.

Choose your data types intelligently.

As a secondary point, id (though it is numeric) happens to be a text field. If you index id in this case, you would have to specify a key length.

mysql> select max(length(id)) from websites;
+—————–+
| max(length(id)) |
+—————–+
| 22 |
+—————–+
1 row in set (0.02 sec)

mysql> select avg(length(id)) from websites;
+—————–+
| avg(length(id)) |
+—————–+
| 8.3315 |
+—————–+
1 row in set (0.00 sec)

mysql>

Based on this, we might decide to set the key length to 22 as it is a relatively small number and allows room to grow. Personally, I would have opted to have the id be an unsigned int which would be much smaller, but, the application developer uses alphanumeric id’s which are exposed externally. With sharding, you could use the id throughout the various tables, or, you could map the text id to a numeric id internally for all of the various tables.

There are a number of possible solutions to help any SQL engine perform better. And your data set will dictate some of the things that you can do to make data access quicker.

Helping MySQL Help You

If you do select * from table where condition_a=1 and condition_b=2 in one place, and select * from table where condition_b=2 and condition_a=1, setting up a single index on condition_a,condition_b and adjusting your second query, reversing the conditions to the same order as the keys on the index will increase performance.

Limit your results

Another thing that will help considerably is using a limit clause. So many times a programmer will do: select * from table where condition_a=1 which returns 2300 rows but only the first few rows are used. A limit clause will prevent a lot of data from being fetched by MySQL and buffered waiting for the response. select * from table where condition_a=1 limit 20 would hand you the first 20 records.

Avoid reading the data file, do all your work from the Index

Additionally, if you have a table and only need three of the columns from the result, select fielda,fieldb,fieldc from table where condition_a=1 will return only the three fields. As an added boost, if the fields you are checking can be answered from the index, the query will never hit the actual data file and will be answered from the index. Many times I’ve added a field that wasn’t needed in the index, just to eliminate the lookup of the key in the index then the corresponding read of the data file.

Let MySQL do the work

MySQL reads tables, filters results, can do some calculations. Going through 40000 records to pick the best 100 is still faster in MySQL than allowing PHP to fetch 40000 rows and do calculations and sorts to come up with that 100 rows. Index, optimize, and allow MySQL to do the database work.

Summary

Making MySQL work more efficiently goes a long way towards making your database driven site work better. Adding six indexes to the system resulted in quicker response times and an increase in the transactions per second.

Uptime: 32405 Threads: 1 Questions: 58729705 Slow queries: 64122 Opens: 2911 Flush tables: 1 Open tables: 295 Queries per second avg: 1812.366

Previously, MySQL was generating 3.26 slow queries per second. Now we’re just beneath 2 slow queries per second and our system is processing 55 more transactions per second. There is still a bit more analysis to do to identify the slow queries that are still running and to alter the queries to reverse the inequality checks, but, even just adding indexes to a few tables has helped noticeably. Once the developer is able to make some changes to the application, I’m sure we’ll see an additional speedup.

For the last 14 months we’ve been developing an application to replace 90000 lines of PHP code. Rewriting the application from scratch to support I18N and many of the enhancements it needed was deemed to be a better long term solution.

When that project was first started, I spent a month with Django and a month with Turbogears writing the same test application so that I could compare the development cycle. Both have matured and I needed to do a rapid turnaround on another project. I decided to give Django another look since it had hit version 1.0 and had added quite a few features that were missing in my preliminary evaluation. What follows is a discussion of the major points from both of the frameworks.

Turbogears has excellent form handling. Except for the Forms Wizard in Django, working with forms is much easier in Turbogears. Validation of the form and the resulting database update methods are much cleaner in Turbogears. Django, while slightly more complex in handling form input, does handle things with a single function which might enhance readability in a large project.

Database handling through SQL Alchemy in Turbogears is much better than the database methods in Django. Yes, you can use SQL Alchemy in Django now, but, their default ORM has plenty of room for improvement.

Turbogears is true MVC. Their terminology and methods are true to the paradigm set forth by Smalltalk. Django is MVC, but they call it MTV, Model, Template, View. The differences are slight and the developers of both have made good decisions. Adapting to either project’s methods is quick and not a hindrance.

Django’s definitely wins with Authentication/Authorization. Methods to handle registration, user creation, login and forgotten passwords are built in and wrapped with very nice templates that can be overridden. For Turbogears, repoze.who and repoze.what have been pulled from Plone and put into place. While Turbogears works with repoze, the decisions made and the lack of full support behind it make it difficult to implement.

Django feels faster. Comparing 14 months of development in Turbogears on an application to an application written in Django this week, the template engine, database access and pageload time seemed faster. Django is a lighter weight framework and you are closer to the data. Turbogears puts a little more insulation in which makes some coding easier at the expense of performance.

Maintainability of code would have to go to Turbogears. IBM once stated that the maximum number of bugfree lines of code that could be written was 23. With Turbogears, much of the heavy lifting is handled by widgets and decorators and your model resulting in a smaller codebase. Django requires more code to do the same task unless you utilize some of the snippets. Turbogears makes certain assumptions and has wrapped many of the libraries that make development easy in the default installation. Django’s default installation lacks those decisions, but, you are not prevented from establishing your own middleware. If you were developing a number of Django projects, you would pick and choose snippets that would replicate the decisions that Turbogears has already made.

URL Mapping is much easier to implement with Django. While routes support in Turbogears is present, Django’s regexp mapping is much easier to manipulate.

Community, hands down, Django wins. With a much larger installed base, bugs are found and fixed much more quickly. While Turbogears was founded on loftier principles, execution and follow through are lacking. Development is done when it is needed by a client project in the core group of developers. There is a definite air of condescension when the project developers handle questions from potential developers. With Django, there are people of all experience levels willing to help on groups.google, IRC, and thorough documentation that far exceeds most of the open source documentation out there.

Documentation, again, Django. Well organized, well thought out and well documented examples on Django’s part show dedication to having a framework that is well understood and well used. Turbogears recently moved to Sphinx, but, documentation generated from poorly documented code still means poor documentation. The tutorials and examples have been improving, but, they have a long way to go.

Genshi and Mako are supported fairly well with Turbogears and are both very good engines. Jinja is also supported which is a bit faster than Genshi and is powerful and very easy to work with. Django’s template language is also very flexible, powerful and easy to work with. Django had some definite advantages with a simpler language, but, neither Django or Turbogears stood out as a clear winner.

If you were looking to write an extremely heavy database or form involved site, I think Turbogears would be a good solution. If you choose Turbogears, be prepared to delve into the code when you are faced with problems. Bugs are not dealt with very promptly even though upgrades are pushed out. Be prepared to patch upgraded packages or hold certain packages once you move into production.

On the other hand, if you are writing a less complicated application, Django would be a good choice.

All told, the application developed this week in Django took about 12 hours and had I been working with Django for the last 14 months, I would estimate the project to have taken roughly 8 hours. Developed in Turbogears, I probably could have written it in 6 hours. PHP, to mimic all of the functionality would have taken 14-16 hours even using one of the numerous frameworks.

There is a definite advantage to using a framework and Python frameworks do appear to offer Rapid Application Development even over most of the PHP frameworks. For me, new development will probably be done in Django unless there is a strong case to use Turbogears.

* Turbogears 2.0
* Django
* Django Snippets