Benchmark 4 - Notes About Caching and Benchmarking
When benchmarking web related cache tools you really should benchmark the application that you want to improve performance upon.
Keep in mind the golden rule of benchmarking: All benchmarks are flawed benchmarks.
If you are running a benchmark on MySQL Cache using a simple SELECT query and your real life application is using prepared statements, your benchmark results will be useless because MySQL Cache doesn't work on prepared statements.
Benchmarking is always a bit difficult. The results may vary depending on both hardware, network, and software. If either one changes the results will change as well.
The individual settings also plays a major role. In case you are running a test on a memory based cache, the size of the memory matters and fragmentation matters, but you must also remember to evaluate the difference between the development environment and the production environment.
If your application is an Open Source application that needs deployment by several people (perhaps even thousands of people) you should always avoid cache features in development. This way you are sure to get a good feeling for how your application performs if the cache features isn't available.
Every time any part of your application runs more than once and produces the exact same result, you have failed a little bit. Every time your entire application runs and produces the exact same result as the last time it ran, you have failed a lot.
Caching is not an optimization to be thought of some time later in development. It is an integrate part of good coding style and development principles.
By this I don't mean that your application should depend on a specific cache feature that may or may not be available, but you should always consider how you can make your application perform optimally and avoid duplication both in code and running results.
Static pages should always be an integrated part of any web application even if the underlying content is dynamically stored.
Sometimes this isn't possible for parts of the application and in such cases a memory based cache such as Memcached is extremely usefull.
The book Advanced PHP Programming by George Schlossnagle (one of the authors of the Alternative PHP Cache - APC) has three nice chapters about PHP performance called: External Performance Tunings, Datacomponent Caching, and Computional Reuse and I highly recommend the book for anyone interested in learning more about PHP performance.
I hope you have found my benchmarking articles useful eventhough I know that real life applications benchmarking and good programing design really is the key issue when it comes to performance.
If you have any comments or corrections feel free to email me.