When I built the original PerformanceTest, I hacked it together quickly to meet a specific need I had at the time. As I used it, I found I needed more capability, and I became increasingly frustrated with both the messy code, and the poorly thought out API.
In response to feedback, and my own frustrations, I’ve completely rebuilt PerformanceTest to be more robust, extensible, feature-rich, *and* easier to use.
Core features:
- Track time, memory usage and retained (unreleased) memory for functions.
- Test rendering time for any display object.
- Write simple one line tests, or build formal test suites.
- Test suites can be built using inheritance or aggregation/composition.
- Trace simple text results, or access robust XML reports.
- Save XML and compare results between different versions of your code, or on different Flash players.
- Easily build logging classes to analyze or visualize results.
- Run multiple iterations of tests to get min, max, and deviation values.
- Run tests synchronously, or queue them to run them asynchronously.
- Listen for events from tests, test suites, and PerformanceTest.
- Integrates easily with TDD by facilitating the creation of tests for time, memory, and retained memory limits.
Here’s a very simple demo:
In it’s simplest form, you can use PerformanceTest in one line:
trace(PerformanceTest.run(myMethod)); trace(PerformanceTest.run(mySprite, "spriteRenderTest", iterations, loops)); |
You can go even simpler when testing methods, by using the ptest function shortcut:
trace(ptest(myMethod)); trace(ptest(myMethod, ["param"], "name", iterations, loops)); |
Want something a bit more robust? You can set up formal collections of tests using TestSuite. You can use these to set up suites of benchmarks to test different player platforms, or you can maintain them over the life of a project to track its performance profile by saving the XML Logs.
This is a beta. The docs are complete, but a little sparse. There are very few sample / demo files. It almost definitely has bugs. I also plan to add at least one more class/tool to it, for comparing XML logs.
Given that, I’d appreciate anyone who’s interested checking it out and giving some feedback. Does it do what you need? Find any bugs? Is the API straightforward? How much more documentation do you need?
You can download PerformanceTest v2 here. As with pretty much all of the source code I release, it is licensed under the MIT license.
UPDATE: I just uploaded a new version with a working FLA. My apologies for the trouble with the original one. It also includes a *very* simple benchmark comparison tool.
Thanks, Grant, this looks awesome. Especially if you don’t use the FlexBuilder.
Hmm. Made me curious. So you use something like BitmapData.draw to test sprite drawing performance? Or you do it differently?
Right, it uses BitmapData.draw(). This gives a pretty good idea of what the render time for a display object is on a single core. Not an exact value, but great for comparative metrics.
I saw your tweet, and if you decide not to publish your thoughts on the future of Flash you will be my hero for about a week.
I value your opinion above most others that I’ve heard, but it would be a pretty pimp move not to beat this topic any further.
cgortz – sorry to disappoint. 🙂
This is just in time for a project I’m working on. I used v1 quite a lot and found it very useful, so this is going to be even better.
For some reason I’m finding the FLA in the zip is failing to open in Flash CS4. Everything else unzipped fine.
So Powerful
Same as Ben, the FLA is not opening in Flash CS4.
Thankyou Grant! I was about to write my own really simple one when I found this, which is far better and really easy to use.
For those who can’t open the flash file, it contains a dynamic text field with the instance name of ‘outFld’ and the document root class of PerformanceTestDemo
Just wanted to say that I just found PerformanceTest tool set (20 minutes ago) and it is both very simple to use and invaluable. Thank you for releasing it!
I tried doing some memory usage checks for my unit tests they way you have, but it seemed like I kept getting false positive. Something as simple as:
var sprite : Sprite;
var(i : int = 0; i
Im sorry to inform you that the Fla fails to open i both CS3 and CS4 on PC ??
In the PerformanceTest class there are a static method and protected member both named “queue”, maybe we need different names?
Hey Grant,
nice classes. But I have a question to you. How can I calculate the minimum system requirements(cpu,ram etc.) for his own flash movie. Could be the perfomancetest class a help for that?? I have already googled the whole day – no solution found!!
Thanks
Great Work Here… I Finally Able to optimize my development and also my FYP research… LOL… Thanks a lot… ^ ^
I was linked to this page by my friend who was struggling to make use of this tool, and after some time of me trying to get some sense of the results it produces I just came to the conclusion, that I don’t know what it does. Sorry to say this, but it’s based on a whole bunch of misconceptions about how the code is translated to the machine codes, or, sometimes, the testing methods themselves are so poorly written that the mistake that they generate is bigger than the actual time you want to measure…
Well, just for the record, I’ve tested some function, of which I was certain that it had to spend about 120-160ms on processing ~600Kb of data, and using this tool I was getting 400+-10ms… This is off by 150%… what’s the point?
Hello,
PerformaceTest v2 looks awesome, thanks for sharing 🙂 But I just found something that possibly invalidates all speed tests.
If you “play” a series of identical loops, the later ones are always fairly quicker than the previous ones. This happens both with while than with for loops.
I tried it by myself (with custom fors and whiles, targeting FlashPlayer 10 and also FlashPlayer 10.1 in both Debug and Release versions), and also using your TestPerformance.
With TestPerformance, I tried to run a simple method which contained only a “var n:int = 2” operation. I ran PerformanceTest.run multiple times (with a 1000000 loops value) with this same method, and it’s evident that the first results are quite slower than the others.
Any idea why this happens?
Hi Daniele,
Yes, this is due to a couple of factors.
The first is that the player has a ramp up period when it first initializes. This slows everything down for the first few seconds.
The second is that some code may not be JITed the first time through (especially if it is run immediately), but subsequent runs may be.
For this reason I recommend having tests run 3s or more after your test SWF initializes (easy to do with a Timer), and running each test multiple times (which PerformanceTest supports directly). This is also why PerformanceTest provides metrics like min, max and deviation values, so you can evaluate the consistency of your results.
Hi Grant 🙂
Yup, I now tried using an iteration value (in PerformanceTest.run) bigger than 0, and the deviation values are very helpful 🙂
Though, I have a button on my stage which calls the performance test (to avoid the player ramp up period), and I ran it many times. I’m quite puzzled by the fact that, each time I ran the same test, the first loops are always slower. This surely has a striking impact when using loops in actual code, and means that actually there’s also a frame ramp up period.
Using this to try and create unit tests for a client.
And quite honesty I don’t really understand the output, or how to use the iterations / loops parameters and what effect they really have on the test.
I was asked to run the RenderTest on some particles, and I just find your documentation incredibly lacking. Like is the displayobject supposed to be added to the stage or not?
And an explanation of how the render test, all of the tests really, actually works, and what is it testing for, would greatly help to understand its output.
Right now I feel like I wasted a few hours setting up a .fla, writting classes, and getting everything to work, for just a bunch of junk output.
-Robert Baindourov
And I just double checked the example I downloaded, and the output you are generating here on this page, and you don’t even run a render test in either. A functional example would have been nice.
it is so great .i am excited to find this useful tool.
and i used it in my code .
But now i find a problem, The tool just can test the function without parameters. So the tool can not be used anywhere.
can you fixed it ?
thank you .
Hey Grant, I just felt like commenting based on some of the previous responses (newbs). I just really want to thank you for building this test harness, I love it when someone out there saves me some programming time, and this certainly did, and will continue to save me tons of time when there’s a doubt or an argument about which direction to take with regards to optimization.
Thanks so much! this will aid great learnings
Thanks for great lib!
I find myself using and using it!
Any possibility of it put on gitHub?