I recently decided to retest my results on uint, int and Number performance to see what had changed in Flash player 10 (more on that in my next post). I got a little carried away, and wound up building a generic performance testing harness for ActionScript 3. I’ve been meaning to do this for awhile, but hadn’t gotten around to it until now. It’s not that its very complicated, it just required a lot of thought into specifically how I would want it to work.
I think the nicest thing about the test harness is how flexible it is. You can easily use it to quickly test a couple of functions on the timeline, or you can build a full set of performance testing suites to support a project under development, similar to unit testing.
Even test suites are super simple. Point it at an instance of a class, and it will test all the public methods on it. Or, use properties or parameters to formalize the tests with names, descriptions, and specific method lists.
You can even use PerformanceTest to test the render time for complex vectors or filters.
You can also easily customize how it logs the results of tests to output them differently, save them to a file, or graph the results.
Here’s a simple demo of it in action.
Note that this is a beta release. I’m considering a few more features, and possibly rearchitecting things a bit. For example, I’d like to make it easier to test renders as part of a suite. I’d also like to add a verbose mode that will run iterations asynchronously, and provide more advanced logging like min, max, and standard deviation for times.
I’ll also be writing a series of blog entries over the next little while outlining (and hopefully explaining) some of the results I’ve been seeing from my tests.
If you have suggestions, or build some useful test suites, please feel free to share them in the comments or by emailing me with the contact button above.
UPDATE: Performance test has been updated with support for synchronous tests and integration with unit testing and test driven development. This release also fixes a bug with testFunction and testRender (pointed out by a few commenters below). You can download it from the link above. I will be posting a new version shortly with many more sample test suites.
UPDATE 2: PerformanceTest v2 beta has been released. It represents a very significant update from v1. Click here for information.
You can download version 1 of TestPerformance package, including API documentation and sample test suites by clicking here.
Nice stuff! 🙂 Good thing that you can watch all different tests live!
Looks awesome Grant! Can’t wait to test this out with a couple projects. See you at FITC!
That looks really great!
About the performance comparison between int, Number and uint: I know your first test is much more extended, but the conclusion was that Number was the overall winner. At your current test I see the following results:
intTest 49 9.80
numberTest 169 33.80
uintTest 40 8.00
Do you think the ‘low’ performance of Number has to do with Flash Player 10?
Can’t wait to see what’s up next, great work!
@Abel this makes sense because working with floating point values (ie Number) is slower than int and uint.
There has been a bunch of blog posts describing how ints and uints sometimes get converted to Numbers, and thus are slower than just using Numbers to begin with. This conversion doesn’t happen when you increment or decrement ints and uints and use them array access.
The way I understand it, any operation that could potentially result in a float must have its members converted to Numbers first.
How timely!
Not only on the API portion from your tweets, but I’ve also been building an internal benchmark suite for testing our API. Unfortunately, I’m developing in Flash Lite and, therefore, AS2, so I may roll up my sleeves and see if I can port this. It’s much more robust than the simple unit test iterator I created.
Great work, can’t wait to play with it more on my next AS3 project.
Until a test has a low-quality code, it will be biased.
I think one very important thing that can not be restated often enough is that you always have to test performance with a release version in the release player. Do not take the values that a debug version or even the debug player gives you as the truth since there can be huge differences.
I always go by the rule: “never trust any ‘faster’ method found in any blog post that you haven’t verified yourself”.
This is fantastic, thanks for the great lib Grant!
Thanks! Was just going to do some performance testing on my app 🙂
etc – not certain if I’m understanding your comment correctly, but if you’re suggesting that .quality must be set to low in order to run tests accurately, that would be incorrect. There is no rendering taking place while the tests are running (except for testRender of course).
mario – completely right. Further, it is important to test in the specific environment(s) you are deploying in. I’ve seen different results on plugin vs. AIR vs. authoring/stand alone, across OS’s, and of course between debug and standard players.
I think I’ll write up a “guide for performance testing”.
I wonder how many of these instance/static var/etc. performance gains can be automatically applied during the compiler’s optimization steps? For that matter, how much of this can be applied by the runtime engine?
Awesome tool! thank’s Grant to share it
A little error occurs “Error #1009…null object reference” When .testFunction() and .testRender() are not following any .testSuite().
A “perfTestInstance.testSuite({})” helps to pass through.
Very nice tool, Grant! I was snooping into some optimization tricks for a trigonometry-heavy app. Found some that sounded nice on http://lab.polygonal.de/2007/07/18/fast-and-accurate-sinecosine-approximation, but in the spirit of Mario’s comment I tested them with your suite before changing all the code. Turns out the benefit of optimizing is next to nothing, if I don’t want to put the sin/cos math inline every time I need it.
In addition, maybe it’s an idea to add a line for the version into the PerformanceTest.logBegin() method. That way you can copy/paste the results for comparison:
out(pad(“Flash version: ” + Capabilities.version + (Capabilities.isDebugger ? ” (debug)” : ” (regular)”), 72));
cool
flash player 10 will however consume a lot of proccessing power….70% like
Hi I got the same bug as above “Error #1009” I simply added the init() method to the PerformanceTest constructor.
In addition i also wanted to pass variables into the testFunction so i modified line 311 from
try{method();}
to
try{
{
method.apply( this, o.variables );
}
and added an extra variables array to the test object.
There is a bug in the PerformanceTest.as. You can’t called testFunction first, you must call testSuite to init the harness. Just need to copy this part from testSuite to testRender and testFunction:
// do initial set up if needed:
init();
Really cool!
I feel like I must be doing something wrong; I’ve read the documentation and looked at the samples, but all of my tests come back with a “TypeError #1010: A term is undefined and has no properties.” I’m running FP 9.0.246.0 (Debug). Nothing I do seems to impact the occurrence of this error. I tried an earlier suggesting to convert “try( method(); )” to “try( method.call(o); )”, because that error is usually a typing issue, but no dice. I’ve tried it on a generic class (inline, no package), and a fully qualified class it uses via composition. I tried running suite just from a fully qualified class, but had the same results. Here is my code:
var performanceTest:PerformanceTest = PerformanceTest.getInstance();
performanceTest.out = trace;
performanceTest.testSuite(this);
And here is some sample output (package is redacted because it contains client’s name):
xxx.xxx.xxx.xxx.PhoneTreeView (100 iterations)
Player version: WIN 9,0,246,0 (debug)
method……………………………………………ttl ms…avg ms
* addChild: TypeError: Error #1010: A term is undefined and has no properties.
* addChildAt: TypeError: Error #1010: A term is undefined and has no properties.
* addEventListener: TypeError: Error #1010: A term is undefined and has no properties….. etc.
BTW, the same thing happens with this simple test:
var performanceTest:PerformanceTest = PerformanceTest.getInstance();
performanceTest.out = trace; performanceTest.testSuite(new Sprite());
I’m running in FB3, is that a possible issue? Does it only run in the Flash IDE?
Think I just realized why most of these are occurring, because the calls are not parameterized… but why am I getting the same thing on methods that take no arguments?
Really nice, helps a lot in performance optimizations.
A thing I’m wondering, don’t know if I’m missing something, but would really want to be able to add parameters to test methods.
e.g.
performanceTester.testFunction( _tileEngine.buildMap, 50);
e.g. with params
performanceTester.testFunction( _tileEngine.buildMap,[“value1″,”value2”], 50);
Once again someone releases a library I have been working on myself before I have had the chance to finish and publish it. 😛 That happens FAR too many times.
And this has been around since April, too. I need someone to inform me when new, useful tools are released, as most of the time, I’m still way behind.
A few months back I raised a few questions regarding performance testing on the kirupa forums (http://www.kirupa.com/forum/showthread.php?t=336669)
Sadly, I didn’t get any replies, but I had a few questions which you (or someone else) here may be able to comment on.
“There needs to be some sort of standard, so no matter how my computer is running, or what computer I am running it from, can still get the same results relative to some standard unit of CPU power/timing.”
I provided one example of this, but it didn’t turn out very successful or accurate…
And my second question “Has anyone already compiled a list of the speed differences of different functions and code snippets?”
I’m still shocked to find out that uint is slower, which I REALLY was not expecting. It takes up the same amount of bytes as int, just with a different offset.
In addition, negative numbers make me nervous… 😉
Cheers, Grant! I’m downloading the library as I write this, and looking forward to it.
Once again someone releases a library I have been working on myself before I have had the chance to finish and publish it. 😛 That happens FAR too many times.
And this has been around since April, too. I need someone to inform me when new, useful tools are released, as most of the time, I’m still way behind.
A few months back I raised a few questions regarding performance testing on the kirupa forums (http://www.kirupa.com/forum/showthread.php?t=336669)
Sadly, I didn’t get any replies, but I had a few questions which you (or someone else) here may be able to comment on.
“There needs to be some sort of standard, so no matter how my computer is running, or what computer I am running it from, can still get the same results relative to some standard unit of CPU power/timing.”
I provided one example of this, but it didn’t turn out very successful or accurate…
And my second question “Has anyone already compiled a list of the speed differences of different functions and code snippets?”
I’m still shocked to find out that uint is slower, which I REALLY was not expecting. It takes up the same amount of bytes as int, just with a different offset.
In addition, negative numbers make me nervous… 😉
Cheers, Grant! I’m downloading the library as I write this, and looking forward to it.
Hello!
I use flex builder 3.0.1 and for some reason I have an error in the PerformanceTest.as.
Type was not found or was not a compile-time constant:Shape
It can’t define the protected var shape:Shape; inside the class.
I don’t really understand why and of course I didn’t change anything inside the class.
I miss something?
Thanks
Any chance that you can post the results of each submission as they come in? It will give all entrants a better idea of what kind of performance they should be looking for… and perhaps drive the bar higher (early submission may need to revise as spankage occurs).
Hey Grant, I was excited to find this post, i was quickly browsing the code and am curious as to your reasoning of using the localConnect to force gc rather than built in gc method in AS? Wouldn’t that be more native speed to collect being that if someone is testing to log they would also be using undoubtedly the debugger version?
Just curious,
Thanks
Hey Grant,i got a “file may be corrupted” error while trying to unzip the library…
can you check?
Thanks anyway 🙂
Hi Grant,
is there a way to use the testFuncion even when the tested a function picks up some arguments?
Thanks Nils
I wonder if this is still true with 10.1
The Kindle Reader is a wireless handheld device for all the book lovers. If there was no kindle reader in the lastdays, well. I would just carry paper magazine reviews of the Sony Reader to fresh’in up with.
I think one very important thing that can not be restated often enough is that you always have to test performance with a release version in the release player. Do not take the values that a debug version or even the debug player gives you as the truth since there can be huge differences.
I always go by the rule: “never trust any ‘faster’ method found in any blog post that you haven’t verified yourself”.
Check this page for some interesting findings but here is the key info
Someone @ Adobe writes us back the explanation:
The AS3 VM uses 32-bit Atom formats to store various types. 3-bits of the Atom describe the type data, leaving 29-bits for actual data. In some cases, like null/undefined/void/etc, we use 3-bits plus a few more bits to define the type.
For signed integers, we have 28-bits to work with plus a sign bit. For unsigned integers we have 29-bits (or maybe 28 with the upper bit always zero?). Once your number gets larger than 28/29 bits, we dynamically switch it to a Number Atom type, which contains a 3-bit type value with a 29-bit ptr. All ptrs are 8-BYTE aligned so we get away with using 29-bits there.
All three formats are basically interchangeable. When storing a number in an Atom, we see if it’s an integer and fits within the 28/29 bits. If so, we store it as a int/uint value – otherwise we store it as a Number value.
Interestingly, I believe that the exact numeric type being used can be determined through a performance test, using an Object as a hash map:
var hashMap:Object = new Object();
var time:int = getTimer();
for( var i:int = 0; i < 1000000; i++ )
hashMap[ 1 << 27 ] = "blah";
trace( getTimer() – time );
If the key's value is between 0 (inclusive) and 1 << 27 (inclusive), this loop takes 35 ms on my hardware. Otherwise (e.g. change 1 << 27 to 1 << 28), it takes a 150 ms.
This leads me to believe that, above 1 << 27 (and below 0), Numbers are used instead of integers.
What surprises me, though, is that 1 << 27 has its first bit set to 1 and the other 27 bits set to 0. Why couldn't ActionScript set all these bits to 1, reaching a value almost twice as great, before switching to the bigger Number type?
Additionally, why are the integer's negative values not used? Why does ActionScript use a Number for valid negative integer values such as -17?
I realize that some (or even all?) of these occurences may apply purely to Object keys rather than to integers in general.
Could anybody share more insight?