Music Visualizer in HTML5 / JS with Source Code

It’s no secret that I like building music visualizers, or that I’ve been playing with HTML5 a fair amount lately. Given that, I thought I’d combine the two interests, and build a music visualizer using JS, the canvas & audio elements in HTML5, and the EaselJS framework.

The primary challenge was that Javascript doesn’t have any built in mechanism for accessing the volume of a playing audio tag. To address this I wrote a little AIR application that will read an MP3 file using Sound.extract() and export peak volume data as a text or JPG image file. I then wrote a JS class called VolumeData.js that reads in these files and provides access to the data via a simple interface (ex. myVolumeData.getVolume(time) ).

With those pieces in place and tested, I started putting together a demo of it in action using EaselJS. Two of the newest features were compositeOperation support (which let me approximate an “add” blend mode), and the drawPolyStar method, both of which I used to excess.

I think the end result is pretty cool, though it requires a fairly modern system, and will still melt your CPU – I was intentionally pushing things hard to try to find the performance limit. It requires an up to date browser to run.

I built two variations: Star Field and Atomic. Occasionally dynamic audio loading seems to break on certain browsers, just reload if it gets stuck on “loading music”.

If you’re interested in building your own music visualizers in HTML5/ Javascript, you can download the demo source, VolumeData.js class (MIT licensed) and VolumeData AIR application here. Also, be sure to check out the latest version of EaselJS (we just released v0.3.2 today).

Please let me know if you build anything cool with the code. I’d love to see it.

EaselJS v0.3.2 Released

I’m happy to announce the release of EaselJS v0.3.2. It provides a few new interaction features, cleans up the docs, and fixes some known issues. Here’s the full list:

– added stage.mouseInBounds
– added DisplayObject.onMouseOver and onMouseOut callbacks
– added stage.enableMouseOver(freq)
– improved support for calculating mouseX/Y in divs with relative positioning
– fixed Graphics.clone()
– fixed an issue with shadows not being reset properly
– migrated to use YUIDocs instead of JSDocs
– fixed an issue that prevented BitmapSequence instances with frameData from working with gotoAndPlay(frameNumber)

I’d like to thank everyone who contributed by providing bug reports, feedback, and edits. In particular I would like to thank Mike Chambers (of Adobe fame) for all his fantastic help on EaselJS. He has been an invaluable resource for bouncing ideas off, providing a critical second perspective, and helping with code / doc edits.

As always, you can get more info and grab the latest zip from easeljs.com, or check it out on the EaselJS GitHub repository.

We’re currently finalizing plans for v0.4 (which looks to be a very aggressive undertaking), and would love any feedback or ideas you can provide here or on GitHub.

Zoë: Export SWF Animation as EaselJS SpriteSheets

Alongside the release of EaselJS v0.3 we’re also releasing the first version of Zoë, a free Adobe AIR application for exporting SWF animations as sprite sheets (single images containing a grid of animation cells), including frame data for use with EaselJS.

This means you can use Flash Pro to lay out your animations then very easily prep them for use with EaselJS and the HTML5 canvas element.

We used an early version of Zoë to prep all of the animations for the Pirates Love Daisies game we released a few weeks ago, which let our illustration team work with a tool they felt comfortable with, using tweens, skeleton constraints, and graphic symbols.

Here’s a quick feature overview:

  • Exports a single sprite sheet image, or individual frames
  • Reads frame labels in the swf to generate frame data.
  • Writes frame data as JSON or EaselJS files
  • Calculates the frame dimensions automatically based on the animation content
  • Saves profiles to make it easy to re-export when art changes

And a screenshot:

You can grab Zoë from easeljs.com/zoe.html. It’s currently not open source, but we’ll likely release the source once we have a chance to fix any major issues that arise with the public release and clean up the code.

EaselJS v0.3 Released!

I’m very happy to announce that version 0.3 of EaselJS has been released. Even better, it’s now on GitHub, so you can fork it to your hearts content.

This version makes it even easier to build compelling content using the HTML5 canvas element and javascript. It has a rewritten rendering engine with support for skewing, an improved and optimized interaction model (5-10x faster), and a slew of other new features and fixes. Here’s the whole list, in no particular order:

  • renamed Tick to Ticker
  • adding a listener to Ticker twice will now move it to the end of the listener list
  • added Ticker.getMeasuredFPS()
  • added Ticker.setFPS()
  • renamed Stage.tick to Stage.update and made Stage.tick point to Stage.update
  • made the pauseable param “true” by default on Ticker.addListener()
  • added Stage.toDataURL(backgroundColor, mimeType)
  • renamed Rectangle.w/h to width/height
  • improvements to commenting and documentation
  • formalized inheritance model to make code more readable
  • changed color to null instead of 0 in Shadow.IDENTITY
  • fixed an error with simple sprite sheets that weren’t set to loop
  • implemented “tiny” instructions for Graphics. (ex. myGraphics.f(“#0FF”).c(x,y,r) )
  • fixed problems with Graphics.clone()
  • fixed bug with passing instructions to a Graphics object constructor
  • complete rewrite of the render engine to be cleaner and more flexible
  • added .decompose() method to Matrix2D
  • added .skew() method to Matrix2D
  • added .skewX/Y on DisplayObject
  • added SpriteSheetUtils.extractFrame(spriteSheet,frame) method
  • added basic multi-line support for Text
  • added DisplayObject.suppressCrossDomainErrors property
  • complete rewrite of the hit testing system to be cleaner and run 5-10x faster
  • improved the interaction model:
    • added .onPress, .onClick callbacks on DisplayObject
    • removed .mouseChildren from Container
    • made .mouseEnabled true by default
    • added .onMouseDown, .onMouseUp, and .onMouseMove callbacks on Stage
    • added MouseEvent
  • Stage.getObjectsUnderPoint() has been moved to Container
  • added Container.contains(child) method
  • added DisplayObject.compositeOperation
  • added appendProperties() and prependProperties methods to Matrix2D
  • added DisplayObject.hitTest() method
  • added stage.snapToPixelEnabled and displayObject.snapToPixel
  • added DisplayObject.updateCache()
  • renamed minified file from easeljs.js to easel.js
  • Re-factored directory layout

Check out EaselJS.com for more info or to grab the latest build.

Update on EaselJS v0.3

EaselJS v0.3 is going to be a very significant update. It will include:

  • a much faster (currently 10-15x) and useful / robust interaction model
  • improved code clarity, documentation, and comments
  • early tool support (ie. the start of an API to allow tooling to export EaselJS content)
  • multi-line text
  • a (theoretically) faster and more flexible drawing engine, with support for skews
  • a public GitHub repo
  • a large number of new features & bug fixes
  • release of a sprite sheet creation tool

I originally expected to have this version released this week, but I’ve broadened the scope of what I’m changing. Given that, I anticipate the release will push out to next week some time, but I’d like to believe it will be worth the wait.

I’m still accepting feedback on bug fixes or minor features for v0.3, and am very open to ideas for v0.4 (which is likely to focus on performance and bounds support).

EaselJS v0.2 Released for HTML5 Canvas

I’m happy to announce that I’ve just released EaselJS v0.2. The focus of this release was on making drawing vector shapes and text to the HTML5 canvas easier with the new Graphics and Text classes. I also added Point and Rectangle classes, some new examples, put in support for multiple params on add/removeChild (ex. addChild(child1, child2, child3) ), added some minor fixes for mobile, and squashed a few bugs.

The next version of Easel will focus on performance improvements, architectural updates (finalizing an inheritance & event model), and improving/formalizing the mouse interaction model. I will likely put 0.3 up on GitHub, and open it for contributions.

As always, you can grab the latest version of the open source EaselJS library from EaselJS.com.

Pirates Love Daisies HTML5 Game Launches!

I’m extremely excited to announce the launch of one of our recent projects: Pirates Love Daisies!

Microsoft approached us a few months ago and asked to work with us to build a best of breed tower defense game in Javascript and HTML5. I was a bit hesitant at first – it’s been quite awhile since I worked with JS and HTML in any great depth, and most of my memories were of fighting browser incompatibilities and cursing the lack of decent developer tools. However, we did some quick code spikes to get a feel for what was possible and decided that this was a great opportunity to gain some applied experience with a new technology while working on a fun project.

Concept

We teamed up with local illustration group Pulp Studios and started brainstorming ideas for the theme of the game. Fairly early on we latched on to the idea of pirates (I mean really, who doesn’t like pirates!), but we needed something for them to defend. We considered going with the typical tower defense model, and simply have the user prevent creeps (enemies) from getting from point A to point B, but I’ve never liked that narrative. What exactly is at point B that’s so important?

We decided that the creeps were out to steal something valuable from the pirates, but we weren’t sure what that should be. Gold was an obvious choice, but it was boring, and it made more sense as the currency for purchasing units and upgrades. Then it hit us… Daisies!

Pirates Love Daisies, Art by Pulp Studios

Why daisies? Because Pirates Love Daisies, of course. The fact that daisies are visually iconic, seemed to be appropriately fun and quirky, and would work well aesthetically may have factored in as well.

With the illustrators working on sketches, we got down to writing code.

Coding

We started out by building core game logic, and a simple library to manage canvas state. We profiled performance, then revisited some of our initial ideas to work with the limitations we found.

Overall, we found working with JS to be a lot easier than we expected. Picking up the language was a breeze, and we were able to apply the processes and approaches we’ve developed from years of working on interactive content with Flash. For me, working with Javascript again actually had a certain pleasant nostalgia to it, as it recalled the days working with ActionScript 1. There is a certain amount of fun and freedom afforded by a dynamic scripting language, though the lack of strict typing, compiler time errors, and language-level support for inheritance (ie. no super keyword) were frustrating but manageable.

We also ran into some browser incompatibilities, though far fewer than we initially feared. The worst of them revolved around audio. It seems that every browser has its own set of irritating quirks related to dynamic audio that we had to stumble our way through. We wound up building an audio manager to resolve these issues that we may clean up and share in the future.

All round, things went fairly smoothly from a coding perspective. We ran into some challenges and had to make some compromises, but we surmounted all the major obstacles and learned a ton.

Pirates Love Daisies

Performance

Overall, I was really impressed by the performance we were able to obtain. Code execution speeds are fairly similar to AS3 (which is impressive if you consider all the extra type data the AS3 JIT has to work with), and graphics performance is, for the most part, reasonable.

IE9 was the big stand out on graphics performance – its GPU-enabled engine is smoking fast, especially with large bitmaps. Interestingly, we found that iOS had terrible performance (when it worked at all) – something like 4-5x slower than Android on comparable hardware.

One other thing we found that’s worth noting is that drop shadows are ridiculously slow in canvas. A few small drop shadows rendered to canvas could drop our framerate in half.

HTML5 Features

HTML5 is actually a collection of standards at different levels of completion, so while it’s easy to refer to this as an HTML5 game, it really just leverages a few of the new features that are included under that umbrella.

Specifically, we used canvas for drawing the game board (terrain, units, creeps, effects, etc), local storage to save local scores, the audio element for all of the sound, and embedded fonts throughout.

Tools and Libraries

One of the challenges we encountered was the lack of good tools. For JS development, we tried a few different options but ultimately settled on Dreamweaver – it’s not a fantastic JS code editor, but it’s decent, cross-platform, and everyone in the office already had it.

The game art was created in Illustrator, then animated in Flash Pro. We built a custom AIR tool for exporting SWF animations to a sprite sheet (a grid of animation frames) which helped out tremendously. We also built a super simple AIR app that preps our JS files, runs level 1 of the google compiler on them, and then runs the result through the YUI compressor. We’re hoping to release both of these apps in the new year, once we have a chance to clean them up.

For testing and debugging, we used the debug tools in every major browser. Roughly half our developers were on Macs using Safari and Chrome, and the other half were using Windows with IE9 and Firefox. This let us spot cross browser issues early on. We also found different problems were more easily solved with different browser’s dev/debug tools.

In our code, we leveraged JQuery, JSON, and a new library we developed called Easel. The Easel library provides a hierarchical display list, similar to the one in AS3, that makes it much easier to work with the canvas element. We released the alpha version of Easel today at EaselJS.com under the MIT license. It should make it much easier for both Flash and JS developers to get started using canvas.

Gameplay

When building a game it’s essential that you don’t underestimate the huge amount of time required to balance gameplay. Our level designer built spreadsheets, graphs, and formula to help model the characteristics of the units, creeps, and maps, but it took hundreds of plays through the game (a sacrifice, to be sure) to get the balance right.

We tried hard to make the game very approachable, with a short tutorial, and easy start, but also layer lots of strategy into it for the more experienced gamer. Our scoring system involves lots of trade-offs. As an example, you get points for defeating creeps and ending waves faster, but you also get points each wave for hoarding gold. You have to decide how to balance these two opposed reward systems.

It was only when, having opened PLD to test some specific feature, I caught myself in my third game trying to beat another developer’s high score (and getting yelled at by my wonderfully patient wife) that I really started to feel like everything was coming together. If you can completely lose track of time while playing your own game, after being immersed in planning, designing, and coding it for a couple months, I think that’s probably a good sign… though it can have a negative impact on office productivity.

PLD-game

Overall

Microsoft was a fantastic client. After suggesting the tower defense genre, they trusted us with all the technical and creative details to make the project a success, while offering great feedback throughout. They put a lot of faith in a company that’s really not known for Javascript or game development, and I’d like to believe it was well placed.

I never thought I’d say this, but Internet Explorer 9 actually looks to be a great browser. It has impressive performance, and seems to be very standards compliant. I would definitely recommend checking it out, you might be surprised.

Everyone involved with the project had a huge amount of fun building it out. We had a great time brainstorming ideas, coming up with ways to tweak gameplay, and spending (way too much) time play testing. We’d love to hear what you think of the game in the comments below.

Easel JS Simplifies Working with Canvas in HTML5

The new Canvas element in HTML5 is powerful, but it can be difficult to work with. It has no internal concept of display objects, so you are required to manage updates manually. The Easel Javascript library provides a full, hierarchical display list, a core interaction model, and helper classes to make working with Canvas much simpler.

We built Easel based on what we learned while building Pirates Love Daisies, then retrofitted the game to use it. Development on the project would have been a lot simpler if we had Easel from the start.

Easel is being released for free under the MIT license, which means you can use it for almost any purpose (including commercial projects). We appreciate credit where possible, but it is not a requirement.

Examples, API documentation, and the latest version of the library can always be found at EaselJS.com.

Easel is currently in early alpha. We will be making significant improvements to the library, samples, and documentation over the coming weeks. Please be aware that this may necessitate changes to the existing API. We will also be moving it to Google code or GitHub and opening it to contributors once we feel like it’s in a decent beta state.

The API is loosely based on Flash’s display list, and should be easy to pick up for both JS and AS3 developers. In our preliminary testing, Easel appears to be fully compatible with iOS, Android, and all major browsers that support the canvas element.

We welcome feedback on the library in the comments below, at least until we get something more permanent in place.