Flash-animation in Unity3D from scratch. Part one, lyrical

In this article series, I will talk about how and why we decided to create our own solution for import of Flash animation to Unity, and about optimization techniques and internal workings of the plug-in. I also have lots of other fascinating stuff to tell about: internals of the SWF format, special features of the Unity editor extension and general matters of animation. You'll find all that inside!

Introduction

When you start working on any project, you always have to face the problem of choosing the technologies for most its components. One of these components is, of course, the animation system. Now, there are a few variables which your choice will depend on. First question you have to ask yourself is what tools are your animators used to (can work with), and which of those are easier for you to find. Sure thing, choosing an extremely specific product doesn't make much sense. In this case, if your employees leave for some reason (or get hit by a bus), it will be almost impossible to replace them. So, basically, your project, whole or in part, will be stopped for an indefinite period, which can be very expensive, especially for a commercial product. The second variable is more technical one, and it's abou integrating the tools into your engine. Are there any third-party solutions and what is the quality of those? Do you have opportunity and ability to create your own solution? Of course, the efficiency and convenience of all of the above is crucial, too. And the third variable is the tools' capabilities. Thus, if you really, really need inverse kinematics, then it would be very weird of you to prefer non-skeletal animation.

Types of 2D-animation

Now, let's look at the most popular animation techniques in games. I will consider only 2D-animation, since 3D animation is a topic for a completely different conversation, with its own approaches and tools. Feel free to skip this section if you are well-versed in this matter (trust me, there are going to be no revelations or eye-openers). Just as well, if you're here only to see the features of Flash animation, you will find them in the article below.

Frame-by-frame animation

Kicking-off our mini-review, the simplest and most ancient type of animation: frame-by-frame animation. In this animation, each frame is represented by an individual picture, and by changing those pictures really quickly, we get the illusion of movement.

Pros:

Elementary implementation, usually available for any engine. Any complexity and style of animation, you can even insert the whole movie frames (in theory, of course). Any tools: almost anything can provide you with a sequence of frames that you can, if you need to, put together into one texture atlas.

Cons:

The main disadvantage is, of course, the amount of memory all this stuff requires. Certainly, you can use all kinds of tricks, such as cutting frames in blocks and then reuse of the repetitive blocks, background loading of frames from disk and removing the ones already shown, and so on. But remember that all of these tricks have their own drawbacks, too. For example, cutting in blocks can be only used in pixel art, background loading places an additional strain on the disk, takes a lot of space on this disk, thus increasing the size of the game itself, as well as the amount of data required for a user to launch or update it.

Bottom line:

Works only for small and few animation frames, perfect for pixel art and retro-style games like NES.

Video animation

Basically, video animation is an apotheosis of the frame-by-frame madness. And yes, it still is applied and sometimes even rightly. We've implemented something like this in Hidden Object games for bulky and complex animation, and also for cutscenes with a hint of realism. Add to this codecs like theora or vp8, and you will be able to create quite nice video animation (even with alpha compositing) of your own.

What's good about this kind of animation is, of course, that you can create almost anything, from stage-managed scenes with real actors to rendered 3D-battles. Bad thing is a great load on the CPU when you're decoding and, of course, the image quality. The compromise between quality and efficiency can be achieved only in limited number of specific cases, like in above-mentioned Hidden Object games.

So, anyway, video animation is not for everyone and not always applicable, almost never, to be precise. Basically, this type of animation works great for rendered cutscenes only. If you manage to make a good implementation of such animation, it sure won't be a piece of cake. There are tons of nuances and tricks to bear in mind, both regarding the quality of the output image and the CPU load. Due to this specificity, there are very few ready-to-use implementations, which either are not good enough, or cost big money (hello, Bink_Video), so most people prefer to write their own ones.

Skeletal animation

So, let's move towards more popular and modern types of 2D-animation. Skeletal animations are winning more and more hearts and minds of 2D-developers, getting closer to being the game animation standard. Funny thing is, they've actually became so popular in 2D relatively lately, unlike 3D, where they go way back, when our ancestors were creating Half-Life. The essence of skeletal animation, you'd be surprised, is a skeleton created by an animator. Basically, a skeleton is a set of bones connected with each other and forming a tree structure. The bones are bound to pieces, represented by individual pictures. This construction starts to move when bones are shifted or rotated relative to one another. The shifts and rotations of the skeleton bones stick to the general animation timeline.

The pros are obvious: we don't have to store every animation frame as a separate image, just the pieces (for example, legs and arms of the character), which will be moved by the bones. There are some really cool implementations of this type of animation with runtimes for different platforms and engines, which are affordable or even free: Spine, Spriter, Anima2D, DragonBones, and others. Using the skeleton and its features, you can achieve incredible smoothness level of your animation by interpolating the positions of the bones. You can and also alternate the transitions between the scenes and even mix two different types of scenes: shooting while running, shooting while creeping and so on. I'm not going to list all the possibilities of skeletal animation, so you just keep in mind that there are really a lot of them and they are really cool. And I'd rather give you a link to the Spine website where you can take a good look at all of them, with pictures and descriptions

Looks like we got our universal soldier, but not just yet. It also has its own drawbacks. And I must say right away that for many projects and animations these disadvantages may not even exist. So, if skeletal animation works for you — that is just great, use it, cause it looks really fine and modern.

Now, let's go back to drawbacks. Skeletal animation requires an additional step — rigging. It's about creation of a skeleton and binding bones and pieces. If your object has just one animation, not a few ones, like, for example, the character of the platform game, then this step makes no use for you. You should also remember that not all scenes can be easily animated with bones. In fact, there are many examples when it's better off without them, both in terms of creating and end game performance. I would also point out the availability of experts on the market as an important factor. Of course, with the increasing popularity of skeletal animation their number is growing too, but still it probably won't be easy for you to get them in your team. The variety of tools is also doing its part. Everyone has its own preferences, and still it takes to be flexible and ready to learn, which creates additional difficulties in recruiting and training your employees.

Timeline animation

I saved this type of animation for last not because it's better than all the others, but because it's mainly represented by Flash animation the insides of which we'll be talking about in the technical part of this article. Now, Flash is not the only representative of this type of animation, but it is certainly the leading one, so from this moment forward, I'm going to describe everything in relation to it. Timeline animation scenes consist of pieces animated in different layers on the timeline. Basically, it's almost the same thing as what we saw in skeletal animation, but without skeletons. Using the keyframes on the timeline, we can move, rotate, replace, and add new pieces into these frames and, of course, interpolate transitions, scale-ups, and rotations between them. It means that in this case, we move not the bones which pieces attached to, but the pieces themselves. Sure, it deprives us of many possibilities of skeletal animation, but like I said before, not everyone needs them. Moreover, we get other advantages, such as:

• ability to add and insert new pieces in the middle of a scene;
• no rigging, no bones that sometimes are just unnecessary;
• Adobe Flash (currently Adobe Animate) the oldest and tested through the years animation tool; a huge number of animators, one way or another, can work with it;
• ability to use old scenes when transferring your Flash projects to Unity, including the scenes you made in vector graphics, not only in raster.

You may ask, isn't Flash dead? As a browser player — yep, but as an animation tool, it's still in the game. And I'll tell you more, there are no alternatives to Flash now or foreseen. Are there any drawbacks? Sure. And the main disadvantage is that… it's not skeletal animation, ha-ha. So, we're losing the ability to mix our scenes with each other, inverse kinematics and a few other features you can use only when you work with skeletons. But we got timelines inside one another, and also raster and vector masks! And animators just love masks! Here it must be noted that Spine has also recently added the clipping option, but so far they're geometric only and severely limited compared to Flash masks.

Animation types: Summary

Of course, I've listed not all types of animation, but just the main ones. For example, I left behind so-called procedural animation, when objects or pieces are driven by code only, but it’s hardly possible to compare this one to others, and I didn't want to make the article too long to read. I guess, now it’s better for us to take a closer look at something more specialized in animation to complete the picture.

So, to sum up. Like always and elsewhere, there's no universal soldier here. You need to choose your solution depending on your project, tasks, and people. After all, who says you've only got to use one type animation at a time? There are lots of projects where we can see all types of animation used, and everyone's happy. Video animation for cutscenes, skeleton for characters, timeline for scenery, procedural for flying of emitters particle systems, and single-frame for the particles themselves. We simply take the best from each type.

Dilemma and decision to write our own solution

Anyway, for some objective reasons, we decided that for our project we need Flash-animation. Then the question aroused, about their integration into the engine. Having tried several plug-in options, we opted for the one that we considered best for its capabilities. Although it was living and well-supported, it also had a completely unaffordable enterprise license, but who really can put a price on a good product? Now, I intentionally will not give you any names, so not to make it look like advertising or anti-advertising.

We spent almost a year of development with this plug-in, during which period it's become clear why we can't continue to use it. In particular, it was the quality of its integration to Unity. This resulted in quite serious bugs which not easily, not quickly and not without a fight were defeated by the developers. The performance on target devices was also terrible, and for us it still means iPad 2 level. It happened because we tried to make an integration to a particular engine, ignoring its specifics and potholes. We were more than satisfied with the functionality, but the developers refused to open the source code for us, even on an individual basis. So, that's how decided to write our own solution.

I had some quite successful experience in writing extensions for Unity and a little experience in converting Flash animation scenes to my own formats. It's been quite some time since the last one, to be fair, but something I still remembered. In the same time, I made a decision to write all this as a pet project, so I would not depend on the main project or the company where I work. And, let's face it, it's nice to have such a production on your own. Anyway, armed with a 250-page SWF format specification, I marched into battle.

Export options

For starters, let's discuss the options we have for exporting animation from Flash editor. There are a few of those, each one having its own advantages and disadvantages. We're going to go over the main ones.

.XFL format

In Flash animation editor you can save the source file in an uncompressed .XFL format instead of the closed .FLA set by default. Now, this format is pretty easy to work with, but it has no documentation. Basically, it consists of a few subdirectories and a whole bunch of .XML files describing all states of the clips stored inside. The description format is also very simple and understandable, as you can see from the example below:

static_clip.xml

<DOMSymbolItem name="static_clip" itemID="5c719f28-00000051" lastModified="1550950184">
  <timeline>
    <DOMTimeline name="static_clip">
      <layers>
        <DOMLayer name="Layer_1" color="#00FFFF" current="true" isSelected="true">
          <frames>
            <DOMFrame index="0" keyMode="9728">
              <elements>
                <DOMBitmapInstance selected="true" libraryItemName="bitmap.png"/>
              </elements>
            </DOMFrame>
          </frames>
        </DOMLayer>
      </layers>
    </DOMTimeline>
  </timeline>
</DOMSymbolItem>

Source link

What we got here is this clip, static_clip with one layer, Layer_1 and one frame storing a raster image called bitmap.png.

movie_clip.xml

<DOMSymbolItem name="movie_clip" itemID="5c719f30-00000053" lastModified="1550950713">
  <timeline>
    <DOMTimeline name="movie_clip">
      <layers>
        <DOMLayer name="Layer_1" color="#00FFFF" current="true" isSelected="true">
          <frames>
            <DOMFrame index="0" duration="4" tweenType="motion" motionTweenSnap="true" keyMode="22017">
              <elements>
                <DOMSymbolInstance libraryItemName="static_clip">
                  <matrix>
                    <Matrix tx="-50" ty="-50"/>
                  </matrix>
                  <transformationPoint>
                    <Point x="28.5" y="27.5"/>
                  </transformationPoint>
                </DOMSymbolInstance>
              </elements>
            </DOMFrame>
            <DOMFrame index="4" tweenType="motion" motionTweenSnap="true" keyMode="22017">
              <elements>
                <DOMSymbolInstance libraryItemName="static_clip" centerPoint3DX="128.5" centerPoint3DY="127.5">
                  <matrix>
                    <Matrix tx="100" ty="100"/>
                  </matrix>
                  <transformationPoint>
                    <Point x="28.5" y="27.5"/>
                  </transformationPoint>
                </DOMSymbolInstance>
              </elements>
            </DOMFrame>
          </frames>
        </DOMLayer>
      </layers>
    </DOMTimeline>
  </timeline>
</DOMSymbolItem>

Source link

Here you can see the description of the clip movie_clip, which consists of two keyframes with indexes 0 and 4, respectively. In those frames our clip static_clip is stored by the coordinates (-50;-50) и (100;100). Here there's motion tween between the frames (a procedural animation, in our case it's just only a shift, without scaling and rotation). Thus, the position of the static clip between the key frames can be calculated using linear interpolation of the coordinates we take from these frames.

Of course, in case of a real animation it's all will be, uh… a little bit more complicated and bulky, but still it can all be understood, even without documentation. And it might seem to you that this is it. And actually, I'm aware of several projects and companies that went this way and was quite successful, but not without some limitations and difficulties. And these difficulties are the real bump of .XFL, which are:

1. Tweens can be very different, both simple — with linear interpolation, and more complex ones — with customized functions and graphs of this interpolation; vector graphics morphing on tweens is performed only according to their own, understandable for Macromedia and Adobe guys only, rules;
2. Vector graphics, just like all animation except for raster graphics, is described in a text file., for example. It means that you need to write your own rasterizer, which, due to the nature and complexity of vector format in Flash, cannot possibly be done;
3. The rules for animation playback, including the nested ones, need to be made up from scratch. There is no documentation for this, and answers to such questions as when and which frame should be shown, when to tween and when leave it without interpolation, you're going to have to find out on your own, at the same time trying to cover all possible cases during test runs, which, believe me, is a very non-trivial exercise for a general, not private, solution.

These are not all drawbacks of this approach, but I think these are enough for you to understand that it would work for a general solution only with strong limitations. A long time ago, I went down this road working on one of the projects of mine. There were Flash cutscenes with classic tweens (with no customized interpolation rules and special interpolation functions), jut raster graphics and no other advanced opportunities provided by the Flash editor. I came up with that proprietary solution pretty quickly and efficient, but it still required some severe limitations on the animators, so that they would not use anything you can call complicated. The approach has the right to exist and there are a few libraries based on it, with varying degrees of shabbiness, but, as I said, it works with a lot of limitations.

.JSFL scripts

Another almost working option to get information about our animations, .JSFL scripts allow you to extend your Flash editor, interact with its environment, edit your animation and, of course, get all the necessary information about the timelines, layers, clips and frames inside. It is also used by animators for automatization of various processes, but this is actually a completely different story. As the whole, the approach possesses all the drawbacks of the previous one, so I'm not going to go deeply into it. Let me just say that it allows you to export your animation scenes in, for example, frame-by-frame form, but we know that this is not what a real Jedi would do (but, of course, it might be done in some projects). We will get back to these scripts later, when we be discussing the approach I chose: for rasterization of vector graphics and optimization of the exported scenes.

AIR application

This approach is really working and can be used as it should. Still, I didn't choose it, but I'm going to tell you about this option, just for the record. The essence of this approach is that we create an AIR application on Flash itself or, for purists, for example, on Haxe. This application will extract all the information from the animation scenes you've already compiled into the SWF format. In the application we play the scene frame by frame, get all the information about frames and save it in the format most suitable for our runtime. Using this method, we address all the problems I mentioned above:

• we don't need to rasterize our vector graphics, since it will be done by Flash runtime. The only thing left for us to do is to get this information and save the raster output of our vector pieces;
• we don't have to invent playback rules and deal with customized tween interpolation, the Flash player integrated in the AIR application knows exactly how it should be done and basically does the work for us;

As a bonus we get an opportunity to use frame scripts (all the play(), stop() and other gotoAndPlay(), some animators really like them). As a drawback, we lose an opportunity to export the scenes looped in Flash itself. But it's not that bad, really, for we can loop them in our runtime, and the animators could make everything ready for that.

It should be noted now that I'm sure that everything will be more or less harder than I describe it, since I personally haven't gone there, so I can't really tell you all the details. I welcome those who have gone this way to share their experience, I'll be glad to read about your successes and failures!

Your own Flash player

This is probably the most fair, obvious and straightforward option among all, but also it is the most difficult one. After all, that is what a real Flash player does, the one which most of us installed in the favorite browser as an add-on. Now, the popular representatives of this genre are: gameswf and, born out of the first one, scaleform. Both of them are dead today. What's fascinating about them is that they were used mostly for GUI implementations in various projects including AAA. But our interest is in the internals, not the applications of dead libraries.

Any decent Flash player has at least two important things hidden inside:

• SWF format parser;
• vector graphics rasterizer;
• virtual machine for launching ActionScript of the code;
• standard library for a custom code;

I know that you think. Each of these points above is screaming that even if something like this is possible to implement, you're going to have to switch more than one team for this task and spend a few years on development of the basic version. Even the implementation of a small part just to play something, will take you a long while, and you can be sure that the further additions in the form of many undocumented parts of this pipeline will double this long period.

Yes, the Internet is full of attempts to implement each of these parts, varying in degrees of shabiness and buggedness. But you can't even take these developments as a basis for your own stuff since you would have to spend years only for bug hunting. Also you would have to add all the unwritten Unit tests before you realize that you never will be able to fully implement every little thing. Those guys who tried to add necessary capabilities to gameswf will understand what I'm talking about. A fair Flash player should also rasterize vector graphics not in advance, but in the runtime, which will make such an impact on the performance that not every project can get through. By the way, I want to say hello to the survivors who tried to use scaleform on mobile devices. Anyway this is the most solemn and hopeless way, and it's definitely not our choice.

Combining the approaches

So, we've finally go to my option. Having studied all the possible ways, I came up with this interesting combination. It's relatively easy to implement, manageable by one person and at the same time has enough functionality to cover all needs of almost any 2D timeline animation production.

First, we're going to use the animation scene compiled in SWF, as not to invent various playback options and almost never guess how the Flash player works on the inside, since we can’t possibly cover all cases, but we're still looking for the general solution. Also, Flash editor gets rid of all the tweens we used in our animation, giving us only the bare positions of the pieces in the compiled SWF.

Second of all, we prohibit the use of scripts in animation. Yes, this requirement is sure very tough and sad, but I don't have an entire team of programmers to implement a decent virtual machine and standard library that they would have to write almost blindfold. Here, of course, you can see a drawback of my approach in comparison with the AIR application method. In the second one we could use some scripts for internal animation playback. At the same time, it doesn’t stop us from creating good animation scenes. For transferring custom information from the scene into the game, you can use so-called frame labels, which you will find out in the code. Also you can pin user events on the particular frames using callback functions of the runtime.

Then, we get rid of writing our own vector graphics rasterizer, since we can't write it for the general case and to cover every option and possibility of vector in Flash. Instead, we will rasterize our graphics before (!) compilation using a JSFL script. Meanwhile, using this very script, we are going to optimize our vector graphics by merging static clips in one image, and zoom the resulting raster images in or out for the sake of our own priority: quality and/or performance. Also here it's worth taking into account art quality/resolution options for devices with various screen pixel density (for example, HD and SD art).

Conclusion of the lyrical part

This concludes the first part of my article. In the second one we're going to discuss technical implementation details, code fragments, pictures (!), optimization, tricks and tweaks. Coming soon, do not miss it!