Dec

Flex Tutorial - An Asynchronous JPEG Encoder

This weekend I needed to encode a rather large image as a jpeg using flex. And you know what I found? It was slow. And cpu hungry. While this was not entirely unexpected, it was disappointing. I could have dealt with the program being slow and unresponsive during the time it took to encode the image, except for one problem - after about 15 seconds, the flex app would run up against the script timeout error. You know, the error that goes along the lines of "A script has executed for longer than the default timeout period of 15 seconds." And because of this, the jpeg encode never actually finished.

Well, that was no good. I poked a bit at extending the default timeout period, but really that was only a partial solution. Who knows how long it might take to encode the image on a slow computer? So I started poking at what it would take to make the jpeg encoding asynchronous. Sadly, actionscript and flex do not have the concept of just saying "hey go do this and get back to me when your done." This is probably because actionscript is single threaded - everything just runs on the browser's main thread.

Download Source
Above we have a small sample app showing off the asynchronous jpeg encoder that we are going to build in this tutorial. You see the spinning lag meter? Right now, it is probably spinning nice and smooth. This is because flex has plenty of cpu cycles to update the spinning circle. When there isn't enough cpu time available, that animation will start to look choppy, as Flex will only be able to update it once every few hundred milliseconds, possibly longer. If there are no spare cycles, the circle will freeze and stop spinning altogether. If you hit "Normal Encode", that is probably exactly what will happen, and your entire browser will freeze along with it. That is because by clicking "normal encode", you told flex to encode the image displayed in the sample app (which is a 2800x2100 pixels) as a jpeg. If your computer is slow on mine, you will eventually get the "script timeout" error, and the encoding will never finish.

On the right hand side, however, is the button to trigger the asynchronous jpeg encoding. When you click that, the app won't freeze and the circle will keep spinning (although not quite as smoothly). You will also get a progress bar that shows the progress of the encoding. By moving the "pixels per iteration" slider bar to the right, you will increase the speed of the encoding, but decrease the responsiveness of the interface, and if you move the slider to the left, you get the opposite effect. A little farther down I will explain how that is accomplished.

So how do you even go about making something like this asynchronous? As I said above, we only have one thread to work with, so it is all hopeless, right? Not quite - there are ways to act like a multi-threaded system, such as using something like setTimeout. You can use setTimeout to emulate threading - essentially do a little work, but then you queue more work for the future (not immediately). The app then has a chance to breathe and deal with things like UI input during the pauses between work items.

It was here that I ran up against another issue. How was I going to break the act of encoding a large jpeg into manageable little pieces? Well, there is no way to do that from outside the jpeg encoding object, so I went into the Flex source code and found the jpeg encoder. And this is where the real meat of this article starts.

Below we have the problem loop in the original jpeg encoder:

for (var ypos:int = 0; ypos <height; ypos += 8)
{
for (var xpos:int = 0; xpos <width; xpos += 8)
{
RGB2YUV(sourceBitmapData, sourceByteArray, xpos,
ypos, width, height);
DCY = processDU(YDU, fdtbl_Y, DCY, YDC_HT, YAC_HT);
DCU = processDU(UDU, fdtbl_UV, DCU, UVDC_HT, UVAC_HT);
DCV = processDU(VDU, fdtbl_UV, DCV, UVDC_HT, UVAC_HT);
}
}

This loop can take a long time on large images - or at least on my computer it does. So the essential idea here is that we don't want to do this whole loop at once. We want to process the image a chunk at a time, giving the rest of the app time to work between chunks. So how do we do this? Well, we write a function that can process the loop a chunk at a time:

private function AsyncLoop(xpos:int, ypos:int):void
{
for(var i:int=0; i <PixelsPerIter; i++)
{
RGB2YUV(Source, xpos, ypos, SrcWidth, SrcHeight);
DCY = processDU(YDU, fdtbl_Y, DCY, YDC_HT, YAC_HT);
DCU = processDU(UDU, fdtbl_UV, DCU, UVDC_HT, UVAC_HT);
DCV = processDU(VDU, fdtbl_UV, DCV, UVDC_HT, UVAC_HT);

xpos += 8;
if(xpos>= SrcWidth)
{
xpos = 0;
ypos += 8;
}

if(ypos>= SrcHeight)
{
setTimeout(FinishEncode, 10);
return;
}

}
setTimeout(AsyncLoop, 10, xpos, ypos);
}

This AsyncLoop function takes a xpos and ypos into the image, and starts processing from that point. But it only loops PixelsPerIter times. After it has processed that many chunks, it leaves the loop, and calls setTimeout on AsyncLoop. In this setTimeout call, it hands the current x and y position in the image - so when the function is called again, it starts up in the right place.

By setting a 10 millisecond timeout, the application will have time to do other things before it gets back to processing this image data. The PixelsPerIter value is very important - it determines how responsive the application is during the image processing. If PixelsPerIter is 1, the application will seem perfectly responsive. But it will also take a long time for a large image to encode, because the encoding is pausing for 10 milliseconds between event chunk. If you set the value to, say, 10000, the app will become unresponsive, because it is taking multiple seconds to process data before it responds to any UI input. Playing around, I've found that 128 is a pretty good value - the app slows down a little bit, and it takes under 2 minutes to encode a 2880x2880 jpeg.

So what do we do here when we are done processing the pixels? Well we do a setTimeout to FinishEncode, and return out of AsyncLoop. FinishEncode holds all the code that was later than the main loop in the original encode function:

private function FinishEncode():void
{
//EOI
if (bytepos>= 0)
{
var fillbits:BitString = new BitString();
fillbits.len = bytepos + 1;
fillbits.val = (1 <<(bytepos + 1)) - 1;
writeBits(fillbits);
}
writeWord(0xFFD9);
}

Ok, well, that makes sense, but how to we signal the rest of the application that the encoding has completed? Its not like we can just wait for a function call to return. Because the encoding is asynchronous, the original encode call will return immediately - long before the actual encoding is finished. So instead, we use an event. In this case, I created my own event, because I wanted the event object to hold the encoded image:

public class JPEGAsyncCompleteEvent extends Event
{
public static const JPEGASYNC_COMPLETE:String
= "JPEGAsyncComplete";

public var ImageData:ByteArray;

public function JPEGAsyncCompleteEvent(data:ByteArray)
{
ImageData = data;
super(JPEGASYNC_COMPLETE);
}
}

To use this event on the new jpeg encoding class, we add an attribute at the top of the class, and make the class extend EventDispatcher:

[Event(name=JPEGAsyncCompleteEvent.JPEGASYNC_COMPLETE,
type="JPEGAsyncCompleteEvent")]
public class JPEGAsyncEncoder extends EventDispatcher
{
...

And now to fire the event, we add a dispatchEvent call to the FinishEncode function:

I also wanted to add a progress event, so that I could show a progress bar to the user as the jpeg was encoding. To do this, I added another event to the class:

[Event(name=JPEGAsyncCompleteEvent.JPEGASYNC_COMPLETE,
type="JPEGAsyncCompleteEvent")]
[Event(name=ProgressEvent.PROGRESS,
type="flash.events.ProgressEvent")]
public class JPEGAsyncEncoder extends EventDispatcher
{
...

And with this progress event, I added a bunch of logic to the AsyncLoop code to fire the progress event at appropriate points:

private function AsyncLoop(xpos:int, ypos:int):void
{
for(var i:int=0; i <ChunksPerIter; i++)
{
RGB2YUV(Source, xpos, ypos, SrcWidth, SrcHeight);
DCY = processDU(YDU, fdtbl_Y, DCY, YDC_HT, YAC_HT);
DCU = processDU(UDU, fdtbl_UV, DCU, UVDC_HT, UVAC_HT);
DCV = processDU(VDU, fdtbl_UV, DCV, UVDC_HT, UVAC_HT);

xpos += 8;
if(xpos>= SrcWidth)
{
xpos = 0;
ypos += 8;
}

if(ypos>= SrcHeight)
{
setTimeout(FinishEncode, 10);
return;
}

CurrentTotalPos += 64;
if(CurrentTotalPos>= NextProgressAt)
{
this.dispatchEvent(new
ProgressEvent(ProgressEvent.PROGRESS,
false, false, CurrentTotalPos, TotalSize));
NextProgressAt += PercentageInc;
}
}

setTimeout(AsyncLoop, 10, xpos, ypos);
}

That piece of code actually uses a bunch of fields that I added to the class to keep track of total progress, so that the progress event gets fired only about once a percent or so.

Those are pretty much all the changes that I needed to make (barring the couple added fields). Below you can see the entirety of what I did to the code. I put "...." where code from the original jpeg encoder class would be - there is a lot of it, so I didn't want to paste it all here:

[Event(name=JPEGAsyncCompleteEvent.JPEGASYNC_COMPLETE,
type="com.pfp.events.JPEGAsyncCompleteEvent")]
[Event(name=ProgressEvent.PROGRESS,
type="flash.events.ProgressEvent")]
public class JPEGAsyncEncoder extends EventDispatcher
{

......

private var DCY:Number = 0;
private var DCU:Number = 0;
private var DCV:Number = 0;
private var SrcWidth:int = 0;
private var SrcHeight:int = 0;
private var Source:Object = null;
private var TotalSize:int = 0;
private var PixelsPerIter:int = 128;
private var PercentageInc:int = 0;
private var NextProgressAt:int = 0;
private var CurrentTotalPos:int = 0;
private var Working:Boolean = false;

.....

public function set PixelsPerIteration(val:int):void
{ PixelsPerIter = val; }

public function get ImageData():ByteArray
{ return byteout; }

public function encodeByteArray(raw:ByteArray,
width:int, height:int):Boolean
{ return internalEncode(raw, width, height); }

public function encode(image:BitmapData):Boolean
{ return internalEncode(image, image.width, image.height); }

private function internalEncode(newSource:Object,
width:int, height:int):Boolean
{
if(Working)
return false;

Working = true;
Source = newSource;
SrcWidth = width;
SrcHeight = height;
TotalSize = width*height;
PercentageInc = TotalSize/100;
NextProgressAt = PercentageInc;
CurrentTotalPos = 0;

setTimeout(StartEncode, 10);
return true;
}

private function StartEncode():void
{
// Initialize bit writer
byteout = new ByteArray();
bytenew = 0;
bytepos = 7;

// Add JPEG headers
writeWord(0xFFD8); // SOI
writeAPP0();
writeDQT();
writeSOF0(SrcWidth, SrcHeight);
writeDHT();
writeSOS();

DCY = 0;
DCV = 0;
DCU = 0;

bytenew = 0;
bytepos = 7;

this.dispatchEvent(new
ProgressEvent(ProgressEvent.PROGRESS,
false, false, 0, TotalSize));

setTimeout(AsyncLoop, 10, 0, 0);
}

private function AsyncLoop(xpos:int, ypos:int):void
{
for(var i:int=0; i <PixelsPerIter; i++)
{
RGB2YUV(Source, xpos, ypos, SrcWidth, SrcHeight);
DCY = processDU(YDU, fdtbl_Y, DCY, YDC_HT, YAC_HT);
DCU = processDU(UDU, fdtbl_UV, DCU, UVDC_HT, UVAC_HT);
DCV = processDU(VDU, fdtbl_UV, DCV, UVDC_HT, UVAC_HT);

xpos += 8;
if(xpos>= SrcWidth)
{
xpos = 0;
ypos += 8;
}

if(ypos>= SrcHeight)
{
setTimeout(FinishEncode, 10);
return;
}

CurrentTotalPos += 64;
if(CurrentTotalPos>= NextProgressAt)
{
this.dispatchEvent(new
ProgressEvent(ProgressEvent.PROGRESS,
false, false, CurrentTotalPos, TotalSize));
NextProgressAt += PercentageInc;
}
}

setTimeout(AsyncLoop, 10, xpos, ypos);
}

private function FinishEncode():void
{
//EOI
if (bytepos>= 0)
{
var fillbits:BitString = new BitString();
fillbits.len = bytepos + 1;
fillbits.val = (1 <<(bytepos + 1)) - 1;
writeBits(fillbits);
}
writeWord(0xFFD9);
this.dispatchEvent(new
ProgressEvent(ProgressEvent.PROGRESS,
false, false, TotalSize, TotalSize));
this.dispatchEvent(new JPEGAsyncCompleteEvent(byteout));
Working = false;
}

.......

}

The methods internalEncode, encode, and encodeByteArray replace methods in the original jpeg encoder class. Everything else shown above is an addition. I added all of those fields show at the top because unlike the original encoder (where almost everything was done in a single function call), I needed to remember things across function calls.

One thing to note, this encoder does not implement IImageEncoder like the original jpeg encoder does. This is because that interface expects the encode and encodeByteArray to return a byte array containing the encoded image. Obviously, since this class does all the encoding asynchronously, it can't return the byte array straight out of the original call, because the encoding hasn't actually been done yet. Instead, the functions return a boolean. If the call to encode returns false, it is because this instance of the class is already encoding an image - so it can't start encoding another one yet.

So we have this awesome asynchronous jpeg encoder class. How do we use it? Well, lets take a look at some sample code:

private function startEncode(imageBitmapData:BitmapData):void
{
var encoder:JPEGAsyncEncoder = new JPEGAsyncEncoder(80);
encoder.PixelsPerIteration = 128;
encoder.addEventListener(
JPEGAsyncCompleteEvent.JPEGASYNC_COMPLETE, encodeDone);
encoder.addEventListener(ProgressEvent.PROGRESS, encodeProg);
encoder.encode(imageBitmapData);
}

private function encodeProg(event:ProgressEvent):void
{
var percentage:String =
((event.bytesLoaded / event.bytesTotal)*100) + "%";
//Display the percentage somewhere
}

private function encodeDone(event:JPEGAsyncCompleteEvent):void
{
var data:ByteArray = event.ImageData;
//Do something with the encoded image
}

And there you go. Thats all you need to use this asynchronous jpeg encoder class. You can grab the source code for the whole example above here, and feel free to use it for whatever you want. If you have any questions or comments, feel free to leave them below.

Posted in Flex, All Tutorials by The Tallest |

20 Responses

Anurag Says:
January 16th, 2008 at 1:15 am
Indeed a gr8 way to work on images, but what about charts.
‘asyncEncoder.encode(Bitmap(img.content).bitmapData);’
for charts we cant get .content,
There we get a bitmap as
var bd:BitmapData = new BitmapData(bar.width,
bar.height);
bd.draw(bar);
But this is not giving advantages of async encoding, Screen still hangs and timeout is reached. let me know if u have any solution/workaround
Ryan Says:
January 29th, 2008 at 2:44 pm
Just want to say that this is an awesome tutorial!
Blake Eaton Says:
March 10th, 2008 at 4:13 pm
This article is great! This helped me solve a huge problem. I had to iterate through several thousand items in a datagrid and the operation was timing out. Not anymore! Thank you very much for posting this!

Thanks,
Blake Eaton
Brian Says:
April 17th, 2008 at 1:13 pm
Hi there:

I’m working on a project in Flash MX 2004 for a client and I’m having a problem that is similar to the one that you described in your tutorial. Unfortunately setTimeout is not supported in MX04 and setInterval requires quite a bit of tricky garbage collection. Do you have any recommendations for what I might be able to do in MX04 to faux-multithread? The application I’m working on is targeted for FlashPlayer 7.
Matt Kane Says:
May 17th, 2008 at 3:06 am
Hi,
Thank you for this! I was about to write something very similar when I ran across yours. I’m using it in the latest version of the CleVR Stitcher and would like to mention you in the credits. How would you like to be credited? Incidentally, the Stitcher uses the same method for fake threading, and it’s an absolute nightmare to keep track of. We have literally dozens of different loops that are broken up like this! Oh how I wish there were an easier way.
Regards,

Matt
Agustin Says:
June 2nd, 2008 at 8:06 am
Hello, thanks for this incredible class, i’ve been using in a personal project and i have a problem to free the memory that i use.
I use it to encode 6 images and in the process i put the byteArray in 6 global variables, so then i pass them to MySQL to store them. But i’ve notice with the profiling that the memory doesn’t free after the process.
If you have any clue, please let me know, thanks very much in advance for your time.
Sory about my english, is not my native language.
Agustin
yigit Says:
September 2nd, 2008 at 1:16 am
good article, good idea to overcome single threaded structure of flash.
for these kind of problems, there is a function named
callLater,
which calls the function after the screen is drawn.
So instead of using a timer, you could use callLater which would be more consistent&efficient.
seme1 Says:
September 3rd, 2008 at 5:24 am
I too have memory issues with the code when being executed on multiple images
How can I free the memory ?
The Fattest Says:
September 3rd, 2008 at 10:25 am
So you are running it multiple times in a row?
seme1 Says:
September 3rd, 2008 at 10:34 am
This code resides in a function that is being called in a for loop multiple times (once for each image:)

var bd =new air.BitmapData(500,400);
bd.draw(tmploader[i]);
var encoder = new window.runtime.com.pfp.utils.JPEGAsyncEncoder();
encoder.PixelsPerIteration = 128;

encoder.addEventListener(
JPEGAsyncCompleteEvent.JPEGASYNC_COMPLETE,
function(e){
encodeDone(e, lastid, propid);
encoder.removeEventListener(
JPEGAsyncCompleteEvent.JPEGASYNC_COMPLETE,
arguments.callee
);
bd.dispose();
tmploader[i].unload();
}
);
encoder.encode(bd);

tmploader is simply a loader (air.loader()). I am using Javascript/HTML by the way for my lack of knowledge in FLEX, and its lack of support for my language.

Thanks in advance for your help.
The Fattest Says:
September 3rd, 2008 at 2:15 pm
My first attempt at this seme1 would be to try a call to System.gc() after the dispose call. Now this is definitely going to slow things down but it is a start in the right direction I think. So something like:

bd.dispose();
System.gc();

Let me know how that works out.
seme1 Says:
September 3rd, 2008 at 3:40 pm
Thank you very much for the fast response.

Adding gc() helped reduce the leakage a lot. Memory used to increase by 100MB when 20 images are processed (each 256KB). Now, it processes at least 50 before it increases to the same amount.

On a side note, I get a vague message sometimes (SyntaxError: Syntax error) in a window titled ActionScript. Since my whole application is in HTML/Javascript, I presume this is caused by the JPEGAsycEncoder ??
The Fattest Says:
September 3rd, 2008 at 7:58 pm
yeah is that all the information it gives. I don’t know of any errors it would throw but it would be something wonky with the combination of AIR and JPEGAsyncEncoder.
seme1 Says:
September 14th, 2008 at 8:18 am
I have noticed that the quality of the images produced by the JPEGAsynencoder does not match that of other encoders available (i.e. php’s built in image compression functions from the gd library)

Any explanation ? or hints/workarounds for improving the quality of compressed images by JPEGAsyncEncoder ??

Please have a look at the following samples:

original image:
http://img212.imageshack.us/img212/5930/67624460qh5.jpg
Size: 110KB

Image encoded with JPEGAsync:
http://img136.imageshack.us/img136/3554/77570198qp0.jpg
Size:46.7KB

Image encoded with Php’s built-in functions (gd library):
http://img382.imageshack.us/img382/7751/1492if8.jpg
Size:33.4KB
Much smoother than the JPEGAsync version.. and smaller too
seme1 Says:
September 14th, 2008 at 8:20 am
In the images I posted in my previous comment,, Please pay attention to the white board in the picture. In one image ,, the word “hotmail” can be easily read,, while this is not the case in the other.

I tried increasing the quality measure to 80 and 100 (as an argument passed to JPEGAsyncEncoder) and it only resulted in increasing the file size.. The picture did not get any smoother..
The Tallest Says:
September 14th, 2008 at 10:31 am
Hmm, yeah, that is a big quality difference.

In any case, I don’t actually know how the code behind the JPEG encoder works - when I wrote this asyc encoder, I just took the bulk of the code from Flex. You might want to try encoding that image with the regular Flex JPEG encoder, and see if it has the same issues - I bet it does.
The Fattest Says:
September 14th, 2008 at 10:47 am
seme1, may I ask how you are creating the encoded image and saving it using the JPEGAsyncEncoder? I am quite curious because I haven’t noticed any quality issues especially like the ones you have.
seme1 Says:
September 14th, 2008 at 12:37 pm
Thanks again for the fast response..

Below is the whole code I have :

Basically, fileList is an array storing several files (all images). They are collected by a drag and drop function.

fileList[fileList.length]= event.dataTransfer.getData(”application/x-vnd.adobe.air.file-list”); // event here is the drop event

Then,, I have:

var imageFile= new air.File(fileList[i].nativePath);
//new file stream
var fileStream= new air.FileStream();
fileStream.open(imageFile, air.FileMode.READ);
//reading image into bytearray and closing stream
var imgBytes = new air.ByteArray();
fileStream.readBytes(imgBytes);
fileStream.close();

//creating loader and injecting image bytes into it
tmploader[i]= new air.Loader();
// events:
tmploader[i].contentLoaderInfo.addEventListener( air.Event.COMPLETE, function(e){
saveTheFile(e,i,propid);
});

// loading the bites
tmploader[i].loadBytes( imgBytes, air.loaderContext );

// Inside saveTheFile,, I have:
function saveTheFile(){
var bd =new air.BitmapData(600,tmploader[i].content.height);
bd.draw(tmploader[i]);

var encoder = new window.runtime.com.pfp.utils.JPEGAsyncEncoder(80);
encoder.PixelsPerIteration = 128;
encoder.addEventListener(
window.runtime.com.pfp.events.JPEGAsyncCompleteEvent.JPEGASYNC_COMPLETE, function(e){
encodeDone(e, lastid, propid);
bd.dispose();
air.System.gc();
tmploader[i].unload();
});
encoder.encode(bd);
}

// This is the last bit that writes the images after being encoded
function encodeDone(e){
var myFileStream = new air.FileStream();
myFileStream.openAsync(myFile, air.FileMode.WRITE);
myFileStream.writeBytes(e.ImageData);
myFileStream.close();
}
seme1 Says:
September 14th, 2008 at 1:32 pm
ok, I missed these two lines,,
=======================
tmploader[i].content.height =(600/ tmploader[i].content.width) * tmploader[i].content.height ;
tmploader[i].content.width =600;
============================
they are right above the line:
var bd =new air.BitmapData(600,tmploader[i].content.height);
The Tallest Says:
October 19th, 2008 at 9:46 am
Seme1, it looks to me that you are resizing the image before you encode it (the lines where you set the height and width). I’m not sure what algorithm is being used underneath the covers to do the resize, but I bet it is that algorithm that is causing the artifacts you see in the encoded image.

Flex Tutorial - An Asynchronous JPEG Encoder

20 Responses

Leave a Comment

Search Posts

Pages

Categories

Recent Posts

Archives

Meta