Flex File Upload to End All Uploads

Back to the Future
I previously said I had implemented an FTP based solution for file upload.  Well I did and it works pretty well but it's a bit fiddly and tricky to get working so I kept looking for solutions.  I found one.  Hooray!  Flex 4 and FileReference in Flash 10 with Blaze provide the solution.
Big Irritation
A big irritation of mine with http uploads using multi part form stuff is that it is an all or nothing proposal.  Once you fire up the connection and start uploading you have to wait for the whole thing to go up.  For small files this isn't such a problem but if you upload a file that takes a minute or two you often don't know if it is working (from a client perspective) until you decide that 3 hours is too long for a file to take.  There are heaps of issues like socket disconnected by peer, socket read timeout and so forth.  To get things working you need to check special upload timeouts and browser timeouts and virus checkers / indexers on the server and other mysterious Mac OS X related muffery.  To use file upload over http in a true enterprise application you need some guarantees that the upload will happen and that data isn't lost.  You need a reliable system so you can tell your paying customers that their data will get to the server and if it doesn't you know what happened and you can fix it and keep going.  Anything less is amateur hour mickey mouse stuff.
Flash 10 FileReference
Flash player 10 allows you to call fileReference.load() to load a selected file.  This puts all the file's bytes into the data property.  You can then send that as a byte[] through AMF via Blaze to the server where you can save the file.  The problem with this simplistic approach is you don't get any progress events and if it is a big file you still end up with all the problems of http upload.  The solution is really simple; break the byte array into manageable chunks and send them up one at a time waiting for a response between each one.  This way you get a progress event and the system doesn't just freeze up until it's finished.  If you are clever you can also enable pause / resume.
I'm Clever
Yeah well so my mother says anyway.  But I wrote a nice little flex library and java library which go together to provide file upload services.  The flex part takes a byteArray and file name and queries the java service to see what progress the file is up to.  The java service replies with a progress event which either says 0 bytes uploaded, some bytes uploaded or complete.  You can use that to initialise your progress bar in flex.  Then the flex library moves to the correct position in the byteArray and sends a chunk up to java which is written to file and a response sent with progress info.  This continues until the file is complete at which point the flex library dispatches a complete event and you can do whatever you like with the file.  The java service has some methods that let you get an input stream to the uploaded file and also delete the file once your done with it.  If the user cancels midway through the upload (either on purpose or due to error) they can re-upload the file and it will resume from where you left off.  Errors are also communicated back to the flex client so you can auto retry or do whatever.  Hopefully I'll attach this stuff to my blog.  If you like it or find bugs (you probably will) then let me know.
LINKS:
applicationContext-upload.xml
FlexFileUpload.swc
reignite-upload.jar
fileupload.xml
HOW?
add the jar file to your lib folder
include the swc in your flex project
fileupload.xml goes in your web root or docroot whatever you call it
applicationContext-upload.xml is a spring context file so put it where you load them from
If you don't use spring you should, but if you don't, you can just create a facade that adapts the upload service and creates an instance and whatever.  Hey you're at least as clever as me so either use spring or take 5 minutes to figure out a way around it (I did for another project that used ejb)
in your remoting-config.xml

<destination id="fileUpload">
<properties>
<factory>spring</factory>
<source>uploadService</source>
</properties>
</destination>
You'll see I'm using the spring factory, if you don't have it do a google search for spring factory for blaze there's also an ejb factory if you are that way inclined.
The fileupload.xml has 2 things in it.  The first is the name of the destination you just configured in the remoting-config.xml and the other is the endpoint context path for your channel of choice.
Now to use it in your code:
  1. Create an instance of the au.com.reignite.upload.FileUploadManager and remember to declare it in a place that will have scope for the duration of your upload.  Same as FileReference below.  You need to pass in some start up arguments when creating FileUploadManager: url to fileupload.xml, securityToken which you wont use so hand in null and repositoryId which is a unique identifier for this user's uploads.  I use user id for this so the same user can resume their upload.  If you don't use a different id for each user then you'll get stuffed up files if more than one user uploads the same file at the same time.
  2. create a FileReference and get the  user to select a file.  Once selected call load().  Once complete you can do the rest.
  3. Once the file reference has done its load and you have the data do this:

uploadManager.addEventListener(FileUploadErrorEvent.UPLOAD_ERROR, handleUploadError);
uploadManager.addEventListener(ProgressEvent.PROGRESS, fileUploadProgressHandler);
uploadManager.addEventListener(FileUploadCompleteEvent.COMPLETE, completeUpload);
These are the events that get thrown so you had better handle them.  Pretty self explanatory really.
  4.  call uploadManager.upload(fileRef.data, fileRef.name); which starts it all off.  You'll get at least one progress event and a maximum of one complete event.
IF a user cancels or there is an error and you want to start over make sure you call uploadManager.cancel() otherwise you'll break it.  So that means if they close the popup your doing the upload in also make sure you call cancel otherwise the upload continues.
Maybe if I get bored I'll post an example app.

Get around file upload in Flex and Java

Http File Upload

My goodness me but I've had trouble with file upload for web sites. All sorts of issues ranging from timeouts to size restrictions to firewalls and now most recently Mac OS X 10.5 server just likes to keep a hold of uploaded file streams before releasing them to Tomcat.

The Scenario

Mac OS X 10.5 server on a mac mini with apache 2 and tomcat. Apache is has my web app as a virtual host which lives in Tomcat (6.18 or something). I've got a flex 3.4 client trying to do an http upload to my app. Seems to work up to a point then my Java app spits the dummy saying stream connection timeout! Bah! I traced it out and found that when my servlet should be reading the file input stream the file is not in my temporary directory yet. Mac OS X has subverted the file for its own nefarious purposes and left me hanging. I tried to stop indexing, stop finder, stop any sort of virus scanning and so on but no good. OS X keeps holding onto the file. Seems to only happen on multi-page pdf and word documents. Jpgs and pngs are passed straight through.

The Fix

So I've had enough with http file upload and its various vagaries and config problems and OSes sticking their noses in where they aren't wanted. I decided to create a custom FTP solution. Everyone loves FTP! It was actually designed to upload files. But how do I do it?

Server Side

Hooray for Apache FTP Server. It's an open source Java ftp server based on the mina project. So I grabbed it source code and all and began butchering. Firstly it uses Spring for configuration and my app uses Spring for configuration so that is nice. Secondly it allows you to implement your own user manager and your own list of commands. So I created a user manager that would authenticate with my app's authentication system and added a custom command that would save uploaded files into my own file system and data base (all files are actually referenced in the app by DB entry and the DB entry points to the file system). That made for a neat little wrap.

UH OH Flex Sucks Big Ones

Flex is a fantastic bit of work from Adobe; an object oriented (mostly) language and framework for developing rich internet applications. Yum. If it's an app your building (not someones home page) then flex kicks seven kinds of poo out of html. But still, in some ways it sucks pretty hard. The hardest suck is its socket API. You can create a socket and connect to a server on any old port then read and write bytes as you'd expect. However the Flash player hides all the good stuff from you. So you can say socket.write(mybytes) and then socket.flush() but do you think the Flash player will actually send the bytes over the wire when you flush? WRONG, FAIL. The bytes go when Flash says they can go and you wont know when that is. More importantly in the case of my FTP solution you can merrily send bytes onto the socket and flush them and then close the socket only to find that the ftp server closes the socket before Flash has finished flushing! That means you get truncated files and no progress bar on the client side.

Hooray For Open Source

"No worries" says I. I've got my FTP server code and Eclipse so I can do anything. I in fact alter the server code to send back status events with the number of bytes actually read. Flex then listens to these on the control socket and updates a progress bar. Once Flex knows that all the bytes have actually been written and received it closes the socket and logs off the FTP session. The really good thing about this is that I've been able to configure the FTP server so that the only commands available are those required to authenticate and my custom upload command. So no mister hacker you can't just start mucking about with my ftp server and create accounts etc... In fact all you can do is upload files that conform to my app's specification and they aren't executed files.

That's It

So that is the solution to my http upload woes. If you or a person you know suffers from file upload issues I can post some code and such. But since no-one follows my blog anyway I can't be bothered unless requested.