Upload and process a spreadsheet without storing it permanently

Question

I want to provide an option to export data via a spreadsheet. I don't want to store it permanently (hence there's no need of storage services like S3). What would the most most efficient and scalable way of doing this? Where can I temporarily store this file while it is being processed? Here's what should happen:

List item
User uploads spreadsheet
My backend processes it and updates the DB
Discard the spreadsheet

My 2 requirements are efficiency and scalability.

score 0 · Answer 1 · edited May 23 '17 at 10:34

0

If I was you i would look for a way to parse the XLS/CSV on the front-end and sending JSON to your backend. This way you pass slow /intensive work to the client (scalability) and process only JSON on the server.

You can start here: https://stackoverflow.com/a/37083658/1540290

edited May 23 '17 at 10:34

Community

1
1

answered Jun 25 '16 at 03:19

Lukasz Muzyka

2,783
1
30
41

I thought of that, but wouldn't it take the client a very long time to process a large file - say 5MB? – user1175969 Jun 25 '16 at 18:13
@user1175969 well, I suppose you should test that out. – Lukasz Muzyka Jun 26 '16 at 00:52

score 0 · Answer 2 · answered Jun 25 '16 at 16:23

I'm assuming you have a form with a file input to pick the xls file you want to process like this:

<input id="my_model_source" type="file" name="my_model[source]">

To process the xls you could use roo gem.

Option 1:

In some controller (where you are processing the file) you can receive the file like this: params[:my_model][:source]. This file will be an ActionDispatch::Http::UploadedFile instance. This class has the instance method path that will give you a temp file to work with.

So, with roo gem, yo can read it like this:

xls = Roo::Spreadsheet.open(params[:my_model][:source].path, extension: :xlsx)

Option 2:

The option one will work if your importing process is not too heavy. If indeed, is too heavy you can use Active Job to handle the processing in background.

If you choose Active Job, you:

will lose the opportunity to use ActionDispatch::Http::UploadedFile's path method. You will need to generate the temp file on your own. To achieve this you could use cp command to copy the ActionDispatch::Http::UploadedFile's path wherever you want. After use it you can deleted with rm commnad
will lose a real time response. To handle this you could use Job Notifier gem

I have tried to show roughly what paths you can take.

Upload and process a spreadsheet without storing it permanently

2 Answers2

Option 1:

Option 2: