32

I am using the gm package for Node.js along with the default ImageMagick installation that is available on AWS Lambda.

const gm = require('gm').subClass({ imageMagick: true });

For some reason, the resize functionality fails for certain images.

I created an EC2 instance with Amazon Linux AMI (ami-hvm-2016.03.3.x86_64-gp2). I installed the (old) 6.x version of ImageMagick that is available from yum. When I run my script with that install on the EC2 instance, it reproduces the failure I see when the code runs on Lambda, confirming it is something with this version of IM that is causing the failure.

If I install GraphicsMagick with sudo yum install GraphicsMagick. This allows my script to perform the resizes without error.

const gm = require('gm').subClass({ imageMagick: false });

However, I'm not sure how to bundle this in my deploy with serverless. If I install GraphicsMagick to the same folder as the script with sudo yum --installroot=/var/task install GraphicsMagick, and run my script using this require statement instead:

const gm = require('gm').subClass({ imageMagick: false, appPath: './usr/bin/' });

The resizes work when I run my script on the EC2 instance. But when I deploy with serverless, and the script runs in Lambda, the executable appears to be broken. gm fails with the following error on a call to gm(buffer).size(/*...*/).

could not get the image size: ERR: {"code":"EPIPE","errno":"EPIPE","syscall":"write"}

How can I build a version of ImageMagick or GraphicsMagick that can be deployed with serverless?

Yves M.
  • 29,855
  • 23
  • 108
  • 144
wprl
  • 24,489
  • 11
  • 55
  • 70

7 Answers7

30

I spun up the latest aws linux and ran the commands below. 

yum -y install gcc-c++ libpng-devel libjpeg-devel libtiff-devel wget
wget https://downloads.sourceforge.net/project/graphicsmagick/graphicsmagick/1.3.26/GraphicsMagick-1.3.26.tar.gz
tar zxvf GraphicsMagick-1.3.26.tar.gz
cd GraphicsMagick-1.3.26
./configure --prefix=/var/task/graphicsmagick --enable-shared=no --enable-static=yes
make
sudo make install
tar zcvf ~/graphicsmagick.tgz /var/task/graphicsmagick/

I scp the dir down into my local and threw it in the package to be zipped and deployed. My layout is similar to the aws repo code linked, but modified for serverless.

Lambda code:

// graphicsmagick dir is at the root of my project
const BIN_PATH = process.env['LAMBDA_TASK_ROOT'] + "/graphicsmagick/bin/";
const Gm = require('gm').subClass({ appPath: BIN_PATH });

// below is inside the handler
process.env['PATH'] = process.env['PATH'] + ':' + BIN_PATH;

serverless.yml

package:
  artifact: /path/to/function.zip

I use the artifact and build my own zip. If you run into the issue below I suggest you do that. https://github.com/serverless/serverless/issues/3215

# -y to keep the symlinks and thus reduce the size from 266M to 73M
cd lambda && zip -FS -q -r -y ../dist/function.zip *

Ideas grabbed from:

https://gist.github.com/bensie/56f51bc33d4a55e2fc9a

https://github.com/awslabs/serverless-image-resizing

Edit: Might want to also check out lambda layers. May only need to do this kind of thing once.

pbsladek
  • 616
  • 7
  • 12
  • 1
    change to ./configure --prefix=/var/task/graphicsmagick --enable-shared=no --enable-static=yes – user732961 Oct 17 '17 at 01:28
  • 3
    I'm struggling with the dependent libs... can't find for example, libjpeg.so.62 when I run gm on the lambda – Michael Wiles Sep 11 '18 at 00:11
  • 1
    In addition to the listed packages I had to install `tar`, `gzip` and `make`. Another option instead of scp is to (on the host) run `docker cp [container id]:/root/graphicsmagick.tgz ./` – Tuure Sep 17 '18 at 14:59
  • Where have you run these command. I have created a lambda function and it gives me same error and runs fine on local. Can you specify the steps how and where you ran the commands? – Osama Khalid Jan 02 '19 at 09:00
  • where is serverless.yml for my simple lambda function? – Osama Khalid Jan 02 '19 at 09:06
  • I realy dont get why to put the `process.env['PATH'] = process.env['PATH'] + ':' + BIN_PATH;` inside the handler. And I followed all this steps and still haven´t been able to make it work. I always get an error telling me `Could not execute GraphicsMagick/ImageMagick: /var/task/graphicsmagick/bin/gm ... ...this most likely means the gm/convert binaries can't be found` – Guzz Jun 12 '19 at 22:11
  • @Guzz Think i pulled it from here back in the day. https://aws.amazon.com/blogs/compute/running-executables-in-aws-lambda/. There have been a lot of advancements since then. Might want to take a look at docker etc. – pbsladek Jun 12 '19 at 22:51
8

I was struggling on this for a couple of days, ended up going through the process myself and it does indeed work.

ImageMagick is no longer bundled with the Node.js 10.x runtime. There are 3 options to get ImageMagick working with your Node.js 10.x function:

1) Package the dependency and include it in your uploaded ZIP file (like this one)

https://image-magick-example.s3-us-west-2.amazonaws.com/image-magick-example.zip

https://github.com/hmagdy/imagemagick-aws-lambda-Node.js10.x/tree/master/option1_image-magick-example-zip

But with option: The deployment package of your Lambda function "image-magick-example-zip-demo" is too large to enable inline code editing. However, you can still invoke your function.

enter image description here

or

2) Create or use a Lambda Layer that includes ImageMagick, to do that:

clone git@github.com:hmagdy/imagemagick-aws-lambda-Node.js10.x.git
cd imagemagick-aws-lambda-2
start Docker services
make all

That would create a layer.zip inside build folder. But to save you some time here’s a zip file you can use to create a Lambda Layer.

https://image-magick-layer.s3-us-west-2.amazonaws.com/layer.zip

When you create the layer make sure you add Node.js 10.x as a supported runtime. You can then set your function to use the latest Node.js 10.x and add the layer you created. The image conversion should then work again!

enter image description here

Then you can create your aws lambda function like this

https://github.com/hmagdy/imagemagick-aws-lambda-Node.js10.x/tree/master/option2_image-magick-example-c_lib_layer/index.js

3) NodeJS Runtime Environment (npm) with AWS Lambda Layers, to do that:

Also, if you want to use

const imageThumbnail = require('image-thumbnail');

and got 

Runtime.ImportModuleError: Error: Cannot find module 'image-thumbnail'

you should follow option 3:

https://github.com/hmagdy/imagemagick-aws-lambda-Node.js10.x/tree/master/option3_image-magick-example-npm_layer

Inspired by:

https://medium.com/@anjanava.biswas/nodejs-runtime-environment-with-aws-lambda-layers-f3914613e20e

HMagdy
  • 3,029
  • 33
  • 54
  • 1
    I just uploaded the layer (step 2) and it started working straight away. Thanks a lot! great work – Jan Sila Aug 26 '20 at 13:08
7

If you want to tackle image resizing, you may also take a look at the serverless sharp image library which uses Sharp, a high performance Node.js library for image resizing which is about 3x - 5x faster compared to GM/IM. You didn't provide enough information to say that it fits your use case requirements but I just wanted to mention it since this library already saved me a lot of AWS Lambda costs so far.

By the way: I am not related to this project (but licences are MIT/Apache License 2.0 anyway).

Tom
  • 1,942
  • 22
  • 35
  • 1
    Keep in mind of Sharp's limits before switching: http://sharp.pixelplumbing.com/en/stable/api-input/#limitinputpixels If you want to process large images, that might be an issue. – Zedzdead Jan 24 '19 at 00:06
  • Sharp has the same problem. You need to have docker to build it. – astroanu Feb 17 '19 at 05:35
1

All dependencies can be packed and uploaded as a part of your AWS Lambda function

You can mostly use any package you want from AWS Lambda, if you can fit it within the allowed size limits and upload the zip file. Take a look at the AWS Lambda Deployment Limits section

Also, here's an example of how to package dependencies (for python code) https://stackoverflow.com/a/36093281/358013

Chenna V
  • 10,185
  • 11
  • 77
  • 104
  • 1
    I have installed imagemagick in the package and I can see it in node_modules folder. But still it gives the same error. On local i have installed the imageMagick on ubuntu and it worked fine. I don't know how to install it on the lambda environment. – Osama Khalid Jan 02 '19 at 09:04
  • 1
    @OsamaKhalid you don't install in the lambda environment. you would package all dependencies i.e. imageMagick in this case. refer to this for example https://stackoverflow.com/questions/36054976/pandas-aws-lambda/36093281#36093281 – Chenna V Jan 13 '19 at 17:48
1

For node.js, you can use node-lambda, it simplifies packaging using a docker image :

node-lambda package -I lambci/lambda:build-nodejs6.10 -A . -x '*.lock *.zip'

The -I argument will launch a docker image and launch npm i in your project so it will compile the binary node_modules against the right architecture.

Jide
  • 1,130
  • 2
  • 12
  • 18
0

If you install Docker on your local device and add this command into your package.json.

"dockerbuild": "rm -rf node_modules/gm && docker run -v \"$PWD\":/var/task lambci/lambda:build-nodejs8.10 npm install"

Run npm run dockerbuild before you deploy your app.

You should change the node version based on the version of your lambda environment.

Onur Yıldırım
  • 341
  • 3
  • 13
-2

As I checked, the imageMagick is existed already on aws lambda environment. So we can use all library graphics, images related to imageMagick. Refer: https://serverless.com/blog/building-a-serverless-screenshot-service-with-lambda/

Thuy Nguyen
  • 1