Python Packaging for AWS Lambda using the Serverless Framework

AWS Lambda

The process of deploying Python functions to AWS Lambda has a few quirks. If you’re doing something simple that doesn’t require packages outside of those included with Python (like json, datetime, pprint, etc.) then you don’t really need to worry about packaging, and can even get away with developing right in their browser editor if you are so inclined. However, as soon as you need to use pip, you’re most likely going to need to change up your tactics a bit.

Whats in the Box?

Let me start off by mentioning that in the Lambda environment, there’s a few non-core package included, such as the amazingly useful requests package. It’s worth nothing that the requests that’s included there has to be imported a little differently:

from botocore.vendored import requests

What’s happening here is that requests is already bundled in the botocore library, which is included in the Lambda environment by default. For those unfamiliar with it, botocore and the higher level boto3 form the AWS Python SDK, and let you work with AWS resources in a much more convenient way. What’s really going on here is that requests is included because it’s a dependency of the botocore library. So the long and short of it is this:

When writing in Python, the AWS Lambda environment contains the core Python libraries + boto3, botocore, and their dependencies.

Some other libraries include:

• docutils
• jmespath
• python-dateutil
• urlgrabber

A much longer list can be found in this gist.

BYOB: Enter Serverless

Okay, so if do need another library you’re going to need to upload it to Lambda along with your code, which pretty frequently prevents you from using the in-browser code editor built into the Lambda console due to size restrictions. There is, of course, the option of zipping everything up yourself and uploading that way, but that’s a post for another time (or you can check the docs).

However, there’s always a better way. One such way is using the Serverless Framework, which I’ve found to be invaluable when deploying to Lambda. The idea is that once you install and configure it using a YAML file, it manages the whole deployment process for your application, soup to nuts. It packages your code, uploads it to an S3 bucket, deploys it to a Lambda function and sets up API Gateway to point to your function(s).

Leaving Things Behind

The last part I’ll be focusing on here is the serverless-python-requirements plug-in which takes your requirements.txt file, and uses that to package your code with all of the necessary dependencies. It does not, however, leave out the libraries that are already included in the Lambda environment, which is a waste of space. Leaving them in is not the end of the world in most cases, but isn’t preferable either. Thankfully that plug-in does come with an option to omit specific packages, called noDeploy:

# serverless.yml
...
plugins:
  - serverless-python-requirements

custom:
  pythonRequirements:
    noDeploy:
      - botocore
      - boto3
      - docutils
      - jmespath
      - python-dateutil
      - six
      - s3transfer
...

When you run sls deploy to start your deployment, the serverless-python-requirements plug-in will read that option, and just leave behind libraries in that list. It’s also useful for leaving behind any libraries that were installed for unit testing/development purposes, that don’t need to be deployed into production either.