diff --git a/Dockerfile b/Dockerfile index 44603b5..8cd0756 100644 --- a/Dockerfile +++ b/Dockerfile @@ -11,9 +11,26 @@ ARG python_package_to_install="mysqlclient" RUN curl -Ls -c cookieJar -O ${mysql_gpg_key_url} RUN rpm --import ${mysql_gpg_key_name} -# prerequisite for getting mysql-devel package +# prerequisite for getting mysql-devel package; this enables the MySQL Yum Repo RUN curl -Ls -c cookieJar -O ${mysql_devel_package_url} RUN yum install -y ${mysql_repo_rpm} -# install mysql-devel package +# get the "yum-config-manager" utility +RUN yum install -y yum-utils + +# enable the right repo for installing packages related to MySql 5.6 +RUN yum-config-manager --disable mysql80-community +RUN yum-config-manager --enable mysql56-community + +# install the "mysql-community-devel" package for MySQL 5.6 +# this installs /usr/lib64/mysql/libmysqlclient.so.18 +# which is what we need for `pip install mysqlclient` later RUN yum install -y ${mysql_devel_package} +# ======================================================================================================= +# Package Arch Version Repository Size +# ======================================================================================================= +# Installing: +# mysql-community-devel x86_64 5.6.49-2.el7 mysql56-community 3.4 M +# Installing for dependencies: +# mysql-community-common x86_64 5.6.49-2.el7 mysql56-community 289 k +# mysql-community-libs x86_64 5.6.49-2.el7 mysql56-community 2.2 M diff --git a/README.md b/README.md index 64de5d5..134f8ef 100644 --- a/README.md +++ b/README.md @@ -1,19 +1,38 @@ -# AWS layer for `mysqlclient` for Python 3.x +# AWS Lambda layer for `mysqlclient` (or any other Python package) + +This project provides: + +1. Ready-made AWS layer zips for the Python [mysqlclient](https://github.com/PyMySQL/mysqlclient-python) (aka MySQLdb) package: for MySQL 5.6 and MySQL 8.0 +2. An easy, docker-based solution for building your own AWS layer: for `mysqlclient` for ANY version of MySQL server and ANY version of Python too. +3. An easy, docker-based, completely generalized solution for building your own AWS layer for ANY Python package for ANY Python version. This is especially useful for importing and using Python packages with platform-specific dependencies (e.g. the package uses `.so` files via FFI) in AWS Lambda. These packages are usually non-trivial to use in AWS Lambda for reasons described below. Example packages that fit these criteria: pandas, numpy, cchardet. If you have this use-case, use the `general-purpose` branch. ## TLDR -- Are you using Python 3.8? -- Are you using MySQL v8.0.x? -- Are you having trouble importing/using [mysqlclient](https://github.com/PyMySQL/mysqlclient-python) in your AWS Lambda function? -- Are you in a hurry? +If you need a ready-made, tested AWS Layer for `mysqlclient`, just use `build_output/layer.zip` according to this table: + + +| Python Version | MySQL Version | Branch to use | +|---|---|---| +| 3.9 | MySQL v8.0.x | master | +| 3.8 | MySQL v8.0.x | mysql8-py3.8 | +| 3.8 | MySQL v5.6.x | mysql5.6-py3.8 | +| 3.x | I want to build an AWS Lambda layer for a non-MySQL Python package| general-purpose | + +If your use-case is not reflected in the table above (for example, you need to target a different version of MySQL and/or a different version of Python) then you can build your own AWS layer with the tools provided in this repo. Read on for more instructions. + + +### Short HowTo -If you answered `yes` to all the above questions, simply upload `layer.zip` from the `build_output` dir as an AWS Layer to your AWS account. +- Switch to the appropriate branch as per the table above. +- Either use the already-provided `build_output/layer.zip` or build your own `layer.zip` as described below. +- Upload `layer.zip` as an AWS Layer to your AWS account. For details on this step, see the [create a new AWS layer with `layer.zip`](https://github.com/nonbeing/mysqlclient-python3-aws-lambda#create-a-new-aws-layer-with-layerzip) section. +- Configure your AWS Lambda Python function to use your newly-created layer. +- Profit! -If you answered `no` to any of the above questions, then read on to figure out if you need to build your own `mysqlclient` AWS layer with the tools provided in this repo. -For details on how to upload a zip file as a new layer to AWS Lambda, see the section below called **create a new AWS layer with `layer.zip`**. +## `mysqlclient`: simple, sufficient test for your new AWS Lambda layer -After you create the required layer, you can configure your Lambda function to use this layer and simply `import MySQLdb` without error in your Lambda function. +After you create the required layer in AWS Lambda, you can configure your Lambda function to use this layer and simply `import MySQLdb` (as an example) in your Lambda function. The import should work just fine, without and errors if you are using the right Lambda layer and you have configured your Lambda function to use your Lambda layer correctly. Here's a barebones example to test if you are able to import and use `mysqlclient` correctly: @@ -27,7 +46,7 @@ def lambda_handler(event, context): } ``` -If you get the success message and don't see an error like `ModuleNotFoundError: No module named 'MySQLdb'` or `ImportError: No module named _mysql`, then you're all set to use `mysqlclient` on AWS Lambda. +If you get the success message and don't see an error like `ModuleNotFoundError: No module named 'MySQLdb'` or `ImportError: No module named _mysql`, then you're all set to use `mysqlclient` on AWS Lambda - the Lambda layer is working just fine for you. ## Motivation @@ -86,23 +105,46 @@ For most use-cases, `mysqlclient` is the preferred choice of DB connector to MyS This project attempts to solve (or at least alleviate) this problem to a large extent by providing a relatively-straightforward path to building an AWS Layer for `mysqlclient` which can then be readily consumed in AWS Lambda Python functions. -## Usage +# Usage + +## Building your own Lambda layer.zip for ANY Python package + +Skip the following steps if you are using any of the provided, ready-to-use `layer.zip` files. Go directly to [create a new AWS layer with `layer.zip`](https://github.com/nonbeing/mysqlclient-python3-aws-lambda#create-a-new-aws-layer-with-layerzip) instead. -### prerequisites +If you need to build your own Lambda layer from scratch, read on... -You need a `*nix` environment where you can run docker commands and bash scripts (such as WSL2 on Windows 10 Pro, which has been tested). Tested on Ubuntu 20.04 +### build prerequisites -Ensure you have docker installed. You will need to pull a [`lambci` image from docker hub](https://hub.docker.com/r/lambci/lambda). This image will be used to build a local docker image using the `Dockerfile` and install the appropriate `mysql-devel` package. +You need a `*nix` environment where you can run docker commands and bash scripts. This project has been tested on Ubuntu 20.04, WSL2 on Windows 10 Pro and MacOS. + +Ensure you have docker installed; you should be able to run `docker --version` without any issues. The script will pull a [`lambci` image from docker hub](https://hub.docker.com/r/lambci/lambda) as the first step of the build. This `lambci` image will be used to build a local docker image using the `Dockerfile`. If you are using one of the `mysql` branches or `master` branch, the script will also install the appropriate `mysql-devel` package which is necessary for building the `libmysqlclient.so`. ### build the layer locally -Simply clone this repo and invoke the `build.sh` shell script; it will perform all the steps required. +- Clone this repo. +- Switch to the appropriate branch (see table above). If you are not building for `mysqlclient`, use the `general-purpose` branch. +- Review the Dockerfile. You may need to edit this as per your needs (e.g. not building a layer for `mysqlclient`, but for some other Python package). +- Review `requirements.txt`. You may need to edit this file - it should have the Python package for which you are building an AWS Lambda layer. +- Review the `pip_and_copy.sh` script. You may need to edit this as per your needs (e.g. not building a layer for `mysqlclient`, but for some other Python package). +- Invoke the `build.sh` shell script (e.g. by `bash build.sh`). + +The `build.sh` script will perform all the necessary steps and if successful, will produce a `layer.zip` file in the `build_output` directory. + +`build.sh` will use the `Dockerfile` to build a docker image based off the `lambci/lambda:build-python3.8` image that very-closely replicates the AWS Lambda environment. Any build dependencies (e.g. RPM packages needed in the build environment) should be specified in the `Dockerfile` beforehand. -`build.sh` will use the `Dockerfile` to build a docker image based off the `lambci/lambda:build-python3.8` image that very-closely replicates the AWS Lambda environment. The `mysql-community-devel` RPM will be downloaded and installed in the image. This is necessary to `pip install mysqlclient` in Amazon Linux 2. After `pip install mysqlclient` in the docker image has succeeded, the correct `.so` file and the python libs are copied out from the docker container and zipped into `build_output/layer.zip`. +After the docker image has been built, `build.sh` runs `pip_and_copy.sh` which in turn runs `pip install -r requirements.txt` and copies over the necessary `.so` file to the output directory. Finally, `build.sh` zips up the build artifacts in the `build_output/python` and `build_output/lib` directories into a zip ready for upload. This `layer.zip` file is the final artifact, ready-to-upload to AWS Lambda for your new layer. -This `layer.zip` file is the final artifact, ready-to-upload to AWS Lambda for your new layer. +If you are building a layer for `mysqlclient`, `build.sh` specifically does the following: +- Downloads and installs the correct, appropriate `mysql-community-devel` RPM in the docker image. This is necessary to `pip install mysqlclient` in Amazon Linux 2. +- Invokes `pip_and_copy.sh` to `pip install mysqlclient` and copy the correct `.so` file and the python libs out from the docker container and into the `build_output/python` directory. +- Zips the `build_output/python` and `build_output/lib` dir into `build_output/layer.zip`. -### create a new AWS layer with `layer.zip` +### full example for building a layer for ANY Python package + +Check out the `cchardet-wheel` branch and view `Dockerfile` and `pip_and_copy.sh` files. This branch was created to build the [cchardet](https://github.com/PyYoshi/cChardet) Python package (https://pypi.org/project/cchardet) for AWS Lambda. This is a different procedure from how `mysqlclient` is built. + + +## Create a new AWS layer with `layer.zip` You should upload `layer.zip` as-is; you don't need to zip or unzip anything. @@ -110,13 +152,34 @@ The easiest way is to use the AWS Lambda web console. Of course, there are many For a nice blog post with screenshots on how to upload a zip file as a new layer to AWS Lambda, read the [Deploying our Layer](https://www.freecodecamp.org/news/lambda-layers-2f80b9211318/) section. -### reference the newly-created layer in your lambda function +## Reference the newly-created layer in your lambda function + +Here's how to add your new layer to your Lambda function's configuration. + +In the AWS Lambda Web Console: + +### step 1 + +Go to the function configuration, click `Layers` and then click the `Add a layer` button. + +![AddLayerStep1](images/add-layer-to-lambda-config/AddLambdaLayer-1.png) + +### step 2 + +On the `Add Layer` page, select `Custom layers`, then select the exact custom layer in the dropdown. + +Next, select the version of the custom layer. If this is a brand-new layer, it will only have one version: select version `1`. + +Click the `Add` button. + +![AddLayerStep2](images/add-layer-to-lambda-config/AddLambdaLayer-2.png) -Add the new layer to your Lambda function's configuration. +### step 3 -[TODO: ADD SCREENSHOTS HERE] +Finally, confirm that the custom layer has been added to your Lambda function as shown here: +![AddLayerStep3](images/add-layer-to-lambda-config/AddLambdaLayer-3.png) -### import and run! +## Import and run! Finally, you can simply `import MySQLdb` in your Python3 Lambda function; here's a barebones example: @@ -132,15 +195,15 @@ def lambda_handler(event, context): If you get the success message and don't see an error like `ModuleNotFoundError: No module named 'MySQLdb'` or `ImportError: No module named _mysql`, then you're all set to use `mysqlclient` on AWS Lambda. -## Feedback and Contributions +# Feedback and Contributions ... are most welcome. Please file PRs and issues as you see fit. Will respond to them as soon as possible. -## Troubleshooting +# Troubleshooting See the [`mysqlclient` FAQ](https://github.com/PyMySQL/mysqlclient-python/blob/a33e1c38363b8c71775394965ca70d576ffd3a90/doc/FAQ.rst) that covers troubleshooting for common error cases, including build errors. -## Credits and Thanks +# Credits and Thanks The work in this repo is largely based off Seungyeon Kim(Acuros Kim)'s project at: https://github.com/StyleShare/aws-lambda-python3-mysql - thanks! diff --git a/build.sh b/build.sh index 902c8e9..bb017b7 100755 --- a/build.sh +++ b/build.sh @@ -3,9 +3,9 @@ PKG_DIR='build_output/python' LIB_DIR='build_output/lib' # set the docker image name here (optional) -IMAGE_NAME='nonbeing/lambda-python38-mysqlclient' +IMAGE_NAME='nonbeing/lambda-python38-mysqlclient-mysql5.6' -sudo rm -rf ${PKG_DIR} ${LIB_DIR} +sudo rm -rf "build_output/*" mkdir -p ${PKG_DIR} && mkdir -p ${LIB_DIR} # build a docker image closely matching the AWS Lambda environment, with mysql-devel installed diff --git a/build_output/layer.zip b/build_output/layer.zip index 94d3ada..a1757d7 100644 Binary files a/build_output/layer.zip and b/build_output/layer.zip differ diff --git a/images/add-layer-to-lambda-config/AddLambdaLayer-1.png b/images/add-layer-to-lambda-config/AddLambdaLayer-1.png new file mode 100644 index 0000000..cfe3996 Binary files /dev/null and b/images/add-layer-to-lambda-config/AddLambdaLayer-1.png differ diff --git a/images/add-layer-to-lambda-config/AddLambdaLayer-2.png b/images/add-layer-to-lambda-config/AddLambdaLayer-2.png new file mode 100644 index 0000000..a75cc39 Binary files /dev/null and b/images/add-layer-to-lambda-config/AddLambdaLayer-2.png differ diff --git a/images/add-layer-to-lambda-config/AddLambdaLayer-3.png b/images/add-layer-to-lambda-config/AddLambdaLayer-3.png new file mode 100644 index 0000000..b78c4cc Binary files /dev/null and b/images/add-layer-to-lambda-config/AddLambdaLayer-3.png differ