Table of Contents

Background

The boto3 SDK and the underlying botocore package are very large Python libraries.

boto3 and botocore are 50MB on disk
Just boto3 and botocore alone are over 50MB

It can highly beneficial to reduce the size of these in a final build, especially in a serverless environment. AWS Lambda functions for instance only allow a maximum extracted size of 250MB, so this is over 1/5th of that alone. Additionally, a smaller package size also can dramatically improve the cold-start time of a function.

Jack over at blog.cubieserver.de has an excellent blog post on how to strip down boto3 to the essentials, using a tool he wrote to facilitate layers in Lambda functions. My application (Linkspring) however is built with the confusingly-named serverless and serverless-python-requirements libraries. I was able to adapt his instructions to work with this stack and want to share for anyone else.

Solution

Really, it’s pretty straight forward. Add a custom key (if you don’t have it already) to your serverless.yml file and add a pythonRequirements key under that. Within that, add slim: true and slimPatternsAppendDefaults: true and then your patterns to exclude AWS services you don’t need. Example lifted directly from my codebase:

  1custom:
  2  pythonRequirements:
  3    slim: true
  4    strip: false
  5    slimPatternsAppendDefaults: true
  6    slimPatterns:
  7      # Commented out lines are INCLUDED
  8      - "boto3/examples/*"
  9      - "botocore/data/accessanalyzer/*"
 10      - "botocore/data/acm-pca/*"
 11      - "botocore/data/acm/*"
 12      - "botocore/data/alexaforbusiness/*"
 13      - "botocore/data/amp/*"
 14      - "botocore/data/amplify/*"
 15      - "botocore/data/amplifybackend/*"
 16      - "botocore/data/apigateway/*"
 17      - "botocore/data/apigatewaymanagementapi/*"
 18      - "botocore/data/apigatewayv2/*"
 19      - "botocore/data/appconfig/*"
 20      - "botocore/data/appflow/*"
 21      - "botocore/data/appintegrations/*"
 22      - "botocore/data/application-autoscaling/*"
 23      - "botocore/data/application-insights/*"
 24      - "botocore/data/appmesh/*"
 25      - "botocore/data/appstream/*"
 26      - "botocore/data/appsync/*"
 27      - "botocore/data/athena/*"
 28      - "botocore/data/auditmanager/*"
 29      - "botocore/data/autoscaling-plans/*"
 30      - "botocore/data/autoscaling/*"
 31      - "botocore/data/backup/*"
 32      - "botocore/data/batch/*"
 33      - "botocore/data/braket/*"
 34      - "botocore/data/budgets/*"
 35      - "botocore/data/ce/*"
 36      - "botocore/data/chime/*"
 37      - "botocore/data/cloud9/*"
 38      - "botocore/data/clouddirectory/*"
 39      - "botocore/data/cloudformation/*"
 40      - "botocore/data/cloudfront/*"
 41      - "botocore/data/cloudhsm/*"
 42      - "botocore/data/cloudhsmv2/*"
 43      - "botocore/data/cloudsearch/*"
 44      - "botocore/data/cloudsearchdomain/*"
 45      - "botocore/data/cloudtrail/*"
 46      - "botocore/data/cloudwatch/*"
 47      - "botocore/data/codeartifact/*"
 48      - "botocore/data/codebuild/*"
 49      - "botocore/data/codecommit/*"
 50      - "botocore/data/codedeploy/*"
 51      - "botocore/data/codeguru-reviewer/*"
 52      - "botocore/data/codeguruprofiler/*"
 53      - "botocore/data/codepipeline/*"
 54      - "botocore/data/codestar-connections/*"
 55      - "botocore/data/codestar-notifications/*"
 56      - "botocore/data/codestar/*"
 57      - "botocore/data/cognito-identity/*"
 58      - "botocore/data/cognito-idp/*"
 59      - "botocore/data/cognito-sync/*"
 60      - "botocore/data/comprehend/*"
 61      - "botocore/data/comprehendmedical/*"
 62      - "botocore/data/compute-optimizer/*"
 63      - "botocore/data/config/*"
 64      - "botocore/data/connect-contact-lens/*"
 65      - "botocore/data/connect/*"
 66      - "botocore/data/connectparticipant/*"
 67      - "botocore/data/cur/*"
 68      - "botocore/data/customer-profiles/*"
 69      - "botocore/data/databrew/*"
 70      - "botocore/data/dataexchange/*"
 71      - "botocore/data/datapipeline/*"
 72      - "botocore/data/datasync/*"
 73      - "botocore/data/dax/*"
 74      - "botocore/data/detective/*"
 75      - "botocore/data/devicefarm/*"
 76      - "botocore/data/devops-guru/*"
 77      - "botocore/data/directconnect/*"
 78      - "botocore/data/discovery/*"
 79      - "botocore/data/dlm/*"
 80      - "botocore/data/dms/*"
 81      - "botocore/data/docdb/*"
 82      - "botocore/data/ds/*"
 83      # - 'botocore/data/dynamodb/*'
 84      # - 'botocore/data/dynamodbstreams/*'
 85      - "botocore/data/ebs/*"
 86      - "botocore/data/ec2-instance-connect/*"
 87      - "botocore/data/ec2/*"
 88      - "botocore/data/ecr-public/*"
 89      - "botocore/data/ecr/*"
 90      - "botocore/data/ecs/*"
 91      - "botocore/data/efs/*"
 92      - "botocore/data/eks/*"
 93      - "botocore/data/elastic-inference/*"
 94      - "botocore/data/elasticache/*"
 95      - "botocore/data/elasticbeanstalk/*"
 96      - "botocore/data/elastictranscoder/*"
 97      - "botocore/data/elb/*"
 98      - "botocore/data/elbv2/*"
 99      - "botocore/data/emr-containers/*"
100      - "botocore/data/emr/*"
101      - "botocore/data/es/*"
102      - "botocore/data/events/*"
103      - "botocore/data/firehose/*"
104      - "botocore/data/fis/*"
105      - "botocore/data/fms/*"
106      - "botocore/data/forecast/*"
107      - "botocore/data/forecastquery/*"
108      - "botocore/data/frauddetector/*"
109      - "botocore/data/fsx/*"
110      - "botocore/data/gamelift/*"
111      - "botocore/data/glacier/*"
112      - "botocore/data/globalaccelerator/*"
113      - "botocore/data/glue/*"
114      - "botocore/data/greengrass/*"
115      - "botocore/data/greengrassv2/*"
116      - "botocore/data/groundstation/*"
117      - "botocore/data/guardduty/*"
118      - "botocore/data/health/*"
119      - "botocore/data/healthlake/*"
120      - "botocore/data/honeycode/*"
121      - "botocore/data/iam/*"
122      - "botocore/data/identitystore/*"
123      - "botocore/data/imagebuilder/*"
124      - "botocore/data/importexport/*"
125      - "botocore/data/inspector/*"
126      - "botocore/data/iot-data/*"
127      - "botocore/data/iot-jobs-data/*"
128      - "botocore/data/iot/*"
129      - "botocore/data/iot1click-devices/*"
130      - "botocore/data/iot1click-projects/*"
131      - "botocore/data/iotanalytics/*"
132      - "botocore/data/iotdeviceadvisor/*"
133      - "botocore/data/iotevents-data/*"
134      - "botocore/data/iotevents/*"
135      - "botocore/data/iotfleethub/*"
136      - "botocore/data/iotsecuretunneling/*"
137      - "botocore/data/iotsitewise/*"
138      - "botocore/data/iotthingsgraph/*"
139      - "botocore/data/iotwireless/*"
140      - "botocore/data/ivs/*"
141      - "botocore/data/kafka/*"
142      - "botocore/data/kendra/*"
143      - "botocore/data/kinesis-video-archived-media/*"
144      - "botocore/data/kinesis-video-media/*"
145      - "botocore/data/kinesis-video-signaling/*"
146      - "botocore/data/kinesis/*"
147      - "botocore/data/kinesisanalytics/*"
148      - "botocore/data/kinesisanalyticsv2/*"
149      - "botocore/data/kinesisvideo/*"
150      - "botocore/data/kms/*"
151      - "botocore/data/lakeformation/*"
152      # - 'botocore/data/lambda/*'
153      - "botocore/data/lex-models/*"
154      - "botocore/data/lex-runtime/*"
155      - "botocore/data/lexv2-models/*"
156      - "botocore/data/lexv2-runtime/*"
157      - "botocore/data/license-manager/*"
158      - "botocore/data/lightsail/*"
159      - "botocore/data/location/*"
160      - "botocore/data/logs/*"
161      - "botocore/data/lookoutequipment/*"
162      - "botocore/data/lookoutmetrics/*"
163      - "botocore/data/lookoutvision/*"
164      - "botocore/data/machinelearning/*"
165      - "botocore/data/macie/*"
166      - "botocore/data/macie2/*"
167      - "botocore/data/managedblockchain/*"
168      - "botocore/data/marketplace-catalog/*"
169      - "botocore/data/marketplace-entitlement/*"
170      - "botocore/data/marketplacecommerceanalytics/*"
171      - "botocore/data/mediaconnect/*"
172      - "botocore/data/mediaconvert/*"
173      - "botocore/data/medialive/*"
174      - "botocore/data/mediapackage-vod/*"
175      - "botocore/data/mediapackage/*"
176      - "botocore/data/mediastore-data/*"
177      - "botocore/data/mediastore/*"
178      - "botocore/data/mediatailor/*"
179      - "botocore/data/meteringmarketplace/*"
180      - "botocore/data/mgh/*"
181      - "botocore/data/mgn/*"
182      - "botocore/data/migrationhub-config/*"
183      - "botocore/data/mobile/*"
184      - "botocore/data/mq/*"
185      - "botocore/data/mturk/*"
186      - "botocore/data/mwaa/*"
187      - "botocore/data/neptune/*"
188      - "botocore/data/network-firewall/*"
189      - "botocore/data/networkmanager/*"
190      - "botocore/data/opsworks/*"
191      - "botocore/data/opsworkscm/*"
192      - "botocore/data/organizations/*"
193      - "botocore/data/outposts/*"
194      - "botocore/data/personalize-events/*"
195      - "botocore/data/personalize-runtime/*"
196      - "botocore/data/personalize/*"
197      - "botocore/data/pi/*"
198      - "botocore/data/pinpoint-email/*"
199      - "botocore/data/pinpoint-sms-voice/*"
200      - "botocore/data/pinpoint/*"
201      - "botocore/data/polly/*"
202      - "botocore/data/pricing/*"
203      - "botocore/data/qldb-session/*"
204      - "botocore/data/qldb/*"
205      - "botocore/data/quicksight/*"
206      - "botocore/data/ram/*"
207      - "botocore/data/rds-data/*"
208      - "botocore/data/rds/*"
209      - "botocore/data/redshift-data/*"
210      - "botocore/data/redshift/*"
211      - "botocore/data/rekognition/*"
212      - "botocore/data/resource-groups/*"
213      - "botocore/data/resourcegroupstaggingapi/*"
214      - "botocore/data/robomaker/*"
215      - "botocore/data/route53/*"
216      - "botocore/data/route53domains/*"
217      - "botocore/data/route53resolver/*"
218      # - 'botocore/data/s3/*'
219      # - 'botocore/data/s3control/*'
220      - "botocore/data/s3outposts/*"
221      - "botocore/data/sagemaker-a2i-runtime/*"
222      - "botocore/data/sagemaker-edge/*"
223      - "botocore/data/sagemaker-featurestore-runtime/*"
224      - "botocore/data/sagemaker-runtime/*"
225      - "botocore/data/sagemaker/*"
226      - "botocore/data/savingsplans/*"
227      - "botocore/data/schemas/*"
228      - "botocore/data/sdb/*"
229      - "botocore/data/secretsmanager/*"
230      - "botocore/data/securityhub/*"
231      - "botocore/data/serverlessrepo/*"
232      - "botocore/data/service-quotas/*"
233      - "botocore/data/servicecatalog-appregistry/*"
234      - "botocore/data/servicecatalog/*"
235      - "botocore/data/servicediscovery/*"
236      - "botocore/data/ses/*"
237      - "botocore/data/sesv2/*"
238      - "botocore/data/shield/*"
239      - "botocore/data/signer/*"
240      - "botocore/data/sms-voice/*"
241      - "botocore/data/sms/*"
242      - "botocore/data/snowball/*"
243      # - 'botocore/data/sns/*'
244      - "botocore/data/sqs/*"
245      - "botocore/data/ssm/*"
246      - "botocore/data/sso-admin/*"
247      - "botocore/data/sso-oidc/*"
248      - "botocore/data/sso/*"
249      - "botocore/data/stepfunctions/*"
250      - "botocore/data/storagegateway/*"
251      - "botocore/data/sts/*"
252      - "botocore/data/support/*"
253      - "botocore/data/swf/*"
254      - "botocore/data/synthetics/*"
255      - "botocore/data/textract/*"
256      - "botocore/data/timestream-query/*"
257      - "botocore/data/timestream-write/*"
258      - "botocore/data/transcribe/*"
259      - "botocore/data/transfer/*"
260      - "botocore/data/translate/*"
261      - "botocore/data/waf-regional/*"
262      - "botocore/data/waf/*"
263      - "botocore/data/wafv2/*"
264      - "botocore/data/wellarchitected/*"
265      - "botocore/data/workdocs/*"
266      - "botocore/data/worklink/*"
267      - "botocore/data/workmail/*"
268      - "botocore/data/workmailmessageflow/*"
269      - "botocore/data/workspaces/*"
270      # - 'botocore/data/xray/*'

This list of patterns tells serverless-python-requirements what folders to ignore. The full list of services inside botocore will obviously differ over time, but it’s trivial to write a Python or shell script to generate this for you.

Unfortunately excluding glob paths doesn’t work, so you have to do it this way.

Sidenote, I added the strip: false as otherwise I was unable to import Pillow.

Conclusion

That’s really it. Thanks again to Jack for creating the original guide. I was able to shave nearly 50MB of size of my Lambda function with this method.

boto3 and botocore are only around 5MB on disk after stripping
After removing all my unneeded services, boto3 and botocore come out to around 5MB.

References