Aws Lambda Function Upload and Download Speed

Introducing Amazon S3 Object Lambda – Use Your Lawmaking to Process Information as It Is Existence Retrieved from S3

When you store data in Amazon Unproblematic Storage Service (Amazon S3), you lot can easily share it for utilize by multiple applications. However, each application has its own requirements and may need a different view of the information. For example, a dataset created past an e-commerce awarding may include personally identifiable data (PII) that is non needed when the same data is processed for analytics and should be redacted. On the other side, if the same dataset is used for a marketing entrada, you may demand to enrich the data with additional details, such every bit information from the customer loyalty database.

To provide different views of data to multiple applications, there are currently two options. You either create, shop, and maintain boosted derivative copies of the information, so that each application has its own custom dataset, or you build and manage infrastructure every bit a proxy layer in forepart of S3 to intercept and process data equally it is requested. Both options add complication and costs, and so the S3 team decided to build a better solution.

Today, I'm very happy to denote the availability of S3 Object Lambda, a new adequacy that allows you to add your own code to process data retrieved from S3 before returning it to an application. S3 Object Lambda works with your existing applications and uses AWS Lambda functions to automatically process and transform your data as it is existence retrieved from S3. The Lambda function is invoked inline with a standard S3 GET asking, then you don't need to change your application code.

In this mode, yous can hands present multiple views from the same dataset, and you lot tin update the Lambda functions to change these views at whatever time.

There are many use cases that can exist simplified by this arroyo, for example:

• Redacting personally identifiable information for analytics or non-production environments.
• Converting across data formats, such as converting XML to JSON.
• Augmenting data with information from other services or databases.
• Compressing or decompressing files every bit they are being downloaded.
• Resizing and watermarking images on the fly using caller-specific details, such as the user who requested the object.
• Implementing custom authorization rules to access data.

Yous tin start using S3 Object Lambda with a few elementary steps:

1. Create a Lambda Function to transform information for your use case.
2. Create an S3 Object Lambda Access Bespeak from the S3 Direction Console.
3. Select the Lambda function that you created above.
4. Provide a supporting S3 Access Point to give S3 Object Lambda access to the original object.
5. Update your awarding configuration to use the new S3 Object Lambda Access Point to remember data from S3.

To get a better understanding of how S3 Object Lambda works, let's put it in practice.

How to Create a Lambda Function for S3 Object Lambda
To create the function, I start by looking at the syntax of the input consequence the Lambda function receives from S3 Object Lambda:

            {     "xAmzRequestId": "1a5ed718-5f53-471d-b6fe-5cf62d88d02a",     "getObjectContext": {         "inputS3Url": "https://myap-123412341234.s3-accesspoint.us-east-i.amazonaws.com/s3.txt?10-Amz-Security-Token=...",         "outputRoute": "io-iad-cell001",         "outputToken": "..."     },     "configuration": {         "accessPointArn": "arn:aws:s3-object-lambda:us-east-one:123412341234:accesspoint/myolap",         "supportingAccessPointArn": "arn:aws:s3:the states-east-1:123412341234:accesspoint/myap",         "payload": "test"     },     "userRequest": {         "url": "/s3.txt",         "headers": {             "Host": "myolap-123412341234.s3-object-lambda.united states of america-eastward-1.amazonaws.com",             "Accept-Encoding": "identity",             "X-Amz-Content-SHA256": "e3b0c44297fc1c149afbf4c8995fb92427ae41e4649b934ca495991b7852b855"         }     },     "userIdentity": {         "blazon": "IAMUser",         "principalId": "...",         "arn": "arn:aws:iam::123412341234:user/myuser",         "accountId": "123412341234",         "accessKeyId": "..."     },     "protocolVersion": "1.00" }          

The getObjectContext holding contains some of the virtually useful data for the Lambda role:

• The inputS3Url is a presigned URL that the function can utilise to download the original object from the supporting Access Point. In this way, the Lambda function doesn't need to have S3 read permissions to retrieve the original object and tin only access the object processed by each invocation.
• The outputRoute and the outputToken are two parameters that are used to send back the modified object using the new WriteGetObjectResponse API.

The configuration property contains the Amazon Resources Name (ARN) of the Object Lambda Access Signal and of the supporting Access Point.

The userRequest property gives more than information of the original asking, such as the path in the URL, and the HTTP headers.

Finally, the userIdentity section returns the details of who made the original asking and can exist used to customize access to the data.

Now that I know the syntax of the effect, I tin can create the Lambda office. To proceed things simple, here'south a function written in Python that changes all text in the original object to uppercase. When creating this office in the Lambda console, I select the Python three.7 runtime.

            import boto3 import requests  def lambda_handler(event, context):     impress(result)      object_get_context = issue["getObjectContext"]     request_route = object_get_context["outputRoute"]     request_token = object_get_context["outputToken"]     s3_url = object_get_context["inputS3Url"]      # Get object from S3     response = requests.get(s3_url)     original_object = response.content.decode('utf-8')      # Transform object     transformed_object = original_object.upper()      # Write object back to S3 Object Lambda     s3 = boto3.client('s3')     s3.write_get_object_response(         Body=transformed_object,         RequestRoute=request_route,         RequestToken=request_token)      return {'status_code': 200}                      

Looking at the code of the function, there are three principal sections:

• Showtime, I use the inputS3Url property of the input outcome to download the original object. Since the value is a presigned URL, the office doesn't need permissions to read from S3.
• And then, I transform the text to be all capital letter. To customize the behavior of the function for your use case, this is the part you need to change. For example, to detect and redact personally identifiable information (PII), I tin use Amazon Comprehend to locate PII entities with the DetectPiiEntities API and supercede them with asterisks or a description of the redacted entity type.
• Finally, I employ the new WriteGetObjectResponse API to send the issue of the transformation dorsum to S3 Object Lambda. In this style, the transformed object can be much larger than the maximum size of the response returned past a Lambda role. For larger objects, the WriteGetObjectResponse API supports chunked transfer encoding to implement a streaming data transfer. The Lambda function just needs to return the condition code (200 OK in this instance), eventual errors, and optionally customize the metadata of the returned object as described in the S3 GetObject API.

I parcel the function and its dependencies, including an updated version of the AWS SDK for Python (boto3) implementing the new write_get_object_response method, and upload it to Lambda. Note that the maximum duration for a Lambda function used by S3 Object Lambda is 60 seconds, and that the Lambda function needs AWS Identity and Access Management (IAM) permissions to call the WriteGetObjectResponse API.

How to Create an S3 Object Lambda Access Point from the Console
In the S3 console, I create an S3 Access Betoken on ane of my S3 buckets:

And then, I create an S3 Object Lambda Admission Point using the supporting Access Point I just created. The Lambda function is going to apply the supporting Access Point to download the original objects.

During the configuration of the S3 Object Lambda Admission Indicate equally shown below, I select the latest version of the Lambda office I created in a higher place. Optionally, I can enable support for requests using a byte range, or using function numbers. For now, I leave them disabled. To empathise how to apply byte range and part numbers with S3 Object Lambda, please come across the documentation.

When configuring the S3 Object Lambda Access Signal, I tin fix a string as a payload that is passed to the Lambda role in all invocations coming from that Access Point, as you tin see in the configuration property of the sample outcome I described before. In this way, I can configure the same Lambda function for multiple S3 Object Lambda Access Points, and use the value of thepayload to customize the behavior for each of them.

Finally, I can set up a policy, similar to what I can practise with normal S3 Access Points, to provide admission to the objects accessible through this Object Lambda Admission Point. For at present, I keep the policy empty. Then, I leave the default pick to block all public admission and create the Object Lambda Admission Signal.

Now that the S3 Object Lambda Access Betoken is gear up, let'due south run into how I can use it.

How to Use the S3 Object Lambda Admission Point
In the S3 panel, I select the newly created Object Lambda Admission Point. In the properties, I copy the ARN to have information technology available later.

With the AWS Command Line Interface (CLI), I upload a text file containing a few sentences to the S3 bucket behind the S3 Object Lambda Access Point:

Using S3 Object Lambda with my existing applications is very simple. I just need to replace the S3 bucket with the ARN of the S3 Object Lambda Admission Indicate and update the AWS SDKs to accept the new syntax using the S3 Object Lambda ARN.

For case, this is a Python script that downloads the text file I just uploaded: commencement, straight from the S3 bucket, and and so from the S3 Object Lambda Access Point. The only difference between the two downloads is the value of the Bucket parameter.

            import boto3  s3 = boto3.client('s3')  impress('Original object from the S3 bucket:') original = s3.get_object(   Saucepan='danilop-data',   Fundamental='s3.txt') print(original['Body'].read().decode('utf-8'))  print('Object processed by S3 Object Lambda:') transformed = s3.get_object(   Bucket='arn:aws:s3-object-lambda:us-east-one:123412341234:accesspoint/myolap',   Fundamental='s3.txt') print(transformed['Torso'].read().decode('utf-8'))          

I start the script on my laptop:

And this is the result I get:

Original object on S3:
Amazon Simple Storage Service (Amazon S3) is an object storage service that offers industry-leading scalability, data availability, security, and performance. This means customers of all sizes and industries can use it to store and protect any amount of data for a range of use cases, such as data lakes, websites, mobile applications, backup and restore, annal, enterprise applications, IoT devices, and big information analytics.


Object processed past S3 Object Lambda:
AMAZON Unproblematic STORAGE SERVICE (AMAZON S3) IS AN OBJECT STORAGE SERVICE THAT OFFERS Industry-LEADING SCALABILITY, Information AVAILABILITY, SECURITY, AND Operation. THIS MEANS CUSTOMERS OF ALL SIZES AND INDUSTRIES Tin can Use It TO Store AND PROTECT ANY Corporeality OF Information FOR A RANGE OF USE CASES, SUCH Equally Information LAKES, WEBSITES, MOBILE APPLICATIONS, BACKUP AND RESTORE, Archive, ENTERPRISE APPLICATIONS, IOT DEVICES, AND BIG DATA ANALYTICS.


The first output is downloaded directly from the source saucepan, and I meet the original content as expected. The 2d time, the object is processed past the Lambda function as it is beingness retrieved and, every bit the result, all text is uppercase!

More Use Cases for S3 Object Lambda
When retrieving an object using S3 Object Lambda, at that place is no demand for an object with the same name to exist in the S3 bucket. The Lambda function tin can use information in the proper noun of the file or in the HTTP headers to generate a custom object.

For example, if you ask to apply an S3 Object Lambda Access Point for an image with name sunset_600x400.jpg, the Lambda office can wait for an image named dusk.jpg and resize it to fit the maximum width and superlative every bit described in the file name. In this instance, the Lambda function would need access permission to read the original image, because the object key is different from what was used in the presigned URL.

Another interesting use case would be to retrieve JSON or CSV documents, such as lodge.json or items.csv, that are generated on the wing based on the content of a database. The metadata in the request HTTP headers can be used to pass the orderId to employ. Equally usual, I expect our customers' creativity to far exceed the use cases I described here.

Here'due south a short video describing how S3 Object Lambda works and how yous can use it:

Availability and Pricing
S3 Object Lambda is available today in all AWS Regions with the exception of the Asia Pacific (Osaka), AWS GovCloud (US-East), AWS GovCloud (U.s.-West), Prc (Beijing), and China (Ningxia) Regions. Y'all tin use S3 Object Lambda with the AWS Management Console, AWS Command Line Interface (CLI), and AWS SDKs. Currently, the AWS CLI loftier-level S3 commands, such equally aws s3 cp, don't back up objects from S3 Object Lambda Access Points, simply you lot can use the depression-level S3 API commands, such as aws s3api go-object.

With S3 Object Lambda, you pay for the AWS Lambda compute and request charges required to process the data, and for the information S3 Object Lambda returns to your awarding. Yous also pay for the S3 requests that are invoked by your Lambda function. For more pricing information, please see the Amazon S3 pricing page.

If you're looking for another example, here's a complete implementation past my colleague Greg Davis using S3 Object Lambda to render a thumbnail version of an image in S3.

This new capability makes information technology much easier to share and convert data across multiple applications.

Outset using S3 Object Lambda to simplify your storage architecture today.

— Danilo

bakerunfue1978.blogspot.com

Source: https://aws.amazon.com/blogs/aws/introducing-amazon-s3-object-lambda-use-your-code-to-process-data-as-it-is-being-retrieved-from-s3/