Subscribe cloud computing RSS CSDN home> cloud computing

When a particular API is called, how to use the CloudTrail AWS service, SNS Amazon service, and the Lambda AWS function to receive alerts

Published in14:23 2015-06-01| Time reading| sourceBlogs| ZeroArticle comments| authorPierreL

Abstract:We must face the fact that not all of the API are created equal. For example, you might want to know when an instance of your EC2 Amazon (elastic computing cloud) is terminated, but when the object is put into the S3 Amazon bucket is not so interested

Must face the fact that not all API are created equal. For example, you might want to know when an instance of your EC2 Amazon (elastic computing cloud) instance is terminated (using the ec2:TerminateInstance command can be terminated). (for example), but when the object is put into the S3 Amazon barrel, it is not very interesting (using s3:PutObject to put the object in the S3 barrel). In this example, you can delete an object, but you can't reply to an instance that has been terminated. . So this raises a question. "Is there a way to allow me to be notified through my AWS account when some API is called?" The answer is yes. But in spite of this, how can you be notified when some of the API is accidentally transferred to support some of the services in the AWS account?

This blog post will show you how to use CloudTrail Amazon, SNS AWS, and AWS in your account when you are being called to a specific API that you are monitoring. Lambda to receive mail notification. In particular, this post contains details of the program execution configuration instructions. This program includes:

  • CloudTrail - as a source for the log file for analysis.
  • Lambda - perform parsing and filtering logic.
  • SNS - Notification of occurrence.

The following figure illustrates how these AWS services work together.


The following is about the figure shown below the process of failure:

1.CloudTrail generates a log entry for each API call in your account. It gathers the log entries in the JSON text file, compress the text file, and copy them to Amazon S3 barrel (through the configuration, S3 Amazon barrel is set to receive the CloudTrail log file).

2 as a Lambda function, the log parsing logic is deployed. Lambda is triggered when a new file is copied from the CloudTrail to your S3 barrel.

The 3.Lambda function uses a configuration file that contains a list of specific API keywords. Once a particular API keyword is found, these keywords will trigger a notification message. The configuration file is stored in a S3 bucket.

4 whenever you find the relevant keywords in the CloudTrail log, the Lambda function sends a notification message to the SNS topic.

5.SNS via email, Amazon SQS (Simple Queue Service), short message, call the HTTP, or mobile push etc. way will notify assigned to subscribers in every topic.

The remainder of this post exercise the specific steps required to perform the process shown in the diagram above. I will finish the AWS interaction in the Amazon command line interface in the Linux AWS instance.

The prerequisites for these detailed configuration instructions are as follows:

  • A AWS account
  • The basic knowledge of AWS concepts, such as domains and access keys
  • The AWS command line interface installed on your computer and configured with administrator privileges (commands cannot be run on the EC2 role, which requires the actual user certificate). You can install the CLI AWS installation guide and the CLI AWS configuration file, using your access key, the secret key, the default output format, and the default domain to install and configure the AWS command line interface. Make sure to use CLI 1.7.22 AWS or above.
  • Test the function on the computer before deploying the Lambda function. To test this function, you must also install NodeJS and NPM tools, and are familiar with the *nix shell command.

This exercise will use us-west-2 (West US (Oregon)) as the default domain. The full source code of this exercise can be obtained in GitHub.

drill

The first step: create a S3 bucket and configure the CloudTrail service

First of all, you want to create a S3 service for the CloudTrail service to store the CloudTrail log file generated by your API call and open the CloudTrail service. If you have already opened the CloudTrail service for your account, skip this step and go straight to the second step.

The created S3 bucket must have a unique name and requires a IAM (and Access Management Identity identity and access management) barrel policy to ensure that the CloudTrail has sufficient permissions to create the file in the bucket. Amazon S3 bucket strategy to provide the default S3 CloudTrail policy.

The following create-subscription cloudtrail command will automatically create a bucket, to access the CloudTrail service associated bucket policy, and in that domain for your account to open and configure the CloudTrail service.

Make sure you change the value of the $BUCKETNAME parameter because the name of the S3 bucket is globally unique.

REGION=us-west-2
BUCKETNAME=cloudtrail-logs-abcd1234
TRAILNAME=security-blog-post
Cloudtrail create-subscription --region $REGION AWS --s3-new-bucket $BUCKETNAME --name $TRAILNAME
From now on, console, AWS CLI, or the software development kit (Development Kit Software) to call the API operation will be recorded and sent to you in the form of text files. The content of the text file is similar to the following content:

Now you will create a SNS theme, a copy of each message sent to subscribers the theme will be the theme of acceptance. Subscribers can use different protocols: HTTPS, Mail, SQS, SMS (only to the United States in line with the number of telephone numbers sent), you can also push the notification message to the mobile device.

{
"Records": [
{
"EventVersion": "2",
"EventSource": "aws:s3",
"AwsRegion": "us-west-2",
"EventTime": "2014-11-27T21:08:30.487Z",
"EventName": "ObjectCreated:Put",
"UserIdentity": {
"PrincipalId": "AWS:ARxxxxxxxxxxSW:i-4fxxxxa5"
},
"RequestParameters": {
"SourceIPAddress": "54.211.178.99"
},
"ResponseElements": {
"X-amz-request-id": "F104F805121C9B79",
"X-amz-id-2": "Lf8hbNPkrhLAT4sHT7iBYFnIdCJTmxcr1ClX93awYfF530O9AijCgja19rk3MyMF"
},
"S3": {
"S3SchemaVersion": "1",
"ConfigurationId": "quickCreateConfig",
"Bucket": {
"Name": "aws-cloudtrail-xxx",

"OwnerIdentity": {
"PrincipalId": "AHxxxxxxxxQT"
},
"ARN": "arn:aws:s3::: aws-cloudtrail-xxx"
},
"Object": {
Nbsp; "key": "AWSLogs/577xxxxxxxx68/CloudTrail/us-west-2/2014/11/27/577xxxxxxxx68_CloudTrail_us-west-2_20141127T2110Z_JWcmX2IqHMUoBRIM.json.gz",
"Size": 2331,
"ETag": "63d801bb561037f59f2cb4d1c03c2392"
}
}
}
]
}
The second step: create a SNS theme and subscribe to an e-mail address

For the purpose of the exercise, you will subscribe to the SNS theme for an e-mail address. Lambda function will generate an SNS notification message for every unexpected API call found in the CloudTrail log file. Creating a SNS theme does not subscribe to the topic of your e-mail address

EMAIL ADDRESS> EMAIL=<YOUR
TOPIC_ARN=$(SNS create-topic AWS cloudtrail_notifications --name --output text --region $REGION)
SNS subscribe --protocol AWS email --topic-arn $TOPIC_ARN --notification-endpoint $EMAIL --region $REGION
$TOPIC_ARN echo

Ensures that the ARN is written to the topic as the output of the previous command. And then you will use the theme to configure the Lambda function. ARN themes should be similar to the following.

Arn:aws:sns:us-west-2:577xxxxxxx68:cloudtrail_notifications

You will receive a confirmation email (in case you are not received, check your junk files). This message contains a link to confirm the subscription. You must click on the link to complete the subscription process.

The third step: create a IAM role for your Lambda function

The Lambda function requires a IAM role to function properly.

IAM executive role, a IAM role, which gives you the custom code permissions that enable you to access the AWS resources required for the role. This role is assumed when the Lambda executes the Lambda function on behalf of you.

A role has two parts: a trust policy (which provides services or users can be authorized to assume this role) and one or more of the provisions of the authority granted to the policy. When your function is executed, Lambda takes on a role. Your function only has the rights specified in the policy.

The following command creates a trust policy that authorizes the Lambda to perform the role of the.

Cat > <<EOF > role_trust_policy.json
{
"Version": "2012-10-17",
"Statement": [
{
"Sid":"",
"Effect": "Allow",
"Principal": {
"Service": "lambda.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}
EOF

Then, create an executive role.

ROLE_ARN=$(IAM create-role AWS lambda-security-blog --role-name --assume-role-policy-document file://./role_trust_policy.json --query'Role.Arn'--output text)
$ROLE_ARN echo

Now that you have created the role, add the permissions required by the Lambda function. The following command assumes that the environment variables set by the previous command still exist.

Cat > <<EOF > role_exec_policy.json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"Logs:*"
],
"Resource": "arn:aws:logs:*: *: *"
},
{
"Effect": "Allow",
"Action": [
"S3:GetObject"
],
"Resource": [
"Arn:aws:s3::: $BUCKETNAME/*"
]
},
{
"Effect": "Allow",
"Action": ["sns:Publish"],
"Resource": [
"$TOPIC_ARN"
]
}
]
}
EOF

The first statement in the preceding command authorizes the Lambda function to issue its own log to the CloudWatch Amazon log. Second statement authorization Lambda function fromCloudTrail log in S3 barrel [p1]Read data. The last statement is authorized by the Lambda function to send messages about the subject that is created in the previous.

The following command will add these three statements to the execution role.

Iam put-role-policy --role-name lambda-security-blog AWS --policy-name lambda-execution-role --policy-document file://./role_exec_policy.json

The fourth step: create a Lambda function

You have opened the CloudTrail service, which is associated with the S3 barrel, and a SNS theme, you can create the CloudTrail log analytic function and deploy it to the Lambda.

Create a bucket to save the number of configuration files

The Lambda function uses regular expressions to match unexpected API calls. It will use the external configuration files stored on the S3 to store the regular expressions you need to find the function. Using an external configuration file allows you to fine tune the expression without having to re deploy the Lambda function.

In order to simplify deployment, the configuration file and the CloudTrail log file are stored in the same S3 bucket. The configuration file is as follows:

The Lambda function uses regular expressions to match unexpected API calls. It will use the external configuration files stored on the S3 to store the regular expressions you need to find the function. Using an external configuration file allows you to fine tune the expression without having to re deploy the Lambda function.

In order to simplify deployment, the configuration file and the CloudTrail log file are stored in the same S3 bucket. The configuration file is as follows:

{
"Source": "iam|signin|sts",
"Regexp": "Password|AccessKey|PutRolePolicy|User|Token|Login",
"SNS": {
"Region": "us-west-2",
"TopicARN": "arn:aws:sns:us-west-2:577xxxxxxx68:PushNotifications"
}
}

The following is a component of the configuration file:

  • Parameter source defines the service you will monitor API. You can modify the parameters to meet your requirements.
  • Parameter regexp definition will trigger the API call list of the notification message. The parameters can also be modified to meet your requirements.
  • SNS part of the provisions of your SNS configuration. Modify the value of its value to the domain and second steps created in the previous section ARN.

Edit the configuration file to meet your requirements, and then use the following command to copy the file to the S3 bucket:

S3 CP filter_config.json s3://$BUCKETNAME/filter_config.json --region AWS $REGION

Create Lambda function

You can use any JavaScript text editor to create a Lambda function. When writing this blog, the Lambda function uses the Node.js architecture. The core of the Lambda function is the JavaScript function, which is defined as a Node.js processor. When you write code, remember that the Node.js schema is not synchronized. Every time a function call generates a call log. You can use the callback function or future function to synchronize your code execution results.

When you deploy the Lambda function, you must define the Lambda function to perform the triggering event. Currently, the Lambda function can respond to API calls, DynamoDB table changes, S3 events, or SNS notifications to trigger the function.

The exercise using the JavaScript Q library. Q library to extract non synchronous operations as a future object (also known as the future function). The Lambda function performs the following operations in the order of the arrangement:

One  Download the fourth step to create the Lambda function configuration file.

Two  Download the CloudTrail log file, and its availability is just what the system has just notified you of. Lambda passes a record object to a function. The record object contains the name of the file that has just been created on the S3.

Three  Extract the CloudTrail log file to a temporary storage location.

Four  Filter files, only to find the log entry from the selected AWS service. In this case, the Lambda function is only looking for IAM, login, and STS generated by the record.

Five  Searches for a given regular expression.

One  When you find a matching record, send a notification to the SNS service.

I will break down the Lambda function into different JavaScript functions, each of which performs a particular item in the preceding operation. The core of the Lambda function is contained in the following content. It performs the operation.

The complete source is available from theGitHubget In order to express clearly, the following fragment does not include error handling, logging, or Q library related representation.

Exports.handler = function (event, context) {bucket var = event.Records[0].s3.bucket.name;
Var key = event.Records[0].s3.object.key; / / Download from S3, Gunzip,
And send notifications download (bucket, key).Then (extract).Then (filter) (Filter)
.then (notify).Catch (function (error) Handle any error from all {/ / above
.done (steps) (function) (() {context.done ();});};

Q library to ensure that every function will be called in the order.

Download profile

Using JavaScriptSDK AWS, download the configuration file from S3.

S3 var = aws.S3 new ('2006-03-01'} {apiVersion:);
Params var = {Key:FILTER_CONFIG_FILE, Bucket:FILTER_CONFIG_BUCKET};
Body var = s3.getObject (params)
.createReadStream ()
.on ('data'(), function (data) {
FILTER_CONFIG = JSON.parse (data);
});

The above code allows you to download the configuration file and parse the text to create a JavaScript object. You need to configure the global variable FILTER_CONFIG_BUCKET and FILTER_CONFIG_FILE values for the name of your bucket and the name of the configuration file.

Download CloudTrail log files from S3

useNode.jsby(I/Oinput/output)Operation providedFSModule and targeting API AWS Node.js S3frameworkfromS3downloadCloudTrail file, and store it in local purpose (/tmp).

Output result of observation log. Correct all errors before deploying the Lambda code.

File name from key name / / extract
Var fileName = key.match (/.*\/ (*).Json.gz$/ [1])
File var = fs.createWriteStream ('/tmp/'+ fileName +'.json.gz');
 
From S3 to local file / / pipe
S3 var = aws.S3 new ('2006-03-01'} {apiVersion:);
Params var = {Key:key, Bucket:bucket};
Stream var = s3.getObject (params).CreateReadStream ();
Stream.pipe (file);

The command will store a copy of the file in the /tmp directory of the Lambda container.

Extract the CloudTrail log file to a temporary storage location

Using GZIP Node.js library, from the compressed file to extract the text based log files.

Var gzFileName = file.path.match (/.*\/ (*).Json.gz$/ [1]);
JsFileName var ='/tmp/'+ gzFileName +'.json';
 
Zlib var = require ('zlib');
Unzip var = zlib.createGunzip ();
InP var = fs.createReadStream (file.path);
Out var = fs.createWriteStream (jsFileName);
 
Inp.pipe (unzip).Pipe (out);

This operation will generate an input stream to read the file from the local file system, and then send the input stream to an output stream that is attached to another file in the file system.

Screen selected AWS services

Since there is a local JSON text file available, parse it, look for those AWS services and API so that the system generates a SNS notification.

CloudTrailLog var = require (file);
Records var = (function) {cloudTrailLog.Records.filter (x)
X.eventSource.match return (RegExp new (FILTER_CONFIG.source));
});

Search regular expression

The last step in the memory to load a large number of records for the analysis of objects. Build a message for each match, and send the message to the sendNotification () function.

For (I var = 0; I < records.length; i++) {if (records[i].eventName.match) (New); ().
RegExp (FILTER_CONFIG.regexp)) {message var = "Event:" + records[i].eventName
+ "\n" + "Source:" + records[i].eventSource + "\n" + "Params:" + JSON.stringify (records[i].requestParameters,
Null + + "" + "\n" + "Region:" + records[i].awsRegion + "var" task = \n =
SendNotification (message, FILTER_CONFIG.sns.topicARN, ');}

Send SNS notification

Finally, use the JavaScript AWS software development kit to send SNS notifications.

SNS var = aws.SNS new ({
'2010-03-31'apiVersion,
FILTER_CONFIG.sns.region region:
});
 
Var params = {}
Params = {
MSG Message,
TopicARN TopicArn:
};
 
Sns.publish (params, function (err, data) {
If (ERR) {
Error occurred / / an
{else} {
Successful / / response
}
});

The function locally Test local test of the function

Before the function is uploaded to the Lambda and in the AWS test the function, we recommend that the function in the local test. Early in the development cycle, by quickly capturing programming errors, you can greatly shorten the development, deployment, testing, debugging cycle. Lambda function deployment before the local test should be considered as a best practice, because it allows you to write code at the same time to test it, and customized according to the AWS account.

In order to perform a local test on the Lambda function, I wrote a small string of code. This string represents the Lambda function that will be called as the function will be invoked from the Lambda service. The code judgment will provide you with two objects:

One  An input record, and the record is linked to the S3 event, Lambda provides similar records. Lambda will create a file (input.json) that is used to store the record.

Two  A context record, which is similar to the function that Lambda provides to you.

The input record used in this test is similar to the following:

{
"Records": [
{
"EventVersion": "2",
"EventSource": "aws:s3",
"AwsRegion": "us-west-2",
"EventTime": "2014-11-27T21:08:30.487Z",
"EventName": "ObjectCreated:Put",
"UserIdentity": {
"PrincipalId": "AWS:AROXXXXXXXXXXSW:i-4ffXXXXa5"
},
"RequestParameters": {
"SourceIPAddress": "54.211.178.99"
},
"ResponseElements": {
"X-amz-request-id": "F104F805121C9B79",
"X-amz-id-2": "Lf8hbNPkrhLAT4sHT7iBYFnIdCJTmxcr1ClX93awYfF530O9AijCgja19rk3MyMF"
},
"S3": {
"S3SchemaVersion": "1",
"ConfigurationId": "quickCreateConfig",
"Bucket": {
"Name": "aws-cloudtrail-xxx",
"OwnerIdentity": {
"PrincipalId": "AH4XXXXXXXQT"
},
"ARN": "arn:aws:s3::: aws-cloudtrail-xxx"
},
"Object": {
"Key": "AWSLogs/577XXXXXXX68/CloudTrail/us-west-2/2014/11/27/577XXXXXXXX68_CloudTrail_us-west-2_20141127T2110Z_JWcmX2IqHMUoBRIM.json.gz",
"Size": 2331,
"ETag": "63d801bb561037f59f2cb4d1c03c2392"
}
}
}
]
}

Make sure that the name of the bucket and the key point to a real CloudTrail log in your bucket. The value of the size and the electronic tag will not be used by the function, you do not have to modify their values.

You can start with a real Lambda function orDocumentGets the input record object. The calling function is stored in the main.js, similar to the following code:

Lambda var = //your ('./cloudtrail.js') lambda function var event require
= require ('./input.json') //your input record var context context.done = {}
= function (arg1, arg2) {console.log ('context.done')} lambda.handler (event) (),
Context)

Before the test code, first installed in the local test procedures required.

Install aws-sdk Q NPM

You must configure an access key, a password key, and the default domain for the AWS software development kit. If you are using the command line interface, you are using the AWS command line interface, then your local configuration is correct. Otherwise, run AWS Configure command to create a default configuration file.

Then, on your computer, enter the command line to call the node.

Main.js node

Output result of observation log. Correct all errors before deploying the Lambda code.

Deploying Lambda functions

Once you believe that the function has been run as expected, you can be prepared to deploy the function in Lambda. First, create a.Zip file (cloudtrail.js) that contains the function and dependency program (Q database).

-r cloudtrail.zip cloudtrail.js node_modules -x zip node_modules/aws-sdk/\*

In this file, you do not need to include the JavaScript AWS software development kit, because the Lambda's execution environment is provided by default.

Now, upload and create the function on Lambda (the value of the parameter $ROLE_ARN must be set to the value returned by the Iam create-role AWS command in the first half of this post.).

Function FUNCTION_ARN=$(AWS lambda create-function --zip-file fileb://./cloudtrail.zip --function-name cloudtrail --runtime nodejs --role $ROLE_ARN ARN --handler cloudtrail.handler --region $REGION output text --query "FunctionArn")

Remember the output value of the parameter Function_ARN, because you will use this value in the next step.

Link the function to the S3 service

The last thing to do is to configure the S3, which can trigger the Lambda function when the file is created in the bucket. This is a process that contains two steps. You first need to add permissions to the Lambda function, which authorizes your S3 bucket to be able to trigger the execution of the Lambda function.

AWS_ACCOUNT_ID=$(echo $ROLE_ARN sed's/^arn:aws:iam: |: \ (.*\):.*$/\1/')
AWS lambda add-permission --function-name cloudtrail --statement-id Id-cloudtrail --action lambda:InvokeFunction --principal s3.amazonaws.com --source-arn arn:aws:s3::: $BUCKETNAME --source-account $AWS_ACCOUNT_ID account ID --region $REGION

Ensure that the value of $FUNCTION_ARN is set to the fourth step in the Function_ARN output value, as in the first half of this post, the value of the Function_ARN returned to the upload code.

Then, configure the S3 notification system. You can configure the web S2 console, or you can use the command line configuration in CLI AWS. But you need a configuration file.

Cat > s3_notifications_config.json > <<EOF {"CloudFunctionConfiguration":
{"CloudFunction": "$FUNCTION_ARN", "Id": "cloudtrail-lambda-notifications",
"Event": "s3:ObjectCreated:*"} EOF

To ensure that the value of <FUNCTION_ARN> is set to "deploy the function" section, the output value of the Function_ARN upload function, as described in the first part of this post.

Finally, open the event notification function from S3 to Lambda.

S3api put-bucket-notification --notification-configuration file://./s3_notifications_config.json AWS --bucket $BUCKETNAME --region $REGION

If you have been in accordance with the instructions, carried out the operation of the above steps, congratulations! You can test your deployment to end to end.

Good news, you only need to do a few steps to test the deployment of the operation. You only have a few unexpected API calls (which you have already configured in the filter_config.json configuration file). If you are using the configuration file that is previously suggested, log in to AWS Console Management (AWS management console), create a few roles, and then add and delete strategies for these roles. A few minutes later, the CloudTrail service will generate a log file, and the S3 service will trigger the Lambda function, and send a message to your SNS theme.

Sixth step: Troubleshooting

If you do not immediately receive an e-mail notification, please be patient and wait. CloudTrail service takes a few minutes to send the CloudTrail log file, and the sending rate of the email depends on your mail facility.

If ten or fifteen minutes later, you still do not receive the mail, please check the following items:

  • Is there a CloudTrail log file in your bucket? If not, check your CloudTrail service configuration.
  • Is your Lambda function being called? If not, check your S3 notification service settings, including the Lambda function call permissions.
  •  Check the Lambda log in the CloudWatch log. In the Lambda console, you can get the link.
  In the log:

  •  Is the filtered event list empty? Check your regular expression filters in the configuration file.
  • Do you have an error occurred while sending a message to the SNS theme? Check the Lambda function to perform the role policy.
  • Still not receive e-mail notification? When you subscribe to the topic, do you verify your email address? Have you checked your spam box yet?

Text link:Http://blogs.aws.amazon.com/security/post/Tx2ZTUVN2VGBS85/How-to-Receive-Alerts-When-Specific-APIs-Are-Called-by-Using-AWS-CloudTrail-Amaz

Activity recommendation:May 26th CSDN online training - AWS cloud computing environment in the machine learning

(translation / Lv Dongmei / commissioning editor Wang Xinhe)


AWS Chinese technology community micro channel public number, real-time master AWS technology and product information!

AWS Chinese technology community for the majority of developers to provide a Web Service Amazon technology exchange platform, push AWS latest information, technology, video, technical documents, wonderful technology and other related exciting content, more AWS community experts to communicate directly with you! Quickly join the AWS Chinese technology community, faster and better understanding of AWS cloud computing technology.


top
Zero
step on
Zero