Failure Analysis Assistant (FA2) Slack App Version

日本語で読む

This is a sample implementation that responds to alarms sent to Slack by Amazon Q Developer in chat applications and helps analyze the root cause of the failure. This sample code provides the following feature:

Failure analysis assist

Logs are retrieved from a predefined log storage location within a time range specified by the user, information is extracted and summarized with LLM, and information useful for failure analysis is returned to Slack. For an example of how the function works, see Failure Analysis Assist.

Metrics Insights

In response to questions given by users, a function has been added to select metrics that require generative AI and answer questions based on that metric data. Please see Metrics Insights Doc

Findings Report

We've added the feature to create a report explaining Security Hub and GuardDuty Findings by LLM. Please see Findings Report Doc

Branches

• main - This branch. This version uses the Slack App.
• chatbot-customaction - Instead of the Slack App, this is a version that implements an input form using a Custom Action of Amazon Q Developer in chat applications. If you don't have an environment where you can't use the Slack App, or if you don't want to manage the Slack App, use this. This is not updated to Agent version.

Architecture & Workflow

The scope of existing workloads is based on the assumption that you have already been implemented. You can try this sample if the log is output to CloudWatch Logs. S3 and X-Ray are optional.

1. Alarms are triggered on the target system, and notifications are sent to Slack via Amazon SNS and Amazon Q Developer in chat applications
2. Show the input form of Slack App when it received an alarm
3. Enter log retrieval time range and event information understood from alarms, and submit a request
4. FA2 runs on AWS Lambda and uses an agent based on the ReACT (Reasoning + Acting) algorithm to perform failure analysis:
5. Thinking Step: The agent analyzes the situation and decides what action to take next
6. Acting Step: The agent executes the decided tool (e.g., retrieving logs from CloudWatch Logs, analyzing metrics)
7. Observing Step: The agent observes the results of the tool execution and collects new information
8. Cycle Repetition: The agent repeats the thinking→acting→observing cycle until sufficient information is gathered
9. Result Generation Phase: The agent generates a final failure analysis report based on the collected information
10. The agent uses the following tools to collect and analyze information:
11. metrics_tool: Retrieves and analyzes CloudWatch metrics
12. logs_tool: Retrieves and analyzes logs from CloudWatch Logs
13. athena_log_tool: Retrieves and analyzes audit logs from CloudTrail and access logs from ALB via Amazon Athena
14. xray_tool: Retrieves and analyzes trace information from X-Ray
15. kb_tool: Searches documents from Knowledge Base
16. When the analysis is complete, the agent generates a detailed failure analysis report that includes:
17. Issue summary
18. Root cause (with severity and confidence levels)
19. Referenced logs/metrics
20. Timeline analysis
21. Recommended actions
22. Prevention measures
23. The generated report is sent to Slack and provided to the user

Detailed ReACT Agent Operation

The ReACT agent, which is the core of FA2, operates with the following detailed process:

1. Session Management:
2. A unique session ID is generated for each analysis request
3. Session state is stored in DynamoDB and maintained between Lambda function executions

4. Session information includes thinking history, executed tools, collected data, etc.

5. Thinking Process:

6. In the initial thinking, the agent develops an analysis strategy based on error content and available tools
7. In subsequent cycles, it decides the next action considering the information collected so far

8. The result of thinking is output in a structured format, and the next action is determined

9. Tool Execution:

10. Selected tools are executed with appropriate parameters
11. Execution results are recorded in the session state, and data collection status is updated

12. The results of each tool execution are used as input for the next thinking cycle

13. Cycle Control:

14. The maximum number of cycles can be controlled with the maxAgentCycles parameter (default: 5)
15. When sufficient information is gathered or the maximum number of cycles is reached, it transitions to the final answer generation phase

16. Appropriate error handling is performed when Bedrock rate limits are reached

17. Final Answer Generation:

18. All collected information is integrated to generate a structured failure analysis report
19. The report is sent to Slack in Markdown format and shared as a file

Requirements

• You can use the AWS Cloud Development Kit (CDK)
• This sample is implemented in CDK
• You must have a CloudWatch Logs log group containing the logs you want to analyze
• Amazon Athena and AWS X-Ray are optional
• If you want to invlude AWS CloudTrail or Application Load Balancer (ALB) access logs, an Amazon Athena database must be created
• If AWS X-Ray trace information is also used, an AWS X-Ray trace for the relevant system must have been obtained
• Specific LLMs access has been granted from model access on Amazon Bedrock. (Ex. Claude 3.7 Sonnet, etc.)
• Confirm that an alarm notification will be sent to Slack from the Amazon Q Developer in chat applications set up in the existing workload
• If you don't have the test envrionment for FA2 or you cannot use it for FA2. You can create test environment as follow How to create a test environment for FA2.
• You must have the permission to register the Slack App to the Slack workspace you want to use.
• Turn on execution logging and access logging.
• If you didn't turn on, please turn on this setting as follow: [Create an IAM role to log to CloudWatch] and [Add the IAM role in the API Gateway console] in How do I turn on CloudWatch Logs for troubleshooting my API Gateway REST API or WebSocket API?

How to Deploy

Registration of Slack App

1. Access to Slack api and click [create New App]
2. Select [From scratch], enter an app name, select the workspace you want to use when developing, and then click [Create App]
3. The app you created will be shown in Slack api. Choose it you created.
4. Click [Basic Information] on the left menu, check [Signing Secret], execute the following command, and register with AWS Secrets Manager
5. $ aws secretsmanager create-secret --name slackSigningSecret --secret-string XXXXXXXXXXXXXXXXXXXX --profile {your_profile}
6. Click [OAuth & Permissions] on the left menu, please add permissons channels:read, chat:write and files:write in [Scopes] section.
7. Click [Install to Workspace] in [OAuth Tokens for Your Workspace] of top of same page to install your Slack App to your workspace.
8. After installing, your browser will be redirected to [OAuth & Permissions] page. You check [Bot User OAuth Token], execute the following command, and register with AWS Secrets Manager
9. $ aws secretsmanager create-secret --name slackAppToken --secret-string xxxx-111111111111111-11111111111111111-xxxxxxxxxxxxxxxxxxxxxxxxx-profile {your_profile}

Setting parameters

Refer to the following description, copy parameter_template.ts, create parameter.ts, and then change each value.

// Example: Settings for the Slack App version when using Claude 3.7 Sonnet and using CloudWatch Logs as search targets
export const devParameter: AppParameter = {
  env: {
    account: "123456789012",
    region: "us-west-2",
  },
  language: "ja",
  envName: "Development",
  modelId: "us.anthropic.claude-3-7-sonnet-20250219-v1:0",
  slackAppTokenKey: "SlackAppToken",
  slackSigningSecretKey: "SlackSigningSecret",
  architectureDescription: "The workload you are responsible for consists of ALB, EC2, and Aurora, and Spring applications are deployed on EC2.",
  cwLogsLogGroups: [
    "/ec2/demoapp",
    "/ec2/messages",
    "/aws/application-signals/data",
  ],
  cwLogsInsightQuery: "fields @message | limit 100",
  xrayTrace: false,
  slashCommands: {
    insight: true,
    findingsReport: true
  },
  knowledgeBase: true,
  embeddingModelId: "amazon.titan-embed-text-v2:0",
  maxAgentCycles: 5 // Maximum number of cycles for ReAct agent
};

Explanation of parameters