Visualizing Amazon DynamoDB data with Amazon OpenSearch Service and Amazon Managed Grafana

High-performance applications with unlimited throughput capabilities pose significant monitoring challenges, especially when tracking real-time metrics, utilization, and throttling events across distributed database workloads. Near real-time visibility into metrics is crucial for application performance and cost optimization.

AWS allows you to seamlessly integrate multiple services to tackle these operational complexities. With Amazon DynamoDB, you can build applications that require fast local reads and write performances. With Amazon OpenSearch Service, you can have real-time search, monitoring, and analysis of business and operational data. With Amazon DynamoDB zero-ETL integration with Amazon OpenSearch Service, you can synchronize the data between Amazon DynamoDB and Amazon OpenSearch Service. The integration is implemented through the DynamoDB plugin for OpenSearch Ingestion, providing a fully managed, no-code experience for data ingestion. This enables you to use the search features of Amazon OpenSearch Service against your data in Amazon DynamoDB. With Amazon Managed Grafana, you can add Amazon OpenSearch Service as a data source to query and correlate metrics and logs, then view and analyze all of that data in a single visualization or dashboard.

Building on the above capabilities, this blog post demonstrates how to create a pipeline connecting DynamoDB storage, OpenSearch analytics, and Grafana visualization. You will learn how to gain insights from your data more quickly and efficiently without complex ETL processes. The solution enables near real-time visualization of DynamoDB application metrics in Grafana dashboards, allowing quick identification of performance issues and easier troubleshooting.

Solution overview

The following diagram illustrates the different components of this architecture.

The diagram depicts the architecture diagram

Figure 1:Architecture diagram

At a high level, the steps can be summarized as follows:

Enable zero-ETL integration to synchronize the data between Amazon DynamoDB and Amazon OpenSearch Service.
Use Amazon Managed Grafana built-in data source for Amazon OpenSearch Service to discover the OpenSearch Service accounts, and manage the configuration of the authentication credentials that are required to access OpenSearch.
Build dashboards in Amazon Managed Grafana to view near real-time health of your Amazon DynamoDB.

Prerequisites

For this walk through, you will need the following:

AWS account
IAM user/role with required permissions to perform tasks in walk through section
AWS CLI
AWS CDK
Git, NPM, Node.js, Python 3.x, boto 3 Python library installed locally
Amazon Managed Grafana workspace

Walk through

The following sections walk you through the solution. For this demonstration we will use movies data as the sample dataset.

Enabling zero-ETL integration between Amazon DynamoDB and Amazon OpenSearch service

To begin, you will deploy the necessary AWS services using CDK-based Infrastructure as Code (IaC), enabling zero-ETL integration between DynamoDB and Amazon OpenSearch. The initial step is to configure your local environment for CDK deployment. For that you need to establish credentials to authenticate with your AWS account from your local environment. The preferred and secured approach is to use temporary credentials as per the instructions in Authenticating with short-term credentials for the AWS CLI.

Setup and CDK bootstrapping:

# Clone the solution repository
git clone https://github.com/aws-samples/sample-grafana-opensearch-dynamodb.git
cd <directory>

# Install dependencies
npm install

# Bootstrap the CDK 
cdk boostrap

Creating a DynamoDB table:
To create the DynamoDB table, run the CDK stack named ‘DynamoDBStack’ using the following command. This will create a DynamoDB table named ‘movies-datastore’ with a primary key named ‘title’. As part of the table creation, the stack also enables “Streams” on your DynamoDB table with the stream view type as “New and old images”.
```
cdk deploy DynamoDBStack
```
Creating an OpenSearch domain:
Next, create the Amazon OpenSearch domain for storing and searching the “movies” data. To do this, run the CDK stack named ‘OpenSearchStack’ using the following command. This stack creates an Amazon OpenSearch domain with required configurations along with the following resources. a/ A KMS key with the policies for encrypting the data in the OpenSearch domain and b/ An IAM Role with the permissions to access the DynamoDB table, its associated streams, and the OpenSearch domain.
```
cdk deploy OpenSearchStack
```
Creating the OpenSearch Ingestion Pipeline for the zero-ETL integration:
Now that you have the DynamoDB table, which is the source, and the OpenSearch domain, which is the destination, create the OpenSearch ingestion pipeline. There are two ways to create the ingestion pipeline:
(a) Using AWS Management Console to configure the “Zero-ETL with DynamoDB” blueprint from the available templates.
(b) Programmatically running the CKD stack ‘PipelineStack’ which creates the following: (1) OpenSearch ingestion pipeline with the following configurations: DynamoDB table as the source, OpenSearch domain as the sink, Associating the IAM role created in step 3, and Streaming ingestion for data updates and (2) CloudWatch log group to write the ingestion log.
```
cdk deploy PipelineStack
```
Configuring authorization within the OpenSearch domain:
OpenSearch authentication is configured at the AWS IAM level and the authorization is configured within the OpenSearch security settings accessed through the OpenSearch domain url. You should always follow the principle of least privilege while granting access.Using the AWS Management Console, navigate to AWS Secrets Manager -> Secrets -> OpenSearchMasterUser to obtain the master user credentials to log in to the OpenSearch domain. Once you have authenticated to the OpenSearch domain, using the hamburger menu on the left side, navigate to Security -> Roles to authorize the IAM role that was created in step 3.

Figure 2: Use the hamburger menu on the left side and navigate to “Security” -> “Roles”

Next, choose the “all_access” role and select “Manage mapping” within the “Mapped users” tab.

Figure 3: Choose the “all_access” role and select “Manage mapping” within the “Mapped users” tab

Next, enter the ARN of the IAM role created in step 3 in “Backend roles” and select “Map”.

Figure 4: Enter the ARN of the IAM role created in step 3 in “Backend roles” and select “Map”

You can validate the above implementation by running the python script using the following command that loads the DynamoDB table with a sample movie data. If everything is configured correctly, the zero-ETL integration will automatically capture the DynamoDB table updates and stream them directly into the OpenSearch domain through the ingestion pipeline. For monitoring, use the CloudWatch logs created in step 4.
```
python UploadData.py
```

Configuring Amazon Managed Grafana

Creating a Grafana workspace:
Grafana is a visualization tool that is popular for its extensible data support. Amazon Managed Grafana is a fully managed and secure data visualization service that is easy to deploy, operate, and scale. To create a Grafana workspace, follow the steps mentioned in Creating a workspace.
Configuring OpenSearch authorization for Amazon Managed Grafana service-linked IAM role:
A service-linked role makes setting up Amazon Managed Grafana easier because you don’t have to manually add the necessary permissions, as these roles are predefined by Amazon Managed Grafana and include all the permissions that the service requires to call other AWS services on your behalf. In our use-case, the defined permissions include the trust policy and the permissions policy to access OpenSearch. You can create a service-linked role via AWS Management Console, the AWS CLI, or the AWS API. Grafana needs additional permissions to access OpenSearch because authorization is configured within the OpenSearch security settings. To configure the necessary permissions, refer to section 1, step 5 where we covered the setup process.
Configuring Amazon OpenSearch Service data source:
Using OpenSearch Service data source, you can perform OpenSearch queries in Grafana in order to visualize data that stored in OpenSearch and build dashboards. To configure Amazon OpenSearch Service as a data source in Grafana workspace, follow the steps mentioned in Use AWS data source configuration to add OpenSearch Service as a data source. The following image shows a successfully configured OpenSearch data source.

Figure 5: Grafana data source configuration to add OpenSearch Service as a data source
Creating a dashboard:
A dashboard is a set of one or more panels that allows you to show your data in a visual form. To create a dashboard, follow the steps mentioned in Creating dashboards. Make sure to select Amazon OpenSearch Service as your data source in the panel. The following image shows a sample dashboard.

Figure 6: Grafana dashboard querying Amazon OpenSearch to visualize data from DynamoDB

A dashboard in Grafana is represented by a JSON object, which stores metadata of its dashboard. You can create a similar dashboard by uploading the provided dashboard.json file and following the steps mentioned in Importing a dashboard.

Cleanup

To avoid ongoing charges in your AWS account, you should delete the AWS resources by running the CDK destroy command from the project directory. Furthermore, log in to the AWS Management Console and delete any manually created resources.

cdk destroy --all

Conclusion

In this post, you learned how to visualize metrics stored in DynamoDB in near real-time Grafana dashboards. We provided you with an infrastructure as code (IaC) template and a sample dashboard JSON file to test the solution. We are here to help and if you need further assistance, reach out to AWS Support and your AWS account team.

AWS Cloud Operations Blog