wifikeron.blogg.se - Redshift data api

#Redshift data api code#

The rule triggers the Lambda function LambdaRedshiftDataApiETL, with the action notify as an input parameter.

When the stored procedure is complete, the EventBridge rule EventBridgeRedshiftEventRule is triggered automatically to capture the event based on the source parameter redshift-data from the Amazon Redshift Data API.

We have pre-populated this with the NYC Yellow Taxi public dataset for the year 2015 to test this solution.

The stored procedure uses the Amazon S3 location event-driven-app-with-lambda-redshift/nyc_yellow_taxi_raw/ as the data source for the ELT process.

The function performs an asynchronous call to the stored procedure run_elt_process in Amazon Redshift, performing ELT operations using the Amazon Redshift Data API.

#Redshift data api code#

The Python code for the Lambda function is available in the GitHub repo. The rule triggers the Lambda function LambdaRedshiftDataApiETL, with the action run_sql as an input parameter.The EventBridge rule EventBridgeScheduledEventRule is initiated based on a cron schedule.The workflow includes the following steps: The following architecture diagram highlights the end-to-end solution, which you can provision automatically with an AWS CloudFormation template. For more information, see Getting Started with Amazon Redshift. PrerequisitesĪs a prerequisite for creating the application in this post, you need to set up an Amazon Redshift cluster and associate it with an AWS Identity and Access Management (IAM) role.

It also simplifies adding other components or third-party products to the application without many changes. This event-driven server-less architecture offers greater extensibility and simplicity, making it easier to maintain and faster to release new features, and also reduces the impact of changes. When complete, it sends an event to EventBridge, which triggers a Lambda function to send notification to end-users through Amazon Simple Notification Service (Amazon SNS) to inform them about the availability of updated data in Amazon Redshift. This stored procedure copies the source data from Amazon Simple Storage Service (Amazon S3) to Amazon Redshift and aggregates the results. In response to a scheduled event defined in EventBridge, this application automatically triggers a Lambda function to run a stored procedure performing extract, load, and transform (ELT) operations in an Amazon Redshift data warehouse, using its out-of-the-box integration with the Amazon Redshift Data API. In this post, we explain how you can easily design a similar event-driven application with Amazon Redshift, AWS Lambda, and Amazon EventBridge.