/
Creating Lambda to enable scaling of external EMR

Creating Lambda to enable scaling of external EMR

Applies to: Kyvos Enterprise  Kyvos Cloud (SaaS on AWS) Kyvos AWS Marketplace

Kyvos Azure Marketplace   Kyvos GCP Marketplace Kyvos Single Node Installation (Kyvos SNI)

Kyvos allows you to use external EMR for relational multidimensional queries. You can also configure scaling for this external EMR. 

For this, you need to create a Lambda role. This section provides the steps and procedure for creating the Lambda role.

Note

The scaling configuration for external EMR is supported for Kyvos versions 2021.1.1 and above.

To deploy the Kyvos using the default CloudFormation template, perform the following steps.

  1. Log in to your AWS Console, with the user having sufficient rights as mentioned in the prerequisites.

  2. From the top-left of the screen, click the Create Stack  >  With New Resources (Standard)  option.

Caution

Please provide the information carefully, as you cannot edit the stack after creation.

Step 1: Specify template

  1. In the Prerequisite - Prepare template, select the Template is ready option.

  2. In the Specify template,  select the Upload a template file option to upload your CloudFormation template using the Choose File button.

  3. Click NEXT.

Step 2: Specify stack details

  1. Enter a Stack name. Stack name can contain letters (A-Z and a-z), numbers (0-9), and dashes (-). The Kyvos cluster will be deployed in this name, and you cannot edit the name after deployment.

  2. The Parameters area displays the configurations defined in your template. It also allows you to enter custom values to create or update a stack.

  3. Enter details as explained in the following table, and then click NEXT.

Configuration

Parameter

Description

Configuration

Parameter

Description

EMR Detail

Enter the Cluster ID of the EMR node

Provide the cluster-ID of your external EMR node.

EMR Scale-Out Schedule

Enter the number of core nodes that you want

The number of core nodes that will keep running during the scale-out event.

Enter the time in hours when the EMR cluster should be scaled-up

Enter the time in UTC 24 hour format.

Enter the time in minutes when the EMR cluster should be scaled-up

Enter the time in UTC 24 hour format.

EMR Scale-In Schedule

Enter the number of core nodes that you want

The number of core nodes that will keep running during the scale-in event.

Enter the time in hours when the EMR cluster should be scaled-down

Enter the time in UTC 24 hour format.

Enter the time in minutes when the EMR cluster should be scaled-down

Enter the time in UTC 24 hour format.

Other parameters

LambdaIamRole

To use an existing Lambda role, provide the name here.

If blank, the role will be created automatically.

Step 3: Configure stack options

  1. Tags: You can specify tags (key-value pairs) to apply to resources in your stack. You can add up to 50 unique tags for each stack.

Caution

If deploying in a secure environment with Ranger deployed on AWS, then you MUST provide the tag as UsedBy=Kyvos

DO NOT use the UsedBy tag for any other deployments.

  1. Permissions: Choose an IAM role to explicitly define how CloudFormation can create, modify, or delete resources in the stack. If you don't choose a role, CloudFormation uses permissions based on your user credentials.

  2. Advanced options: Optionally, you can set additional options for your stack, like notification options and a stack policy. 
    Enter details as:

Parameter

Description/Remarks

Parameter

Description/Remarks

Stack policy

Defines the resources that you want to protect from unintentional updates during a stack update. Kyvos does not allow any updates in the stack, so you can leave this blank.

Rollback configuration

Specify alarms for CloudFormation to monitor when creating and updating the stack. If the operation breaches an alarm threshold, CloudFormation rolls it back.

  • Monitoring time (optional): The number of minutes after the operation completes that CloudFormation should continue monitoring the specified alarms.

  • CloudWatch alarm (optional): Amazon Resource Name (ARN) of the alarm to monitor.

Notification options

Specify SNS topic ARN (optional).

Stack creation options

  • Rollback on failure: Specifies whether the stack should be rolled back if stack creation fails.

  • Timeout: The number of minutes before a stack creation times out.

  • Termination protection: Prevents the stack from being accidentally deleted. Once created, you can update this through stack actions.

Step 4: Review

Review the settings selected so far. Click Previous if you need to change any configurations.

Caution

Review the settings and information thoroughly. Once the stack is created, you CANNOT update any information. You will have to delete the stack and start over again.

Step 5: Create Stack

To proceed, select the I acknowledge checkbox and click Create Stack to create and deploy the Kyvos cluster on your AWS environment.

Once the stack is created, you will see the status in the info as shown in the picture below.

 

Copyright Kyvos, Inc. 2026. All rights reserved.