Deploying Kyvos Resources for GCP using Terraform Script

Applies to: Kyvos Enterprise Kyvos Cloud (SaaS on AWS) Kyvos AWS Marketplace

Kyvos Azure Marketplace Kyvos GCP Marketplace Kyvos Single Node Installation (Kyvos SNI)

Before you begin

In addition to the prerequisites, please ensure the following settings are enabled on your GCP project.

Project Billing: For this, search Billing on your Google Cloud project.
1. Click Link a Billing Account, and configure the billing information.
  Once your billing is enabled, you will see an estimate as shown in the following figure.
Compute Engine APIs: Search for Compute Engine APIs on your project, and click the Enable button.
Once the API is enabled, the corresponding status is displayed, as shown in the following figure.
Cloud Key Management Service (KMS) API: Enable this API, which extends customer control over encryption keys.
Cloud Resource Manager API: Search for Cloud Resource Manager API on your project, and click the Enable button.
Once the API is enabled, the API Enabled status is displayed, as shown in the following figure.
Enable the following APIs on your project. Refer to the GCP documentation for details.
1. Cloud Functions
2. Cloud Build
3. Cloud Scheduler
Create an App Engine project and select the region where you want to deploy your resources.
To the default Google APIs Service Agent service account, add the storage.buckets.get role. This is required to delete deployment through the Deployment Manager.
Kubernetes Engine API: Search for Kubernetes Engine API on your project and click the Enable button. Once the API is enabled, the API Enabled status is displayed.

Creating resources using script

Download the terraform.tar (available in GCP Installation Files) file on your workstation.
On your workstation, install the gcloud command-line tool.
To execute Terraform on Google Cloud Platform's Cloud Shell, activate Cloud Shell.
Configure the gcloud command-line tool to use your project using the following command.
gcloud config set project [MY_PROJECT]
Here, replace [MY_PROJECT] with your project ID.

Note

After opening the terminal in Cloud Shell, ensure that Cloud Shell is configured to operate within the same project where you intend to deploy your resources.

Copy terrform.tar file and untar it. The following subdirectories and files are displayed.
Access the kyvosparams.tfvars file located in the conf directory, and configure the parameters as needed for your deployment
Enter details for Kyvos resources with Kubernetes and Dataproc in the kyvosparams.tfvars file:

Parameter	Description

Parameter	Description
projectId	Enter the project Id. NOTE: This project Id will be used for the Kyvos deployment.
region	Enter the region to deploy Kyvos resources.
existingVpcName	Enter name of existing VPC. NOTE: If this field is left blank, a new VPC will be created.
existingvpcProjectId	Enter the project ID of the existing VPC.
existingSubnetworkName	Enter name of existing VPC subnet.
createNetworkFirewall	To create firewall rules, set the value of this parameter to true. NOTE: If the value of createVPC is set to true, firewall rules will be created unconditionally.
gkeSubnetName	Enter the name of an existing Subnet in which you want to deploy GKE Cluster. If left blank subnetwork name will be used.
secondaryRangeName1	Enter the Secondary IPv4 ranges name for GKE Cluster creation. NOTE: This must be preconfigured if using an existing VPC. The range should have a minimum masking of /22. For more information, see Google documentation.
secondaryRangeName2	Enter the Secondary IPv4 ranges name for GKE Cluster creation. NOTE: This must be preconfigured if using an existing VPC. The range should have a minimum masking of /22. For more information, see Google documentation.
secondaryRange1	Enter the Secondary IPv4 ranges for GKE Cluster creation in the case of existing VPC.
secondaryRange2	Enter the Secondary IPv4 ranges for GKE Cluster creation in the case of existing VPC.
customPrefixNameVPC	If the value of the existing VPC name is left blank, enter the prefix name to be used for VPC to be created.
customPrefixNameSubNetwork	If the value of the existing subnetwork name is left blank, enter the prefix name to be used for VPC to be created.
ipCidrRange	Enter the value of ipCidrRange if value of existing VPC name is left blank.
vpcConnectorName	Enter the name of the VPC Connector to be used with GCP functions.
customImage	Set the value to true to use custom image for deployment.
customSourceImage	Enter the source of customSourceImage in format “project/project-name/global/images/image-name”
multiAzDeployment	Set to true for a multi-Availability Zone deployment.
kmCount	The number of Kyvos Manager instances to be launched.
kmInstanceType	Instance type of Kyvos Manager (n4-standard-4 and n2-standard-4).
kmVolumeSizeGB	Size of the disk to be attached to the Kyvos Manager.
kmVolumeType	Type of the disk for KM (pd-ssd). Note: The hyperdisk-balanced disk type is supported for n4 instance type
hostNameBasedDeployment	Change the value to true to use the hostname for the cluster deployment.
createLoadBalancer	Set the value as true to create load balancer. By default, the value is set as false.
enableWebServerHA	Set the value as true for enabling webserver and Kyvos manager High Availability By default, the value is set as false.
webServerInstanceType	Configure the Web Server instance type.
webServerVolumeSizeGB	Size of the disk to be attached for the Web Server.
webServerVolumeType	Type of the disk for the Web Server (pd-ssd). Note: The hyperdisk-balanced disk type is supported for n4 instance type
qeCount	The number of instances to be used as query engines.
qeInstanceType	Instance type of query engine (n4-highmem-4 and n2-highmem-4).
qeDataVolumeSizeGB	Size of the disk to be attached with query engines.
qeCacheVolumeSizeGB	Size of the disk to be attached for the cache.
qeCacheVolumeCount	The number of disks to be attached to the cache.
qeCacheVolumeType	Type of the disk for the Web Server (pd-ssd). Note: The hyperdisk-balanced disk type is supported for n4 instance type
biCount	Enter the number of instances to be used as the BI server.
biInstanceType	Instance type of BI Server (n4-standard-8 and n2-standard-8).
biVolumeCount	The number of disks to be attached to the BI Server.
biVolumeSizeGB	Size of the disk to be attached to the BI Server.
biVolumeType	Type of the disk for the Web Server (pd-ssd). Note: The hyperdisk-balanced disk type is supported for n4 instance type
createGcpFunctions	Set the value as true to configure GCP Functions in Kyvos.
dataprocMetastoreURI	Enter the Metastore URI if you want to deploy Kyvos with no Spark configuration.
createGKE	Enter the value as True or False. True: To create Kubernetes cluster.
gkeWorkerInstancetype	Enter the worker node instance type for the Kubernetes cluster. NOTE: n2-standard-16 is the minimum configuration. Instance type smaller than this aren't supported.
existingGkeClusterName	Enter the name of existing GKE cluster. Note: Set the value of createGKE' parameter to ‘false’
existingNodePoolName	Enter the node pool name of existing GKEcluster.
sharedK8sNodePool	Select the value as true to use shared K8s node pool.
existingGKERange	Enter the secondary IP range used in the existing GKE cluster if the VPC used by the GKE cluster different from the one used for the deployment.
existingGKEserviceAccountName	Enter the name of service account used in the existing GKE cluster if the service account used by the GKE cluster differs from the one used for the deployment.
minPodCount	Enter minimum pod count. NOTE: If the values of MinPodCount and MaxWorkerNodeCount are the same, then scaling will be disabled. The value of MinPodCount parameter cannot be greater than the value of MaxWorkerNodeCount parameter. If the value of minPodCount is less than maxWorkerNodeCount, then scaling will be enabled.
maxWorkerNodeCount	Enter the maximum worker node count.
kyvosComputeWorkerNamespace	Enter the name of the Kyvos Compute Worker namespace.
minPodCountExistingNodePool	Enter minimum pod count of existing GKEcluster.
maxWorkerNodeCountExistingNodePool	Enter the maximum worker node count of the existing cluster.
createDataProc	Enter the value as True or False. True: If you want to deploy Kyvos with Spark Configuration.
enableComponentGateway	Set the value of ‘enableComponentGateway’ to True to get publicly accessible URL for Dataproc.
sharedDataprocCluster	Select true to use the shared Dataproc cluster. In this case, Kyvos will not manage the Dataproc cluster.Select false to use the on-demand Dataproc cluster. In this case, the Dataproc cluster will automatically start or stop.
dataProcNetworkTags	Provide a list of comma-separated network tags to be added to the Dataproc cluster. Example: dataProcNetworkTags : ["abc","xyz"]
enableSshFlag	Set the value to true to enable SSH to the Dataproc cluster.
enableLivy	Set the value of Livy to True if using Dataproc version 2.2.41-debian12
masterInstanceCount	The number of master nodes. For example, 1
masterInstanceType	Instance type of master node (n2-highmem-4)
masterInstanceVolumeType	Type of the disk for master node (pd-ssd)
workerInstanceCount	The number of worker instances.
workerInstanceType	Instance type of worker node (n2-highmem-8)
workerInstanceVolumeType	Type of the disk for worker node (pd-ssd)
enableDataProcMetastore	Set the value as true to allow external Dataproc metastore. NOTE: Existing metastore is not supported if the value of the createVPC is true
dataProcMetastoreProjectId	If enable DataProcMetastore is set as true, provide the name of the metastore project ID.
dataProcMetaStoreName	Provide the name of the metastore name.
dataProcVersion	Kyvos is certified with 2.2.41-debian12
enableAutoScaling	Set the value as true to enable the autoscaling of cluster nodes.
existingAutoScalingPolicyName	Provide the name of the existing autoscaling property, if any. NOTE: Use this configuration only if enableAutoScaling is set as true.
secondaryWorkerMinInstanceCount	Specify the number of minimum worker instances to be kept running while scaling. NOTE: Use this configuration only if enableAutoScaling is set as true.
secondaryWorkerMaxInstanceCount	Specify the number of maximum worker instances to be kept running while scaling. NOTE: Use this configuration only if enableAutoScaling is set as true.
existingDataprocClusterName	Enter the name of the existing Dataproc cluster. NOTE: Use these Configurations if you want to use the existing Dataproc and set the value of the parameter (createDataProc) to false.
sshPrivateKeyDataproc	The private key of existing Dataproc. NOTE: The private key should be base64 encoded
dataprocUsername	Name of the user.
dedicatedCompute	Set the value of a dedicated Compute cluster to true, if you want to create a new instance group. The default value is set to false.
dedicatedComputeMinInstanceCount	By default, the minimum instance group for scaling dedicated compute server is set to 1. You can change the value as needed.
dedicatedComputeMaxInstanceCount	By default, the maximum instance group for scaling dedicated compute server is set to 5. You can change the value as needed.
dedicatedComputeInstanceGroupVmType	Specify the required configuration. However, the ‘n2-standard-16’ and ‘n4-standard-16’ is the minimum configuration.
dedicatedComputeInstanceVolumeSize	Enter the dedicated compute instance volume size.
dedicatedComputeInstanceVolumeType	Change the value of below parameter to use a different disk type. For more information regarding disk types, visit Google documentation.
createCloudSql	By default, the value is set to ‘false’. To create CloudSQL, set the value to ‘true'. NOTE: The CloudSQL is created only with the PostgreSQL 16.9.
cloudSqlInstanceType	Enter the cloudSQL instance type. The recommended instance type is “db-custom-2-4096”
kyvosManagerDbPassword	Enter the Kyvos Manager database password. This parameter is mandatory if the ‘createCloudSql’ parameter is set to true.
kyvosDbPassword	Enter the Kyvos database password. This parameter is mandatory if the ‘createCloudSql’ parameter is set to true.
createServiceAccount	Change the value to false if you want to use the existing Service Account.
serviceAccountName	Enter the service account name to be attached to all Kyvos Virtual Machines.
enableNodeScaling	Set the value to false to disable the addition of permission required for Node Scaling on Service Account.
createSecretManager	Set value of the parameter to true to create a new secret manager for Kyvos. The Secret Manager will be explicitly created only when createCloudSql is set to true.
secretManagerName	Provide the name of the existing Secret Manager.
secretManagerProjectId	Provide the name of the Project ID in which the existing secret manager is created.
customPrefixNameDataproc	A prefix is to be added before the name of Dataproc.
customPrefixNameGKE	A prefix is to be added before GKE.
customPrefixNameBI	A prefix is to be added before the name of BI virtual machines.
customPrefixNameBIDisk	A prefix is to be added before the name of BI Disks.
customPrefixNameQE	A prefix is to be added before the name of Query Engine virtual machines.
customPrefixNameQEDisk	A prefix is to be added before the name of Query Engine disks.
customPrefixNameKM	A prefix is to be added before the name of the Kyvos Manager virtual machine.
customPrefixNameKMDataDisk	A prefix is to be added before the name of the Kyvos Manager disk.