Deploying Kyvos Resources for GCP using Terraform Script

Applies to: Kyvos Enterprise Kyvos Cloud (SaaS on AWS) Kyvos AWS Marketplace

Kyvos Azure Marketplace Kyvos GCP Marketplace Kyvos Single Node Installation (Kyvos SNI)

Before you begin

In addition to the prerequisites, please ensure the following settings are enabled on your GCP project.

Project Billing: For this, search Billing on your Google Cloud project.
1. Click Link a Billing Account, and configure the billing information.
  Once your billing is enabled, you will see an estimate as shown in the following figure.
Compute Engine APIs: Search for Compute Engine APIs on your project, and click the Enable button.
Once the API is enabled, the corresponding status is displayed, as shown in the following figure.
Cloud Key Management Service (KMS) API: Enable this API, which extends customer control over encryption keys.
Cloud Resource Manager API: Search for Cloud Resource Manager API on your project, and click the Enable button.
Once the API is enabled, the API Enabled status is displayed, as shown in the following figure.
Enable the following APIs on your project. Refer to the GCP documentation for details.
1. Cloud Functions
2. Cloud Build
3. Cloud Scheduler
Create an App Engine project and select the region where you want to deploy your resources.
To the default Google APIs Service Agent service account, add the storage.buckets.get role. This is required to delete deployment through the Deployment Manager.
Kubernetes Engine API: Search for Kubernetes Engine API on your project and click the Enable button. Once the API is enabled, the API Enabled status is displayed.

Creating resources using script

Download the terraform.tar (available in GCP Installation Files) file on your workstation.
On your workstation, install the gcloud command-line tool.
To execute Terraform on Google Cloud Platform's Cloud Shell, activate Cloud Shell.
Configure the gcloud command-line tool to use your project using the following command.
gcloud config set project [MY_PROJECT]
Here, replace [MY_PROJECT] with your project ID.

Note

After opening the terminal in Cloud Shell, ensure that Cloud Shell is configured to operate within the same project where you intend to deploy your resources.

Copy terrform.tar file and untar it. The following subdirectories and files are displayed.
Access the kyvosparams.tfvars file located in the conf directory, and configure the parameters as needed for your deployment
Enter details for Kyvos resources with Kubernetes and Dataproc in the kyvosparams.tfvars file:

Parameter	Description

Parameter	Description
projectId	Enter the project Id. NOTE: This project Id will be used for the Kyvos deployment.
region	Enter the region to deploy Kyvos resources.
enableDr	Set the value as true or false. true: Set this value to enable disaster recovery. false: Set this value to disable disaster recovery.
drRegion	Specify the region so that the replica should create in the disaster recovery region. Note: To ensure the selected region and disaster region form a valid dual-region pair, see the Google documentation.
recoverkyvoscluster	Setting the value to true will enable complete disaster recovery as secondary resources will create in the specified disaster recovery region.
existingVpcName	Enter name of existing VPC. NOTE: If this field is left blank, a new VPC will be created.
existingvpcProjectId	Enter the project ID of the existing VPC.
existingSubnetworkName	Enter name of existing VPC subnet.
existingSubnetworkDr	Enter the name of the existing subnetwork that must reside in the drRegion. Note: This is applicable only when the value of the enableDr is set to true.
createNetworkFirewall	To create firewall rules, set the value of this parameter to true. NOTE: If the value of createVPC is set to true, firewall rules will be created unconditionally. Firewall rules cannot be created in cross-project VPC
gkeSubnetName	Enter the name of an existing Subnet in which you want to deploy GKE Cluster. If left blank subnetwork name will be used.
secondaryRangeName1	Enter the Secondary IPv4 ranges name for GKE Cluster creation. NOTE: This must be preconfigured if using an existing VPC. The range should have a minimum masking of /22. For more information, see Google documentation.
secondaryRangeName2	Enter the Secondary IPv4 ranges name for GKE Cluster creation. NOTE: This must be preconfigured if using an existing VPC. The range should have a minimum masking of /22. For more information, see Google documentation.
secondaryRange1	Enter the Secondary IPv4 ranges for GKE Cluster creation in the case of existing VPC.
secondaryRange2	Enter the Secondary IPv4 ranges for GKE Cluster creation in the case of existing VPC.
customPrefixNameVPC	If the value of the existing VPC name is left blank, enter the prefix name to be used for VPC to be created.
customPrefixNameSubNetwork	If the value of the existing subnetwork name is left blank, enter the prefix name to be used for VPC to be created.
ipCidrRange	Enter the value of ipCidrRange if value of existing VPC name is left blank.
ipCidrRangeReplicate	Enter the value of ipCidrRange. Note: This is applicable if the value of ‘existingVpcName’ is left blank and ‘enableDr’ is set to true.
vpcConnectorName	Enter the name of the VPC Connector to be used with GCP functions.
customImage	Set the value to true to use custom image for deployment.
customSourceImage	Enter the source of customSourceImage in format “project/project-name/global/images/image-name”
multiAzDeployment	Set to true for a multi-Availability Zone deployment.
kmCount	The number of Kyvos Manager instances to be launched.
kmInstanceType	Instance type of Kyvos Manager (n4-standard-4 and n2-standard-4).
kmVolumeSizeGB	Size of the disk to be attached to the Kyvos Manager.
kmVolumeType	Type of the disk for KM (pd-ssd). Note: The hyperdisk-balanced disk type is supported for n4 instance type
hostNameBasedDeployment	Change the value to true to use the hostname for the cluster deployment.
createLoadBalancer	Set the value as true to create load balancer. By default, the value is set as false.
enableWebServerHA	Set the value as true for enabling webserver and Kyvos manager High Availability By default, the value is set as false.
webServerInstanceType	Configure the Web Server instance type.
webServerVolumeSizeGB	Size of the disk to be attached for the Web Server.
webServerVolumeType	Type of the disk for the Web Server (pd-ssd). Note: The hyperdisk-balanced disk type is supported for n4 instance type
qeCount	The number of instances to be used as query engines.
qeInstanceType	Instance type of query engine (n4-highmem-4 and n2-highmem-4).
qeDataVolumeSizeGB	Size of the disk to be attached with query engines.
qeCacheVolumeSizeGB	Size of the disk to be attached for the cache.
qeCacheVolumeCount	The number of disks to be attached to the cache.
qeCacheVolumeType	Type of the disk for the Web Server (pd-ssd). Note: The hyperdisk-balanced disk type is supported for n4 instance type
biCount	Enter the number of instances to be used as the BI server.
biInstanceType	Instance type of BI Server (n4-standard-8 and n2-standard-8).
biVolumeCount	The number of disks to be attached to the BI Server.
biVolumeSizeGB	Size of the disk to be attached to the BI Server.
biVolumeType	Type of the disk for the Web Server (pd-ssd). Note: The hyperdisk-balanced disk type is supported for n4 instance type
createGcpFunctions	Set the value as true to configure GCP Functions in Kyvos.
dataprocMetastoreURI	Enter the Metastore URI if you want to deploy Kyvos with no Spark configuration.
createGKE	Enter the value as True or False. True: To create Kubernetes cluster.
gkeWorkerInstancetype	Enter the worker node instance type for the Kubernetes cluster. NOTE: n2-standard-16 is the minimum configuration. Instance type smaller than this aren't supported.
existingGkeClusterName	Enter the name of existing GKE cluster. Note: Set the value of createGKE' parameter to ‘false’
existingNodePoolName	Enter the node pool name of existing GKEcluster.
sharedK8sNodePool	Select the value as true to use shared K8s node pool.
existingGKERange	Enter the secondary IP range used in the existing GKE cluster if the VPC used by the GKE cluster different from the one used for the deployment.
existingGKEserviceAccountName	Enter the name of service account used in the existing GKE cluster if the service account used by the GKE cluster differs from the one used for the deployment.
minPodCount	Enter minimum pod count. NOTE: If the values of MinPodCount and MaxWorkerNodeCount are the same, then scaling will be disabled. The value of MinPodCount parameter cannot be greater than the value of MaxWorkerNodeCount parameter. If the value of minPodCount is less than maxWorkerNodeCount, then scaling will be enabled.
maxWorkerNodeCount	Enter the maximum worker node count.
kyvosComputeWorkerNamespace	Enter the name of the Kyvos Compute Worker namespace.
minPodCountExistingNodePool	Enter minimum pod count of existing GKEcluster.
maxWorkerNodeCountExistingNodePool	Enter the maximum worker node count of the existing cluster.
createDataProc	Enter the value as True or False. True: If you want to deploy Kyvos with Spark Configuration.
enableComponentGateway	Set the value of ‘enableComponentGateway’ to True to get publicly accessible URL for Dataproc.
sharedDataprocCluster	Select true to use the shared Dataproc cluster. In this case, Kyvos will not manage the Dataproc cluster.Select false to use the on-demand Dataproc cluster. In this case, the Dataproc cluster will automatically start or stop.
dataProcNetworkTags	Provide a list of comma-separated network tags to be added to the Dataproc cluster. Example: dataProcNetworkTags : ["abc","xyz"]
enableSshFlag	Set the value to true to enable SSH to the Dataproc cluster.
enableLivy	Set the value of Livy to True if using Dataproc version 2.2.41-debian12
masterInstanceCount	The number of master nodes. For example, 1
masterInstanceType	Instance type of master node (n2-highmem-4)
masterInstanceVolumeType	Type of the disk for master node (pd-ssd)
workerInstanceCount	The number of worker instances.
workerInstanceType	Instance type of worker node (n2-highmem-8)
workerInstanceVolumeType	Type of the disk for worker node (pd-ssd)
enableDataProcMetastore	Set the value as true to allow external Dataproc metastore. NOTE: Existing metastore is not supported if the value of the createVPC is true
dataProcMetastoreProjectId	If enable DataProcMetastore is set as true, provide the name of the metastore project ID.
dataProcMetaStoreName	Provide the name of the metastore name.
dataProcVersion	Kyvos is certified with 2.2.41-debian12
enableAutoScaling	Set the value as true to enable the autoscaling of cluster nodes.
existingAutoScalingPolicyName	Provide the name of the existing autoscaling property, if any. NOTE: Use this configuration only if enableAutoScaling is set as true.
secondaryWorkerMinInstanceCount	Specify the number of minimum worker instances to be kept running while scaling. NOTE: Use this configuration only if enableAutoScaling is set as true.
secondaryWorkerMaxInstanceCount	Specify the number of maximum worker instances to be kept running while scaling. NOTE: Use this configuration only if enableAutoScaling is set as true.
existingDataprocClusterName	Enter the name of the existing Dataproc cluster. NOTE: Use these Configurations if you want to use the existing Dataproc and set the value of the parameter (createDataProc) to false.
sshPrivateKeyDataproc	The private key of existing Dataproc. NOTE: The private key should be base64 encoded
dataprocUsername	Name of the user.
dedicatedCompute	Set the value of a dedicated Compute cluster to true, if you want to create a new instance group. The default value is set to false.
dedicatedComputeMinInstanceCount	By default, the minimum instance group for scaling dedicated compute server is set to 1. You can change the value as needed.
dedicatedComputeMaxInstanceCount	By default, the maximum instance group for scaling dedicated compute server is set to 5. You can change the value as needed.
dedicatedComputeInstanceGroupVmType	Specify the required configuration. However, the ‘n2-standard-16’ and ‘n4-standard-16’ is the minimum configuration.
dedicatedComputeInstanceVolumeSize	Enter the dedicated compute instance volume size.
dedicatedComputeInstanceVolumeType	Change the value of below parameter to use a different disk type. For more information regarding disk types, visit Google documentation.
createCloudSql	By default, the value is set to ‘false’. To create CloudSQL, set the value to ‘true'. NOTE: The CloudSQL is created only with the PostgreSQL 16.9.
cloudSqlInstanceType	Enter the cloudSQL instance type. The recommended instance type is “db-custom-2-4096”
existingKyvosCloudSqlName	Enter existing Kyvos cloud SQL name. Note: Use these configuration if you want to use existingCloudSql.
existingKyvosManagerCloudSqlName	Enter existing Kyvos Manager cloud SQL name. Note: Use these configuration if you want to use existingCloudSql.
sharedRepo	Set the value as true or false. The default value of the shared repository is true. This indicates that the repository is used in multiple applications (two Kyvos application cannot share the same repository instance) If the value of the sharedrepo is set to false, it becomes a dedicated repository and associated to a Kyvos application.
kyvosManagerDbPassword	Enter the Kyvos Manager database password. This parameter is mandatory if the ‘createCloudSql’ parameter is set to true.
kyvosDbPassword	Enter the Kyvos database password. This parameter is mandatory if the ‘createCloudSql’ parameter is set to true.
createServiceAccount