Prerequisites for Azure
Before performing the prerequisite steps in this section, perform the following.
All the other disaster resources should be created in the DR region.
Resource Groups and Virtual networks are required in the DR region.
Read replica of the Flexible server that should be created in the DR region.
The service principle attached to the Databricks cluster must have storage blob data contributor permission on the Primary’s storage container.
To create a read replica of the Postgres Flexible Server, an existing Virtual Network peered with the Primary region’s Virtual Network is required in the DR region.
A subnet with delegation to flexible Servers in the Virtual network of the DR region is required.
Step 1: Replica Resources creation in the other region
Kyvos Manager and Kyvos External Repository (Azure Postgres Flexible Server)
Create Read Replicas for Kyvos Manager and Kyvos: Cross-region read replicas can be deployed to protect your databases from region-level failures. Read replicas are updated asynchronously using PostgreSQL physical replication technology. Follow Microsoft documentation to configure Replication in an existing Postgres Flexible Server.
Promote replicas to an independent server: In case of a disaster, the user must promote a read replica to a Standalone Postgres using the Azure document.
Step 2: Replication configuration for syncing data from primary to secondary
Storage Account
Migrate Redundancy: To configure DR for the Storage Account, the user has to migrate the Redundancy option of the Storage Account. (Follow the Steps to change the Storage Redundancy). After the Redundancy option has been migrated to RA-GRS, a Secondary Storage Blob Service Endpoint will be available where data is continuously Replicated.
The following image depicts the Primary region as Central US and the DR region as East US 2.
Note
Azure provides a failover mechanism where the Storage Account shifts to the Secondary region in case of a disaster. Still, this feature is currently unavailable for the Storage Accounts with Hierarchical namespace enabled, and it's a prerequisite in Kyvos to use a Storage Account with Hierarchical namespace enabled.
Key Vault
Key Vault automatically manages Disaster recovery: If you're in a region that automatically replicates your key vault to a secondary region, then in the rare event that an entire Azure region is unavailable, your requests of Azure Key Vault in that region are automatically routed (failed over) to a secondary region. When the primary region is available again, requests are routed back (failed back) to the primary region. Again, you don't need to take any action because this happens automatically. See Microsoft documentation to know more about failover across regions.
Note
If Networking is enabled on the Key Vault used in the DR deployment, allow the subnets of Kyvos instances & Functions in the Networking. For more details, see Microsoft documentation to configure Azure Storage firewalls and virtual networks.
Only follow this step if the Primary region’s template was created with Enhanced security.
Step 3: Replicated resource information configuration in Kyvos Manager
To save replication information on primary cluster for disaster recovery, click here.
Additional Information for Disaster Recovery.
If DR happens, then you cannot move to the Original installation. The DR cluster will be the Primary cluster.
If you have configured additional settings for the primary cluster, in this case you need to perform the following settings on the secondary cluster as Primary cluster.
If the primary deployment was on a private network (tunneling established between Customer and Kyvos AZURE VNET), you must repeat the same procedure after DR deployment.
Once the deployment is complete, you must wait for Cuboid replication on all the query engines to execute queries.
If any additional IPs were allowed in the Security group of primary installation, you MUST configure the same in the DR Security Group, too.
Once the DR deployment is complete, you must create the custom URL and DNS mapping again.
Storage Account Template (optional)
{
"$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"StorageAccountName": {
"type": "string",
"defaultValue": "drbucket",
"metadata": {
"description": "Name of Storage Account to be used."
}
},
"StorageAccountContainerName": {
"type": "string",
"defaultValue": "kyvoscontainer",
"metadata": {
"description": "Name of Container in Storage Account."
}
},
"MultiAzStorageAccount": {
"type": "bool",
"defaultValue": false,
"allowedValues": [ true, false ],
"metadata": {
"description": "Select True to Create New Managed Identity for kyvos."
}
},
"AdditionalTags": {
"type": "object",
"metadata": {
"description": "Additional tags to put on all resources. Syntax: {\"Key1\": \"Value1\", \"Key2\" : \"Value2\"}"
},
"defaultValue": {
"UsedBy": "Kyvos"
}
}
},
"variables": {
"TagMap": {
"LayerTag": {
"WebServer": "Kyvos_WebPortal",
"OlapEngine": "Service",
"QueryEngine": "Query",
"StorageAccount": "Persistent_Storage",
"KyvosManager": "KM_Service",
"Function": "Scale_Layer",
"Vault": "Secrets",
"ManagedIdentity": "Authentication",
"AzurePostgresServer": "Metadata_Storage",
"LogsStorageAccount": "Logs_Storage",
"CreditInfoPostgres": "CreditInfo_Metadata_Storage",
"CreditInfoKeyVault": "CreditInfo_Secrets_Storage",
"Vnet": "Networking",
"LogWorkspace": "Logging",
"PrivateEndpoint": "Connection"
},
"RoleTag": {
"WebServer": "WP_CLUSTER",
"OlapEngine": "BI_CLUSTER",
"QueryEngine": "QE_CLUSTER",
"StorageAccount": "STORAGE",
"KyvosManager": "KM",
"Function": "KYVOS_FUNCTION",
"Vault": "SECRETS_MANAGER",
"ManagedIdentity": "RESOURCES_ACCESS",
"AzurePostgresServer": "DATABASE",
"AzurePostgresServerKmRepo": "DATABASE_KM",
"LogsStorageAccount": "LOGS_DATA",
"CreditInfoPostgres": "CREDITINFO_DATABASE",
"CreditInfoKeyVault": "CREDITINFO_PASSWORDS",
"Vnet": "NETWORK",
"LogWorkspace": "LOGGING",
"PrivateEndpoint": "CONNECTION"
}
}
},
"resources": [
{
"type": "Microsoft.Storage/storageAccounts",
"apiVersion": "2022-09-01",
"name": "[parameters('StorageAccountName')]",
"location": "[resourceGroup().location]",
"sku": {
"name": "[if(parameters('MultiAzStorageAccount'), 'Standard_ZRS', 'Standard_LRS')]",
"tier": "Standard"
},
"tags": "[union(parameters('AdditionalTags'),json(concat('{\"CLUSTER_ID\": \"kyvos-', deployment().name, '\" , \"CreatedBy\": \"Kyvos\", \"Name\": \"kyvos-storage-', deployment().name, '\" , \"ROLE\": \"', variables('TagMap').RoleTag.StorageAccount, '\" , \"LAYER\": \"', variables('TagMap').LayerTag.StorageAccount, '\"')))]",
"kind": "StorageV2",
"properties": {
"largeFileSharesState": "Disabled",
"isHnsEnabled": true,
"networkAcls": {
"bypass": "AzureServices",
"virtualNetworkRules": [],
"ipRules": [],
"defaultAction": "Allow"
},
"supportsHttpsTrafficOnly": true,
"encryption": {
"services": {
"file": {
"keyType": "Account",
"enabled": true
},
"blob": {
"keyType": "Account",
"enabled": true
}
},
"keySource": "Microsoft.Storage"
},
"minimumTlsVersion": "TLS1_2",
"accessTier": "Hot",
"allowBlobPublicAccess": false
}
},
{
"type": "Microsoft.Storage/storageAccounts/blobServices",
"apiVersion": "2022-09-01",
"name": "[concat(parameters('StorageAccountName'), '/default')]",
"dependsOn": [
"[resourceId('Microsoft.Storage/storageAccounts', parameters('StorageAccountName'))]"
],
"properties": {
"cors": {
"corsRules": []
},
"deleteRetentionPolicy": {
"enabled": false
}
},
"tags": "[union(parameters('AdditionalTags'),json('{}'))]"
},
{
"type": "Microsoft.Storage/storageAccounts/blobServices/containers",
"apiVersion": "2022-09-01",
"name": "[concat(parameters('StorageAccountName'), '/default/',parameters('StorageAccountContainerName'))]",
"dependsOn": [
"[resourceId('Microsoft.Storage/storageAccounts/blobServices', parameters('StorageAccountName'), 'default')]",
"[resourceId('Microsoft.Storage/storageAccounts', parameters('StorageAccountName'))]"
],
"properties": {
"publicAccess": "None"
},
"tags": "[union(parameters('AdditionalTags'),json('{}'))]"
}
],
"outputs": {
}
}