No-Spark Semantic model processing through Kyvos Native Compute Engines

This section details how to process semantic models using Kyvos Native Compute Engines without relying on Spark or external compute clusters, enabling cost-efficient and simplified semantic model processing.

Compute type platform for processing semantic models

You can choose Kyvos Native as a compute engine or External engines for semantic model processing.

	Compute Type
Platform	External	Kyvos Native
AWS	EMR / Databricks	SHARED QE or Dedicated Compute (K8S)
AZURE	Databricks	SHARED QE or Dedicated Compute (K8S)
GCP	Dataproc	SHARED QE or Dedicated Compute (K8S)
ON PREM	Hadoop	SHARED QE or Dedicated Compute

Semantic model processing through Kyvos Native Compute

Kyvos Native Compute operates independently of any external compute clusters when processing semantic models. It uses its proprietary Kyvos Analytical Store, which reduces costs, bolsters security, and removes the dependency on permissions.

Note

From Kyvos 2024.11 onwards, you can process a flexible semantic model allows the summary function of a dimension or measure to be modified at any time during SQL queries.
To do this,
- Set the value of the kyvos.process.compute.type property must be Kyvos Compute.
- Enable Raw Data Querying
When you process a semantic model with no Spark, cuboids are stored at persistent storage. However, a copy of these cuboids is kept at the local storage (local disk).

Shared Query Engine: In this mode, the query engine not only performs queries but also handles semantic model processing. This dual role is named SHARED because the same process undertakes both activities.

Note

If you use Query Engines as a compute server:

For Load Based Scaling: Query Engines will be automatically started when the semantic model is processed.
For Schedule based scaling: Query Engines will start automatically when the semantic model is processed. However, if you have enabled schedule-based scaling, the Query Engines will not start automatically. In such cases, Kyvos recommends switching to load-based scaling.

Dedicated Compute: In this mode, the semantic model is processed via dedicated service. In cloud-based deployment, the semantic model is processed using Kubernetes (K8S) cluster-based nodes while in ON PREM environment, models are processed on dedicated nodes.

You can change from an external compute cluster to Kyvos Native for processing semantic models through Kyvos Manager on the Compute Cluster page.

From Kyvos 2024.10 onwards, you can

Process semantic model with no-Spark using the Shared Query Engine and dedicated Kubernetes cluster on AWS Managed Services.
Resume failed or canceled semantic model process.
Run Process Test Data job on the semantic model.

Supported Platforms

	Supported Environments
	AWS			AZURE		GCP		ON PREM
Supported Native Types	Kyvos Enterprise	Marketplace	Managed Services	Enterprise	Marketplace	Enterprise	Marketplace
SHARED QE
Dedicated Compute (K8S)

AWS: For Kubernetes, Kyvos processes the semantic model using Amazon Elastic Kubernetes Service (Amazon EKS). You can select Query Engine or Kubernetes as a compute engine using no Spark model processing.
For further details about deployment, see the Automated deployment for AWS via CloudFormation with Kyvos Native section.
Azure: For Kubernetes, Kyvos processes the semantic model using Azure’s managed service AKS (Azure Kubernetes Service).
- The Azure cluster is deployed via ARM templates. You can create a cluster without Spark or process the semantic model using Spark mode within ARM templates.
  For further details about deployment, see the Automated deployment on Azure with Kyvos Native section.
- From Kyvos 2024.3 onwards, you can select the compute cluster as the Query engine or Kubernetes when deploying Kyvos through Azure Template Specs.
GCP: For Kubernetes, Kyvos processes the semantic model using Google Cloud's managed service GKE (Google Kubernetes Engine). The GKE cluster is deployed through GCP Installation Files. Using the scripts, you can select a No-Spark-based cluster or process the semantic model using Spark Mode.
Optionally, for no-Spark deployments, you can either use new or existing Dataproc cluster.
For further details about Kyvos deployment on GCP using the no-Spark model, see the following section:
- Prerequisites to deploy Kyvos using Kubernetes
- Creating Kyvos Resources for GCP using Script
On Premises: For On-premises deployment, you can deploy using No Spark types: SHARED_QE and Compute Server. For further details about on premises deployment with no-Spark, see the Deploying no Hadoop no Spark.

Important points to know

To process semantic model without spark, you must do the following:

To execute the sanity suite for no-spark and Kubernetes (K8S) supported deployments, refer to the Prerequisites for no-spark deployments section.
Modify the value of the following semantic model advanced properties on Kyvos Web Portal:
- kyvos.sqlwarehouse.catalog = ‘Your catalog name’
- kyvos.sqlwarehouse.tempdb = ‘Your Database name’
- kyvos.process.compute.type=KYVOS_COMPUTE
  This property is required only if the Default Compute Cluster Type is External at the application level
- kyvos.build.aggregate.type=TABULAR
  This property is not required if the semantic model is configured for Kyvos compute-based processing.
You can also set subtype to select no-Spark model processing for semantic model via property.
To do this, navigate to the Kyvos Properties page and update the KYVOS_PROCESS_COMPUTE_SUBTYPE property and restart Kyvos services.
For Kyvos Compute (no-Spark) deployments with enhanced security, when creating a Kyvos warehouse connection of a Databricks managed workspace, ensure that the firewall settings on the storage account are disabled.