You can configure the "Bring Your Own LLM Service" feature in Kyvos. This feature enables AI/BI developers to integrate their custom Large Language Models (LLMs) service into the Kyvos platform for Kyvos Dialogs to work.

This framework potentially enables Kyvos Dialogs to use any LLM service for Conversational Analytics and MDX based KPI Generation.

This document covers steps to upload, configure, and validate your LLM service integration code.

Prerequisites

Ensure you have the following before starting:

Kyvos Manager Access: Required for uploading and configuring files.
Custom LLM Implementation: A JAR file having code to integrate with custom LLM service for all the operations that Kyvos supports.
Metadata JSON: A JSON file having all the required details for LLM service configuration and properties.
API Keys and Credentials Required for connecting to your LLM service.

Configuring Java Code And Metadata for Kyvos-LLM Integration

Step 1: Implement the Callback interfaces
Step 2: Creating the metadata JSON file for connection
Step 3: Structure of the zip file

Step 1: Implement the Callback interfaces

GenAICallbackService Interface: To enable communication between Kyvos and your custom LLM service, you must implement the necessary methods defined in the GenAICallbackService interface.
The following methods need to be implemented:

public interface GenAICallbackService {

/**

This method is used to generate sample questions which will be displayed in copilot screen. It has arguments as user prompt which contains prompt and key elements which are marked on semantic model.
GenAIResponse contains a JSON array which has questions and its explanation.
@param userPrompt
@param genAIConnection
@return
@throws IOException
@throws InterruptedException
@throws GenAiServiceException

*/
GenAIResponse generateSampleQuestions(String userPrompt, GenAIConnection genAIConnection) throws IOException, InterruptedException, GenAiServiceException;

/**

This method is used to get list of semantic model ids having user prompt as semantic model details and NL text through which we get similar id's
@param userPrompt
@param genAIConnection
@return
@throws GenAiServiceException
@throws IOException

*/
GenAIResponse getSemanticModelsBasedOnNLText(String userPrompt, GenAIConnection genAIConnection) throws GenAiServiceException, IOException;

/**

This method is used to determine the type for natural language query, as Kyvos Copilot supports both business queries and command executions.
It helps determine the user's requested natural language query and decide how to execute it.
@param userPrompt
@param systemPrompt
@param genAIConnection
@return
@throws GenAiServiceException
@throws IOException

*/
GenAIResponse getNlQueryType(String userPrompt, String systemPrompt, GenAIConnection genAIConnection) throws GenAiServiceException, IOException;

/**

This method is used to execute commands like viz settings or other metadata commands supported (other then business queries) in Kyvos copilot.
@param userPrompt
@param systemPrompt
@param genAIConnection
@return
@throws GenAiServiceException
@throws IOException

*/
GenAIResponse executeCommand(String userPrompt, String systemPrompt, GenAIConnection genAIConnection) throws GenAiServiceException, IOException;

/**

This method is used to generate summary title based on spark SQL generated and result data after execution of SQL. It accepts user prompt as argument.
@param userPrompt
@param connection
@return
@throws GenAiServiceException
@throws IOException
@throws InterruptedException

*/
GenAIResponse generateSummaryTitle(String userPrompt, GenAIConnection connection) throws GenAiServiceException, IOException, InterruptedException;

/**

This method is used to generate summary and key insights based on spark SQL generated and result data after execution of SQL. It accepts user prompt and system prompt as argument.
@param userPrompt
@param systemPrompt
@param genAIConnection
@param jsonResponse
@return
@throws GenAiServiceException
@throws IOException

*/
GenAIResponse generateSummary(String userPrompt, String systemPrompt, GenAIConnection genAIConnection, boolean jsonResponse) throws GenAiServiceException, IOException;

/**

This method is used to get SQL from NL text passed by user. It takes user prompt and system prompt as arguments and return a JSON containing SQL.
@param userPrompt
@param systemPrompt
@param genAIConnection
@return
@throws GenAiServiceException
@throws IOException

*/
GenAIResponse getSQLFromNLText(String userPrompt, String systemPrompt, GenAIConnection genAIConnection) throws GenAiServiceException, IOException;

/**

This method is used to get SQL from NL text passed by use in case the first generated SQL fails it reattempts to get SQL from LLM. Here user gets sqlRetryJson as
an extra argument which contains last generated SQL and exception.
@param userPrompt
@param systemPrompt
@param genAIConnection
@param sqlRetryJson
@return
@throws GenAiServiceException
@throws IOException

*/
GenAIResponse getSQLFromRetry(String userPrompt, String systemPrompt, GenAIConnection genAIConnection, String sqlRetryJson) throws GenAiServiceException, IOException;
}

GenAICallbackService Interface: To integrate the custom embedding, you need to implement the GenAIEmbeddingCallbackService interface

public interface GenAIEmbeddingCallbackService {

/**

This core method is used to generate embeddings.
Implementation of this method should include below basic implementations:
1. input can be a text for whose embedding should be generated and returned in the form of text only.
2. input can be a List<String> for whose embeddings need to be generated for each column. A Map<String, Vector<Double>> has to be returned containing column names and embeddings.
3. input can be list of RAG records where it contains all information related to columns of a semantic model. User needs to return same List<RAGRecords> with adding a embedding against a new entry "embeddingColumn"
If any input is missed flow can be impacted.
@param input
@param embeddingColumn
@param genAIConnection
@param logResponse
@return
@throws IOException
@throws GenAiServiceException

     */
    Object processEmbeddings(Object input, String embeddingColumn,
                          GenAIConnection genAIConnection, boolean logResponse) throws IOException, GenAiServiceException;
}

Important

Both the interfaces should be implemented in a single class which will be required to configure as an endpoint on Kyvos Manager.
For example: public class BedrockCallbackServiceImpl implements GenAICallbackService, GenAIEmbeddingCallbackService { }

Step 2: Creating the metadata JSON file for connection

Create a metadata JSON file which holds details about all the configuration needed to integrate your custom LLM service with Kyvos. Using this file Kyvos Manager will create a connection UI specific to this LLM service.
The metadata file must use the following:

[
  {

      "params": [

       {
    "key": "endPoint",
    "fullyQualifiedName": "endPoint",
    "label": "URL",
    "inputType": "TEXTAREA",
    "dataType": "STRING",
    "defaultValue": "https://api.openai.com/",
    "validation": {
        "required": true
    },
    "description": {
        "Description": "The property provides the URL of the provider-specific endpoint for generating output."
    }
},
{
    "key": "authKey",
    "fullyQualifiedName": "authKey",
    "label": "Authentication Key",
    "inputType": "PASSWORD",
    "dataType": "STRING",
    "defaultValue": "",
    "validation": {
        "required": true
    },
    "description": {
        "Description": "The property specifies a unique key to authenticate and authorize requests to the provider's endpoint."
    }
},
{
    "key": "model",
    "fullyQualifiedName": "model",
    "label": "Model",
    "dataType": "STRING",
    "defaultValue": "gpt-4o-2024-08-06",
    "inputType": "TEXTBOX",
    "validation": {
        "required": true,
        "leadingWhiteSpace": true,
        "trailingWhiteSpace": true
    },
    "description": {
        "Description": "The property specifies the name of the GenAI Large Language Model (LLM) used to generate the output."
    }
},
{
    "key": "is_model_fine_tuned",
    "fullyQualifiedName": "is_model_fine_tuned",
    "label": "Is Model Fine Tuned",
    "inputType": "SELECTLIST",
    "dataType": "BOOLEAN",
    "defaultValue": false,
    "options": [
        {
            "display": "Yes",
            "value": true
        },
        {
            "display": "No",
            "value": false
        }
    ],
    "validation": {},
    "description": {
        "Description": "The property specifies whether the model used is fine tuned or not."
    }
},
{
    "key": "embeddingConnectionName",
    "fullyQualifiedName": "embeddingConnectionName",
    "label": "Embedding Connection",
    "inputType": "SELECTLIST",
    "applicableFor": "LLMProvider",
    "dataType": "STRING",
    "defaultValue": "",
    "visible": true,
    "validation": {
        "required": true
    },
    "description": {
        "Description": "The property specifies the name of the GenAI provider that the system will use to generate output."
    }
},
{
    "key": "applicableUsecases",
    "fullyQualifiedName": "applicableUsecases",
    "label": "Usage",
    "inputType": "MULTISELECTLIST",
    "dataType": "STRING",
    "applicableFor": "LLMProvider",
    "defaultValue": [
        "CONVERSATIONAL_AI"
    ],
    "options": {
        "CALCULATIONS": {
            "display": "MDX Generation",
            "value": "CALCULATIONS"
        },
        "CONVERSATIONAL_AI": {
            "display": "Conversational Analytics",
            "value": "CONVERSATIONAL_AI"
        }
    },
    "visible": true,
    "validation": {
        "required": true
    },
    "description": {
        "Description": "The property provides a list of use cases for which this connection can generate output."
    },
    "dependency": {
        "BASE": {
            "top_p": false,
            "frequency_penalty": false,
            "presence_penalty": false,
            "prompt_token_limit": false,
            "similarity_threshold": false,
            "allow_values_for_nlp_generation": false,
            "max_retry_count_for_sql_execution": false,
            "max_rows_summary": false,
            "generate_content": false,
            "is_model_fine_tuned": false
        },
        "CALCULATIONS": {
            "top_p": true,
            "frequency_penalty": true,
            "presence_penalty": true
        },
        "CONVERSATIONAL_AI": {
            "prompt_token_limit": true,
            "similarity_threshold": true,
            "allow_values_for_nlp_generation": true,
            "max_retry_count_for_sql_execution": true,
            "max_rows_summary": true,
            "generate_content": true,
            "is_model_fine_tuned": true
        }
    }
},
{
    "key": "defaultConnectionMdxGeneration",
    "fullyQualifiedName": "defaultConnectionMdxGeneration",
    "label": "Default Connection For MDX Generation",
    "inputType": "CHECKBOX",
    "applicableFor": "LLMProvider",
    "dataType": "BOOLEAN",
    "defaultValue": false,
    "options": {},
    "visible": true,
    "validation": {
        "checkIfDefaultValueValid": true
    },
    "description": {
        "Description": "Sets the default connection for MDX Generation usage."
    }
},
{
    "key": "defaultConnectionConversationalAI",
    "fullyQualifiedName": "defaultConnectionConversationalAI",
    "label": "Default Connection For Conversational Analytics",
    "inputType": "CHECKBOX",
    "dataType": "BOOLEAN",
    "applicableFor": "LLMProvider",
    "defaultValue": false,
    "options": {},
    "visible": true,
    "validation": {
        "checkIfDefaultValueValid": true
    },
    "description": {
        "Description": "Sets the default connection for Conversational AI usage."
    }
},
{
    "key": "allow_values_for_nlp_generation",
    "fullyQualifiedName": "allow_values_for_nlp_generation",
    "label": "Allow Sending Data to LLM",
    "inputType": "SELECTLIST",
    "dataType": "BOOLEAN",
    "defaultValue": true,
    "options": [
        {
            "display": "Yes",
            "value": true
        },
        {
            "display": "No",
            "value": false
        }
    ],
    "validation": {},
    "description": {
        "Description": "The property specifies whether the generated questions should include values."
    }
},
{
    "key": "generate_content",
    "fullyQualifiedName": "generate_content",
    "label": "Generate Content",
    "inputType": "MULTISELECTLIST",
    "dataType": "STRING",
    "defaultValue": [
        "TITLE",
        "OVERVIEW",
        "KEY_INSIGHTS"
    ],
    "visible": true,
    "options": {
        "TITLE": {
            "display": "Title",
            "value": "TITLE"
        },
        "OVERVIEW": {
            "display": "Summary",
            "value": "OVERVIEW"
        },
        "KEY_INSIGHTS": {
            "display": "Key Insights",
            "value": "KEY_INSIGHTS"
        }
    },
    "description": {
        "Description": "This property determines the content to be generated, such as the title, summary, and key insights. For summary and key insights, the value for 'Allow Sending Data for NLG', should be set to 'Yes'."
    }
},
{
    "key": "max_rows_summary",
    "fullyQualifiedName": "max_rows_summary",
    "label": "Max Rows Summary",
    "inputType": "TEXTBOX",
    "dataType": "NUMBER",
    "defaultValue": 1000,
    "validation": {
        "required": true,
        "numeric": true,
        "integer": true,
        "minValue": 10
    },
    "description": {
        "Description": "The property configures the maximum number of rows for generating summary and key insights.",
        "Default Value": 1000
    }
},
{
    "key": "prompt_token_limit",
    "fullyQualifiedName": "prompt_token_limit",
    "label": "Input Prompt Token Limit",
    "inputType": "TEXTBOX",
    "dataType": "NUMBER",
    "defaultValue": 16000,
    "validation": {},
    "description": {
        "Description": "The property specifies the maximum number of tokens permitted for a prompt in a single request to the current provider.",
        "Default Value": 16000,
        "Minimum Value": 0
    }
},
{
    "key": "max_tokens",
    "fullyQualifiedName": "max_tokens",
    "label": "Output Prompt Token Limit",
    "inputType": "TEXTBOX",
    "dataType": "NUMBER",
    "defaultValue": 2000,
    "validation": {
        "required": true,
        "integer": true,
        "minValue": 0,
        "maxValue": 10000
    },
    "description": {
        "Description": "The property defines the maximum number of tokens shared between the prompt and output, which varies by model. One token is approximately four characters for English text.",
        "Default Value": 2000,
        "Minimum Value": 0,
        "Maximum Value": 10000
    }
},
{
    "key": "max_retry_count_for_sql_execution",
    "fullyQualifiedName": "max_retry_count_for_sql_execution",
    "label": "Max Retry Count",
    "inputType": "TEXTBOX",
    "dataType": "NUMBER",
    "defaultValue": 2,
    "validation": {
        "required": true,
        "numeric": true,
        "integer": true,
        "minValue": 0
    },
    "description": {
        "Description": "This property specifies the maximum number of retry counts attempted to obtain the correct query.",
        "Default Value": 2
    }
}, {
    "key": "similarity_threshold",
    "fullyQualifiedName": "similarity_threshold",
    "label": "Summary Records Threshold",
    "inputType": "TEXTBOX",
    "dataType": "NUMBER",
    "defaultValue": 0.2,
    "stepCount" : 0.1,
    "validation": {},
    "description": {
        "Description": "The property specifies the similarity threshold for autocorrecting queries.",
        "Default Value": 0.2,
        "Minimum Value": 0.2
    },
    "isEditable": false
}, {
    "key": "temperature",
    "fullyQualifiedName": "temperature",
    "label": "LLM Temperature",
    "inputType": "TEXTBOX",
    "dataType": "NUMBER",
    "defaultValue": 1,
    "validation": {
        "required": true,
        "numeric": true,
        "minValue": 0,
        "maxValue": 2
    },
    "description": {
        "Description": "The property configures the temperature, which controls the level of randomness in the output. Lowering the temperature results in less random completions. The responses of the model become increasingly deterministic and repetitive as it approaches zero. It is recommended to adjust either the temperature or top-p, but not both simultaneously.",
        "Default Value": 1,
        "Minimum Value": 0,
        "Maximum Value": 2
    }
}

    ] 

}

]

Step 3: Structure of the zip file

Once have the JAR file (including java code to integrate your LLM service), create a zip file that must contain two folders at the root:

lib: Create a lib folder and place your main LLM JAR file and all supporting JAR files required for the LLM service to function.
conf: Create a conf folder and place the metadata.json file inside it. This file must follow the required format and include necessary details like API keys, and other configurations.

Compress the folders

Ensure that the ZIP file does not include a parent folder.
For example, if the folder structure resides in my-llm/, the ZIP should only include the lib/ and conf/ folders directly, not my-llm/lib/ or my-llm/conf/.
Select the lib and conf folders.
Compress them into a ZIP file without including a parent directory.

Configuration

Upload the zip file from Kyvos Manager:

To integrate your LLM service into Kyvos, follow these steps

Login to Kyvos Manager.
Navigate to the GenAI configuration page.
After configuring the LLM service, Kyvos Manager will validate and store the configuration in the designated folder on the BI Server engine.

Verification of the integration

Once your files are uploaded and the callbacks are implemented, perform the following tests:

Verify LLM Integration: Ensure that your LLM is successfully integrated with Kyvos by running test queries on Conversational Analytics Interface and in logs user can verify logging used in his interface using Kyvos Logger.
Check Metadata: Ensure the metadata is being correctly parsed and applied by Kyvos Manager.
Execute SQL Generation: Test the generation of SQL queries, summaries, and embeddings to verify they work as expected.

Configure the Bring Your Own LLM Service in Kyvos

Prerequisites

Configuring Java Code And Metadata for Kyvos-LLM Integration

Step 1: Implement the Callback interfaces

Step 2: Creating the metadata JSON file for connection

Step 3: Structure of the zip file

Compress the folders

Configuration

Verification of the integration