Overview

This guide provides step-by-step instructions on configuring Vector search with ElasticSearch. It includes logging into the environments, creating keys for API access, setting up a machine learning instance, downloading necessary models, creating source and destination indices, and ingesting data to generate text embeddings. The guide also covers manual data loading steps and utilizing client helper functions for data ingestion. It concludes with verifying the data and content embeddings in the destination index.

Log into Environments

Begin by logging in to your IBM Cloud account

• To provision in IBM Cloud:,
  • Navigate to Databases for ElasticSearch.
  • Select the Platinum Database Edition.
• Otherwise, provision within Elastic Cloud as normal.

There are two environments to work from.

• ElasticSearch Cloud console. Notice the icons in the top right corner.
• Kibana console
  • Users may be taken directly to the Kibana console after creating a deployment. If not, navigate there by selecting Open on the deployment page from the ElasticSearch Cloud console.

Creating Keys

• Select the circle icon in the top right of the Kibana screen.
• Select Connection Details
• Here, you will see the ElasticSearch endpoint and the Cloud ID.
• Select Create and Manage API Keys.
• To create a new API key, click Create API Key.
  • Add a unique name.
  • Select the type as User API Key.
  • Click Create API Key button at the bottom of the dialog.

Save these values in a safe place for later use.

Create a Machine Learning Instance

Elastic requires a machine learning instance to run the NLP models required for vectorizing the data for indexing.

• Navigate to the Home screen of your ElasticSearch instance.
• Navigate to the newly created deployment and select Manage.
• On the side menu, select Edit.
• Scroll down to the Machine Learning Instances section.
• Select Add Capacity.
• Select 4 GB RAM.
• Click Save at the bottom of the page.

Download Models

    eland_import_hub_model
    --cloud-id <cloud-id> \
    -u <username> -p <password> \
    --hub-model-id elastic
    distilbert-base-cased-finetuned-conll03-english \
    --task-type ner

Create Source Index and Upload Data

Indices can be created by either manually loading data using the _bulk API, or by using a client helper function which will create the index and load the data.

    PUT /search-gs-docs-src
    {
    "mappings": {
        "properties": {
        "title": { 
            "type": "text" 
        },
        "content": { 
            "type": "text" 
        },
        "source": { 
            "type": "text" 
        },
        "url": { 
            "type": "text" 
        },
        "public_record": { 
            "type": "boolean" 
        }
        }
    }
    }

    POST _bulk
    { "index" : { "_index" : "search-gs-docs-src", "_id" : "1" } }
    { "title" : "Top 3 Best Practices to Secure Your Gainsight PX Subscription",
    "content" : "We should all protect what has been entrusted…”,
    "url" : "https://support.gainsight.com/...",
    "source" : "docs”,
    “public_record”:true,
    “objectID”: “https://support.gainsight.com/...”
    }
    { "index" : { "_index" : "search-gs-docs-src", "_id" : "2" } }
    { "title" : "Using PX with Content Security Policy",
    "content" : "This article describes the steps to allow a Content Security Policy…”,
    "url" : "https://support.gainsight.com/...",
    "source" : "docs”,
    “public_record”:true,
    “objectID”: “https://support.gainsight.com/...”
    }
    …

    'use strict'

    require('array.prototype.flatmap').shim()
    const { Client } = require('@elastic/elasticsearch')
    const client = new Client({
    cloud: { id: '<cloud_id>'},
    auth: { apiKey: '<api_key>' }
    })
    const dataset = require('./gainsight_documentation_data/gainsight-en-federated.json')

    // Create and load the source index
    async function run () {
    await client.indices.create({
        index: 'search-gs-docs-src',
        operations: {
        mappings: {
            properties: {
            title: { type: 'text' },
            content: { type: 'text' },
            url: { type: 'text' },
            source: { type: 'text' },
            public_record: { type: 'boolean' },
            objectID: { type: 'text' }
            }
        }
        }
    }, { ignore: [400] })

    const operations = dataset.flatMap(doc => [{ index: { _index: 'search-gs-docs-src' } }, doc])

    const bulkResponse = await client.bulk({ refresh: true, operations })

    if (bulkResponse.errors) {
        const erroredDocuments = []
        // The items array has the same order of the dataset we just indexed.
        // The presence of the `error` key indicates that the operation
        // that we did for the document has failed.
        bulkResponse.items.forEach((action, i) => {
        const operation = Object.keys(action)[0]
        if (action[operation].error) {
            erroredDocuments.push({
            // If the status is 429 it means that you can retry the document,
            // otherwise it's very likely a mapping error, and you should
            // fix the document before to try it again.
            status: action[operation].status,
            error: action[operation].error,
            operation: operations[i * 2],
            document: operations[i * 2 + 1]
            })
        }
        })
        console.log(erroredDocuments)
    }

    const count = await client.count({ index: 'search-gs-docs-src' })
    console.log(count)
    }

    run().catch(console.log)

Once the data is loaded, either manually or programmatically, verify that it appears properly in the index.

• Navigate to the Kibana console.
• Navigate to Search > Content > Indices.
• Open the search-gs-docs-src index.
• Open the Documents tab to see the data for verification.

Create destination Index

Create a destination index using the same schema as the source index. Add a field to store the content embeddings.

• Enter the following code, then hit the run icon.

    PUT /search-gs-docs-dest
    {
    "mappings": {
        "properties": {
        "content_embedding": { 
            "type": "sparse_vector" 
        },
        "title": { 
            "type": "text" 
        },
        "content": { 
            "type": "text" 
        },
        "source": { 
            "type": "text" 
        },
        "url": { 
            "type": "text" 
        },
        "public_record": { 
            "type": "boolean" 
        }
        }
    }
    }

Ingest the Data to Generate Text Embeddings

• Create an ingest pipeline with an inference processor. Enter the following code:

    PUT _ingest/pipeline/my-content-embedding-pipeline
    {
    "processors": [
        {
        "inference": {
            "model_id": ".elser_model_2_linux-x86_64",
            "input_output": [ 
            {
                "input_field": "content",
                "output_field": "content_embedding"
            }
            ]
        }
        }
    ]
    }

• Click the run icon.

• Ingest the data through the inference index pipeline to create the text embeddings. Enter the following code into the dev console:

    POST _reindex?wait_for_completion=false
    {
    "source": {
        "index": "search-gs-docs-src",
        "size": 50 
    },
    "dest": {
        "index": "search-gs-docs-dest",
        "pipeline": "my-content-embedding-pipeline"
    }
    }

• To get the name of the pipeline with the model loaded, navigate to Kibana > Machine Learning > Trained Models.
• Expand the Deployed model.
• Navigate to the Pipelines tab to view the my-content-embesddings-pipeline created in the above step.

To confirm the task was run successfully, run the following command using the task ID produced in the response from the previous command. GET _tasks/<task_id>.

• Verify the content embeddings are in the new destination index.
  • Navigate to Kibana.
  • Navigate to Search > Content > Indices.
  • Open the search-gs-docs-dest index.
  • Open the Documents tab to see the data.

Map a Field

Models compatible with ElasticSearch NLP generate dense vectors as output, so the dense_vector field type for the index is suitable for storing. This field type must be configured with the same number of dimensions using the dims option.

• Enter the following code into the dev console to create an index mapping that defines field containing the model output.

      PUT my-index
      {
      "mappings": {
          "properties": {
          "my_embeddings.predicted_value": { 
              "type": "dense_vector", 
              "dims": 384 
          },
          "my_text_field": { 
              "type": "text" 
          }
          }
      }
      }
  

• my_embeddings.predicted_value is equal to the name of the field containing the embeddings generated by the model.

• The "type" field must be "dense_vector".
• The "dims" field contains the number of dimensions of the embeddings produced by the model. Be sure that this number is configured in the dense_vector field.
• The "my_text_field" field is equal to the name of the field from which to create the dense vector representation.
• The "type" field is text.

Test the Semantic Search

        GET search-gs-docs-dest/_search
    {
    "query":{
        "text_expansion":{
            "content_embedding":{
                "model_id":".elser_model_2_linux-x86_64",
                "model_text":"Put sample query here"
            }
        }
    }
    }

    GET my-index/_search
    {
    "knn": {
        "field": "my_embeddings.predicted_value",
        "k": 10,
        "num_candidates": 100,
        "query_vector_builder": {
        "text_embedding": {
            "model_id": "sentence-transformers__msmarco-minilm-l-12-v3",
            "model_text": "the query string"
        }
        }
    }
    }

Connect NeuralSeek to Elasticsearch

• Navigate to your IBM Cloud account.
• Open the NeuralSeek service instance.
• Navigate to the Configure screen.
• Save your current setting by clicking the Download Settings button at the bottom of the screen.
• Open the KnowledgeBase Connection accordion and update the following fields.
  • Set KnowledgeBase Type to ElasticSeach
  • Set the ElasticSearch Endpoint.
  • Set the ElasticSearch Private API Key.
  • Set the ElasticSearch Index Name to the destination index. In this case, search-gs-docs-dest.
  • Set the Curation Data Field to content.
  • Set the Documentation Name Field to title.
  • Set the Link Field to url.
• Click the Save button at the bottom of the page.

Enable Vector Search in NeuralSeek

In the NeuralSeek Configure screen, open the Hybrid and Vector Search Settings accordion to update the following fields.

• Set Elastic Query Type to Hybrid.
  • This will allow for both Lucene (exact match) and Vector (semantic) searching to achieve a more robust response.
• Set the Model ID to ".elser_model_2_linux-x86_64"
• Set the Embedding Field to content_embedding
• Set the Use the Elastic ELSER Model field to True for ELSER Model Use, or set to False to allow NeuralSeek to expect JSON format for a kNN search query.
• Click Save at the bottom of the screen.

    PUT /<INDEX_NAME>/_settings
    {
        "index" : {
            "highlight.weight_matches_mode.enabled" : "false"
        }
    }