Vector Analytics

Vector analytics is useful when you are interested in determining the quality of your vectors.

Once we have our multi-dimensional vectors (in 128D, 256D, 768D etc), we may want some way to visually grasp how good our encodings are and to understand if there’s any anomalies in our data and if we may want to try a different model. Vi supports visualising your vectors in 2D using high-level libraries such as Plotly.

Vector analytics include clustering the different vectors, reducing the dimensionality of the vectors and then visualising.

# %pip install -e ..

%load_ext autoreload
%autoreload 2

from vectorai import ViClient
import os
username = os.environ['VI_USERNAME']
api_key = os.environ['VI_API_KEY']
collection_name = 'nlp-qa'

collection_name = 'nlp-qa'
vi_client = ViClient(username, api_key)

Clustering

Quickstart using Vi client. Download Google QUEST Kaggle data from: https://www.kaggle.com/c/google-quest-challenge/data Once you download data, move train.csv into local directory.

import pandas as pd
data = pd.read_csv('train.csv')
data.head(2)

	qa_id	question_title	question_body	question_user_name	question_user_page	answer	answer_user_name	answer_user_page	url	category	...	question_well_written	answer_helpful	answer_level_of_information	answer_plausible	answer_relevance	answer_satisfaction	answer_type_instructions	answer_type_procedure	answer_type_reason_explanation	answer_well_written
0	0	What am I losing when using extension tubes in...	After playing around with macro photography on...	ysap	https://photo.stackexchange.com/users/1024	I just got extension tubes, so here's the skin...	rfusca	https://photo.stackexchange.com/users/1917	http://photo.stackexchange.com/questions/9169/...	LIFE_ARTS	...	1.000000	1.000000	0.666667	1.000000	1.000000	0.800000	1.0	0.0	0.000000	1.000000
1	1	What is the distinction between a city and a s...	I am trying to understand what kinds of places...	russellpierce	https://rpg.stackexchange.com/users/8774	It might be helpful to look into the definitio...	Erik Schmidt	https://rpg.stackexchange.com/users/1871	http://rpg.stackexchange.com/questions/47820/w...	CULTURE	...	0.888889	0.888889	0.555556	0.888889	0.888889	0.666667	0.0	0.0	0.666667	0.888889

2 rows × 41 columns

Setup ~ Insert documents

# Change into document level first.
documents = data.to_dict(orient='records')

# Run encoding
from vectorai.models.transformer_models import Transformer2Vec
encoder = Transformer2Vec('distilbert')

# Quick test to see if this works on 1 document
# vector = encoder.encode(document=documents[0], document_fields=['question_title', 'question_body'])

vectors = encoder.bulk_encode(documents,
          document_fields=['question_title', 'question_body'],
        vector_output_field='question_vector_')

Some weights of the model checkpoint at distilbert-base-cased were not used when initializing TFDistilBertModel: ['vocab_transform', 'vocab_projector', 'vocab_layer_norm', 'activation_13']
- This IS expected if you are initializing TFDistilBertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPretraining model).
- This IS NOT expected if you are initializing TFDistilBertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
All the weights of TFDistilBertModel were initialized from the model checkpoint at distilbert-base-cased.
If your task is similar to the task the model of the ckeckpoint was trained on, you can already use TFDistilBertModel for predictions without further training.

Finished updating documents with additional field.

vi_client.insert_documents(collection_name=collection_name, documents=documents, chunksize=50)

{'inserted_successfully': 6079, 'failed': 0, 'failed_document_ids': []}

Quickstart Clustering

We support clustering on the client-side in order to help improve the product. On the backend-clustering, we are using KMeans.

# Our main clustering is done via advanced_clustering_job. This allows us to easily determine IDs.
vi_client.advanced_clustering_job(
    collection_name=collection_name, vector_field='question_vector_',
    alias='1st_cluster', n_clusters=50, n_init=5)

{'job_name': 'advanced_clustering',
 'job_id': '19235f50-fa36-4c4f-8b7e-ba76be800a2a'}

That’s it! Clustering is now running! Now, we wait till the job is complete.

vi_client.wait_till_jobs_complete(collection_name=collection_name,
    job_id='7ed0ae66-1b44-43ce-929d-5eea1a887785', job_name='advanced_clustering')

{'status': 'Finished'}

'Done'

Now, when we preview the collection, we can see that documents now belong to a cluster.

vi_client.retrieve_documents(collection_name, page_size=1)['documents'][0]['_clusters_']

{'question_vectors_': {'1st_cluster': 43},
 'question_vector_': {'1st_cluster': 39}}

vi_client.head(collection_name, return_as_pandas_df=True, page_size=2)

	_id	qa_id	question_title	question_body	question_user_name	question_user_page	answer	answer_user_name	answer_user_page	url	...	answer_relevance	answer_satisfaction	answer_type_instructions	answer_type_procedure	answer_type_reason_explanation	answer_well_written	insert_date_	_clusters_	_dr_	question_vector_
0	3e80C3QBW7TQbeJX1K3V	5899	Eigenvalues of a transition probability matrix	I have read that, for\n$$I - \alpha P$$\nwhere...	user243716	https://math.stackexchange.com/users/243716	As stated, the result is FALSE. Presuming P i...	Mark L. Stone	https://math.stackexchange.com/users/240387	http://math.stackexchange.com/questions/130034...	...	1.000000	1.000000	0.0	0.000000	1.000000	1.000000	2020-08-21T05:20:49.072579	{'question_vectors_': {'1st_cluster': 43}, 'qu...	{'default': {'2': {'question_vectors_': [1.229...	[0.15878267586231232, 0.2284708470106125, -0.0...
1	dn00C3QBv-xqKfY1H0KD	2660	Does adding a comma before "or" change the mea...	For example, the definition given from the OAL...	kiamlaluno	https://ell.stackexchange.com/users/95	Regarding general usage (in the U.S., at least...	Scott	https://ell.stackexchange.com/users/357	http://ell.stackexchange.com/questions/3208/do...	...	0.888889	0.733333	0.0	0.333333	0.666667	0.888889	2020-08-21T05:20:49.073579	{'question_vectors_': {'1st_cluster': 42}, 'qu...	{'default': {'2': {'question_vectors_': [-0.19...	[0.24962793290615082, 0.05536836013197899, -0....

2 rows × 46 columns

Custom clustering of the vectors

Sometimes, we may want to use our own clustering functions because KMeans doesn’t do exactly what we want. Here, we show a simple implementation of clustering to view what this is like for the popular HDBSCAN package.

%%capture
%pip install hdbscan

documents = vi_client.retrieve_all_documents(collection_name)

import hdbscan
import numpy as np
clusterer = hdbscan.HDBSCAN()

# get a numpy array of values
question_vectors = np.array([x['question_vector_'] for x in documents])
clusterer.fit(question_vectors)
clusterer.labels_

array([-1, -1, -1, ..., -1, 28, 41])

# Add the cluster labels below.
{x.update({'_cluster_label': label}) for x, label in zip(documents, clusterer.labels_)}

{None}

Dimensionality Reduction

Quickstart Dimensionality Reduction

Dimensionality reduction from client side is as simple as running the steps below. This runs Principal Component Analysis on dimensionality reduction and allows us to visualise what the end result looks like.

vi_client.dimensionality_reduction_job(
    collection_name=collection_name,
    vector_field='question_vector_',
    n_components=2, alias="default")

{'job_name': 'dimensionality_reduction',
 'job_id': 'b56b00a1-9fc2-4eb3-b467-260d96715ae3'}

Custom Dimensionality Reduction

Below shows how we utilised a custom implementation of IVIS Dimensionality Reduction!

%%capture
%pip install ivis

from vectorai.analytics.dimensionality_reduction import ViDimensionalityReductionBase
from sklearn.preprocessing import MinMaxScaler
from ivis import Ivis
import numpy as np

class IVISDimensionalityReduction(ViDimensionalityReductionBase):
    """IVIS Dimensionality Reduction.
    """
    def __init__(self, batch_size=120, k=15, embedding_dims=2):
        self.scaler = MinMaxScaler()
        self.model = Ivis(embedding_dims=embedding_dims, k=k, batch_size=batch_size)

    def fit_transform(self, documents, field_vector):
        """Fit transform at an document level
        """
        if isinstance(documents[0], dict):
            if 'documents' in documents[0].keys():
                vectors = [self.get_field(field_vector, document) for document in documents['documents']]
            else:
                vectors = [self.get_field(field_vector, document) for document in documents]

        X = np.stack(vectors)
        X_scaled = self.scaler.fit_transform(X)
        return self.model.fit_transform(X_scaled)

    def transform(self, X):
        if isinstance(documents[0], dict):
            if 'documents' in documents[0].keys():
                vectors = [document[field_vector] for document in documents['documents']]
            else:
                vectors = [document[field_vector] for document in documents]
        X_scaled = self.scaler.transform(vectors)
        return self.model.transform(X_scaled)

dim_reducer = IVISDimensionalityReduction()
red_output = dim_reducer.fit_transform(documents, field_vector='question_vector_')

{x.update({'question_reduced_': q_red_}) for q_red_, x in zip(red_output, documents)}

{None}

Visualisations (Advanced)

Visualisation with Vector AI is much different to embedding visualisations in other libraries. Our focus is on practicality for deep learning practitioners to better understand the embeddings they want to explore and focus on what matters - the quality of their embeddings in as practical of a situation as possible. A few of our guiding principles when we are building tools:

• Is it immediately obvious what the graph shows?

• Is as much information captured as possible and as clear as possible?

• What is the status quo in regards to the tools being used?

Our focus is on ensuring that practitioners have a lot of good tools at their disposal when they are evaluating the quality of their vectors.

vi_client.collection_schema(collection_name)

{'_clusters_': 'dict',
 '_clusters_.question_vector_': 'dict',
 '_clusters_.question_vector_.1st_cluster': 'numeric',
 '_clusters_.question_vectors_': 'dict',
 '_clusters_.question_vectors_.1st_cluster': 'numeric',
 '_dr_': 'dict',
 '_dr_.default': 'dict',
 '_dr_.default.2': 'dict',
 '_dr_.default.2.question_vector_': 'vector',
 '_dr_.default.2.question_vectors_': 'numeric',
 'answer': 'text',
 'answer_helpful': 'numeric',
 'answer_level_of_information': 'numeric',
 'answer_plausible': 'numeric',
 'answer_relevance': 'numeric',
 'answer_satisfaction': 'numeric',
 'answer_type_instructions': 'numeric',
 'answer_type_procedure': 'numeric',
 'answer_type_reason_explanation': 'numeric',
 'answer_user_name': 'text',
 'answer_user_page': 'text',
 'answer_well_written': 'numeric',
 'category': 'text',
 'host': 'text',
 'insert_date_': 'date',
 'qa_id': 'numeric',
 'question_asker_intent_understanding': 'numeric',
 'question_body': 'text',
 'question_body_critical': 'numeric',
 'question_conversational': 'numeric',
 'question_expect_short_answer': 'numeric',
 'question_fact_seeking': 'numeric',
 'question_has_commonly_accepted_answer': 'numeric',
 'question_interestingness_others': 'numeric',
 'question_interestingness_self': 'numeric',
 'question_multi_intent': 'numeric',
 'question_not_really_a_question': 'numeric',
 'question_opinion_seeking': 'numeric',
 'question_title': 'text',
 'question_type_choice': 'numeric',
 'question_type_compare': 'numeric',
 'question_type_consequence': 'numeric',
 'question_type_definition': 'numeric',
 'question_type_entity': 'numeric',
 'question_type_instructions': 'numeric',
 'question_type_procedure': 'numeric',
 'question_type_reason_explanation': 'numeric',
 'question_type_spelling': 'numeric',
 'question_user_name': 'text',
 'question_user_page': 'text',
 'question_vector_': 'vector',
 'question_well_written': 'numeric',
 'url': 'text'}

Plotting Dimensionality-Reduced Vectors

Plotting dimensionality-reduced vectors allows you understand your clusters better. It is a common (and pretty) technique to understand if your clustering technique was reflective of groups.

When you visualise the vectors, you should see clear groups where the points are separated. This means that the vectors have been trained well for their respective clusters - an example of having high quality vector representations if they learn groups on their own.

documents = vi_client.retrieve_all_documents(collection_name)

cluster_label = 'category'
dim_reduction_field = '_dr_.default.2.question_vector_' # the '.' in names implies nested dictionaries in Vi
cluster_field = 'question_vector_'
alias = '1st_cluster'
collection_name = 'nlp-qa'

vi_client.plot_dimensionality_reduced_vectors(
    collection=documents,
    cluster_field=cluster_field,
    cluster_label=cluster_label,
    point_label='question_title',
    dim_reduction_field=dim_reduction_field,
    # include_centroids=True,
    alias=alias)

# plotting the IVIS dimensionality reduction
# documents = vi_client.random_documents(collection_name, 30)['documents']
point_label = 'question_title'
cluster_label = 'category'
dim_reduction_field = 'question_reduced_'
vi_client.plot_dimensionality_reduced_vectors(collection=documents,
    cluster_field=cluster_field, cluster_label=cluster_label,
    dim_reduction_field=dim_reduction_field, point_label=point_label)