MongoDB

Scott Kurowski

Staff Engineer

There might be hidden business knowledge in your MongoDB vector database.

When business data is vectorized, its embedding enables non-retrieval-augmented generation (RAG) retrieval or document search. Example data could be customer text inputs, large language model (LLM) classification annotation strings or string arrays, or other varying text content. Applications which store changing or cumulative embedded vector database content can develop strategic latent semantic knowledge as a population, and this article shows a method to extract it. 

You are probably familiar with vector embedding of text using an embedding model. OpenAI text-embedding-3-small and text-embedding-ada-002 semantically encode a vector’s input text *meaning*, not merely its token sequence. Given a text string less than 8 kt (kilotokens) long, text-embedding-ada-002 returns an array vector of 1536 floats or doubles, each valued in range [-1.0, 1.0]. Each vector is the neural network weight’s output from the first few layers of a partial LLM model. The model-encoded vector maps a *specific point* in the model’s *normalized conceptual vector (hyper-)space for its embedded text* — and this map is model-specific: We cannot mix vectors embedded by text-embedding-ada-002 and text-embedding-3-small in the same analysis.

![Map text strings into LLM "semantic space"](https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blta284b10897d4d230/66e16957f7198c0a1bb2b929/Screenshot_2024-09-11_at_10.48.31.png)

A model’s embedding vector space is normalized in values range [-1.0, 1.0], scaled to its standard deviation (sigma) as unit 1.0, making old-fashioned analytic geometry a convenient toolbox. Yet it is the model which gives the vector space its meaning. For our AI embedding model, vector (point) location indicates (maps) specific semantic content. Casting this meaning through our toolbox further tells us that vector proximity indicates semantic similarity, and vectors separated by greater than or equal to 3.0 (3 unit sigmas) can be considered un-matching (“distinct” semantic content) to a confidence of 99.7%.

MongoDB $vectorSearch aggregation takes advantage of the semantic proximity of vector points to an input’s vector point location to return “nearby” results, effectively up to a cutoff distance in vector space (an inverse function of the match score), or max_k count limit.

## Vector clustering: Unsupervised, unbiased vector classification ##

Let’s see how this further applies to semantic vector ensembles. Assume we somehow observed a possible pattern in the source texts embedded as vectors and want to try an unsupervised semantic-density self-grouping of its vectors.

A powerful example strategy to discover unsupervised and unbiased semantic structure in a vector database is to group the vectors into a hierarchy of central concept clusters, leveraging analytic geometry and statistical properties of AI embedding model vector spaces. In other words, the model makes the vector space semantically meaningful. Note it is the mathematical treatment of vector space data which is unbiased, not the model’s embedding encoding which, by definition, must reflect its intrinsic training biases.

In Python 3.11, one can use the sklearn.scikit.org OPTICS density-clustering class [1]. Clustering is an unsupervised pre-LLM AI technique. The main tunable parameter, *min_per_cluster*, which I call its “granularity,” adjusts the *minimum number of vector (point) members for a semantic cluster* to configure how generalized to distinct clustered concept centroids become — a lower minimum groups many distinct smaller clusters, a higher minimum groups fewer generalized larger clusters. The other tunables are how many total vectors to cluster and their common vector length.

```python
sklearn.cluster.OPTICS(
   vectors,
   min_per_cluster=512
)
```
It took about six hours to cluster N = 100K embedding vectors 1536 length on Apple silicon, reflecting its O(N^2) performance.

![Example semantic vector-density clustered concept tree. Python networkx circular digraph with matplotlib](https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt01129eccde0d2b99/66e16ac001b09c4e32813d7f/Screenshot_2024-09-11_at_10.57.51.png)

The result of OPTICS clustering is a hierarchy tree of density-grouped vectors (semantic concepts), where each cluster has a vector membership list. In addition to the “leaf” cluster concepts, identified is how they branch from a more generalized concept (also having a vector membership list) to a “root” concept cluster at the tree root.

It directly reveals the semantic structure within the vector set. Now, having the membership of which vectors form which conceptual clusters of the tree, we finally have the data to create a *semantic centroid* from each identified cluster.

Almost magically, we again leverage the math properties of the embedded vector-space to derive these by simply *averaging the vectors of a cluster’s membership list*, creating the c*luster’s centroid concept vector*.

The resultant centroid vector’s encoding “describes” its entire cluster as a group. But exactly what text does each centroid vector represent?

## Reversing semantic clustered concept vectors back into text ##

We need the text the clustered mean centroid concept vector represents. This is somewhat more difficult than embedding a text’s vector.

Three basic methods of reversing a clustered centroid embedding vector back into its conceptual source text were effective in my limited testing:
<ol>
   <li>Brute-force iterative error gradient descent of embedding + LLM calls</li>
      <ol type="a">
         <li> Very slow, expensive $$$$$, ~20K iterations, a few hours</li>
         <li>Very simple, LLM next-guess + embed and diff its guess for error</li>
            <li>Faster when initial guess text is somewhat “close” to target</li>
      <li>Faster when LLM given best three guesses+errors, last eight guesses+errors</li>
      </ol>


![Un-map LLM "semantic space" vector into a text string](https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt90df6af236c729c7/66e16ac1de1781e07d3d72aa/Screenshot_2024-09-11_at_10.58.00.png)
<ol start="2">
<li>Trained Embedding Dictionary Probe Model (vec2text)</li>
       <ol type="a">
        <li>Fast-ish, ~1 minute to reverse a vector into model-generated text</li>
        <li>Python module on Pytorch + GPUs</li>
        <li>Pre-trained on text-embedding-ada-002 semantic vector space</li>
        <li>Re-trainable on other embedding models and semantic content</li>
<li>Up to a few dozen text-embedding-ada-002 model calls </li>
<li>Ideal without vector source texts for Method 3</li>
<li>Not very good when only a few vectors are averaged as a centroid</li>
</ol>

<ol start="3">
<li>LLM-merging of the source texts of clustered embedded vectors</li>
<ol type="a">
<li>Fast, 1x LLM call, cheap $, at most a few seconds </li>
<li>Text-frequency table of clustered member vector texts + prompt </li>
<li>Tested qualitatively superior to Method 1</li>
<li>On average, provided better detail and fewer errors than Method 2</li>
</ol>
</ol>

Methods 1 and 2 generate new (model-specific) conceptual language text directly from a *computed* or *unknown source text’s embedding vector*. Method 1 is not very practical yet demonstrates effective, if tediously/costly, text reversal. Method 2 (vec2text) uses a custom model trained to unmap embedding model vectors to texts to inform generating its iterative text guesses [2, 3].

When the embedding source texts are available for clustered vectors, Method 3 skips reversing the cluster centroid vector by simply asking an LLM to merge a text-frequency table of the cluster’s members’ source texts into a new, single centroid text. Qualitatively, in my experience, this method generated superior centroid texts relative Methods 1 and 2.  I tested Method 2 using its pre-trained model, yet with subject-domain content training, it might be competitive with Method 3.

## Example semantic structure extraction demo ##
Let’s take a look at a real example of embedding vector clustering with a “toy-sized” set of 26 short texts selected for their obvious semantically-distinct conceptual grouping into five broader topics, having two (the MongoDB and Gödel topics) rich enough to potentially split into semantic sub-groups. You can run it yourself on Python 3.11 using the demo code linked below.

Here are the demo input texts, which are vectorized and then clustered as centroids:

```json
TEXTS = [
    "Be at one with your power, joy and peace.",
    "Know the flow of the much greater oneness we share.",
    "Let one's superpower be considered choices in the network of life.",

    "MongoDB Ops Manager",
    "MongoDB Cloud Manager",
    "MongoDB Cloud Manager Backups",
    "MongoDB Atlas Database",
    "MongoDB Atlas Stream Processing",
    "MongoDB Atlas Vector Search",
    "MongoDB Atlas Data Lake",
    "MongoDB Enterprise Database Server",

    "Gödel, Escher, Bach: An Eternal Golden Braid",
    "How Gödel's Theorems Shape Quantum Physics as explored by Wheeler and Hawking",
    "Bach, Johann Sebastian - Six Partitas BWV 825-830 for Piano",
    "M.C. Escher, the Graphic Work",
    "Bach's baroque style features recursion or self-referencing iterated functions like the artwork of Escher.",
    "In 1931, Gödel proved the profound duality that formal systems cannot be both self-consistent and complete.",
    "John Von Neumann was able to derive Gödel's 2nd theorem from his 1st before Gödel published it.",

    "My cat is a fun black and white half-sized tuxedo.",
    "Some people prefer the company of dogs instead of cats.",
    "My friend has a large saltwater aquarium with colorful and exotic tropical fish.",
    "My clever dog opens locked windows and doors to follow me.",

    "Mesopotamian tablets record a fantastic version of human history.",
    "North American burial mounds often held deceased local royal families.",
    "Mayan pyramids predated most Aztec pyramids.",
    "The Aztecs Quetzalcoatl closely resembles the Egyptian god Thoth.",
]
```
The specific embedding model and vector length can be varied; the demo uses text-embedding-ada-002 at 1536 dimensions.

Can it find and describe the expected semantic structure? Yes! Here is the generated tree, post-annotated with its LLM-merged (Method 3) centroid texts:

![Generated tree, post-annotated with its LLM-merged (Method 3) centroid texts](https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt28fc9a3ed1361982/66e16ac1b9afadeae9e73bd0/Screenshot_2024-09-11_at_10.58.18.png)

The tree and clustered centroid texts beautifully reveal the semantic structure of the input texts’ concepts at a clustering “granularity” minimum of two vectors.

For a larger vector set having additional associated metadata, it may be of further interest to examine those in each cluster’s member vectors, as grouped, for any related insights.

For a final thought, it should be obvious that Atlas Vector Search and clustering of *normalized scaled vectors* from any data model (not just AI models) works the same way. A world of possibilities.

## What Next ##

Ready to give semantic vector clustering a try? [Find the demo code](https://github.com/mongodb-developer/semantic-vector-clustering).

For effective manipulation of vector data, it is crucially important that embedding source texts be “cleaned” strings or arrays of strings omitting any formatting syntax and newlines, keeping only sentence separators.

When clustering vectors, it can be useful to test a range of “granularities” starting with a large minimum number of vectors per cluster setting (say, 512), then lowering it in large steps until clustering yields a useful balance between the generality and distinctness of the reversed centroid texts, and how many centroids were classified. If only one (the hierarchical root) cluster develops, use a lower minimum. If more than a few dozen clusters form, try a higher minimum.

The demo code uses Python *openai* API module 1.36 or later and is configured with (Method 2) vec2text disabled as there are too few and dissimilar vectors per cluster in this demo to create a meaningful mean centroid vector to reverse. To enable it, set ```CONFIGURE_VEC2TEXT = True``` in *demo_vector_clustering.py*. When enabled in the demo, vec2text centroid texts are somewhat nonsensical due to the sparse few member texts of dissimilar words and tenses among the texts of each cluster. A larger centroid membership should better converge to a perfectly valid central concept text using vec2text.

To adapt the demo to a collection in an existing MongoDB vector database, follow these steps:
<ol>
<li>Verify you have the demo working with highly-similar results as shown above before proceeding. You might need to do step (3)(e) below for this.</li>
<li>Skip using `load_vector_db.py` since we’re using an existing vector database.</li>
<ol type="a">
   <li>Edit the MongoDB instance URI string including required auth credentials, name of the target vector database, and collection name on lines 39, 40, and 43:</li>
<li>Edit the embedded source text field name and its embedding field name to cluster upon as a dictionary object *{ "text_field_name": "embedding_field_name" }* in line 49. You can get this info using *db.collection.getSearchIndexses()* in a *mongosh* shell. Add more than one if there are multiple embedded fields to cluster in separate passes in each vector. Here, the single vector text field name is *text* and its embedding vector is stored in the array field *embedding*</li>
<li>Edit the output fitted cluster centroids collection name (in the same vector database), line 53. It does not need to start with *VECTOR_COLLECTION* as a prefix</li>
<li>To use *vec2text* for a second version of each semantic centroid text in addition to LLM-merged centroid texts, first verify it functions correctly using the demo data, and then edit ```CONFIGURE_VEC2TEXT = True``` on line 57 here.</li>
<li>Edit the OpenAI completion model to merge centroid texts, line 62. gpt35-turbo might perform okay, but I recommend gpt-4, gpt-4o, or gpt-4o-mini. For Azure OpenAI models, this is the deployment name.</li>
<li>Edit the OpenAI completion model token budget, line 63. This limits the text-frequency table size for centroid text merging. You could consider increasing it above 16000; I recommend trying 16000 first.</li>
<li> Edit the collection vector index path field’s embedding model used, line 67.  I found *"text-embedding-3-small"* works well, too, if that’s what the search index is embedded upon. If using vec2text, however, you must use *text-embedding-ada-002*.</li>
<li>Edit the cluster grouping "granularity" settings, where each *MINIMUM_VECTORS_PER_CLUSTER* list integer value is clustered one at a time in an outer loop (line 72); here is a guess which starts with coarse granularity at 512 and progressively clusters finer and finer granularities. If only one semantic cluster is discovered, the granularity is too high; cut it in half and try again until you start seeing clustered centroids at a useful level of detail.
	```MINIMUM_VECTORS_PER_CLUSTER = [512,256,128,64,32,16]```
</li>
<li>If you’d like to keep the centroids data between repeated clustering code runs, comment-out line 81:
```vector_db[fitted_collection].delete_many({})```</li>
<li>Unless you are using the environment variable *OPENAI_API_KEY* for the *OpenAI()* class instance declaration in line 84, edit it with your api_key parameter and value. If using *AzureOpenAI()*, use that class instead and populate any other parameters required. If ```CONFIGURE_VEC2TEXT = True```, repeat this in *reverse_vector_vec2text.py* in line 56, also:
```model_client = OpenAI(api_key=”my api key string”)```</li>
<li>Edit and comment-out *load_demo_vectors()* in line 260.</li>
</ol>

<ol start="3">
<li>Run the semantic vector clustering code. For ~100K vectors length 1536, an M2 Apple required just over six hours. And half as many (~50K) vectors would take a quarter as long, ~1.5 hours, because clustering time scales O(N^2): </li>

    $ python  demo_vector_clustering.py
<ol start="4">
<li>Inspect the output data between each clustering iteration of values in *MINIMUM_VECTORS_PER_CLUSTER* for the level of detail in their centroid texts, and stop the code run when there is “too much” detail. In a *mongosh*, try:</li>


    db.<your_fitted_semanatic_clusters_collection>.find({
    		min_vectors_per_cluster: 512 // etc.
    	},{centroid_embedding:0,_id:0});
<ol start="5">
<li>Inspect each generated cluster hierarchy structure chart png file, which tends to branch increasingly as *min_vectors_per_cluster* gets lower:</li>

    demo_vectors_clustered_<<source_text_field_name>>_<<min_vectors_per_cluster>>minPerCluster.png
  
<ol start="6">
<li>Optional: If there is another metadata field in each cluster’s member vector for which there is further value to accumulate those into a list for each fitted centroid’s further external analysis, complete these steps also:</li>
<ol type="a">
<li>Edit and un-comment the code lines which refer to a non-vector field *field_to_accumulate* in lines 221, 243, 249, 273, 279, 295, 357, and 388.</li>
<li>Edit and replace *field_to_accumulate* to use the actual metadata field name everywhere it is referenced in the uncommented code.</li>
</ol>
</ol>

## Summary ##

The mathematical properties of AI embedding vectors enable us to directly use “old school AI” clustering techniques on semantic content. By modifying the demo code, you can experimentally apply this technique to your own embedding vector dbs. You just might reveal new, hidden structure and intelligence in your business data.

If you have questions, want to share what you’re building, or want to see what other developers are up to, visit the [MongoDB Developer Community](https://www.mongodb.com/community/forums/) next.

### For further reading: ###
[1] [OPTICS: Ordering Points To Identify the Clustering Structure](https://dl.acm.org/doi/pdf/10.1145/304181.304187)
[2] [Text Embeddings Reveal (Almost) As Much As Text](https://arxiv.org/pdf/2310.06816)

[Python vec2text clustering module]( https://github.com/jxmorris12/vec2text)

[3] [Steerable Language Generation from [Reversed] Text Embedding Vectors](http://reports-archive.adm.cs.cmu.edu/anon/2024/CMU-CS-24-109.pdf)
     See: 1.2 Case Study 2
[4] [Atlas Search MongoDB University Course](https://learn.mongodb.com/learning-paths/atlas-search)
[5] [Vector Search MongoDB University Course](https://learn.mongodb.com/courses/introduction-to-ai-and-vector-search)



Leverage the mathematical properties of a population of db AI-embedded vectors to extract potential novel business intelligence.

Scott Kurowski

Content by Scott Kurowski

Discover Latent Semantic Structure With Vector Clustering