How to Create Vector Embeddings

Initialize your Go project.

In a terminal window, run the following commands to create a new directory named my-embeddings-project and initialize your project:

mkdir my-embeddings-project
cd my-embeddings-project
go mod init my-embeddings-project

Install and import dependencies.

In a terminal window, run the following commands:

go get github.com/joho/godotenv
go get go.mongodb.org/mongo-driver/mongo
go get github.com/tmc/langchaingo/llms

Create a .env file to manage secrets.

In your project, create a .env file to store your Atlas connection string and Hugging Face access token.

HUGGINGFACEHUB_API_TOKEN = "<access-token>"
ATLAS_CONNECTION_STRING = "<connection-string>"

Replace the <access-token> placeholder value with your Hugging Face access token.

Replace the <connection-string> placeholder value with the SRV connection string for your Atlas cluster.

Your connection string should use the following format:

mongodb+srv://<db_username>:<db_password>@<clusterName>.<hostname>.mongodb.net

Define a function to generate vector embeddings.

1. Create a directory in your project called common to store common code that you'll use in later steps:

   mkdir common && cd common

2. Create a file named get-embeddings.go and paste the following code. This code defines a function named GetEmbeddings to generate an embedding for a given input. This function specifies:

   • The feature-extraction task using the Go port of the LangChain library. To learn more, see the Tasks documentation in the LangChain JavaScript documentation.

   • The mxbai-embed-large-v1 embedding model.

   get-embeddings.go

   package common
   import (
   	"context"
   	"log"
   	"github.com/tmc/langchaingo/embeddings/huggingface"
   )
   func GetEmbeddings(documents []string) [][]float32 {
   	hf, err := huggingface.NewHuggingface(
   		huggingface.WithModel("mixedbread-ai/mxbai-embed-large-v1"),
   		huggingface.WithTask("feature-extraction"))
   	if err != nil {
   		log.Fatalf("failed to connect to Hugging Face: %v", err)
   	}
   	embs, err := hf.EmbedDocuments(context.Background(), documents)
   	if err != nil {
   		log.Fatalf("failed to generate embeddings: %v", err)
   	}
   	return embs
   }

   Note

   503 when calling Hugging Face models

   You may occasionally get 503 errors when calling Hugging Face model hub models. To resolve this issue, retry after a short delay.

3. Move back into the main project root directory.

   cd ../

Initialize your Go project.

In a terminal window, run the following commands to create a new directory named my-embeddings-project and initialize your project:

mkdir my-embeddings-project
cd my-embeddings-project
go mod init my-embeddings-project

Install and import dependencies.

In a terminal window, run the following commands:

go get github.com/joho/godotenv
go get go.mongodb.org/mongo-driver/mongo
go get github.com/milosgajdos/go-embeddings/openai

Create a .env file to manage secrets.

In your project, create a .env file to store your connection string and OpenAI API token.

OPENAI_API_KEY = "<api-key>"
ATLAS_CONNECTION_STRING = "<connection-string>"

Replace the <api-key> and <connection-string> placeholder values with your OpenAI API key and the SRV connection string for your Atlas cluster. Your connection string should use the following format:

mongodb+srv://<db_username>:<db_password>@<clusterName>.<hostname>.mongodb.net

mongodb+srv://<db_username>:<db_password>@<clusterName>.<hostname>.mongodb.net

Define a function to generate vector embeddings.

1. Create a directory in your project called common to store code you'll use in multiple steps:

   mkdir common && cd common

2. Create a file named get-embeddings.go and paste the following code. This code defines a function named GetEmbeddings that uses OpenAI's text-embedding-3-small model to generate an embedding for a given input.

   get-embeddings.go

   package common
   import (
   	"context"
   	"log"
   	"github.com/milosgajdos/go-embeddings/openai"
   )
   func GetEmbeddings(docs []string) [][]float64 {
   	c := openai.NewClient()
   	embReq := &openai.EmbeddingRequest{
   		Input:          docs,
   		Model:          openai.TextSmallV3,
   		EncodingFormat: openai.EncodingFloat,
   	}
   	embs, err := c.Embed(context.Background(), embReq)
   	if err != nil {
   		log.Fatalf("failed to connect to OpenAI: %v", err)
   	}
   	var vectors [][]float64
   	for _, emb := range embs {
   		vectors = append(vectors, emb.Vector)
   	}
   	return vectors
   }

3. Move back into the main project root directory.

   cd ../

Create your Java project and install dependencies.

1. From your IDE, create a Java project using Maven or Gradle.

2. Add the following dependencies, depending on your package manager:

   Maven

   Gradle

   If you are using Maven, add the following dependencies to the dependencies array in your project's pom.xml file:

   pom.xml

   <dependencies>
      <!-- MongoDB Java Sync Driver v5.2.0 or later -->
      <dependency>
         <groupId>org.mongodb</groupId>
         <artifactId>mongodb-driver-sync</artifactId>
         <version>[5.2.0,)</version>
      </dependency>
      <!-- Java library for working with Hugging Face models -->
      <dependency>
         <groupId>dev.langchain4j</groupId>
         <artifactId>langchain4j-hugging-face</artifactId>
         <version>0.35.0</version>
      </dependency>
   </dependencies>

   If you are using Gradle, add the following to the dependencies array in your project's build.gradle file:

   build.gradle

   dependencies {
      // MongoDB Java Sync Driver v5.2.0 or later
      implementation 'org.mongodb:mongodb-driver-sync:[5.2.0,)'
      // Java library for working with Hugging Face models
      implementation 'dev.langchain4j:langchain4j-hugging-face:0.35.0'
   }

3. Run your package manager to install the dependencies to your project.

Set your environment variables.

In your IDE, create a new configuration template and add the following variables to your project:

• If you are using IntelliJ IDEA, create a new Application run configuration template, then add your variables as semicolon-separated values in the Environment variables field (for example, FOO=123;BAR=456). Apply the changes and click OK.

  To learn more, see the Create a run/debug configuration from a template section of the IntelliJ IDEA documentation.

• If you are using Eclipse, create a new Java Application launch configuration, then add each variable as a new key-value pair in the Environment tab. Apply the changes and click OK.

  To learn more, see the Creating a Java application launch configuration section of the Eclipse IDE documentation.

HUGGING_FACE_ACCESS_TOKEN=<access-token>
ATLAS_CONNECTION_STRING=<connection-string>

Update the placeholders with the following values:

• Replace the``<access-token>`` placeholder value with your Hugging Face access token.

• Replace the <connection-string> placeholder value with the SRV connection string for your Atlas cluster.

  Your connection string should use the following format:

  mongodb+srv://<db_username>:<db_password>@<clusterName>.<hostname>.mongodb.net

Define a method to generate vector embeddings.

Create a file named EmbeddingProvider.java and paste the following code.

This code defines two methods to generate embeddings for a given input using the mxbai-embed-large-v1 open-source embedding model:

• Multiple Inputs: The getEmbeddings method accepts an array of text inputs (List<String>), allowing you to create multiple embeddings in a single API call. The method converts the API-provided arrays of floats to BSON arrays of doubles for storing in your Atlas cluster.

• Single Input: The getEmbedding method accepts a single String, which represents a query you want to make against your vector data. The method converts the API-provided array of floats to a BSON array of doubles to use when querying your collection.

import dev.langchain4j.data.embedding.Embedding;
import dev.langchain4j.data.segment.TextSegment;
import dev.langchain4j.model.huggingface.HuggingFaceEmbeddingModel;
import dev.langchain4j.model.output.Response;
import org.bson.BsonArray;
import org.bson.BsonDouble;
import java.util.List;
import static java.time.Duration.ofSeconds;
public class EmbeddingProvider {
    private static HuggingFaceEmbeddingModel embeddingModel;
    private static HuggingFaceEmbeddingModel getEmbeddingModel() {
        if (embeddingModel == null) {
            String accessToken = System.getenv("HUGGING_FACE_ACCESS_TOKEN");
            if (accessToken == null || accessToken.isEmpty()) {
                throw new RuntimeException("HUGGING_FACE_ACCESS_TOKEN env variable is not set or is empty.");
            }
            embeddingModel = HuggingFaceEmbeddingModel.builder()
                    .accessToken(accessToken)
                    .modelId("mixedbread-ai/mxbai-embed-large-v1")
                    .waitForModel(true)
                    .timeout(ofSeconds(60))
                    .build();
        }
        return embeddingModel;
    }
    /**
     * Takes an array of strings and returns a BSON array of embeddings to
     * store in the database.
     */
    public List<BsonArray> getEmbeddings(List<String> texts) {
        List<TextSegment> textSegments = texts.stream()
                .map(TextSegment::from)
                .toList();
        Response<List<Embedding>> response = getEmbeddingModel().embedAll(textSegments);
        return response.content().stream()
                .map(e -> new BsonArray(
                        e.vectorAsList().stream()
                                .map(BsonDouble::new)
                                .toList()))
                .toList();
    }
    /**
     * Takes a single string and returns a BSON array embedding to
     * use in a vector query.
     */
    public BsonArray getEmbedding(String text) {
        Response<Embedding> response = getEmbeddingModel().embed(text);
        return new BsonArray(
                response.content().vectorAsList().stream()
                        .map(BsonDouble::new)
                        .toList());
    }
}

Create your Java project and install dependencies.

1. From your IDE, create a Java project using Maven or Gradle.

2. Add the following dependencies, depending on your package manager:

   Maven

   Gradle

   If you are using Maven, add the following dependencies to the dependencies array in your project's pom.xml file:

   pom.xml

   <dependencies>
      <!-- MongoDB Java Sync Driver v5.2.0 or later -->
      <dependency>
         <groupId>org.mongodb</groupId>
         <artifactId>mongodb-driver-sync</artifactId>
         <version>[5.2.0,)</version>
      </dependency>
      <!-- Java library for working with OpenAI models -->
      <dependency>
         <groupId>dev.langchain4j</groupId>
         <artifactId>langchain4j-open-ai</artifactId>
         <version>0.35.0</version>
      </dependency>
   </dependencies>

   If you are using Gradle, add the following to the dependencies array in your project's build.gradle file:

   build.gradle

   dependencies {
      // MongoDB Java Sync Driver v5.2.0 or later
      implementation 'org.mongodb:mongodb-driver-sync:[5.2.0,)'
      // Java library for working with OpenAI models
      implementation 'dev.langchain4j:langchain4j-open-ai:0.35.0'
   }

3. Run your package manager to install the dependencies to your project.

Set your environment variables.

In your IDE, create a new configuration template and add the following variables to your project:

• If you are using IntelliJ IDEA, create a new Application run configuration template, then add your variables as semicolon-separated values in the Environment variables field (for example, FOO=123;BAR=456). Apply the changes and click OK.

  To learn more, see the Create a run/debug configuration from a template section of the IntelliJ IDEA documentation.

• If you are using Eclipse, create a new Java Application launch configuration, then add each variable as a new key-value pair in the Environment tab. Apply the changes and click OK.

  To learn more, see the Creating a Java application launch configuration section of the Eclipse IDE documentation.

OPEN_AI_API_KEY=<api-key>
ATLAS_CONNECTION_STRING=<connection-string>

Update the placeholders with the following values:

• Replace the``<api-key>`` placeholder value with your OpenAI API key.

• Replace the <connection-string> placeholder value with the SRV connection string for your Atlas cluster.

  Your connection string should use the following format:

  mongodb+srv://<db_username>:<db_password>@<clusterName>.<hostname>.mongodb.net

Define a method to generate vector embeddings.

Create a file named EmbeddingProvider.java and paste the following code.

This code defines two methods to generate embeddings for a given input using the text-embedding-3-small OpenAI embedding model:

• Multiple Inputs: The getEmbeddings method accepts an array of text inputs (List<String>), allowing you to create multiple embeddings in a single API call. The method converts the API-provided arrays of floats to BSON arrays of doubles for storing in your Atlas cluster.

• Single Input: The getEmbedding method accepts a single String, which represents a query you want to make against your vector data. The method converts the API-provided array of floats to a BSON array of doubles to use when querying your collection.

import dev.langchain4j.data.embedding.Embedding;
import dev.langchain4j.data.segment.TextSegment;
import dev.langchain4j.model.openai.OpenAiEmbeddingModel;
import dev.langchain4j.model.output.Response;
import org.bson.BsonArray;
import org.bson.BsonDouble;
import java.util.List;
import static java.time.Duration.ofSeconds;
public class EmbeddingProvider {
    private static OpenAiEmbeddingModel embeddingModel;
    private static OpenAiEmbeddingModel getEmbeddingModel() {
        if (embeddingModel == null) {
            String apiKey = System.getenv("OPEN_AI_API_KEY");
            if (apiKey == null || apiKey.isEmpty()) {
                throw new IllegalStateException("OPEN_AI_API_KEY env variable is not set or is empty.");
            }
            return OpenAiEmbeddingModel.builder()
                    .apiKey(apiKey)
                    .modelName("text-embedding-3-small")
                    .timeout(ofSeconds(60))
                    .build();
        }
        return embeddingModel;
    }
    /**
     * Takes an array of strings and returns a BSON array of embeddings to
     * store in the database.
     */
    public List<BsonArray> getEmbeddings(List<String> texts) {
        List<TextSegment> textSegments = texts.stream()
                .map(TextSegment::from)
                .toList();
        Response<List<Embedding>> response = getEmbeddingModel().embedAll(textSegments);
        return response.content().stream()
                .map(e -> new BsonArray(
                        e.vectorAsList().stream()
                                .map(BsonDouble::new)
                                .toList()))
                .toList();
    }
    /**
     * Takes a single string and returns a BSON array embedding to
     * use in a vector query.
     */
    public BsonArray getEmbedding(String text) {
        Response<Embedding> response = getEmbeddingModel().embed(text);
        return new BsonArray(
                response.content().vectorAsList().stream()
                        .map(BsonDouble::new)
                        .toList());
    }
}

Initialize your Node.js project.

In a terminal window, run the following commands to create a new directory named my-embeddings-project and initialize your project:

mkdir my-embeddings-project
cd my-embeddings-project
npm init -y

Update your package.json file.

Configure your project to use ES modules by adding "type": "module" to your package.json file and then saving it.

{
  "type": "module",
  // other fields...
}

Install and import dependencies.

In a terminal window, run the following command:

npm install mongodb @xenova/transformers

Create a .env file.

In your project, create a .env file to store your Atlas connection string.

ATLAS_CONNECTION_STRING = "<connection-string>"

Replace the <connection-string> placeholder value with the SRV connection string for your Atlas cluster. Your connection string should use the following format:

mongodb+srv://<db_username>:<db_password>@<clusterName>.<hostname>.mongodb.net

Define a function to generate vector embeddings.

Create a file named get-embeddings.js and paste the following code. This code defines a function named to generate an embedding for a given input. This function specifies:

• The feature-extraction task from Hugging Face's transformers.js library. To learn more, see Tasks.

• The nomic-embed-text-v1 embedding model.

import { pipeline } from '@xenova/transformers';
// Function to generate embeddings for a given data source
export async function getEmbedding(data) {
    const embedder = await pipeline(
        'feature-extraction', 
        'Xenova/nomic-embed-text-v1');
    const results = await embedder(data, { pooling: 'mean', normalize: true });
    return Array.from(results.data);
}

Initialize your Node.js project.

In a terminal window, run the following commands to create a new directory named my-embeddings-project and initialize your project:

mkdir my-embeddings-project
cd my-embeddings-project
npm init -y

Update your package.json file.

Configure your project to use ES modules by adding "type": "module" to your package.json file and then saving it.

{
  "type": "module",
  // other fields...
}

Install and import dependencies.

In a terminal window, run the following command:

npm install mongodb openai

Create a .env file.

In your project, create a .env file to store your Atlas connection string and OpenAI API key.

OPENAI_API_KEY = "<api-key>"
ATLAS_CONNECTION_STRING = "<connection-string>"

Replace the <api-key> and <connection-string> placeholder values with your OpenAI API key and the SRV connection string for your Atlas cluster. Your connection string should use the following format:

mongodb+srv://<db_username>:<db_password>@<clusterName>.<hostname>.mongodb.net

Define a function to generate vector embeddings.

Create a file named get-embeddings.js and paste the following code. This code defines a function named getEmbedding that uses OpenAI's text-embedding-3-small model to generate an embedding for a given input.

import OpenAI from 'openai';
// Setup OpenAI configuration
const openai = new OpenAI({apiKey: process.env.OPENAI_API_KEY});
// Function to get the embeddings using the OpenAI API
export async function getEmbedding(text) {
    const results = await openai.embeddings.create({
        model: "text-embedding-3-small",
        input: text,
        encoding_format: "float",
    });
    return results.data[0].embedding;
}

Set up the environment.

Create an interactive Python notebook by saving a file with the .ipynb extension, and then run the following command in the notebook to install the dependencies:

pip install --quiet sentence-transformers pymongo einops

Define a function to generate vector embeddings.

Paste and run the following code in your notebook to create a function that generates vector embeddings by using an open-source embedding model from Nomic AI. This code does the following:

• Loads the nomic-embed-text-v1 embedding model.

• Creates a function named get_embedding that uses the model to generate an embedding for a given text input.

• Tests the function by generating a single embedding for the string foo.

from sentence_transformers import SentenceTransformer
# Load the embedding model (https://huggingface.co/nomic-ai/nomic-embed-text-v1")
model = SentenceTransformer("nomic-ai/nomic-embed-text-v1", trust_remote_code=True)
# Define a function to generate embeddings
def get_embedding(data):
   """Generates vector embeddings for the given data."""
   embedding = model.encode(data)
   return embedding.tolist()
# Generate an embedding
get_embedding("foo")

[-0.029808253049850464, 0.03841473162174225, -0.02561120130121708, -0.06707508116960526, 0.03867151960730553, ... ]

Set up the environment.

Create an interactive Python notebook by saving a file with the .ipynb extension, and then run the following command in the notebook to install the dependencies:

pip install --quiet openai pymongo

Define a function to generate vector embeddings.

Paste and run the following code in your notebook to create a function that generates vector embeddings by using a proprietary embedding model from OpenAI. Replace <api-key> with your OpenAI API key. This code does the following:

• Specifies the text-embedding-3-small embedding model.

• Creates a function named get_embedding that calls the model's API to generate an embedding for a given text input.

• Tests the function by generating a single embedding for the string foo.

import os
from openai import OpenAI
# Specify your OpenAI API key and embedding model
os.environ["OPENAI_API_KEY"] = "<api-key>"
model = "text-embedding-3-small"
openai_client = OpenAI()
# Define a function to generate embeddings
def get_embedding(text):
   """Generates vector embeddings for the given text."""
   embedding = openai_client.embeddings.create(input = [text], model=model).data[0].embedding
   return embedding
# Generate an embedding
get_embedding("foo")

[-0.005843308754265308, -0.013111298903822899, -0.014585349708795547, 0.03580040484666824, 0.02671629749238491, ... ]

Create a file named create-embeddings.go and paste the following code.

Use the following code to generate embeddings from an existing collection in Atlas.

Specifically, this code uses the GetEmbeddings function that you defined and the MongoDB Go Driver to generate embeddings from an array of sample texts and ingest them into the sample_db.embeddings collection in Atlas.

package main
import (
	"context"
	"fmt"
	"log"
	"my-embeddings-project/common"
	"os"
	"github.com/joho/godotenv"
	"go.mongodb.org/mongo-driver/mongo"
	"go.mongodb.org/mongo-driver/mongo/options"
)
var data = []string{
	"Titanic: The story of the 1912 sinking of the largest luxury liner ever built",
	"The Lion King: Lion cub and future king Simba searches for his identity",
	"Avatar: A marine is dispatched to the moon Pandora on a unique mission",
}
type TextWithEmbedding struct {
	Text      string
	Embedding []float32
}
func main() {
	ctx := context.Background()
	if err := godotenv.Load(); err != nil {
		log.Println("no .env file found")
	}
	// Connect to your Atlas cluster
	uri := os.Getenv("ATLAS_CONNECTION_STRING")
	if uri == "" {
		log.Fatal("set your 'ATLAS_CONNECTION_STRING' environment variable.")
	}
	clientOptions := options.Client().ApplyURI(uri)
	client, err := mongo.Connect(ctx, clientOptions)
	if err != nil {
		log.Fatalf("failed to connect to the server: %v", err)
	}
	defer func() { _ = client.Disconnect(ctx) }()
	// Set the namespace
	coll := client.Database("sample_db").Collection("embeddings")
	embeddings := common.GetEmbeddings(data)
	docsToInsert := make([]interface{}, len(embeddings))
	for i, string := range data {
		docsToInsert[i] = TextWithEmbedding{
			Text:      string,
			Embedding: embeddings[i],
		}
	}
	result, err := coll.InsertMany(ctx, docsToInsert)
	if err != nil {
		log.Fatalf("failed to insert documents: %v", err)
	}
	fmt.Printf("Successfully inserted %v documents into Atlas\n", len(result.InsertedIDs))
}

package main
import (
	"context"
	"fmt"
	"log"
	"my-embeddings-project/common"
	"os"
	"github.com/joho/godotenv"
	"go.mongodb.org/mongo-driver/mongo"
	"go.mongodb.org/mongo-driver/mongo/options"
)
var data = []string{
	"Titanic: The story of the 1912 sinking of the largest luxury liner ever built",
	"The Lion King: Lion cub and future king Simba searches for his identity",
	"Avatar: A marine is dispatched to the moon Pandora on a unique mission",
}
type TextWithEmbedding struct {
	Text      string
	Embedding []float64
}
func main() {
	ctx := context.Background()
	if err := godotenv.Load(); err != nil {
		log.Println("no .env file found")
	}
	// Connect to your Atlas cluster
	uri := os.Getenv("ATLAS_CONNECTION_STRING")
	if uri == "" {
		log.Fatal("set your 'ATLAS_CONNECTION_STRING' environment variable.")
	}
	clientOptions := options.Client().ApplyURI(uri)
	client, err := mongo.Connect(ctx, clientOptions)
	if err != nil {
		log.Fatalf("failed to connect to the server: %v", err)
	}
	defer func() { _ = client.Disconnect(ctx) }()
	// Set the namespace
	coll := client.Database("sample_db").Collection("embeddings")
	embeddings := common.GetEmbeddings(data)
	docsToInsert := make([]interface{}, len(data))
	for i, movie := range data {
		docsToInsert[i] = TextWithEmbedding{
			Text:      movie,
			Embedding: embeddings[i],
		}
	}
	result, err := coll.InsertMany(ctx, docsToInsert)
	if err != nil {
		log.Fatalf("failed to insert documents: %v", err)
	}
	fmt.Printf("Successfully inserted %v documents into Atlas\n", len(result.InsertedIDs))
}

Save and run the file.

go run create-embeddings.go

Successfully inserted 3 documents into Atlas

go run create-embeddings.go

Successfully inserted 3 documents into Atlas

You can also view your vector embeddings in the Atlas UI by navigating to the sample_db.embeddings collection in your cluster.

Create a file named create-embeddings.go and paste the following code.

Use the following code to generate embeddings from an existing collection in Atlas. Specifically, this code does the following:

• Connects to your Atlas cluster.

• Gets a subset of documents from the sample_airbnb.listingsAndReviews collection that have a non-empty summary field.

• Generates embeddings from each document's summary field by using the GetEmbeddings function that you defined.

• Updates each document with a new embeddings field that contains the embedding value by using the MongoDB Go Driver.

package main
import (
	"context"
	"log"
	"my-embeddings-project/common"
	"os"
	"github.com/joho/godotenv"
	"go.mongodb.org/mongo-driver/bson"
	"go.mongodb.org/mongo-driver/mongo"
	"go.mongodb.org/mongo-driver/mongo/options"
)
func main() {
	ctx := context.Background()
	if err := godotenv.Load(); err != nil {
		log.Println("no .env file found")
	}
	// Connect to your Atlas cluster
	uri := os.Getenv("ATLAS_CONNECTION_STRING")
	if uri == "" {
		log.Fatal("set your 'ATLAS_CONNECTION_STRING' environment variable.")
	}
	clientOptions := options.Client().ApplyURI(uri)
	client, err := mongo.Connect(ctx, clientOptions)
	if err != nil {
		log.Fatalf("failed to connect to the server: %v", err)
	}
	defer func() { _ = client.Disconnect(ctx) }()
	// Set the namespace
	coll := client.Database("sample_airbnb").Collection("listingsAndReviews")
	filter := bson.D{
		{"$and",
			bson.A{
				bson.D{
					{"$and",
						bson.A{
							bson.D{{"summary", bson.D{{"$exists", true}}}},
							bson.D{{"summary", bson.D{{"$ne", ""}}}},
						},
					}},
				bson.D{{"embeddings", bson.D{{"$exists", false}}}},
			}},
	}
	opts := options.Find().SetLimit(50)
	cursor, err := coll.Find(ctx, filter, opts)
	if err != nil {
		log.Fatalf("failed to retrieve documents: %v", err)
	}
	var listings []common.Listing
	if err = cursor.All(ctx, &listings); err != nil {
		log.Fatalf("failed to unmarshal retrieved documents to Listing object: %v", err)
	}
	var summaries []string
	for _, listing := range listings {
		summaries = append(summaries, listing.Summary)
	}
	log.Println("Generating embeddings.")
	embeddings := common.GetEmbeddings(summaries)
	docsToUpdate := make([]mongo.WriteModel, len(listings))
	for i := range listings {
		docsToUpdate[i] = mongo.NewUpdateOneModel().
			SetFilter(bson.D{{"_id", listings[i].ID}}).
			SetUpdate(bson.D{{"$set", bson.D{{"embeddings", embeddings[i]}}}})
	}
	bulkWriteOptions := options.BulkWrite().SetOrdered(false)
	result, err := coll.BulkWrite(context.Background(), docsToUpdate, bulkWriteOptions)
	if err != nil {
		log.Fatalf("failed to write embeddings to existing documents: %v", err)
	}
	log.Printf("Successfully added embeddings to %v documents", result.ModifiedCount)
}

Create a file that contains Go models for the collection.

To simplify marshalling and unmarshalling Go objects to and from BSON, create a file that contains models for the documents in this collection.

1. Move into the common directory.

   cd common

2. Create a file named models.go, and paste the following code into it:

   Open-Source

   OpenAI

   models.go

   package common
   import (
   	"time"
   	"go.mongodb.org/mongo-driver/bson/primitive"
   )
   type Image struct {
   	ThumbnailURL string `bson:"thumbnail_url"`
   	MediumURL    string `bson:"medium_url"`
   	PictureURL   string `bson:"picture_url"`
   	XLPictureURL string `bson:"xl_picture_url"`
   }
   type Host struct {
   	ID                 string   `bson:"host_id"`
   	URL                string   `bson:"host_url"`
   	Name               string   `bson:"host_name"`
   	Location           string   `bson:"host_location"`
   	About              string   `bson:"host_about"`
   	ThumbnailURL       string   `bson:"host_thumbnail_url"`
   	PictureURL         string   `bson:"host_picture_url"`
   	Neighborhood       string   `bson:"host_neighborhood"`
   	IsSuperhost        bool     `bson:"host_is_superhost"`
   	HasProfilePic      bool     `bson:"host_has_profile_pic"`
   	IdentityVerified   bool     `bson:"host_identity_verified"`
   	ListingsCount      int32    `bson:"host_listings_count"`
   	TotalListingsCount int32    `bson:"host_total_listings_count"`
   	Verifications      []string `bson:"host_verifications"`
   }
   type Location struct {
   	Type            string    `bson:"type"`
   	Coordinates     []float64 `bson:"coordinates"`
   	IsLocationExact bool      `bson:"is_location_exact"`
   }
   type Address struct {
   	Street         string   `bson:"street"`
   	Suburb         string   `bson:"suburb"`
   	GovernmentArea string   `bson:"government_area"`
   	Market         string   `bson:"market"`
   	Country        string   `bson:"Country"`
   	CountryCode    string   `bson:"country_code"`
   	Location       Location `bson:"location"`
   }
   type Availability struct {
   	Thirty         int32 `bson:"availability_30"`
   	Sixty          int32 `bson:"availability_60"`
   	Ninety         int32 `bson:"availability_90"`
   	ThreeSixtyFive int32 `bson:"availability_365"`
   }
   type ReviewScores struct {
   	Accuracy      int32 `bson:"review_scores_accuracy"`
   	Cleanliness   int32 `bson:"review_scores_cleanliness"`
   	CheckIn       int32 `bson:"review_scores_checkin"`
   	Communication int32 `bson:"review_scores_communication"`
   	Location      int32 `bson:"review_scores_location"`
   	Value         int32 `bson:"review_scores_value"`
   	Rating        int32 `bson:"review_scores_rating"`
   }
   type Review struct {
   	ID           string    `bson:"_id"`
   	Date         time.Time `bson:"date,omitempty"`
   	ListingId    string    `bson:"listing_id"`
   	ReviewerId   string    `bson:"reviewer_id"`
   	ReviewerName string    `bson:"reviewer_name"`
   	Comments     string    `bson:"comments"`
   }
   type Listing struct {
   	ID                   string               `bson:"_id"`
   	ListingURL           string               `bson:"listing_url"`
   	Name                 string               `bson:"name"`
   	Summary              string               `bson:"summary"`
   	Space                string               `bson:"space"`
   	Description          string               `bson:"description"`
   	NeighborhoodOverview string               `bson:"neighborhood_overview"`
   	Notes                string               `bson:"notes"`
   	Transit              string               `bson:"transit"`
   	Access               string               `bson:"access"`
   	Interaction          string               `bson:"interaction"`
   	HouseRules           string               `bson:"house_rules"`
   	PropertyType         string               `bson:"property_type"`
   	RoomType             string               `bson:"room_type"`
   	BedType              string               `bson:"bed_type"`
   	MinimumNights        string               `bson:"minimum_nights"`
   	MaximumNights        string               `bson:"maximum_nights"`
   	CancellationPolicy   string               `bson:"cancellation_policy"`
   	LastScraped          time.Time            `bson:"last_scraped,omitempty"`
   	CalendarLastScraped  time.Time            `bson:"calendar_last_scraped,omitempty"`
   	FirstReview          time.Time            `bson:"first_review,omitempty"`
   	LastReview           time.Time            `bson:"last_review,omitempty"`
   	Accommodates         int32                `bson:"accommodates"`
   	Bedrooms             int32                `bson:"bedrooms"`
   	Beds                 int32                `bson:"beds"`
   	NumberOfReviews      int32                `bson:"number_of_reviews"`
   	Bathrooms            primitive.Decimal128 `bson:"bathrooms"`
   	Amenities            []string             `bson:"amenities"`
   	Price                primitive.Decimal128 `bson:"price"`
   	WeeklyPrice          primitive.Decimal128 `bson:"weekly_price"`
   	MonthlyPrice         primitive.Decimal128 `bson:"monthly_price"`
   	CleaningFee          primitive.Decimal128 `bson:"cleaning_fee"`
   	ExtraPeople          primitive.Decimal128 `bson:"extra_people"`
   	GuestsIncluded       primitive.Decimal128 `bson:"guests_included"`
   	Image                Image                `bson:"images"`
   	Host                 Host                 `bson:"host"`
   	Address              Address              `bson:"address"`
   	Availability         Availability         `bson:"availability"`
   	ReviewScores         ReviewScores         `bson:"review_scores"`
   	Reviews              []Review             `bson:"reviews"`
   	Embeddings           []float32            `bson:"embeddings,omitempty"`
   }

   models.go

   package common
   import (
   	"time"
   	"go.mongodb.org/mongo-driver/bson/primitive"
   )
   type Image struct {
   	ThumbnailURL string `bson:"thumbnail_url"`
   	MediumURL    string `bson:"medium_url"`
   	PictureURL   string `bson:"picture_url"`
   	XLPictureURL string `bson:"xl_picture_url"`
   }
   type Host struct {
   	ID                 string   `bson:"host_id"`
   	URL                string   `bson:"host_url"`
   	Name               string   `bson:"host_name"`
   	Location           string   `bson:"host_location"`
   	About              string   `bson:"host_about"`
   	ThumbnailURL       string   `bson:"host_thumbnail_url"`
   	PictureURL         string   `bson:"host_picture_url"`
   	Neighborhood       string   `bson:"host_neighborhood"`
   	IsSuperhost        bool     `bson:"host_is_superhost"`
   	HasProfilePic      bool     `bson:"host_has_profile_pic"`
   	IdentityVerified   bool     `bson:"host_identity_verified"`
   	ListingsCount      int32    `bson:"host_listings_count"`
   	TotalListingsCount int32    `bson:"host_total_listings_count"`
   	Verifications      []string `bson:"host_verifications"`
   }
   type Location struct {
   	Type            string    `bson:"type"`
   	Coordinates     []float64 `bson:"coordinates"`
   	IsLocationExact bool      `bson:"is_location_exact"`
   }
   type Address struct {
   	Street         string   `bson:"street"`
   	Suburb         string   `bson:"suburb"`
   	GovernmentArea string   `bson:"government_area"`
   	Market         string   `bson:"market"`
   	Country        string   `bson:"Country"`
   	CountryCode    string   `bson:"country_code"`
   	Location       Location `bson:"location"`
   }
   type Availability struct {
   	Thirty         int32 `bson:"availability_30"`
   	Sixty          int32 `bson:"availability_60"`
   	Ninety         int32 `bson:"availability_90"`
   	ThreeSixtyFive int32 `bson:"availability_365"`
   }
   type ReviewScores struct {
   	Accuracy      int32 `bson:"review_scores_accuracy"`
   	Cleanliness   int32 `bson:"review_scores_cleanliness"`
   	CheckIn       int32 `bson:"review_scores_checkin"`
   	Communication int32 `bson:"review_scores_communication"`
   	Location      int32 `bson:"review_scores_location"`
   	Value         int32 `bson:"review_scores_value"`
   	Rating        int32 `bson:"review_scores_rating"`
   }
   type Review struct {
   	ID           string    `bson:"_id"`
   	Date         time.Time `bson:"date,omitempty"`
   	ListingId    string    `bson:"listing_id"`
   	ReviewerId   string    `bson:"reviewer_id"`
   	ReviewerName string    `bson:"reviewer_name"`
   	Comments     string    `bson:"comments"`
   }
   type Listing struct {
   	ID                   string               `bson:"_id"`
   	ListingURL           string               `bson:"listing_url"`
   	Name                 string               `bson:"name"`
   	Summary              string               `bson:"summary"`
   	Space                string               `bson:"space"`
   	Description          string               `bson:"description"`
   	NeighborhoodOverview string               `bson:"neighborhood_overview"`
   	Notes                string               `bson:"notes"`
   	Transit              string               `bson:"transit"`
   	Access               string               `bson:"access"`
   	Interaction          string               `bson:"interaction"`
   	HouseRules           string               `bson:"house_rules"`
   	PropertyType         string               `bson:"property_type"`
   	RoomType             string               `bson:"room_type"`
   	BedType              string               `bson:"bed_type"`
   	MinimumNights        string               `bson:"minimum_nights"`
   	MaximumNights        string               `bson:"maximum_nights"`
   	CancellationPolicy   string               `bson:"cancellation_policy"`
   	LastScraped          time.Time            `bson:"last_scraped,omitempty"`
   	CalendarLastScraped  time.Time            `bson:"calendar_last_scraped,omitempty"`
   	FirstReview          time.Time            `bson:"first_review,omitempty"`
   	LastReview           time.Time            `bson:"last_review,omitempty"`
   	Accommodates         int32                `bson:"accommodates"`
   	Bedrooms             int32                `bson:"bedrooms"`
   	Beds                 int32                `bson:"beds"`
   	NumberOfReviews      int32                `bson:"number_of_reviews"`
   	Bathrooms            primitive.Decimal128 `bson:"bathrooms"`
   	Amenities            []string             `bson:"amenities"`
   	Price                primitive.Decimal128 `bson:"price"`
   	WeeklyPrice          primitive.Decimal128 `bson:"weekly_price"`
   	MonthlyPrice         primitive.Decimal128 `bson:"monthly_price"`
   	CleaningFee          primitive.Decimal128 `bson:"cleaning_fee"`
   	ExtraPeople          primitive.Decimal128 `bson:"extra_people"`
   	GuestsIncluded       primitive.Decimal128 `bson:"guests_included"`
   	Image                Image                `bson:"images"`
   	Host                 Host                 `bson:"host"`
   	Address              Address              `bson:"address"`
   	Availability         Availability         `bson:"availability"`
   	ReviewScores         ReviewScores         `bson:"review_scores"`
   	Reviews              []Review             `bson:"reviews"`
   	Embeddings           []float64            `bson:"embeddings,omitempty"`
   }

3. Move back into the project root directory.

   cd ../

Generate embeddings.

go run create-embeddings.go

2024/10/10 09:58:03 Generating embeddings.
2024/10/10 09:58:12 Successfully added embeddings to 50 documents

You can view your vector embeddings as they generate by navigating to the sample_airbnb.listingsAndReviews collection in the Atlas UI.

Define code to generate embeddings from an existing collection in Atlas.

Create a file named CreateEmbeddings.java and paste the following code.

This code uses the getEmbeddings method and the MongoDB Java Sync Driver to do the following:

1. Connect to your Atlas cluster.

2. Get the array of sample texts.

3. Generate embeddings from each text using the getEmbeddings method that you defined previously.

4. Ingest the embeddings into the sample_db.embeddings collection in Atlas.

import com.mongodb.MongoException;
import com.mongodb.client.MongoClient;
import com.mongodb.client.MongoClients;
import com.mongodb.client.MongoCollection;
import com.mongodb.client.MongoDatabase;
import com.mongodb.client.result.InsertManyResult;
import org.bson.BsonArray;
import org.bson.Document;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
public class CreateEmbeddings {
    static List<String> data = Arrays.asList(
            "Titanic: The story of the 1912 sinking of the largest luxury liner ever built",
            "The Lion King: Lion cub and future king Simba searches for his identity",
            "Avatar: A marine is dispatched to the moon Pandora on a unique mission"
    );
    public static void main(String[] args){
        String uri = System.getenv("ATLAS_CONNECTION_STRING");
        if (uri == null || uri.isEmpty()) {
            throw new RuntimeException("ATLAS_CONNECTION_STRING env variable is not set or is empty.");
        }
        // establish connection and set namespace
        try (MongoClient mongoClient = MongoClients.create(uri)) {
            MongoDatabase database = mongoClient.getDatabase("sample_db");
            MongoCollection<Document> collection = database.getCollection("embeddings");
            System.out.println("Creating embeddings for " + data.size() + " documents");
            EmbeddingProvider embeddingProvider = new EmbeddingProvider();
            // generate embeddings for new inputted data
            List<BsonArray> embeddings = embeddingProvider.getEmbeddings(data);
            List<Document> documents = new ArrayList<>();
            int i = 0;
            for (String text : data) {
                Document doc = new Document("text", text).append("embedding", embeddings.get(i));
                documents.add(doc);
                i++;
            }
            // insert the embeddings into the Atlas collection
            List<String> insertedIds = new ArrayList<>();
            try {
                InsertManyResult result = collection.insertMany(documents);
                result.getInsertedIds().values()
                        .forEach(doc -> insertedIds.add(doc.toString()));
                System.out.println("Inserted " + insertedIds.size() + " documents with the following ids to " + collection.getNamespace() + " collection: \n " + insertedIds);
            } catch (MongoException me) {
                throw new RuntimeException("Failed to insert documents", me);
            }
        } catch (MongoException me) {
            throw new RuntimeException("Failed to connect to MongoDB ", me);
        } catch (Exception e) {
            throw new RuntimeException("Operation failed: ", e);
        }
    }
}

Generate the embeddings.

Save and run the file. The output resembles:

Creating embeddings for 3 documents
Inserted 3 documents with the following ids to sample_db.embeddings collection:
 [BsonObjectId{value=6735ff620d88451041f6dd40}, BsonObjectId{value=6735ff620d88451041f6dd41}, BsonObjectId{value=6735ff620d88451041f6dd42}]

You can also view your vector embeddings in the Atlas UI by navigating to the sample_db.embeddings collection in your cluster.

Define code to generate embeddings from an existing collection in Atlas.

Create a file named CreateEmbeddings.java and paste the following code.

This code uses the getEmbeddings method and the MongoDB Java Sync Driver to do the following:

1. Connect to your Atlas cluster.

2. Get a subset of documents from the sample_airbnb.listingsAndReviews collection that have a non-empty summary field.

3. Generate embeddings from each document's summary field using the getEmbeddings method that you defined previously.

4. Update each document with a new embeddings field that contains the embedding value.

import com.mongodb.MongoException;
import com.mongodb.bulk.BulkWriteResult;
import com.mongodb.client.MongoClient;
import com.mongodb.client.MongoClients;
import com.mongodb.client.MongoCollection;
import com.mongodb.client.MongoCursor;
import com.mongodb.client.MongoDatabase;
import com.mongodb.client.model.BulkWriteOptions;
import com.mongodb.client.model.Filters;
import com.mongodb.client.model.UpdateOneModel;
import com.mongodb.client.model.Updates;
import com.mongodb.client.model.WriteModel;
import org.bson.BsonArray;
import org.bson.Document;
import org.bson.conversions.Bson;
import java.util.ArrayList;
import java.util.List;
public class CreateEmbeddings {
    public static void main(String[] args){
        String uri = System.getenv("ATLAS_CONNECTION_STRING");
        if (uri == null || uri.isEmpty()) {
            throw new RuntimeException("ATLAS_CONNECTION_STRING env variable is not set or is empty.");
        }
        // establish connection and set namespace
        try (MongoClient mongoClient = MongoClients.create(uri)) {
            MongoDatabase database = mongoClient.getDatabase("sample_airbnb");
            MongoCollection<Document> collection = database.getCollection("listingsAndReviews");
            Bson filterCriteria = Filters.and(
                    Filters.and(Filters.exists("summary"),
                            Filters.ne("summary", null),
                            Filters.ne("summary", "")),
                    Filters.exists("embeddings", false));
            try (MongoCursor<Document> cursor = collection.find(filterCriteria).limit(50).iterator()) {
                List<String> summaries = new ArrayList<>();
                List<String> documentIds = new ArrayList<>();
                int i = 0;
                while (cursor.hasNext()) {
                    Document document = cursor.next();
                    String summary = document.getString("summary");
                    String id = document.get("_id").toString();
                    summaries.add(summary);
                    documentIds.add(id);
                    i++;
                }
                System.out.println("Generating embeddings for " + summaries.size() + " documents.");
                System.out.println("This operation may take up to several minutes.");
                EmbeddingProvider embeddingProvider = new EmbeddingProvider();
                List<BsonArray> embeddings = embeddingProvider.getEmbeddings(summaries);
                List<WriteModel<Document>> updateDocuments = new ArrayList<>();
                for (int j = 0; j < summaries.size(); j++) {
                    UpdateOneModel<Document> updateDoc = new UpdateOneModel<>(
                            Filters.eq("_id", documentIds.get(j)),
                            Updates.set("embeddings", embeddings.get(j)));
                    updateDocuments.add(updateDoc);
                }
                int updatedDocsCount = 0;
                try {
                    BulkWriteOptions options = new BulkWriteOptions().ordered(false);
                    BulkWriteResult result = collection.bulkWrite(updateDocuments, options);
                    updatedDocsCount = result.getModifiedCount();
                } catch (MongoException me) {
                    throw new RuntimeException("Failed to insert documents", me);
                }
                System.out.println("Added embeddings successfully to " + updatedDocsCount + " documents.");
            }
        } catch (MongoException me) {
            throw new RuntimeException("Failed to connect to MongoDB", me);
        } catch (Exception e) {
            throw new RuntimeException("Operation failed: ", e);
        }
    }
}

Generate the embeddings.

Save and run the file. The output resembles:

Generating embeddings for 50 documents.
This operation may take up to several minutes.
Added embeddings successfully to 50 documents.

You can also view your vector embeddings in the Atlas UI by navigating to the sample_airbnb.listingsAndReviews collection in your cluster.

Create a file named create-embeddings.js and paste the following code.

Use the following code to generate embeddings from an existing collection in Atlas.

Specifically, this code uses the getEmbedding function that you defined and the MongoDB Node.js Driver to generate embeddings from an array of sample texts and ingest them into the sample_db.embeddings collection in Atlas.

import { MongoClient } from 'mongodb';
import { getEmbedding } from './get-embeddings.js';
// Data to embed
const data = [ 
    "Titanic: The story of the 1912 sinking of the largest luxury liner ever built",
    "The Lion King: Lion cub and future king Simba searches for his identity",
    "Avatar: A marine is dispatched to the moon Pandora on a unique mission"
]
async function run() {
    // Connect to your Atlas cluster
    const client = new MongoClient(process.env.ATLAS_CONNECTION_STRING);
    
    try {
        await client.connect();
        const db = client.db("sample_db");
        const collection = db.collection("embeddings");
        await Promise.all(data.map(async text => {
            // Check if the document already exists
            const existingDoc = await collection.findOne({ text: text });
            // Generate an embedding by using the function that you defined
            const embedding = await getEmbedding(text);
            // Ingest data and embedding into Atlas
            if (!existingDoc) {
                await collection.insertOne({
                    text: text,
                    embedding: embedding
                });
                console.log(embedding);
            }
        }));
    } catch (err) {
        console.log(err.stack);
    }
    finally {
        await client.close();
    }
}
run().catch(console.dir);

Save and run the file.

node --env-file=.env create-embeddings.js

[ -0.04323853924870491, -0.008460805751383305, 0.012494648806750774, -0.013014335185289383, ... ]
[ -0.017400473356246948, 0.04922063276171684, -0.002836339408531785, -0.030395228415727615, ... ]
[ -0.016950927674770355, 0.013881809078156948, -0.022074559703469276, -0.02838018536567688, ... ]

node --env-file=.env create-embeddings.js

[ 0.031927742, -0.014192767, -0.021851597, 0.045498233, -0.0077904654, ... ]
[ -0.01664538, 0.013198251, 0.048684783, 0.014485021, -0.018121032, ... ]
[ 0.030449908, 0.046782598, 0.02126599, 0.025799986, -0.015830345, ... ]

You can also view your vector embeddings in the Atlas UI by navigating to the sample_db.embeddings collection in your cluster.

Create a file named create-embeddings.js and paste the following code.

Use the following code to generate embeddings from an existing collection in Atlas. Specifically, this code does the following:

• Connects to your Atlas cluster.

• Gets a subset of documents from the sample_airbnb.listingsAndReviews collection that have a non-empty summary field.

• Generates embeddings from each document's summary field by using the getEmbedding function that you defined.

• Updates each document with a new embedding field that contains the embedding value by using the MongoDB Node.js Driver.

import { MongoClient } from 'mongodb';
import { getEmbedding } from './get-embeddings.js';
// Connect to your Atlas cluster
const client = new MongoClient(process.env.ATLAS_CONNECTION_STRING);
async function run() {
    try {
        await client.connect();
        const db = client.db("sample_airbnb");
        const collection = db.collection("listingsAndReviews");
        // Filter to exclude null or empty summary fields
        const filter = { "summary": { "$nin": [ null, "" ] } };
        // Get a subset of documents from the collection
        const documents = await collection.find(filter).limit(50).toArray();
        // Create embeddings from a field in the collection
        let updatedDocCount = 0;
        console.log("Generating embeddings for documents...");
        await Promise.all(documents.map(async doc => {
           // Generate an embedding by using the function that you defined
           const embedding = await getEmbedding(doc.summary);
           // Update the document with a new embedding field
           await collection.updateOne({ "_id": doc._id },
               {
                   "$set": {
                       "embedding": embedding
                   }
               }
           );
           updatedDocCount += 1;
       }));
       console.log("Count of documents updated: " + updatedDocCount);
            
    } catch (err) {
        console.log(err.stack);
    }
    finally {
        await client.close();
    }
}
run().catch(console.dir);

Save and run the file.

node --env-file=.env create-embeddings.js

Generating embeddings for documents...
Count of documents updated: 50

You can view your vector embeddings as they generate by navigating to the sample_airbnb.listingsAndReviews collection in the Atlas UI and expanding the fields in a document.

Paste the following code in your notebook.

Use the following code to generate embeddings from new data.

Specifically, this code uses the get_embedding function that you defined and the MongoDB PyMongo Driver to generate embeddings from an array of sample texts and ingest them into the sample_db.embeddings collection.

import pymongo
# Connect to your Atlas cluster
mongo_client = pymongo.MongoClient("<connection-string>")
db = mongo_client["sample_db"]
collection = db["embeddings"]
# Sample data
data = [
   "Titanic: The story of the 1912 sinking of the largest luxury liner ever built",
   "The Lion King: Lion cub and future king Simba searches for his identity",
   "Avatar: A marine is dispatched to the moon Pandora on a unique mission"
]
# Ingest data into Atlas
inserted_doc_count = 0
for text in data:
   embedding = get_embedding(text)
   collection.insert_one({ "text": text, "embedding": embedding })
   inserted_doc_count += 1
print(f"Inserted {inserted_doc_count} documents.")

Inserted 3 documents.

Specify the connection string.

Replace <connection-string> with your Atlas cluster's SRV connection string.

mongodb+srv://<db_username>:<db_password>@<clusterName>.<hostname>.mongodb.net

Run the code.

You can verify your vector embeddings by viewing them in the Atlas UI by navigating to the sample_db.embeddings collection in your cluster.

Paste the following code in your notebook.

Use the following code to generate embeddings from a field in an existing collection. Specifically, this code does the following:

• Connects to your Atlas cluster.

• Gets a subset of documents from the sample_airbnb.listingsAndReviews collection that have a non-empty summary field.

• Generates embeddings from each document's summary field by using the get_embedding function that you defined.

• Updates each document with a new embedding field that contains the embedding value by using the MongoDB PyMongo Driver.

import pymongo
# Connect to your Atlas cluster
mongo_client = pymongo.MongoClient("<connection-string>")
db = mongo_client["sample_airbnb"]
collection = db["listingsAndReviews"]
# Filter to exclude null or empty summary fields
filter = { "summary": {"$nin": [ None, "" ]} }
# Get a subset of documents in the collection
documents = collection.find(filter).limit(50)
# Update each document with a new embedding field
updated_doc_count = 0
for doc in documents:
    embedding = get_embedding(doc["summary"])
    collection.update_one( { "_id": doc["_id"] }, { "$set": { "embedding": embedding } } )
    updated_doc_count += 1
print(f"Updated {updated_doc_count} documents.")

Updated 50 documents.

Specify the connection string.

Replace <connection-string> with your Atlas cluster's SRV connection string.

mongodb+srv://<db_username>:<db_password>@<clusterName>.<hostname>.mongodb.net

Run the code.

You can view your vector embeddings as they generate by navigating to the sample_airbnb.listingsAndReviews collection in the Atlas UI and expanding the fields in a document.