How to Create Vector Embeddings

Define a DataService class.

Create a new class in a same-named file named DataService.cs and paste the following code. This code defines an async Task named AddDocumentsAsync add documents to Atlas. This function uses the Collection.InsertManyAsync() C# Driver method to insert a list of the BsonDocument type. Each document contains:

• A text field that contains the movie summary.

• An embedding field that contains the array of floats from generating the vector embeddings.

namespace MyCompany.Embeddings;
using MongoDB.Driver;
using MongoDB.Bson;
public class DataService
{
    private static readonly string? ConnectionString = Environment.GetEnvironmentVariable("ATLAS_CONNECTION_STRING");
    private static readonly MongoClient Client = new MongoClient(ConnectionString);
    private static readonly IMongoDatabase Database = Client.GetDatabase("sample_db");
    private static readonly IMongoCollection<BsonDocument> Collection = Database.GetCollection<BsonDocument>("embeddings");
    
    public async Task AddDocumentsAsync(Dictionary<string, float[]> embeddings)
    {
        var documents = new List<BsonDocument>();
        foreach( KeyValuePair<string, float[]> var in embeddings )
        {
            var document = new BsonDocument
            {
                {
                    "text", var.Key
                },
                {
                    "embedding", new BsonArray(var.Value)
                }
            };
            documents.Add(document);
        }
        await Collection.InsertManyAsync(documents);
        Console.WriteLine($"Successfully inserted {embeddings.Count} documents into Atlas");
        documents.Clear();
    }
}

Update the Program.cs in your project.

Use the following code to generate embeddings from an existing collection in Atlas.

Specifically, this code uses the GetEmbeddingsAsync function that you defined to generate embeddings from an array of sample texts and ingest them into the sample_db.embeddings collection in Atlas.

using MyCompany.Embeddings;
var aiService = new AIService();
var texts = new string[]
{
    "Titanic: The story of the 1912 sinking of the largest luxury liner ever built",
    "The Lion King: Lion cub and future king Simba searches for his identity",
    "Avatar: A marine is dispatched to the moon Pandora on a unique mission"
};
var embeddings = await aiService.GetEmbeddingsAsync(texts);
var dataService = new DataService();
await dataService.AddDocumentsAsync(embeddings);

Compile and run your project.

dotnet run MyCompany.Embeddings.csproj

Successfully inserted 3 documents into Atlas

You can also view your vector embeddings in the Atlas UI by navigating to the sample_db.embeddings collection in your cluster.

Define a DataService class.

Create a new class in a same-named file named DataService.cs and paste the following code. This code creates two functions that do the following:

• Connect to your Atlas cluster.

• The GetDocuments method gets a subset of documents from the sample_airbnb.listingsAndReviews collection that have a non-empty summary field.

• The AddEmbeddings async Task creates a new embeddings field on documents in the sample_airbnb.listingsAndReviews collection whose _id matches one of the documents retrieved in the GetDocuments method.

namespace MyCompany.Embeddings;
using MongoDB.Driver;
using MongoDB.Bson;
public class DataService
{
    private static readonly string? ConnectionString = Environment.GetEnvironmentVariable("ATLAS_CONNECTION_STRING");
    private static readonly MongoClient Client = new MongoClient(ConnectionString);
    private static readonly IMongoDatabase Database = Client.GetDatabase("sample_airbnb");
    private static readonly IMongoCollection<BsonDocument> Collection = Database.GetCollection<BsonDocument>("listingsAndReviews");
    public List<BsonDocument>? GetDocuments()
    {
        var filter = Builders<BsonDocument>.Filter.And(
            Builders<BsonDocument>.Filter.And(
                Builders<BsonDocument>.Filter.Exists("summary", true),
                Builders<BsonDocument>.Filter.Ne("summary", "")
            ),
            Builders<BsonDocument>.Filter.Exists("embeddings", false)
        );
        return Collection.Find(filter).Limit(50).ToList(); 
    }
    public async Task<long> AddEmbeddings(Dictionary<string, float[]> embeddings)
    {
        var listWrites = new List<WriteModel<BsonDocument>>();
        foreach( var kvp in embeddings )
        {
            var filterForUpdate = Builders<BsonDocument>.Filter.Eq("summary", kvp.Key);
            var updateDefinition = Builders<BsonDocument>.Update.Set("embeddings", kvp.Value);
            listWrites.Add(new UpdateOneModel<BsonDocument>(filterForUpdate, updateDefinition));
        }
        var result = await Collection.BulkWriteAsync(listWrites);
        listWrites.Clear();
        return result.ModifiedCount;
    }
}

Update the Program.cs in your project.

Use the following code to generate embeddings from an existing collection in Atlas.

Specifically, this code uses the GetEmbeddingsAsync function that you defined to generate embeddings from an array of sample texts and ingest them into the sample_db.embeddings collection in Atlas.

using MyCompany.Embeddings;
var dataService = new DataService();
var documents = dataService.GetDocuments();
if (documents != null)
{
    Console.WriteLine("Generating embeddings.");
    var aiService = new AIService();
    var summaries = new List<string>();
    foreach (var document in documents)
    {
        var summary = document.GetValue("summary").ToString();
        if (summary != null)
        {
            summaries.Add(summary);
        }
    }
    
    try
    {
        if (summaries.Count > 0)
        {
            var embeddings = await aiService.GetEmbeddingsAsync(summaries.ToArray());
        
            try
            {
                var updatedCount = await dataService.AddEmbeddings(embeddings);
                Console.WriteLine($"{updatedCount} documents updated successfully.");
            } catch (Exception e)
            {
                Console.WriteLine($"Error adding embeddings to MongoDB: {e.Message}");
            }
        }
    }
    catch (Exception e)
    {
        Console.WriteLine($"Error creating embeddings for summaries: {e.Message}");
    }
}
else
{
    Console.WriteLine("No documents found");
}

Compile and run your project.

dotnet run MyCompany.Embeddings.csproj

Generating embeddings.
50 documents updated successfully.

Create a file named create-embeddings.go and paste the following code.

Use the following code to generate embeddings from an existing collection in Atlas.

Specifically, this code uses the GetEmbeddings function that you defined and the MongoDB Go Driver to generate embeddings from an array of sample texts and ingest them into the sample_db.embeddings collection in Atlas.

package main
import (
	"context"
	"fmt"
	"log"
	"my-embeddings-project/common"
	"os"
	"github.com/joho/godotenv"
	"go.mongodb.org/mongo-driver/v2/mongo"
	"go.mongodb.org/mongo-driver/v2/mongo/options"
)
var data = []string{
	"Titanic: The story of the 1912 sinking of the largest luxury liner ever built",
	"The Lion King: Lion cub and future king Simba searches for his identity",
	"Avatar: A marine is dispatched to the moon Pandora on a unique mission",
}
type TextWithEmbedding struct {
	Text      string
	Embedding []float32
}
func main() {
	ctx := context.Background()
	if err := godotenv.Load(); err != nil {
		log.Println("no .env file found")
	}
	// Connect to your Atlas cluster
	uri := os.Getenv("ATLAS_CONNECTION_STRING")
	if uri == "" {
		log.Fatal("set your 'ATLAS_CONNECTION_STRING' environment variable.")
	}
	clientOptions := options.Client().ApplyURI(uri)
	client, err := mongo.Connect(clientOptions)
	if err != nil {
		log.Fatalf("failed to connect to the server: %v", err)
	}
	defer func() { _ = client.Disconnect(ctx) }()
	// Set the namespace
	coll := client.Database("sample_db").Collection("embeddings")
	embeddings := common.GetEmbeddings(data)
	docsToInsert := make([]interface{}, len(embeddings))
	for i, string := range data {
		docsToInsert[i] = TextWithEmbedding{
			Text:      string,
			Embedding: embeddings[i],
		}
	}
	result, err := coll.InsertMany(ctx, docsToInsert)
	if err != nil {
		log.Fatalf("failed to insert documents: %v", err)
	}
	fmt.Printf("Successfully inserted %v documents into Atlas\n", len(result.InsertedIDs))
}

package main
import (
	"context"
	"fmt"
	"log"
	"my-embeddings-project/common"
	"os"
	"github.com/joho/godotenv"
	"go.mongodb.org/mongo-driver/v2/mongo"
	"go.mongodb.org/mongo-driver/v2/mongo/options"
)
var data = []string{
	"Titanic: The story of the 1912 sinking of the largest luxury liner ever built",
	"The Lion King: Lion cub and future king Simba searches for his identity",
	"Avatar: A marine is dispatched to the moon Pandora on a unique mission",
}
type TextWithEmbedding struct {
	Text      string
	Embedding []float64
}
func main() {
	ctx := context.Background()
	if err := godotenv.Load(); err != nil {
		log.Println("no .env file found")
	}
	// Connect to your Atlas cluster
	uri := os.Getenv("ATLAS_CONNECTION_STRING")
	if uri == "" {
		log.Fatal("set your 'ATLAS_CONNECTION_STRING' environment variable.")
	}
	clientOptions := options.Client().ApplyURI(uri)
	client, err := mongo.Connect(clientOptions)
	if err != nil {
		log.Fatalf("failed to connect to the server: %v", err)
	}
	defer func() { _ = client.Disconnect(ctx) }()
	// Set the namespace
	coll := client.Database("sample_db").Collection("embeddings")
	embeddings := common.GetEmbeddings(data)
	docsToInsert := make([]interface{}, len(data))
	for i, movie := range data {
		docsToInsert[i] = TextWithEmbedding{
			Text:      movie,
			Embedding: embeddings[i],
		}
	}
	result, err := coll.InsertMany(ctx, docsToInsert)
	if err != nil {
		log.Fatalf("failed to insert documents: %v", err)
	}
	fmt.Printf("Successfully inserted %v documents into Atlas\n", len(result.InsertedIDs))
}

Save and run the file.

go run create-embeddings.go

Successfully inserted 3 documents into Atlas

go run create-embeddings.go

Successfully inserted 3 documents into Atlas

You can also view your vector embeddings in the Atlas UI by navigating to the sample_db.embeddings collection in your cluster.

Create a file named create-embeddings.go and paste the following code.

Use the following code to generate embeddings from an existing collection in Atlas. Specifically, this code does the following:

• Connects to your Atlas cluster.

• Gets a subset of documents from the sample_airbnb.listingsAndReviews collection that have a non-empty summary field.

• Generates embeddings from each document's summary field by using the GetEmbeddings function that you defined.

• Updates each document with a new embeddings field that contains the embedding value by using the MongoDB Go Driver.

package main
import (
	"context"
	"log"
	"my-embeddings-project/common"
	"os"
	"github.com/joho/godotenv"
	"go.mongodb.org/mongo-driver/v2/bson"
	"go.mongodb.org/mongo-driver/v2/mongo"
	"go.mongodb.org/mongo-driver/v2/mongo/options"
)
func main() {
	ctx := context.Background()
	if err := godotenv.Load(); err != nil {
		log.Println("no .env file found")
	}
	// Connect to your Atlas cluster
	uri := os.Getenv("ATLAS_CONNECTION_STRING")
	if uri == "" {
		log.Fatal("set your 'ATLAS_CONNECTION_STRING' environment variable.")
	}
	clientOptions := options.Client().ApplyURI(uri)
	client, err := mongo.Connect(clientOptions)
	if err != nil {
		log.Fatalf("failed to connect to the server: %v", err)
	}
	defer func() { _ = client.Disconnect(ctx) }()
	// Set the namespace
	coll := client.Database("sample_airbnb").Collection("listingsAndReviews")
	filter := bson.D{
		{"$and",
			bson.A{
				bson.D{
					{"$and",
						bson.A{
							bson.D{{"summary", bson.D{{"$exists", true}}}},
							bson.D{{"summary", bson.D{{"$ne", ""}}}},
						},
					}},
				bson.D{{"embeddings", bson.D{{"$exists", false}}}},
			}},
	}
	opts := options.Find().SetLimit(50)
	cursor, err := coll.Find(ctx, filter, opts)
	if err != nil {
		log.Fatalf("failed to retrieve documents: %v", err)
	}
	var listings []common.Listing
	if err = cursor.All(ctx, &listings); err != nil {
		log.Fatalf("failed to unmarshal retrieved documents to Listing object: %v", err)
	}
	var summaries []string
	for _, listing := range listings {
		summaries = append(summaries, listing.Summary)
	}
	log.Println("Generating embeddings.")
	embeddings := common.GetEmbeddings(summaries)
	docsToUpdate := make([]mongo.WriteModel, len(listings))
	for i := range listings {
		docsToUpdate[i] = mongo.NewUpdateOneModel().
			SetFilter(bson.D{{"_id", listings[i].ID}}).
			SetUpdate(bson.D{{"$set", bson.D{{"embeddings", embeddings[i]}}}})
	}
	bulkWriteOptions := options.BulkWrite().SetOrdered(false)
	result, err := coll.BulkWrite(context.Background(), docsToUpdate, bulkWriteOptions)
	if err != nil {
		log.Fatalf("failed to write embeddings to existing documents: %v", err)
	}
	log.Printf("Successfully added embeddings to %v documents", result.ModifiedCount)
}

Create a file that contains Go models for the collection.

To simplify marshalling and unmarshalling Go objects to and from BSON, create a file that contains models for the documents in this collection.

1. Move into the common directory.

   cd common

2. Create a file named models.go, and paste the following code into it:

   Open-Source

   OpenAI

   models.go

   package common
   import (
   	"time"
   	"go.mongodb.org/mongo-driver/v2/bson"
   )
   type Image struct {
   	ThumbnailURL string `bson:"thumbnail_url"`
   	MediumURL    string `bson:"medium_url"`
   	PictureURL   string `bson:"picture_url"`
   	XLPictureURL string `bson:"xl_picture_url"`
   }
   type Host struct {
   	ID                 string   `bson:"host_id"`
   	URL                string   `bson:"host_url"`
   	Name               string   `bson:"host_name"`
   	Location           string   `bson:"host_location"`
   	About              string   `bson:"host_about"`
   	ThumbnailURL       string   `bson:"host_thumbnail_url"`
   	PictureURL         string   `bson:"host_picture_url"`
   	Neighborhood       string   `bson:"host_neighborhood"`
   	IsSuperhost        bool     `bson:"host_is_superhost"`
   	HasProfilePic      bool     `bson:"host_has_profile_pic"`
   	IdentityVerified   bool     `bson:"host_identity_verified"`
   	ListingsCount      int32    `bson:"host_listings_count"`
   	TotalListingsCount int32    `bson:"host_total_listings_count"`
   	Verifications      []string `bson:"host_verifications"`
   }
   type Location struct {
   	Type            string    `bson:"type"`
   	Coordinates     []float64 `bson:"coordinates"`
   	IsLocationExact bool      `bson:"is_location_exact"`
   }
   type Address struct {
   	Street         string   `bson:"street"`
   	Suburb         string   `bson:"suburb"`
   	GovernmentArea string   `bson:"government_area"`
   	Market         string   `bson:"market"`
   	Country        string   `bson:"Country"`
   	CountryCode    string   `bson:"country_code"`
   	Location       Location `bson:"location"`
   }
   type Availability struct {
   	Thirty         int32 `bson:"availability_30"`
   	Sixty          int32 `bson:"availability_60"`
   	Ninety         int32 `bson:"availability_90"`
   	ThreeSixtyFive int32 `bson:"availability_365"`
   }
   type ReviewScores struct {
   	Accuracy      int32 `bson:"review_scores_accuracy"`
   	Cleanliness   int32 `bson:"review_scores_cleanliness"`
   	CheckIn       int32 `bson:"review_scores_checkin"`
   	Communication int32 `bson:"review_scores_communication"`
   	Location      int32 `bson:"review_scores_location"`
   	Value         int32 `bson:"review_scores_value"`
   	Rating        int32 `bson:"review_scores_rating"`
   }
   type Review struct {
   	ID           string    `bson:"_id"`
   	Date         time.Time `bson:"date,omitempty"`
   	ListingId    string    `bson:"listing_id"`
   	ReviewerId   string    `bson:"reviewer_id"`
   	ReviewerName string    `bson:"reviewer_name"`
   	Comments     string    `bson:"comments"`
   }
   type Listing struct {
   	ID                   string          `bson:"_id"`
   	ListingURL           string          `bson:"listing_url"`
   	Name                 string          `bson:"name"`
   	Summary              string          `bson:"summary"`
   	Space                string          `bson:"space"`
   	Description          string          `bson:"description"`
   	NeighborhoodOverview string          `bson:"neighborhood_overview"`
   	Notes                string          `bson:"notes"`
   	Transit              string          `bson:"transit"`
   	Access               string          `bson:"access"`
   	Interaction          string          `bson:"interaction"`
   	HouseRules           string          `bson:"house_rules"`
   	PropertyType         string          `bson:"property_type"`
   	RoomType             string          `bson:"room_type"`
   	BedType              string          `bson:"bed_type"`
   	MinimumNights        string          `bson:"minimum_nights"`
   	MaximumNights        string          `bson:"maximum_nights"`
   	CancellationPolicy   string          `bson:"cancellation_policy"`
   	LastScraped          time.Time       `bson:"last_scraped,omitempty"`
   	CalendarLastScraped  time.Time       `bson:"calendar_last_scraped,omitempty"`
   	FirstReview          time.Time       `bson:"first_review,omitempty"`
   	LastReview           time.Time       `bson:"last_review,omitempty"`
   	Accommodates         int32           `bson:"accommodates"`
   	Bedrooms             int32           `bson:"bedrooms"`
   	Beds                 int32           `bson:"beds"`
   	NumberOfReviews      int32           `bson:"number_of_reviews"`
   	Bathrooms            bson.Decimal128 `bson:"bathrooms"`
   	Amenities            []string        `bson:"amenities"`
   	Price                bson.Decimal128 `bson:"price"`
   	WeeklyPrice          bson.Decimal128 `bson:"weekly_price"`
   	MonthlyPrice         bson.Decimal128 `bson:"monthly_price"`
   	CleaningFee          bson.Decimal128 `bson:"cleaning_fee"`
   	ExtraPeople          bson.Decimal128 `bson:"extra_people"`
   	GuestsIncluded       bson.Decimal128 `bson:"guests_included"`
   	Image                Image           `bson:"images"`
   	Host                 Host            `bson:"host"`
   	Address              Address         `bson:"address"`
   	Availability         Availability    `bson:"availability"`
   	ReviewScores         ReviewScores    `bson:"review_scores"`
   	Reviews              []Review        `bson:"reviews"`
   	Embeddings           []float32       `bson:"embeddings,omitempty"`
   }

   models.go

   package common
   import (
   	"time"
   	"go.mongodb.org/mongo-driver/v2/bson"
   )
   type Image struct {
   	ThumbnailURL string `bson:"thumbnail_url"`
   	MediumURL    string `bson:"medium_url"`
   	PictureURL   string `bson:"picture_url"`
   	XLPictureURL string `bson:"xl_picture_url"`
   }
   type Host struct {
   	ID                 string   `bson:"host_id"`
   	URL                string   `bson:"host_url"`
   	Name               string   `bson:"host_name"`
   	Location           string   `bson:"host_location"`
   	About              string   `bson:"host_about"`
   	ThumbnailURL       string   `bson:"host_thumbnail_url"`
   	PictureURL         string   `bson:"host_picture_url"`
   	Neighborhood       string   `bson:"host_neighborhood"`
   	IsSuperhost        bool     `bson:"host_is_superhost"`
   	HasProfilePic      bool     `bson:"host_has_profile_pic"`
   	IdentityVerified   bool     `bson:"host_identity_verified"`
   	ListingsCount      int32    `bson:"host_listings_count"`
   	TotalListingsCount int32    `bson:"host_total_listings_count"`
   	Verifications      []string `bson:"host_verifications"`
   }
   type Location struct {
   	Type            string    `bson:"type"`
   	Coordinates     []float64 `bson:"coordinates"`
   	IsLocationExact bool      `bson:"is_location_exact"`
   }
   type Address struct {
   	Street         string   `bson:"street"`
   	Suburb         string   `bson:"suburb"`
   	GovernmentArea string   `bson:"government_area"`
   	Market         string   `bson:"market"`
   	Country        string   `bson:"Country"`
   	CountryCode    string   `bson:"country_code"`
   	Location       Location `bson:"location"`
   }
   type Availability struct {
   	Thirty         int32 `bson:"availability_30"`
   	Sixty          int32 `bson:"availability_60"`
   	Ninety         int32 `bson:"availability_90"`
   	ThreeSixtyFive int32 `bson:"availability_365"`
   }
   type ReviewScores struct {
   	Accuracy      int32 `bson:"review_scores_accuracy"`
   	Cleanliness   int32 `bson:"review_scores_cleanliness"`
   	CheckIn       int32 `bson:"review_scores_checkin"`
   	Communication int32 `bson:"review_scores_communication"`
   	Location      int32 `bson:"review_scores_location"`
   	Value         int32 `bson:"review_scores_value"`
   	Rating        int32 `bson:"review_scores_rating"`
   }
   type Review struct {
   	ID           string    `bson:"_id"`
   	Date         time.Time `bson:"date,omitempty"`
   	ListingId    string    `bson:"listing_id"`
   	ReviewerId   string    `bson:"reviewer_id"`
   	ReviewerName string    `bson:"reviewer_name"`
   	Comments     string    `bson:"comments"`
   }
   type Listing struct {
   	ID                   string          `bson:"_id"`
   	ListingURL           string          `bson:"listing_url"`
   	Name                 string          `bson:"name"`
   	Summary              string          `bson:"summary"`
   	Space                string          `bson:"space"`
   	Description          string          `bson:"description"`
   	NeighborhoodOverview string          `bson:"neighborhood_overview"`
   	Notes                string          `bson:"notes"`
   	Transit              string          `bson:"transit"`
   	Access               string          `bson:"access"`
   	Interaction          string          `bson:"interaction"`
   	HouseRules           string          `bson:"house_rules"`
   	PropertyType         string          `bson:"property_type"`
   	RoomType             string          `bson:"room_type"`
   	BedType              string          `bson:"bed_type"`
   	MinimumNights        string          `bson:"minimum_nights"`
   	MaximumNights        string          `bson:"maximum_nights"`
   	CancellationPolicy   string          `bson:"cancellation_policy"`
   	LastScraped          time.Time       `bson:"last_scraped,omitempty"`
   	CalendarLastScraped  time.Time       `bson:"calendar_last_scraped,omitempty"`
   	FirstReview          time.Time       `bson:"first_review,omitempty"`
   	LastReview           time.Time       `bson:"last_review,omitempty"`
   	Accommodates         int32           `bson:"accommodates"`
   	Bedrooms             int32           `bson:"bedrooms"`
   	Beds                 int32           `bson:"beds"`
   	NumberOfReviews      int32           `bson:"number_of_reviews"`
   	Bathrooms            bson.Decimal128 `bson:"bathrooms"`
   	Amenities            []string        `bson:"amenities"`
   	Price                bson.Decimal128 `bson:"price"`
   	WeeklyPrice          bson.Decimal128 `bson:"weekly_price"`
   	MonthlyPrice         bson.Decimal128 `bson:"monthly_price"`
   	CleaningFee          bson.Decimal128 `bson:"cleaning_fee"`
   	ExtraPeople          bson.Decimal128 `bson:"extra_people"`
   	GuestsIncluded       bson.Decimal128 `bson:"guests_included"`
   	Image                Image           `bson:"images"`
   	Host                 Host            `bson:"host"`
   	Address              Address         `bson:"address"`
   	Availability         Availability    `bson:"availability"`
   	ReviewScores         ReviewScores    `bson:"review_scores"`
   	Reviews              []Review        `bson:"reviews"`
   	Embeddings           []float64       `bson:"embeddings,omitempty"`
   }

3. Move back into the project root directory.

   cd ../

Generate embeddings.

go run create-embeddings.go

2024/10/10 09:58:03 Generating embeddings.
2024/10/10 09:58:12 Successfully added embeddings to 50 documents

You can view your vector embeddings in the Atlas UI by navigating to the sample_airbnb.listingsAndReviews collection in your cluster and expanding the fields in a document.

Define code to generate embeddings from an existing collection in Atlas.

Create a file named CreateEmbeddings.java and paste the following code.

This code uses the getEmbeddings method and the MongoDB Java Sync Driver to do the following:

1. Connect to your Atlas cluster.

2. Get the array of sample texts.

3. Generate embeddings from each text using the getEmbeddings method that you defined previously.

4. Ingest the embeddings into the sample_db.embeddings collection in Atlas.

import com.mongodb.MongoException;
import com.mongodb.client.MongoClient;
import com.mongodb.client.MongoClients;
import com.mongodb.client.MongoCollection;
import com.mongodb.client.MongoDatabase;
import com.mongodb.client.result.InsertManyResult;
import org.bson.BsonArray;
import org.bson.Document;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
public class CreateEmbeddings {
    static List<String> data = Arrays.asList(
            "Titanic: The story of the 1912 sinking of the largest luxury liner ever built",
            "The Lion King: Lion cub and future king Simba searches for his identity",
            "Avatar: A marine is dispatched to the moon Pandora on a unique mission"
    );
    public static void main(String[] args){
        String uri = System.getenv("ATLAS_CONNECTION_STRING");
        if (uri == null || uri.isEmpty()) {
            throw new RuntimeException("ATLAS_CONNECTION_STRING env variable is not set or is empty.");
        }
        // establish connection and set namespace
        try (MongoClient mongoClient = MongoClients.create(uri)) {
            MongoDatabase database = mongoClient.getDatabase("sample_db");
            MongoCollection<Document> collection = database.getCollection("embeddings");
            System.out.println("Creating embeddings for " + data.size() + " documents");
            EmbeddingProvider embeddingProvider = new EmbeddingProvider();
            // generate embeddings for new inputted data
            List<BsonArray> embeddings = embeddingProvider.getEmbeddings(data);
            List<Document> documents = new ArrayList<>();
            int i = 0;
            for (String text : data) {
                Document doc = new Document("text", text).append("embedding", embeddings.get(i));
                documents.add(doc);
                i++;
            }
            // insert the embeddings into the Atlas collection
            List<String> insertedIds = new ArrayList<>();
            try {
                InsertManyResult result = collection.insertMany(documents);
                result.getInsertedIds().values()
                        .forEach(doc -> insertedIds.add(doc.toString()));
                System.out.println("Inserted " + insertedIds.size() + " documents with the following ids to " + collection.getNamespace() + " collection: \n " + insertedIds);
            } catch (MongoException me) {
                throw new RuntimeException("Failed to insert documents", me);
            }
        } catch (MongoException me) {
            throw new RuntimeException("Failed to connect to MongoDB ", me);
        } catch (Exception e) {
            throw new RuntimeException("Operation failed: ", e);
        }
    }
}

Generate the embeddings.

Save and run the file. The output resembles:

Creating embeddings for 3 documents
Inserted 3 documents with the following ids to sample_db.embeddings collection:
 [BsonObjectId{value=6735ff620d88451041f6dd40}, BsonObjectId{value=6735ff620d88451041f6dd41}, BsonObjectId{value=6735ff620d88451041f6dd42}]

You can also view your vector embeddings in the Atlas UI by navigating to the sample_db.embeddings collection in your cluster.

Define code to generate embeddings from an existing collection in Atlas.

Create a file named CreateEmbeddings.java and paste the following code.

This code uses the getEmbeddings method and the MongoDB Java Sync Driver to do the following:

1. Connect to your Atlas cluster.

2. Get a subset of documents from the sample_airbnb.listingsAndReviews collection that have a non-empty summary field.

3. Generate embeddings from each document's summary field using the getEmbeddings method that you defined previously.

4. Update each document with a new embeddings field that contains the embedding value.

import com.mongodb.MongoException;
import com.mongodb.bulk.BulkWriteResult;
import com.mongodb.client.MongoClient;
import com.mongodb.client.MongoClients;
import com.mongodb.client.MongoCollection;
import com.mongodb.client.MongoCursor;
import com.mongodb.client.MongoDatabase;
import com.mongodb.client.model.BulkWriteOptions;
import com.mongodb.client.model.Filters;
import com.mongodb.client.model.UpdateOneModel;
import com.mongodb.client.model.Updates;
import com.mongodb.client.model.WriteModel;
import org.bson.BsonArray;
import org.bson.Document;
import org.bson.conversions.Bson;
import java.util.ArrayList;
import java.util.List;
public class CreateEmbeddings {
    public static void main(String[] args){
        String uri = System.getenv("ATLAS_CONNECTION_STRING");
        if (uri == null || uri.isEmpty()) {
            throw new RuntimeException("ATLAS_CONNECTION_STRING env variable is not set or is empty.");
        }
        // establish connection and set namespace
        try (MongoClient mongoClient = MongoClients.create(uri)) {
            MongoDatabase database = mongoClient.getDatabase("sample_airbnb");
            MongoCollection<Document> collection = database.getCollection("listingsAndReviews");
            Bson filterCriteria = Filters.and(
                    Filters.and(Filters.exists("summary"),
                            Filters.ne("summary", null),
                            Filters.ne("summary", "")),
                    Filters.exists("embeddings", false));
            try (MongoCursor<Document> cursor = collection.find(filterCriteria).limit(50).iterator()) {
                List<String> summaries = new ArrayList<>();
                List<String> documentIds = new ArrayList<>();
                int i = 0;
                while (cursor.hasNext()) {
                    Document document = cursor.next();
                    String summary = document.getString("summary");
                    String id = document.get("_id").toString();
                    summaries.add(summary);
                    documentIds.add(id);
                    i++;
                }
                System.out.println("Generating embeddings for " + summaries.size() + " documents.");
                System.out.println("This operation may take up to several minutes.");
                EmbeddingProvider embeddingProvider = new EmbeddingProvider();
                List<BsonArray> embeddings = embeddingProvider.getEmbeddings(summaries);
                List<WriteModel<Document>> updateDocuments = new ArrayList<>();
                for (int j = 0; j < summaries.size(); j++) {
                    UpdateOneModel<Document> updateDoc = new UpdateOneModel<>(
                            Filters.eq("_id", documentIds.get(j)),
                            Updates.set("embeddings", embeddings.get(j)));
                    updateDocuments.add(updateDoc);
                }
                int updatedDocsCount = 0;
                try {
                    BulkWriteOptions options = new BulkWriteOptions().ordered(false);
                    BulkWriteResult result = collection.bulkWrite(updateDocuments, options);
                    updatedDocsCount = result.getModifiedCount();
                } catch (MongoException me) {
                    throw new RuntimeException("Failed to insert documents", me);
                }
                System.out.println("Added embeddings successfully to " + updatedDocsCount + " documents.");
            }
        } catch (MongoException me) {
            throw new RuntimeException("Failed to connect to MongoDB", me);
        } catch (Exception e) {
            throw new RuntimeException("Operation failed: ", e);
        }
    }
}

Generate the embeddings.

Save and run the file. The output resembles:

Generating embeddings for 50 documents.
This operation may take up to several minutes.
Added embeddings successfully to 50 documents.

You can also view your vector embeddings in the Atlas UI by navigating to the sample_airbnb.listingsAndReviews collection in your cluster.

Create a file named create-embeddings.js and paste the following code.

Use the following code to generate embeddings from an existing collection in Atlas. This code uses the getEmbedding function that you defined and the Node.js Driver to generate embeddings from an array of sample texts and ingest them into the sample_db.embeddings collection in Atlas.

If you defined the convertEmbeddingsToBSON function, uncomment lines 3 and 32-33 to convert your embeddings to BSON binData vectors.

import { MongoClient } from 'mongodb';
import { getEmbedding } from './get-embeddings.js';
// import { convertEmbeddingsToBSON } from './convert-embeddings.js';
// Data to embed
const texts = [ 
    "Titanic: The story of the 1912 sinking of the largest luxury liner ever built",
    "The Lion King: Lion cub and future king Simba searches for his identity",
    "Avatar: A marine is dispatched to the moon Pandora on a unique mission"
]
async function run() {
    // Connect to your Atlas cluster
    const client = new MongoClient(process.env.ATLAS_CONNECTION_STRING);
    
    try {
        await client.connect();
        const db = client.db("sample_db");
        const collection = db.collection("embeddings");
        console.log("Generating embeddings and inserting documents...");
        const insertDocuments = [];
        await Promise.all(texts.map(async text => {
            // Check if the document already exists
            const existingDoc = await collection.findOne({ text: text });
            // Generate an embedding using the function that you defined
            var embedding = await getEmbedding(text);
            
            // Uncomment the following lines to convert the generated embedding into BSON format
            // const bsonEmbedding = await convertEmbeddingsToBSON([embedding]); // Since convertEmbeddingsToBSON is designed to handle arrays
            // embedding = bsonEmbedding; // Use BSON embedding instead of the original float32 embedding
                      
            // Add the document with the embedding to array of documents for bulk insert
            if (!existingDoc) {
                insertDocuments.push({
                    text: text,
                    embedding: embedding
                })
            }
        }));
        // Continue processing documents if an error occurs during an operation
        const options = { ordered: false };
        // Insert documents with embeddings into Atlas
        const result = await collection.insertMany(insertDocuments, options);  
        console.log("Count of documents inserted: " + result.insertedCount); 
    } catch (err) {
        console.log(err.stack);
    }
    finally {
        await client.close();
    }
}
run().catch(console.dir);

Save and run the file.

node --env-file=.env create-embeddings.js

Generating embeddings and inserting documents...
Count of documents inserted: 3

You can view your vector embeddings in the Atlas UI by navigating to the sample_db.embeddings collection in your cluster.

Create a file named create-embeddings.js and paste the following code.

Use the following code to generate embeddings from an existing collection in Atlas. Specifically, this code does the following:

• Connects to your Atlas cluster.

• Gets a subset of documents from the sample_airbnb.listingsAndReviews collection that have a non-empty summary field.

• Generates embeddings from each document's summary field by using the getEmbedding function that you defined.

• Updates each document with a new embedding field that contains the embedding value by using the MongoDB Node.js Driver.

If you defined the convertEmbeddingsToBSON function, uncomment lines 3 and 29-30 to convert your embeddings to BSON binData vectors.

import { MongoClient } from 'mongodb';
import { getEmbedding } from './get-embeddings.js';
// import { convertEmbeddingsToBSON } from './convert-embeddings.js';
async function run() {
    // Connect to your Atlas cluster
    const client = new MongoClient(process.env.ATLAS_CONNECTION_STRING);
    try {
        await client.connect();
        const db = client.db("sample_airbnb");
        const collection = db.collection("listingsAndReviews");
        // Filter to exclude null or empty summary fields
        const filter = { "summary": { "$nin": [ null, "" ] } };
        // Get a subset of documents from the collection
        const documents = await collection.find(filter).limit(50).toArray();
        console.log("Generating embeddings and updating documents...");
        const updateDocuments = [];
        await Promise.all(documents.map(async doc => {
            // Generate an embedding using the function that you defined
            var embedding = await getEmbedding(doc.summary);
            // Uncomment the following lines to convert the generated embedding into BSON format
            // const bsonEmbedding = await convertEmbeddingsToBSON([embedding]); // Since convertEmbeddingsToBSON is designed to handle arrays
            // embedding = bsonEmbedding; // Use BSON embedding instead of the original float32 embedding
             
            // Add the embedding to an array of update operations
            updateDocuments.push(
                {
                    updateOne: { 
                        filter: { "_id": doc._id },
                        update: { $set: { "embedding": embedding } }
                    }
                }
           )
       }));
       // Continue processing documents if an error occurs during an operation
       const options = { ordered: false };
       // Update documents with the new embedding field
       const result = await collection.bulkWrite(updateDocuments, options); 
       console.log("Count of documents updated: " + result.modifiedCount); 
            
    } catch (err) {
        console.log(err.stack);
    }
    finally {
        await client.close();
    }
}
run().catch(console.dir);

Save and run the file.

node --env-file=.env create-embeddings.js

Generating embeddings and updating documents...
Count of documents updated: 50

You can view your vector embeddings in the Atlas UI by navigating to the sample_airbnb.listingsAndReviews collection in your cluster and expanding the fields in a document.

Load your data.

The following code defines an array of sample texts.

# Sample data
texts = [
  "Titanic: The story of the 1912 sinking of the largest luxury liner ever built",
  "The Lion King: Lion cub and future king Simba searches for his identity",
  "Avatar: A marine is dispatched to the moon Pandora on a unique mission"
]

Generate embeddings from your data.

Use the get_embedding function to generate embeddings from your data. Use the following code generates embeddings from the sample texts.

If you defined the generate_bson_vector function, uncomment the line that calls this function to compress your embeddings to binData vectors. Your embeddings will appear in binary format.

# Generate embeddings from the sample data
embeddings = []
for text in texts:
 embedding = get_embedding(text)
 # Uncomment the following line to convert to BSON
 # embedding = generate_bson_vector(embedding, BinaryVectorDtype.FLOAT32)
 embeddings.append(embedding)
 # Print the embeddings
 print(f"\nText: {text}")
 print(f"Embedding: {embedding[:3]}... (truncated)")

Generated embeddings:
Text: Titanic: The story of the 1912 sinking of the largest luxury liner ever built
Embedding: [-0.01089042  0.05926645 -0.00291325]... (truncated)
Text: The Lion King: Lion cub and future king Simba searches for his identity
Embedding: [-0.05607051 -0.01360618  0.00523855]... (truncated)
Text: Avatar: A marine is dispatched to the moon Pandora on a unique mission
Embedding: [-0.0275258   0.01144342 -0.02360895]... (truncated)

Ingest the embeddings into Atlas.

Perform the following steps to create documents with the embeddings and ingest them into your Atlas cluster:

1. Define a function to create documents.

   def create_docs_with_embeddings(embeddings, data):
      docs = []
      for i, (embedding, text) in enumerate(zip(embeddings, data)):
         doc = {
               "_id": i,
               "text": text,
               "embedding": embedding,
         }
         docs.append(doc)
      return docs

2. Create the documents with your embeddings.

   # Create documents with embeddings and sample data
   docs = create_docs_with_embeddings(embeddings, texts)

3. Ingest the documents into Atlas.

   Paste and run the following code in your notebook, replacing <connection-string> with your Atlas cluster's SRV connection string.

   Note

   Your connection string should use the following format:

   mongodb+srv://<db_username>:<db_password>@<clusterName>.<hostname>.mongodb.net

   This code does the following:

   • Connects to your Atlas cluster.

   • Inserts the documents into the specified database and collection.

   import pymongo
   # Connect to your Atlas cluster
   mongo_client = pymongo.MongoClient("<connection-string>")
   db = mongo_client["sample_db"]
   collection = db["embeddings"]
   # Ingest data into Atlas
   collection.insert_many(docs)

   HIDE OUTPUT

   InsertManyResult([0, 1, 2], acknowledged=True)

   You can verify your vector embeddings by viewing them in the Atlas UI for the sample_db.embeddings namespace in your cluster.