Realm Trigger: Duplicate Key Error

User Groups & Events MongoDB Conferences
, ,
6

I also have the same error, and deleted my collections and create again but the issue was the same.

The problem is that the Realm Function gdeltUpdater in Sample Code is wrong (from the video Use Realm Functions to Keep Your GDELT Up To Date with @MarK_Smith and @Shane_McAllister)

Wrong Function gdeltUpdater
Wrong Function gdeltUpdater1199×582 260 KB

To get an expected result I modified a bit the code to request always the same zip file:

http://data.gdeltproject.org/gdeltv2/20220518214500.export.CSV.zip

It contains the file 20220518214500.export.CSV <== 1359 Lines (The last is empty)

In the sample code the way to get the lines from this file is:

const csvLines = csvData.split("/n");
console.log(`csvLines.length: ${csvLines.length}`) // IT SHOULD BE 1359 BUT THE RESULT IS csvLines.length: 818

The correct way to do it is:

const csvLines = csvData.split(/\r\n|\r|\n/);
console.log(`csvLines.length: ${csvLines.length}`) // IT SHOULD BE 1359 AND THE RESULT IS csvLines.length: 1359

Here is the complete gdeltUpdater I used to test if the code works ok or not:

exports = async function(){
const AdmZip = require(“adm-zip”);
const http = context.http;
const csvURL = ‘http://data.gdeltproject.org/gdeltv2/20220518214500.export.CSV.zip’;

const latestCSV = (await http.get({ url: csvURL })).body;
// VSCode Warning:
// var Buffer: BufferConstructor new (str: string, encoding?: BufferEncoding) => Buffer (+5 overloads)
//  @deprecated — since v10.0.0 - Use Buffer.from(string[, encoding]) instead.
//const zip = new AdmZip(new Buffer(latestCSV.toBase64(), 'base64')); // SAMPLE CODE
const zip = new AdmZip(new Buffer.from(latestCSV.toBase64(), 'base64')); // Changed by me
const csvData = zip.getEntries()[0].getData().toString('utf-8');

//const csvLines = csvData.split("/n");
//console.log(`csvLines.length: ${csvLines.length}`) // IT SHOULD BE 1359 BUT THE RESULT IS csvLines.length: 818

const csvLines = csvData.split(/\r\n|\r|\n/);
console.log(`csvLines.length: ${csvLines.length}`) // IT SHOULD BE 1359 AND THE RESULT IS csvLines.length: 1359

if (csvLines[csvLines.length - 1] === ""){ // Remove last line
    csvLines.pop();
}
const rows = csvLines.map((line) => line.split("\t"));
console.log(`rows.length: ${rows.length}`) // IT SHOULD BE 1358 BUT THE RESULT IS rows.length: 818 (After changing for split(/\r\n|\r|\n/) is rows.length: 1358)
console.log(`rows: ${rows}`);
// await context.functions.execute("insertCSVRows", rows, downloadId); // COMENTED TO SEE THE LOGS OF THIS FILE

};

3 Likes