What a blast that was!? Thanks for the invite @Shane_McAllister… awesome job @Mark_Smith… If you’re reading this and want to skip around… I put this handy time index together.
0:00 - 13:48
- Introduction and Information about the Hackathon (Mike)
13:49 - 22:10
- Introduction to GDELT Dataset
22:11 - 25:46
- Launch a MongoDB Atlas Cluster
25:47 - 27:05
- Connecting to the MongoDB Atlas Database (Obtaining the connection string)
27:06 - 27:33
- Username and password information
27:49 - 28:40
- Network security information - IP Access List Entry
30:50 - 31:55
- Installing and using python venv
32:13 - 32: 55
- gdelttools install
32:55 - 33:30
- pip tip * don’t skip this
33:59 - 34:29
- Running gdelttools - getting help, obtaining master file with gdeltloader
34:30 - 36:11
- Master file (77mb)
36:58 - 37:32
- Looking at exports - last 20 files with gdeltloader --master --download --overwrite --last 20 --filter export
38:08 - 39:49
- Looking at the data from a GDELT export in TSV format
41:05 - 41:30
- Cloning the gdelttools repo - to get some additional helpful tools
41:31 - 42:26
- Looking at the gdelt_field_file.ff
- formats and field types for the GDELT data
42:27 - 42:52
- mongoimport.sh
- Script to import GDELT data into mongodb
42:53 - 45:33
- Modifying mongoimport.sh
to remove hard reference to database name
45:34 - 45:54
- Running mongoimport.sh
- importing data into your MongoDB Atlas cluster
45:55 - 48:42
- Viewing data in Atlas - Browse Collections
48:43 - 51:30
- Optimizing mongoimport
51:31
- Transforming the imported data into improved, document-style shapes using gdelt_reshaper.js