The MongoDB Web Shell

MongoDB
August 14, 2013

About

The MongoDB Web Shell is a web application designed to emulate some of the features of the mongo terminal shell. This project has three main uses: try.mongodb.org, 10gen Education online classes, and the MongoDB API documentation.

In these three different contexts, users will be able to familiarize themselves with the MongoDB interface and basic commands available both independently and as part of a 10gen education homework assignment in the education program.

See a screenshot of the state of the browser shell prior to this summer below:

Architecture

The user interfaces with the new MongoDB Web Shell through any web browser. A majority of the codebase for the project is written in Javascript executed in the browser. It provides a responsive command line interface similar to the MongoDB desktop shell. Because the MongoDB console is also implemented in Javascript, it is convenient to evaluate user-written code locally.

The browser-based shell interacts with a backing mongod instance through a RESTful interface implemented on top of Flask, a Python web application microframework. The backend provides

1) sandboxing capability to manage sessions and access to resources on the backing database 2) a framework for preloading data into a given resource 3) ability to verify data state for use in the online classes.

One interesting problem we ran into is allowing the blocking paradigm of code used by the mongo shells without actually blocking in the web browser. Browser tabs run on a single thread, so if Javascript code blocks on that thread, you can’t scroll or select things or interact with the tab at all. However, running code such as length = db.foo.aggregate(pipeline).results.length traditionally requires that the aggregate call is a blocking request so that it can fetch the remote data and return it in a single function call. Normally, we should just rewrite out function calls to take callbacks, but since this is a web shell, we don’t have control over the code being written. Our solution was to write a Javascript evaluator that could pause execution at any time, similar to coroutines. This solution manifested itself in a library called suspend.js evaluates Javascript program strings and can pause execution, perform some arbitrary, possibly asynchronous actions, then resume execution with the result of those actions. Thus users can still write code in the blocking paradigm that the mongo shell uses without locking up the browser.

1) try.mongodb.org

One of the primary goals of the project is to replace the existing browser shell on try.mongodb.org. The newer version provides better support for javascript syntax and mongoDB features in a cleaner UI. Among the features that will be present in the shell are syntax highlighting (replacing the green on black styling), fully-featured Javascript, and a new tutorial system that integrates beyond the shell window in the containing page.

A screenshot of the new shell:

One of the challenges in developing a fully featured shell was the ability to execute user-written code in a sandboxed environment within the browser. Originally, the shell relied on parsing the code into an abstract syntax tree (AST) and mutating the source to namespace variables in a manner where var x would become mongo.shells[index].vars['x']. We quickly realized that this implementation would be difficult to maintain given potential feature growth and the number of edge case scenarios. Instead, the latest version of the shell delegates the scoping of variables to the browser by executing in an iframe element that intrinsically provides many sandboxing features.

2) 10gen Education

The MongoDB Web Shell will also make its debut on the 10gen Education platform where users learn to develop for and administer MongoDB quickly and efficiently. Currently, the assignments for the online classes must be downloaded and solved locally on an individual installation of MongoDB. With the web shell, students will be able to complete and submit assignments entirely within the education platform without needing to download and install their own copy of MongoDB or sample data. This online solution makes it more convenient for users to take courses wherever they may be and makes it easier to validate the correct completion of an assignment. Furthermore, users will no longer be required to load their own datasets using mongorestore since it is handled directly by the shell. Finally, since all the work is stored on their online account, students will be able to continue their work from where they left off from any computer.

3) MongoDB Documentation

Finally, the MongoDB Web Shell will be embedded into the API reference pages as part of the database’s documentation. This will allow a user to immediately test the commands detailed on the page and to evaluate its effects against provided sample data sets.

Resources

This project is Apache licensed open source software and is freely available through this Github repository. We have provided documentation using the Github wiki and welcome any feedback or contributions through issues and pull requests. The mentioned suspend.js library can be found at this Github repository.

About Us

The MongoDB Web Shell driver was developed by Ryan Chan and Danny Weinberg, who are currently interns at 10gen.

Acknowledgements

Special thanks to 10gen for hosting us as interns, Stacy Ferranti and Ian Whalen for managing the internship program, and our mentors Ian Bentley and Emily Stolfo for their help and support throughout the project.

← Previous

Do Hadoop and the Cloud Mean the End of the Closed Data Warehouse?

The data warehouse business has been great for at least a decade as almost every sizable company has been finding itself awash in data that it could not bring itself to part with, and was willing to pay a lot to have it stored, organized, and analyzed. But in an era of open tools – such as Hadoop - plus a push to lower costs by using commodity hardware, has the clock started to tick down on closed data warehouses? Data warehouses are fairly simple. They are built to accommodate a business’s data, often its structured data, although with a little tweaking unstructured data can be stored in them too. Everything is usually kept inside a relational database and - as always with relational databases - the upside is speed of processing and the downsides are some inflexibility about kinds of data that can be handled along with heavy demands for costly hardware to run the SQL databases. But as businesses look to cut costs, particularly with open source, and as commodity servers in the cloud become a new standard for running IT operations, this basic structure is beginning to be questioned. Over the long term, private warehouses will continue to thrive, but they likely will make themselves open to manipulation by open tools such as Hadoop and NoSQL databases. It’s increasingly said that enterprise needs a new data fabric that accommodates the growing diversity of data types. Hybrid kinds of arrangements - where relational databases blend in with NoSQL databases - will likely emerge as solutions, at least in the short term. But the other reality is that ours is an era of humongous data - really, really big data - and inevitably that will break many, perhaps not all, data warehouses as presently conceived. What’s keenly needed is big and fast data and the latter of course is not always easy to achieve with Hadoop, but there’s a stampede of plug-ins and tweaks designed to coax Hadoop into realtime, fast processing. MongoDB of course already has announced integration with Hadoop that allows MongoDB data sets also to be manipulated by Hadoop and a plus is that data set size - for all practical purpose - ceases to be much of an issue. However big the data, bring it on. The key of course is that - in their essences - Hadoop and MongoDB are fundamentally very different. MongoDB was written for data storage, not processing, and also data retrieval, while Hadoop was written to process data, particularly massive data sets. But those differences are why they can work together. Processing can be done in Hadoop, then sent for storage in MongoDB, building on the strengths of each. And that is exactly the kind of strengths needed in a world where the new normal is really big data. Ask yourself this, do you envision data warehousing needs diminishing, holding the same, or growing? If the last, get to know Hadoop because it will become a part of your life. At least you will hope it does.

August 13, 2013
Next →

Staff Engineering at MongoDB: Your Path to Making Broad Impact

Andrew Whitaker is a Senior Staff Engineer at MongoDB. His previous experience spans tiny startups to enormous organizations like AWS, where he held several different roles focusing on databases. Before joining MongoDB, he worked at a startup building optimized machine learning models in the cloud. Read on to learn more about why Andrew decided to join MongoDB in a senior-level engineering role and how his work is driving improvement within our engineering organization. Why MongoDB I have long been a fan of MongoDB’s products and services. MongoDB the database has always been a pleasure to work with – the system “brings joy” to quote a phrase. As a Python developer, I appreciate how the Python driver feels “Pythonic” in a completely natural way. The programmer interacts with the database using Python constructs: dictionaries, lists, and primitive types. By contrast, SQL databases force me to change my mental model, and the query language feels like an add-on that does not blend with the core language. As an engineer, I am always looking to expand my knowledge and grow my skills. The scope of challenges engineers face at MongoDB is what triggered my interest in the company. We obviously have people working on core databases and distributed systems. But, we also have teams dedicated to machine learning, streaming data, analytics, networking, developer tooling, drivers, and many more areas. It is very hard to get bored working at MongoDB. Finally, I would be remiss if I did not mention the people. Overall, MongoDB’s engineering culture prioritizes intelligence, low ego, and an ability to get stuff done. CL/CI (Continuous Learning, Continuous Improvement) Working at MongoDB has provided me with opportunities for continued learning and growth. Though I do not program as much as I did earlier in my career, I have recently been exploring the Rust language. I’m excited by Rust because it avoids the tradeoffs between predictable performance and safety. My work in the search space has given me exposure to the fast moving world of AI: vector embeddings, RAG, etc. For various reasons, I think MongoDB is uniquely positioned to do well in this area. On top of this, I’m working on some initiatives that are not fully public. I can say that one focus area is improving the sharding experience for our customers. We believe MongoDB sharding is best-in-breed. Still, the process requires more manual configuration than we think is ideal: customers select the shard key, cluster type, shard count, etc. We give guidance here, but I think we can raise the bar in terms of offering a seamless experience with less “futz”. I’m also working with the search team. We believe there is a natural affinity between MongoDB’s document model and AI/ML workloads. We have some features in the works that extend this integration in new and interesting ways. I also spend a fair bit of time driving quality improvements across our suite of products. Our CTO Jim Scharf frequently refers to our “ big 4 ” goals: security, durability, availability, and performance. These goals are more important than any feature we build. I’ve been working across the company to help teams define their availability SLO/SLAs. It turns out that measuring availability is a subtle topic. For example, a naive approach of counting the percentage of failed requests can underestimate downtime because customers make fewer requests when a service is unavailable. So, the first step is to clarify the definition of availability. Finally, as a lapsed academic (in a distant life, I was a graduate student at the University of Washington Department of Computer Science and Engineering), I’m always interested in finding ways to bridge theory and practice. I’ve been collaborating with some folks in our research team to drive improvements to our replication protocols. There are theoretical results that suggest it is impossible to simultaneously achieve low latency and strong consistency (“linearizability” in the technical jargon). However, we believe there are intermediate points in the consistency/latency spectrum that have not been fully explored. This work hasn't been made into a product yet, but stay tuned. Flexible working MongoDB is a hybrid company. Like many of our engineers, I work outside the company headquarters in New York City (I live in Seattle). I appreciate MongoDB’s approach to hybrid working and that company leadership, starting with Dev , cares about the well-being of their employees. It seems there are companies that don’t seem to trust their employees to make decisions, such as which days to come into the office, so I’m thankful for the autonomy I receive at MongoDB to work in a way that’s best for me. Remote work has its challenges, but I would say that the benefit for my work/life balance has been transformative. Final thoughts I have found MongoDB engineers demonstrate a strong mix of technical depth, pragmatism, and empathy. I have yet to find the “smart jerk” prototype that seems to exist throughout the tech industry. Overall, I have found MongoDB is open to change and growth at both the team level and the individual level. There is a willingness to evolve and improve that aligns with the company’s values and leadership principles and enables the success of our technology and people. Find out more about MongoDB culture and career opportunities by joining our talent community .

November 20, 2024