MongoDB Microservices Help HM Revenue & Customs Cut Wait Times in Half and Save £8.2m in Operational Costs

Mat Keep
March 30, 2016 | Updated: December 8, 2016

Taxes and death. The two things in life we can’t escape. So what on god’s green earth could ever compel someone to tweet positive comments about a tax self assessment process? Well, let me explain.

HM Revenue & Customs, the UK’s tax body, has been a leading agency in the government’s digital transformation programme. The aim of which is “building digital services that are simpler, clearer and faster to use”. It’s fair to say that tax services in the UK, were not widely regarded as being simple, clear or fast.

To create the underlying infrastructure that could power that transformation, HMRC, with its partner Equal Experts, built a network of microservices that use MongoDB’s scalability, data model flexibility and ease of development. This project also ushered in a move to agile development and a default preference for open source technology, like Docker containers.

Earlier this month Mike Potter, Digital Transformation Director at HM Revenue & Customs, revealed the full impact of those transformed services can have:

“We received an unprecedented number of Self Assessment returns online this year - 89% or 9.24m customers. This is fantastic and I wanted to share with you some other numbers I’m excited by.”

The online form replaced a paper process that users had accused of being difficult and complex, which led to a high volume of phone calls to the support staff. That’s now changed. Mike continued:

“So how did the 2016 peak compare to 2015? Well, this year we received:

480k more returns online

300k less paper returns - down 21%

41k fewer calls - down 7% - and, in addition, a further 88k customers got the help they needed through webchat instead of having to phone us

And we halved phone waiting times, down to just over 5 minutes.”

These are not minor improvements. Waiting times cut in half! That means more time and dramatically less stress in the lives of UK citizens. Mike also highlighted how this had been positively received on social media:

How many times have you seen someone tweet their affection for a tax service, right after filling out a self assessment?

The Digital Transformation director also looked at how the Multi-channel Digital Tax Platform or MDTP had facilitated these dramatic changes. He explained that “[MDTP] hosts the digital tax accounts at the heart of our digital future, plus all of our new online services”.

MDTP has launched 28 services, handled 55 million visits and, get this, saved £8.5 million in operating costs. That’s right, the services are vastly improved and the tax papers seeing savings in excess of, for our American friends, $12 million USD.

These impressive numbers are the result of many people doing incredible work. One part of that is the microservice architecture that supports HMRC’s digital transformation. With the help of partner Equal Experts, HMRC has used modern technologies such as MongoDB and Docker to make services more agile, flexible and scalable. Before the microservices transformation, HMRC were pushing two releases a year. Now it’s 50 releases a week.

Simple, clear and fast. And it saves the taxpayers millions. Digital transformation at HMRC is a demonstration of what’s possible when people take a completely new look at how to solve a problem. As Ben Franklin said: we’ll never escape tax or death. But it’s a relief to know at least one of life’s constants is getting a little less painful.

Find out how microservices can help your business. Read more about enabling microservices. Enabling Microservices: Containers & Orchestration Explained

About the Author - Mat Keep

Mat is a director within the MongoDB product marketing team, responsible for building the vision, positioning and content for MongoDB’s products and services, including the analysis of market trends and customer requirements. Prior to MongoDB, Mat was director of product management at Oracle Corp. with responsibility for the MySQL database in web, telecoms, cloud and big data workloads. This followed a series of sales, business development and analyst / programmer positions with both technology vendors and end-user companies.

← Previous

Secure MongoDB with X.509 Authentication

Disclaimer - the following article is intended for a test environment. Overview In this tutorial, I will be describing the detailed process of setting up X.509 based authentication, both for cluster inter-member authentication as well as for client authentication, using a local CA (Certificate Authority). X.509 is one of the multiple authentication mechanisms supported by MongoDB. An X.509 certificate is a digital certificate that uses the widely accepted international X.509 public key infrastructure (PKI) standard to verify that a public key, presented by a client or another member of the cluster, belongs to that said client or member. One of the main benefits compared to conventional password based authentication is it’s more secure in a sense that each machine would need a dedicated key to join the cluster. So stealing an existing key from another machine isn't going to be very helpful for those with an evil agenda. In MongoDB, we need to understand the distinction between member authentication and client authentication. MongoDB is a distributed database and deployments almost always consist of multiple mongod or mongos processes running on multiple machines. Member authentication refers to the fact that these machines need to verify each other to ensure a node attempting to replicate data is indeed part of the current cluster. On the other hand, client authentication refers to those MongoDB clients, including mongo shell, export/import tools and MongoDB drivers. Below is a deployment of standard 3 node replica set and a client. To enable X.509 certificate based authentication, essentially you will need the following set of certificates: 3x server certificates, one for each MongoDB member 1x client certificate for one client 1x certificate for Root CA and 1x certificate for Signing CA All the server certificates and client certificates must be signed by same CA. The bulk part of the work for setting up X.509 authentication would be to create these certificates. The creation of Root CA and Signing CA and the relevant parts of the signing process is a one time effort, while the creation of the server/client certificates are as needed. For simplicity, I am going to use one machine and I will run multiple mongods on different ports for this exercise. I also put the steps into an executable script if you want to jump to the end. You can download and run the script, and you will have a test setup just like the diagram above in no time. This way you can quickly get things working, boost your confidence, and then come back to study the details. Preparation One linux box, I tested on RedHat 7 and Ubuntu 14.04 but it should work with other flavors as well. Download MongoDB Enterprise 3.2 Install MongoDB following instructions documented here Download the demo shell script and save to a clean directory Running the script The script will setup a test replica set on your machine and configure X.509 authentication mechanism for that replica set. Before you run the script, double check: OpenSSL is installed (typically comes with most Linux distributions) Make sure mongod/mongo are in the executable path Make sure no mongod is already running on 27017 port, or change the port numbers in the shell script To run the script, go to the directory that contains the script and execute the following: # chmod +x setup-x509.sh # ./setup-x509.sh If everything goes smoothly, you should have a 3 nodes replica set running with X.509 as member auth. Then in the current directory, you may connect to the primary node with a newly generated client certificate client1.pem: mongo --ssl --sslPEMKeyFile client1.pem --sslCAFile root-ca.pem --sslAllowInvalidHostnames Note above step just allows you to connect to the MongoDB shell. You will not have any permission at this point. To do anything meaningful, you need to authenticate yourself using following command: $mongo ... > db.getSiblingDB("$external").auth( { mechanism: "MONGODB-X509", user: "CN=client1,OU=MyClients,O=MongoDB China,L=Shenzhen,ST=GD,C=CN" } ); > db.test.find() If you are able to execute the last find statement, congratulations, the X.509 authentication is working! Now it's time to perform an anatomy of the script to understand what are the key steps involved in setting up the X.509 authentication mechanism. Main Parts of the Script Initialization Create local CA and signing keys Generate and sign server certificates for member authentication Generate and sign client certificates for client authentication Start MongoDB cluster in non-auth mode Setup replica set and initial users Restart MongoDB replica set in X.509 mode using server certificates 0. Variable initialization First initialize some variables. Feel free to modify the values as appropriate. dn_prefix="/C=CN/ST=GD/L=Shenzhen/O=MongoDB China" ou_member="MyServers" ou_client="MyClients" mongodb_server_hosts=( "server1" "server2" "server3" ) mongodb_client_hosts=( "client1" "client2" ) mongodb_port=27017 Here dn_prefix will be used to construct the full DN name for each of the certificate. ou_member is used to have a different OU than the client certificates. Client certificates uses ou_client in its OU name. mongodb_server_hosts should list the hostname (FQDN) for all the MongoDB servers while mongodb_client_hosts should list the hostnames for all of the client machines. For a clean start, let’s kill the running mongods and clean up the working directory (note: don’t use -9 to kill mongod per the manual ): kill $(ps -ef | grep mongod | grep set509 | awk '{print $2}') mkdir -p db 1. Create local root CA A root CA (Certificate Authority) is at the top of the certificate chain. This is the ultimate source of the trust. Ideally a third party CA should be used. However in the case of an isolated network (very typical in large enterprise environment), or for testing purpose, we need to use local CA to test the functionality. echo "##### STEP 1: Generate root CA " openssl genrsa -out root-ca.key 2048 # !!! In production you will want to password protect the keys # openssl genrsa -aes256 -out root-ca.key 2048 openssl req -new -x509 -days 3650 -key root-ca.key -out root-ca.crt -subj "$dn_prefix/CN=ROOTCA" mkdir -p RootCA/ca.db.certs echo "01" >> RootCA/ca.db.serial touch RootCA/ca.db.index echo $RANDOM >> RootCA/ca.db.rand mv root-ca* RootCA/ Above we first created a key pair root-ca.key with AES256 encryption and 2048 bits strength. Then using openssl req command to generate a self-signed certificate with a validity of 3650 days. One thing to call out here is the argument -x509 which tells openssl to self sign the certificate instead of generating a signing request (as what we will do below). The output is a crt file, a certificate file that contains the public key of the root CA. 2. Create CA config A CA config file is used to provide some default settings during the certificate signing process, such as the directories to store the certificates etc. You may change the defaults in root-ca.cfg file after it is generated or simply change them within the script. echo "##### STEP 2: Create CA config" cat >> root-ca.cfg ${host}.pem done This script is in a for loop to generate multiple certificates. 3 key steps are involved with each certificate: Use openssl genrsa command to create a new key pair Use openssl req command to generate a signing request for the key Use openssl ca command to sign the key and output a certificate, using the Signing CA we created earlier as the signer Notice the variable $ou_member . This signifies the difference between server certificates and client certificates. Server and client certificates must differ in the organization part of the Distinguished Names, or in another word and must differ at least in one of the O, OU, or DC values. 5. Generate and Sign client certificates These certificates are used by clients, such as mongo shell, mongodump, and Java/python/C# drivers to connect to a MongoDB cluster. This step is essentially the same as step 4 except for the use of $ou_client . This will make the combination of the DC/OU/O for these certificates will be different from the server certs above. echo "##### STEP 5: Create client certificates" # Pay attention to the OU part of the subject in "openssl req" command for host in "${mongodb_client_hosts[@]}"; do echo "Generating key for $host" openssl genrsa -out ${host}.key 2048 openssl req -new -days 365 -key ${host}.key -out ${host}.csr -subj "$dn_prefix/OU=$ou_client/CN=${host}" openssl ca -batch -name SigningCA -config root-ca.cfg -out ${host}.crt -infiles ${host}.csr cat ${host}.crt ${host}.key > ${host}.pem done 6. Bring up replicaset in non-auth mode MongoDB does not create a default root/admin user when enabling authentication, and there is no exception with X.509 mode. Instead, the best practice is to create an initial admin user first, then to enable authentication after the admin user has been created. Here we're starting a replica set in non-auth mode. echo "##### STEP 6: Start up replicaset in non-auth mode" mport=$mongodb_port for host in "${mongodb_server_hosts[@]}"; do echo "Starting server $host in non-auth mode" mkdir -p ./db/${host} mongod --replSet set509 --port $mport --dbpath ./db/$host \ --fork --logpath ./db/${host}.log let "mport++" done sleep 3 Now our replica set is up, we need to initialize the replica set and add a user. 7. Initialize replicaset and add initial user When using X.509 client authentication, each client must have a user created in MongoDB and the user must be granted the necessary permissions. The username must be same as the client's DN (Distinguished Name), which can be obtained by running an openssl command: # obtain the subject from the client key: client_subject=`openssl x509 -in client1.pem -inform PEM -subject -nameopt RFC2253 | grep subject | awk '{sub("subject= ",""); print}'` This would return something like: CN=client1,OU=MyClients,O=MongoDB China,L=Shenzhen,ST=GD,C=CN Obviously it's not mandatory to use the openssl command. It's fairly straightforward to deduce the subject string by concatenating the relevant parts of the DN as shown above. Once we have the DN name, let's initialize the replica set and add a user: myhostname=`hostname` cat > setup_auth.js

March 29, 2016

Next →

Hanabi Technologies Uses MongoDB to Power AI Assistant, Hana

For all the hype surrounding generative AI, cynics tend to view the few real-world implementations as little more than “fancy chatbots.” But for Abhinav Aggarwal, CEO of Hanabi Technologies, the idea of a generative AI-powered bot that is more than just an assistant was intriguing. “I’d been using ChatGPT since it launched,” said Aggarwal. “That got me thinking: How could we make a chatbot that was like a team member?” And with that concept, Hana was born. The problem with bots “Most generative AI chatbots do not act like people; they wait for a command and give a response,” said Aggarwal. “We wanted to create a human-like chatbot that would proactively help people based on what they wanted—automating reminders, for example, or fetching time zones from your calendar to correctly schedule meetings.” Hanabi’s flagship product, Hana, is an AI assistant designed to enhance team collaboration within Google Chat, working in concert with Google Workspace and its suite of products. “Our target customers are smaller companies of between 10 and 50 people. At this size you’re not going to build your own agent from scratch,” he said. Hana integrates with Google APIs to deliver a human-like assistant that chimes in with helpful interventions, such as automatically setting reminders and making sure meetings are booked in the right time zone for each participant. “Hana is designed to bring AI to smaller companies and help them collaborate in a space where they are already working—Google Workspace,” Aggarwal explained. The MongoDB Atlas solution For Hana to act like a member of the team, Hanabi needed to process massive amounts of data to support advanced features like retrieval-augmented generation (RAG) for better information retrieval across Google Docs and many other sources. And with a rapidly growing user base of over 600 organizations and 17,000+ installs, Hanabi also required a secure, scalable, and high-performing data storage solution. MongoDB Atlas provided a flexible document model, built-in vector database, and scalable cloud-based infrastructure, freeing Hanabi engineers to build new features for Hana rather than focusing on rote tasks like data extract, transform, and load processes or manual scaling and provisioning. Now, MongoDB Atlas handles a variety of responsibilities: Scalability and security: MongoDB Atlas’s auto-scaling and automatic backup features have enabled Hanabi to seamlessly grow its user base without the need for manual database management. RAG: MongoDB Atlas plays a critical role in Hana’s RAG functionality. The platform enables Hanabi to split Google Docs into small sections, create embeddings, and store these sections in Atlas’s vector database. Development Processes: According to Aggarwal, MongoDB’s flexibility in managing changing schemas has been essential to the company’s fast-paced development cycle. Data Visualization: Using MongoDB Atlas Charts has enabled Hanabi to create comprehensive dashboards for real-time data visualization. This has helped the team track usage, set reminders, and optimize performance without needing to build a manual dashboard. Impact and results With MongoDB Atlas, Hanabi can successfully scale Hana to meet the demands of its rapidly expanding user base. The integration is also enabling Hana to offer powerful features like automatic interactions with customers, advanced information retrieval from Google Docs, and manually added memory snippets, making it an essential tool for teams around the world. Next steps Hanabi plans to continue integrating more tools into Hana while expanding its reach to personal Gmail users. The company is also rolling out a new automatic-interaction feature, further enhancing Hana’s ability to proactively assist users without direct commands. MongoDB Atlas remains a key component of Hanabi’s stack, alongside Google Kubernetes Engine, NestJS, and LangChain, enabling Hanabi to focus on innovating to improve the customer experience. Tech Stack MongoDB Atlas Google Kubernetes Engine NestJS LangChain Are you building AI apps? Join the MongoDB AI Innovators Program today! Successful participants gain access to free MongoDB Atlas credits, technical enablement, and invaluable connections within the broader AI ecosystem. If your company is interested in being featured, we’d love to hear from you. Connect with us at ai_adopters@mongodb.com.

November 21, 2024