MongoDB al servizio dei cittadini: Polo Strategico Nazionale, Cloud ed Healthcare

MongoDB
June 30, 2023

Il Piano Nazionale di Ripresa e Resilienza il nostro Paese è pronto a ripartire. La costruzione dei nuovi Data Center made in Italy è un’occasione per accelerare la digitalizzazione della pubblica amministrazione, in linea con gli indirizzi del PNRR. Il Polo Strategico Nazionale (PSN) può essere uno dei catalizzatori della trasformazione digitale della Pubblica Amministrazione (PA), che da diversi anni ha tre le file eccellenze tecnologiche e skills di primissimo livello. Questo processo sta innescando anche un cambio di tendenza delle giovani menti che vedono ora la PA come un luogo in cui può, e deve, essere fatta innovazione.

Il nuovo PSN sta cambiando, senza dubbio, il modello di relazione tra la PA ed i cittadini, che usufruiranno a breve di servizi digitali, sicuri ed innovativi. Le tecnologie e le scelte architetturali del PSN sono improntate ai principi del multicloud, inclusione di innovativi servizi, e naturalmente la sicurezza, forse il tema che più ha dato un impulso decisivo a questa iniziativa. Uno degli obiettivi è quello di garantire soluzioni per la PA in linea con le evoluzioni di mercato, la sicurezza del controllo dei dati, l’accesso e la gestione in territorio Italiano di tutta l’infrastruttura, la scalabilità e la varietà offerta. Per MongoDB è stato un grande riconoscimento, oltre ad una grande opportunità, essere considerato il primo, ed al momento l’unico, DB NoSQL inserito come servizio essenziale del nuovo PSN. Questa scelta strategica cambierà la vita di noi cittadini, usufruiremo di nuovi servizi come anagrafe digitale, sanità digitale, carte d'identità digitali e tante altre cose che ci semplificheranno la vita. Al centro di tutto ci sono i dati, che per questo tipo di servizi sono caratterizzati da una evoluzione dinamica e flessibile, che un DB NoSQL basato su Document Model come MongoDB può assicurare.

Tra i primi servizi che contribuiranno al riammodernamento dell’Italia ci sono quelli appartenenti al mondo della sanità con Il Fascicolo Sanitario Elettronico del cittadino, un accesso unico a tutti i dati sanitari dei cittadini, e con la Telemedicina, che consentirà visite, monitoraggio e diagnosi remote.

Nel contesto dell'evoluzione digitale del settore sanitario, MongoDB e il framework FHIR (Fast Healthcare Interoperability Resources) stanno rivoluzionando la gestione dei dati sanitari in Italia. Questi strumenti offrono nuove possibilità, consentendo una maggiore interoperabilità e una migliore assistenza sanitaria per i cittadini italiani. Inoltre, l'adozione del Public Cloud e privato offre vantaggi significativi per la gestione dei dati sanitari sia per i clienti istituzionali che per la Pubblica Amministrazione.

Il FHIR ed il Fascicolo Sanitario Elettronico rappresentano un'importante innovazione nella sanità italiana, consentendo la raccolta e l'accesso centralizzato ai dati sanitari dei cittadini. MongoDB, con la sua flessibilità e scalabilità, si adatta perfettamente all'archiviazione e alla gestione dei dati del FSE. L'implementazione di FHIR permette l'interoperabilità tra diversi sistemi e applicazioni, facilitando lo scambio sicuro di informazioni sanitarie tra i fornitori di assistenza sanitaria e migliorando la continuità delle cure.

La telemedicina diventerà sempre più importante nel contesto sanitario italiano, consentendo ai pazienti di ricevere cure mediche a distanza. MongoDB offre una soluzione ideale per supportare la telemedicina, consentendo la gestione sicura e scalabile dei dati dei pazienti. Attraverso MongoDB, i dati generati durante le visite e le interazioni online possono essere archiviati e analizzati in modo efficace, consentendo una migliore comprensione dei pazienti e una personalizzazione delle cure.

Ma quali sono le feature di MongoDB per la gestione dei dati sanitari:

  • Scalabilità orizzontale: MongoDB consente di scalare orizzontalmente l'archiviazione dei dati, adattandosi alle crescenti esigenze del sistema sanitario italiano. Ciò garantisce che il database possa gestire grandi volumi di dati in modo efficiente e affidabile.

  • Flessibilità dello schema: MongoDB offre uno schema flessibile, consentendo di adattarsi ai cambiamenti dei requisiti e di integrare nuove informazioni senza dover effettuare modifiche significative allo schema del database. Ciò è particolarmente vantaggioso nel contesto sanitario, in cui i modelli di dati evolvono nel tempo.

  • High availability: MongoDB offre funzionalità avanzate per garantire l'alta disponibilità dei dati. Ciò significa che i sistemi sanitari possono affidarsi a MongoDB per conservare e recuperare i dati in modo affidabile, garantendo una continuità delle cure senza interruzioni.

  • Sicurezza: MongoDB offre funzionalità per gestire i dati sanitari secondo le best practice di sicurezza per garantire la protezione delle informazioni sensibili. Implementando l'accesso sicuro, la crittografia dei dati, il monitoraggio costante, gli aggiornamenti regolari e i processi di backup e ripristino, le organizzazioni sanitarie possono garantire la sicurezza dei dati dei pazienti e la conformità alle normative vigenti.

L'implementazione di MongoDB, il FHIR e l'adozione del cloud pubblico, privato ed il PSN stanno trasformando il mondo dei dati, a partire proprio della sanità.

Il Fascicolo Sanitario Elettronico e la telemedicina sono solo alcuni degli ambiti in cui queste tecnologie offrono vantaggi significativi, migliorando l'interoperabilità, l'efficienza e la qualità dell'assistenza sanitaria.

Nel futuro dei dati, MongoDB si prospetta come una soluzione affidabile per la gestione, la protezione e l'analisi dei dati. Con le sue caratteristiche avanzate, MongoDB è in grado di soddisfare le sfide che ci aspettano, consentendo alle organizzazioni di sfruttare al meglio il potenziale dei loro dati.

← Previous

SQL Migrations Made Easy with PeerIslands SQL Migration Factory

More and more customers are migrating their SQL workloads to MongoDB due to multiple advantages offered by MongoDB. The advantages of MongoDB Atlas include but are not limited to : Increased development velocity Intuitive data representation using the document format Horizontal scalability High performance Elimination of point solutions and simplification of data landscape PeerIslands SQL Migration factory brings together deep hands on keyboard expertise in migrating SQL workloads and a complete suite of tools for handling each stage of the SQL migration journey. PeerIslands SQL migration factory aims at providing a seamless experience to customers looking to migrate out of SQL DBs into MongoDB. The dimensions of a typical SQL migration SQL migrations require a holistic approach across multiple facets. The typical areas which we need to address as part of migrations are below: PeerIslands has helped multiple customers across each of these dimensions. Based on our experience with customers, PeerIslands has created a suite of tools to help address common problems faced by customers. PeerIslands SQL migration tool suite PeerIslands has developed a portfolio of tools that can help you with each stage of the SQL migration journey. MongoDB schema design We have seen customers struggle with developing the right MongoDB schema when the number of source tables exceeds even 10s of tables. Understanding the lay of the land and the access patterns takes significant time commitment from the SMEs and MongoDB team. The below tools help get a clear understanding of the SQL footprint and get guidance on developing the right MongoDB schema very quickly. SQL intelligence This tool provides rich consumable insights of the source SQL workload. The insights help in designing target MongoDB schema accurately. The tool analyzes 2 broad areas: SQL data profile The tool queries against the source SQL profile to derive: List of tables, no of columns and row count Table relationships (1 to 1, 1 to N, M to N) Read heavy tables, Write heavy tables Cardinality of relationships The tool does this by querying the Data dictionary tables / Information schema and indexes. Data access profile The tool analyzes the SQL query logs to provide information about Frequently joined tables Frequently queried fields Types of joins The tool does this by running a parser on SQL query logs Based on the insights gathered from these tools, we can rapidly design the target MongoDB schema. Application impact analysis One of the core impediments to a successful migration is not having a good sense of the effort and complexity of migrating to MongoDB. We have seen customers struggle on how to handle stored procedures that they have developed over the years. Customers also find it helpful to get a sense of how much code refactoring is involved when they move to MongoDB. PeerIslands has tools that address both these issues: Stored procedures analyzer This tool provides rich consumable insights about the stored procedures. The insights help in designing the right approaches for migrating the stored procedures. The tool analyzes the SQL stored procedures to provide information about: Frequently joined tables Frequently queried fields Types of joins Operators used : Logical, relational and others Subqueries The tool does this by running a parser on the stored procedures. The tool also annotates each of the stored procedures with corresponding MongoDB operators. Application analyzer Applications that have a clear DAL layer are easier to migrate to other databases. However actual production applications can have multiple deviations from a standard architecture pattern. We look at the application code to identify: Entity models Inline SQL queries Data type conversions required Date formats Number of APIs Access patterns We leverage existing cyclometric analysis tools that customer may already have to understand the complexity. These tools helps is getting a clear understanding of the amount of effort required to migrate application code. Design validation and performance analyzer One of the common asks from customers is to verify MongoDB performance for their particular use case. Customers want to simulate specific nuances around workloads, schema and queries. This is required to build confidence on MongoDB internally within the organization and to clearly understand & articulate cost performance benefits of MongoDB to other stakeholders. The below set of tools help address this critical need. Test data generator and performance analyzer Test data generator is a solution accelerator that helps customers generate large volume, customizable, close to real world test data for specific customer schemas. The tooling also helps performance test MongoDB with custom load profiles and run specific queries against the data to understand MongoDB performance. This helps us understand the performance of specific application queries and fine tune MongoDB schema / indexes appropriately. Key highlights of the solution are given below: Test data generator & Performance analyzer is one integrated tool - test data generation, load profiling and testing for performance are all available as part of the tooling. There is no need for multiple tooling. The solution generates close to real word data Test data is customizable to specific customer schema With load profiling, you can test MongoDB for specific load testing scenarios MongoDB queries that are used in the application can be tested for performance under specific load conditions MongoDB Sizer MongoDB Sizer helps you define your own MongoDB schema and understand the size implications of that schema. MongoDB Sizer is a fully UI driven tool. Using the tool, we can: Define our schema or import a MongoDB schema Provide sample documents and look at document sizes Define indexes and collections Provide number of documents expected Calculate raw data size The sizer also takes into account various parameters that drive Atlas sizing like IOPS, RAM and storage to calculate Atlas instance sizing. We will be able to understand the number of instances that are required, the cost implications of those instances as well as the number of shards that we need to have. With the MongoDB sizer, users can get clear visibility into Atlas topology that will work for them. Application refactoring Application migrator The application migrator is a set of libraries that ease migrating applications to use MongoDB. The libraries help with: Quickly bootstrapping read / write interfaces with MongoDB collections Decorators for data type conversions from SQL to MongoDB Support for feature flag implementations to help in risk mitigated migrations. The libraries can help with 2 modes of migration: Dual database approach - SQL and MongoDB DBs coexist for a period of time. A subset of data resides in SQL while another subset of data is persisted in MongoDB. This is required in situations where mission critical data is in SQL and one time migration will be very risky. One shot migration - Complete cutover of SQL DB and the data will be persisted only in MongoDB. Stored procedures annotator The annotator helps address one of the major pain points with SQL migrations : Conversion of stored procedures. The annotator takes a SP as a input and provides you with : MongoDB operators that are equivalent to the given SQL operators. Target collection names & attribute names for the given SQL table names. This is based on the schema mapping between SQL and MongoDB. The annotator helps in rapidly rewriting the existing stored procedures into MongoDB queries. MongoDB code scanner In large enterprises, development teams are not fully conversant with MongoDB and consequently do not use the right coding practices on MongoDB. The MongoDB code scanner is a Semgrep based tool that highlights MongoDB coding issues. This helps the developers in understanding the MongoDB coding best practices and also prevents bad code from getting into production. Data migration & synchronization Migrating from a SQL store to MongoDB brings multiple challenges: Migration requires code for schema transformation. This code can become quite complex. Ongoing change capture requires custom programming. Handling dependencies between migration jobs and managing migration infrastructure can become cumbersome. Validation of migrated data takes significant time if the record set becomes large. The MongoDB Relational migrator addresses all these pain points and can be used to rapidly perform the data migration from SQL to MongoDB Spark migrator Mainframe databases like DB2, Big data stores like Hadoop are not supported yet on Relational migrator. We have also seen cases where transformations can become very complex - multi level hierarchical mapping on SQL DBs etc. For these cases PeerIslands has built a fully configuration driven Spark based migration tool which can be used to support migration from any Spark supported source databases. 1Data For microservices transformation, there is a frequent need to perform real time data synchronization between source SQL workloads and MongoDB for extended periods of time. In some cases we would also need reverse data pipelines from MongoDB to SQL workloads. PeerIslands has done extensive work with customers on building real time data pipelines and brings to the table field tested architectural patterns for addressing this requirement. This complete suite of tools enables PeerIslands to rapidly deliver on SQL migrations. Typical engagement approach PeerIslands follows a standardized approach designed to mitigate migration risks and deliver seamless migration experience. We follow a 4 step process that provides: Migration to MongoDB using MongoDB best practices. Getting migrations First time right. table, th, td { border: 1px solid black; } Stages Pre Consult Discovery Solution blueprint Migration factory Description Understand scope of migration and define pre requisites Leverage PI tools for application and data analysis Develop and validate solution blueprints for various components of the migration Finalize migration strategy and end state architecture Implementation of solution blueprints and solution delivery Deliverables Migration scope definition Pre requisites definition SQL footprint analysis Stored procedures analysis Understand application complexity Workload characteristics NFRs Target MongoDB schema Data migration strategy Target technical architecture SP migration strategy Services refactoring templates & PoC DAL layer refactoring templates & PoC Data pipeline PoCs Detailed work package plan & estimates Application development Data migration Data validation Performance tuning PeerIslands provides hands on keyboard expertise with flexibility to independently deliver on application migration with minimal oversight from customers or leverage a co development model where PeerIslands resources work alongside customer engineering teams on migrations. Application refacoring & new application development table, th, td { border: 1px solid black; } Customer Customer Situation PeerIslands Work Top 4 Accounting firm #App Refactoring Hampered by COSMOS SQL DB inability to scale, customer was unable to serve its extra-large clients. Customer could not perform big bang migration to MongoDB as it would entail significant risk PeerIslands rapidly delivered a low impact, risk mitigated migration in less than 8 weeks Fully refactored customer application to work with both MongoDB and existing SQL DB Successfully enabled MongoDB for specific clients based on configuration and designed a solution that had 0 impact on existing customers One of the world’s largest airlines #App Development Faced with lack of agility, high costs and limited talent availability, customer wanted to move out of mainframes and deliver read workloads using MongoDB. PeerIslands is involved in multiple projects Single pane of glass for all crew related tasks Created micro-frontends using model federation to enable standardization and reuse Watchdog UI to resolve data issues arising out of mainframe change feeds Query tuning & optimization of MongoDB to improve application performance GraphQL for servicing data needs of consuming applications Large financial services firm #App Development Delivered near real time understanding of customer interactions with a unified interaction analytics platform. New platform operates at a lower cost than the existing solution with the customer The new platform enables integration of multi channel customer interactions - voice, chat, SMS, video and brings real time insights with a single unified architecture Platform provides customers with lower operational costs and stronger data governance due to simplified tech stack Platform is future proof and designed to flexibly serve multiple user personas without significant need for development One of the world’s largest telcos #App Development Designed and proved out a distributed database architecture to support expected 3X increase in customer device footprint while delivering results 100x faster Helped customer understand the performance of the platform under difficult to simulate future load scenarios Delivered 5 9’s uptime guarantee required for the mission critical platform Showcased platform’s ability to adhere to stringent query SLAs required to satisfy end customer requirements - 40K msgs / sec, QPS < 1s Data migrations table, th, td { border: 1px solid black; } Customer Customer Situation PeerIslands Work US based state health organization #SQL data migration Helped customers realize better cost performance ratio by migrating from Oracle 11g to MongoDB Atlas Helped deliver a fully reusable migration pipeline for moving data from Oracle to MongoDB and enabled customer to rapidly realize business benefits of using MongoDB Highly complex migration - 100s of tables with complex data transformation requirements Large scale : 100’s million of rows in source SQL tables Data synchronization table, th, td { border: 1px solid black; } Customer Customer Situation PeerIslands Work One of the world’s largest hotel chain #SQL MongoDB real time data pipelines Customer was struggling with syncing data from source SQL database to MongoDB Helped rapidly deliver a real time data pipeline with Kafka tooling Pipeline provides capability for schema versioning. Data transformations were implemented using SMTs Debezium based CDC from source SQL A large urgent care provider #SQL MongoDB real time data pipelines Struggling with COVID restrictions, customer had to rapidly implement Digital Front Door to accelerate contactless patient onboarding, digital waiting room and analytics. The DFD solution had to integrate in real time with SQL based EMR systems Helped rapidly deliver a real time data pipeline with Kafka tooling Streaming window based joins for schema transformation Debezium based CDC from source SQL Data profiling to fine tune Kafka & Debezium configuration Please reach out to partners@mongodb.com to fast track your SQL migrations.

June 29, 2023
Next →

Staff Engineering at MongoDB: Your Path to Making Broad Impact

Andrew Whitaker is a Senior Staff Engineer at MongoDB. His previous experience spans tiny startups to enormous organizations like AWS, where he held several different roles focusing on databases. Before joining MongoDB, he worked at a startup building optimized machine learning models in the cloud. Read on to learn more about why Andrew decided to join MongoDB in a senior-level engineering role and how his work is driving improvement within our engineering organization. Why MongoDB I have long been a fan of MongoDB’s products and services. MongoDB the database has always been a pleasure to work with – the system “brings joy” to quote a phrase. As a Python developer, I appreciate how the Python driver feels “Pythonic” in a completely natural way. The programmer interacts with the database using Python constructs: dictionaries, lists, and primitive types. By contrast, SQL databases force me to change my mental model, and the query language feels like an add-on that does not blend with the core language. As an engineer, I am always looking to expand my knowledge and grow my skills. The scope of challenges engineers face at MongoDB is what triggered my interest in the company. We obviously have people working on core databases and distributed systems. But, we also have teams dedicated to machine learning, streaming data, analytics, networking, developer tooling, drivers, and many more areas. It is very hard to get bored working at MongoDB. Finally, I would be remiss if I did not mention the people. Overall, MongoDB’s engineering culture prioritizes intelligence, low ego, and an ability to get stuff done. CL/CI (Continuous Learning, Continuous Improvement) Working at MongoDB has provided me with opportunities for continued learning and growth. Though I do not program as much as I did earlier in my career, I have recently been exploring the Rust language. I’m excited by Rust because it avoids the tradeoffs between predictable performance and safety. My work in the search space has given me exposure to the fast moving world of AI: vector embeddings, RAG, etc. For various reasons, I think MongoDB is uniquely positioned to do well in this area. On top of this, I’m working on some initiatives that are not fully public. I can say that one focus area is improving the sharding experience for our customers. We believe MongoDB sharding is best-in-breed. Still, the process requires more manual configuration than we think is ideal: customers select the shard key, cluster type, shard count, etc. We give guidance here, but I think we can raise the bar in terms of offering a seamless experience with less “futz”. I’m also working with the search team. We believe there is a natural affinity between MongoDB’s document model and AI/ML workloads. We have some features in the works that extend this integration in new and interesting ways. I also spend a fair bit of time driving quality improvements across our suite of products. Our CTO Jim Scharf frequently refers to our “ big 4 ” goals: security, durability, availability, and performance. These goals are more important than any feature we build. I’ve been working across the company to help teams define their availability SLO/SLAs. It turns out that measuring availability is a subtle topic. For example, a naive approach of counting the percentage of failed requests can underestimate downtime because customers make fewer requests when a service is unavailable. So, the first step is to clarify the definition of availability. Finally, as a lapsed academic (in a distant life, I was a graduate student at the University of Washington Department of Computer Science and Engineering), I’m always interested in finding ways to bridge theory and practice. I’ve been collaborating with some folks in our research team to drive improvements to our replication protocols. There are theoretical results that suggest it is impossible to simultaneously achieve low latency and strong consistency (“linearizability” in the technical jargon). However, we believe there are intermediate points in the consistency/latency spectrum that have not been fully explored. This work hasn't been made into a product yet, but stay tuned. Flexible working MongoDB is a hybrid company. Like many of our engineers, I work outside the company headquarters in New York City (I live in Seattle). I appreciate MongoDB’s approach to hybrid working and that company leadership, starting with Dev , cares about the well-being of their employees. It seems there are companies that don’t seem to trust their employees to make decisions, such as which days to come into the office, so I’m thankful for the autonomy I receive at MongoDB to work in a way that’s best for me. Remote work has its challenges, but I would say that the benefit for my work/life balance has been transformative. Final thoughts I have found MongoDB engineers demonstrate a strong mix of technical depth, pragmatism, and empathy. I have yet to find the “smart jerk” prototype that seems to exist throughout the tech industry. Overall, I have found MongoDB is open to change and growth at both the team level and the individual level. There is a willingness to evolve and improve that aligns with the company’s values and leadership principles and enables the success of our technology and people. Find out more about MongoDB culture and career opportunities by joining our talent community .

November 20, 2024