How to Migrate Your Flask App From SQL To MongoDB
Rate this tutorial
Migrating a Flask application from SQL to MongoDB involves several steps, as the two database systems have different paradigms. This article will guide you through converting an existing Flask blog application that uses SQLAlchemy to Flask-PyMongo to integrate MongoDB with the Flask application. We will explain the changes made to each file and the reasons behind them.
MongoDB is a NoSQL database known for its document-style kind of structure and ease of use.
Key concepts
- Collections: Analogous to tables in SQL databases, collections in MongoDB store documents. However, unlike tables, collections do not enforce a schema, meaning each document can have different fields.
- Documents: This is the primary data structure in MongoDB, similar to rows in SQL databases, but more complex. Documents are JSON-like objects that can contain arrays and nested subdocuments.
- BSON: Binary representation of JSON is used internally by MongoDB. It extends JSON with additional data types, such as Date and Binary.
Feature | SQL | MongoDB |
Schema | Enforces a fixed schema with predefined tables and columns | Schema-less, allowing for flexible and dynamic data models |
Data Structure | Stores data in tables with rows and columns | Stores data in collections of documents |
Relationships | Uses foreign keys and joins to establish relationships between tables | Embeds documents within other documents or references them, but lacks joins |
Query Language | Uses Structured Query Language (SQL) | Uses a rich, JSON-based query language |
Before migrating the Flask application from SQL to MongoDB, thorough planning is necessary to ensure a smooth transition and preserve data integrity.
Proper planning helps identify potential blockers and develop strategies to address them.
- Set up MongoDB environment
- Assess current SQL schema and relationships: Review the existing database schema, including tables, relationships, and data types. Identify complex queries and relationships that might require special handling in MongoDB.
- Define the MongoDB schema: Design the MongoDB schema considering the document-oriented nature of the database. Map SQL tables and relationships to MongoDB collections and documents.
- Backup existing data: Create a complete backup of the current SQL database to prevent data loss. Ensure backup processes are reliable and can be restored if needed.
- Data transformation and migration strategy: Choose a migration tool or write custom scripts to handle data migration, as we will do in this article.
- Update Flask application code: Modify Flask models to use MongoDB instead of SQLAlchemy. Update routes, forms, and templates to work with the new data structure. Test each module of the application to ensure compatibility with MongoDB.
- Post-migration testing: Thoroughly test the application to ensure all functionalities are working as expected. Monitor performance and optimize queries and indexes in MongoDB.
The application to be migrated is a Flask blog application with the following features:
- Creating, editing, and viewing blog posts
- Searching for blog posts
Before integrating MongoDB with a Flask application, the first step is to install MongoDB on your local machine. MongoDB provides detailed installation guides for various operating system versions.
After installing MongoDB, the next step is to configure it to work on your Flask application. This involves installing the
Flask-PyMongo
package and setting up the connection with your local mongo db server.Flask-PyMongo is a Flask extension that makes it easy to use MongoDB instances in your Flask app.
Install it using pip:
1 pip install Flask-PyMongo Flask
You can also install Flask if you haven’t already.
- models.py: Defines the Post model using SQLAlchemy with attributes for id, title, content, and date_posted
- __init__.py: Initializes the Flask application and SQLAlchemy
- forms.py: Contains form definitions using Flask-WTF
- manage.py: Manages database migrations using Flask-Script and Flask-Migrate
- routes.py: Contains the route definitions and logic for handling requests
What you will do is make a copy of the folder containing the old project. We will use the current file structure and just replace the current code with some new code.
Go to models.py and paste in the following code:
1 from datetime import datetime 2 from bson.objectid import ObjectId 3 class Post: 4 def __init__(self, title, content, date_posted=None, _id=None): 5 self.title = title 6 self.content = content 7 self.date_posted = date_posted if date_posted else datetime.utcnow() 8 self._id = _id if _id else ObjectId() 9 10 def to_dict(self): 11 return { 12 "title": self.title, 13 "content": self.content, 14 "date_posted": self.date_posted, 15 "_id": self._id 16 } 17 18 @staticmethod 19 def from_dict(data): 20 return Post( 21 title=data.get('title'), 22 content=data.get('content'), 23 date_posted=data.get('date_posted'), 24 _id=data.get('_id') 25 ) 26 27 def __repr__(self): 28 return f"Post('{self.title}', '{self.date_posted}')"
This code defines a
Post
class that models a blog post for use with a MongoDB database, utilizing PyMongo's BSON types. The class includes an initializer (__init__
method) that sets the title, content, date posted, and a unique identifier for each post. If the date posted or identifier is not provided during instantiation, it defaults to the current UTC time and a new ObjectId
, respectively. The class also provides a to_dict
method to convert an instance into a dictionary format suitable for MongoDB storage, and a from_dict
static method to create a Post instance from a dictionary. Additionally, the __repr__
method is defined to offer a clear string representation of the post, useful for debugging and logging.The use of this is for the handling of blog post data within a Flask web application that uses MongoDB for its database. Converting the
Post
instances to and from dictionary format allows easy interaction with MongoDB's document-oriented storage.Paste the following code into your __init__.py file. The Flask application is initialized to use MongoDB through the Flask-PyMongo extension. The PyMongo instance (mongo) is set up with the Flask application which enables MongoDB support. The application routes are imported to define the URL endpoints and their associated handlers.
1 from flask import Flask 2 from flask_pymongo import PyMongo 3 from config import Config 4 5 6 app = Flask(__name__) 7 app.config.from_object(Config) 8 mongo = PyMongo(app) 9 10 11 from app import routes
Additionally, update the
Config
class in your config.py to include MongoDB settings:1 import os 2 class Config: 3 SECRET_KEY = os.urandom(24) 4 MONGO_URI = 'mongodb://localhost:27017/blogdb'
manage.py is used for database migrations in SQL, which isn't directly applicable to MongoDB so you can delete the file. If you need migrations, the Beanie ODM provides some support for migrations with MongoDB.
1 from flask import render_template, url_for, flash, redirect, request 2 from app import app, mongo 3 from bson.objectid import ObjectId 4 from app.models import Post 5 from app.forms import PostForm 6 7 8 9 def home(): 10 """ 11 Render the home page with a list of all posts. 12 """ 13 posts_data = mongo.db.posts.find() 14 posts = [Post.from_dict(post) for post in posts_data] 15 return render_template("index.html", posts=posts) 16 17 18 19 def new_post(): 20 """ 21 Handle the creation of a new post. Render the new post form and process the form submission. 22 23 """ 24 form = PostForm() 25 if form.validate_on_submit(): 26 post = Post(title=form.title.data, content=form.content.data) 27 mongo.db.posts.insert_one(post.to_dict()) 28 flash("Your post has been created!", "success") 29 return redirect(url_for("home")) 30 return render_template("post.html", title="New Post", form=form) 31 32 33 34 def post(post_id): 35 """ 36 Display a single post by its ID. 37 """ 38 post_data = mongo.db.posts.find_one_or_404({"_id": ObjectId(post_id)}) 39 post = Post.from_dict(post_data) 40 return render_template("detail.html", post=post) 41 42 43 44 def edit_post(post_id): 45 """ 46 Handle the editing of an existing post. Render the edit post form and process the form submission. 47 """ 48 post_data = mongo.db.posts.find_one_or_404({"_id": ObjectId(post_id)}) 49 post = Post.from_dict(post_data) 50 form = PostForm() 51 if form.validate_on_submit(): 52 updated_post = { 53 "$set": {"title": form.title.data, "content": form.content.data} 54 } 55 mongo.db.posts.update_one({"_id": ObjectId(post_id)}, updated_post) 56 flash("Your post has been updated!", "success") 57 return redirect(url_for("post", post_id=post_id)) 58 elif request.method == "GET": 59 form.title.data = post.title 60 form.content.data = post.content 61 return render_template("edit_post.html", title="Edit Post", form=form) 62 63 64 65 def delete_post(post_id): 66 """ 67 Handle the deletion of a post by its ID. 68 """ 69 mongo.db.posts.delete_one({"_id": ObjectId(post_id)}) 70 flash("Your post has been deleted!", "success") 71 return redirect(url_for("home")) 72 73 74 75 def search(): 76 """ 77 Handle the search functionality for posts. Search for posts by title or content. 78 """ 79 query = request.args.get("query") 80 posts_data = mongo.db.posts.find( 81 { 82 "$or": [ 83 {"title": {"$regex": query, "$options": "i"}}, 84 {"content": {"$regex": query, "$options": "i"}}, 85 ] 86 } 87 ) 88 posts = [Post.from_dict(post) for post in posts_data] 89 return render_template("index.html", posts=posts)
To make the Flask application compatible with MongoDB, several modifications were made to the routes to interact with MongoDB instead of SQLAlchemy. Firstly, the Post class instances are now converted to dictionaries using the to_dict method for storage in MongoDB. The home route retrieves all posts from the MongoDB collection using
mongo.db.posts.find()
, converts them to Post instances using the from_dict
method, and then renders them.For creating new posts in the new_post route, the post data is inserted into MongoDB with
mongo.db.posts.insert_one(post.to_dict())
. For retrieving, editing, and deleting posts, routes, post, edit_post
, and delete_post
use MongoDB's query methods (find_one_or_404
, update_one
, and delete_one
) with ObjectId
to handle the unique identifiers. The search route performs a text search on titles and content using MongoDB's regex capabilities, ensuring posts matching the query are fetched and rendered.The migration process involves exporting data from the existing SQL database and importing it into MongoDB. We will start by exporting the data from SQLite. On GitHub, you will find the db file we will use in this tutorial.
If you are using my db file, run the following code to query the database to list all tables and determine their names.
1 import sqlite3 2 3 4 # Connect to the SQLite database 5 db_path = 'relative/path/to/site.db' 6 conn = sqlite3.connect(db_path) 7 cursor = conn.cursor() 8 9 # Query to get all table names 10 cursor.execute("SELECT name FROM sqlite_master WHERE type='table';") 11 tables = cursor.fetchall() 12 13 # Close the connection 14 conn.close() 15 16 print(tables)
This Python code above connects to your existing SQLite database specified in the path
db_path and
retrieves the names of all tables within it, then closes the connection. The SQL query SELECT name FROM sqlite_master WHERE type='table';
is executed to fetch the names of all tables in the database. You should have post as your output.After that, run the following code to export the data to a JSON file.
1 import sqlite3 2 import json 3 4 # Connect to the SQLite database 5 db_path = 'relative/path/to/site.db' 6 conn = sqlite3.connect(db_path) 7 cursor = conn.cursor() 8 9 # Export data from the 'post' table 10 cursor.execute("SELECT * FROM post") 11 rows = cursor.fetchall() 12 13 # Get the column names 14 column_names = [description[0] for description in cursor.description] 15 16 # Convert to list of dictionaries 17 data = [dict(zip(column_names, row)) for row in rows] 18 19 # Save to a JSON file 20 with open('post.json', 'w') as f: 21 json.dump(data, f, indent=4) 22 23 # Close the connection 24 conn.close()
This Python code connects to your SQLite database, extracts all data from the post table, converts it to a JSON format, and saves it to a file. A cursor object is created from this connection to execute SQL commands. The SQL query
SELECT * FROM post
is used to fetch all rows from the post table, and the results are stored in the rows variable. The variable column_names
is used to extract the column names from the cursor's description attribute. Each row is then combined with the column names to create a list of dictionaries, where each dictionary represents a row of the table. And finally, the list is saved as a JSON file named post.json using the json.dump()
function.After running, you will then be provided with a JSON file (post.json) that looks like this:
1 [ 2 { 3 "id": 1, 4 "title": " Superworm (Zophobas morio)", 5 "content": "Superworms, scientifically known as Zophobas morio, are the larval stage of a darkling beetle species...", 6 "date_posted": "2024-05-14 18:03:43.219562" 7 }, 8 { 9 "id": 2, 10 "title": "MongoDB article", 11 "content": "MongoDB is a widely-used NoSQL database known for its flexibility...", 12 "date_posted": "2024-05-14 18:04:57.884181" 13 }, 14 { 15 "id": 3, 16 "title": "Unique Techniques in Farming", 17 "content": "Modern farming has evolved in various ways…", 18 "date_posted": "2024-05-14 18:07:12.665001" 19 }, 20 { 21 "id": 4, 22 "title": " Examples of NoSQL Databases", 23 "content": "NoSQL databases have improved the way...", 24 "date_posted": "2024-05-14 18:08:28.759601" 25 }, 26 { 27 "id": 5, 28 "title": "d", 29 "content": "d", 30 "date_posted": "2024-05-18 18:06:59.767954" 31 } 32 ]
You can now import this JSON file into your MongoDB database. Here's a script to help you with that:
1 from pymongo import MongoClient 2 import json 3 from datetime import datetime 4 5 def import_json_to_mongo(db_name, collection_name, json_path): 6 client = MongoClient("mongodb://localhost:27017/") 7 db = client[db_name] 8 9 def convert_date(data): 10 # Update 'date_posted' field to datetime object 11 for item in data: 12 if 'date_posted' in item: 13 item['date_posted'] = datetime.strptime(item['date_posted'], '%Y-%m-%d %H:%M:%S.%f') 14 return data 15 16 with open(json_path, 'r') as f: 17 data = json.load(f) 18 data = convert_date(data) 19 db[collection_name].insert_many(data) 20 21 client.close() 22 23 if __name__ == "__main__": 24 db_name = 'blogdb' 25 collection_name = 'posts' 26 json_path = 'flask-mongo/post.json' 27 import_json_to_mongo(db_name, collection_name, json_path)
The connection string
mongodb://localhost:27017/blogdb
connects to your MongoDB instance running on localhost at port 27017 and uses the "blogdb" database. If the specified database does not exist, MongoDB will create it automatically when you first store data.For the templates, you only need to change the variable for id. You will change all instances of
post_id=post.id
to post_id=post._id
.You can now run the app with the following command:
1 python run.py
Migrating a Flask application from SQL to MongoDB involves a deep understanding of the differences between both database systems and then planning to ensure a smooth transition. This article provided a step-by-step guide on how to convert an existing Flask blog application from using SQLAlchemy to using Flask-PyMongo to integrate MongoDB with the Flask application. We focused mainly on covering the necessary changes to each file. In a real-world scenario, there would be a need to construct a document schema from a tabular schema with joins and several queries. Even though this is a simple application, this tutorial covers the necessary steps to have a successful migration.
Top Comments in Forums
There are no comments on this article yet.