Overview
- Inserting a piece of data or loads of data into a database server is a common operation. For example, when a user signs up for a web service in Internet, the details of the user are stored into some database server.
- In MongoDB, a JSON document corresponds to a row of data or a data record.
- These records are stored in binary JSON format into the MongoDB collections. MongoDB collections are like what tables are to RDBMSes.
- Since JSON is a schema + data format, technically it is possible to insert documents with different schema into a same MongoDB collection.
- However, it is always better to place the documents with different structures in designated collections.
Inserting documents into a MongoDB collection
- Import the pymongo module.
- Create an instance of pymongo.MongoClient.
- MongoClient takes the port number and the host of the MongoDB server as parameters.
- From the MongoClient instance obtain a database instance.
- The collections of a MongoDB database, are available as attributes of the database instance.
- Using the object.attribute notation access the collection and call insert() method on the collection instance.
- insert() takes a JSON document as a string and inserts into the specified MongoDB collection.
Example
# import MongoClient from pymongo
from pymongo import MongoClient
# Get a MongoClient object
connectionObject = MongoClient('mongodb://localhost:27017/')
#Access the database using object.attribute notation
databaseObject = connectionObject.sample
#Access the mongodb collection using object.attribute notation
collectionObject = databaseObject.test
# insert a simple json document into the test collection
collectionObject.insert({"red":123, "green":223, "blue":23})
collectionObject.insert({"red":146, "green":46, "blue":246})
# Using find() query all the documents from the collection
for document in collectionObject.find():
# print each document
print(document)Output
{'_id': ObjectId('5ab9fb8302334a031cc2fc13'), 'red': 123, 'green': 223, 'blue': 23}
{'_id': ObjectId('5ab9fb8302334a031cc2fc14'), 'red': 146, 'green': 46, 'blue': 246}Inserting multiple documents into MongoDB concurrently using MongoClient's socket pool
- MongoClient is an abstraction of "1 to n" number of database connections to a MongoDB server from a process.
- A MongoClient has a connection pool with a default size of 100.
- Within a process one or more threads can make use of the pool of socket connections to the database utilizing up to maxpoolsize connections in accordance with the parameters waitQueueMultiple and waitQueueTimeoutMS parameters.
- When the maxpoolsize has reached and no more requests can wait in queue MongoClient starts raising exceptions.
- The Python example below tries to insert 400 documents concurrently using 400 threads using the MongoClient and its connection pool.
Example
from pymongo import MongoClient
from threading import Thread
THREAD_COUNT = 400
# Derive from Threading.thread to create a specialised insert thread
class DataInsertThread(Thread):
database = None
threadNumber = None
def __init__(self, database_in, threadNumber):
self.database = database_in
self.threadNumber = threadNumber
Thread.__init__(self)
def run(self):
self.database.test.insert({"Data inserted by thread":self.threadNumber})
# Get a MongoClient instance
mongoClient = MongoClient("mongodb://localhost:27017/",
maxPoolSize=200, # connection pool size is 200
waitQueueTimeoutMS=200, # how long a thread can wait for a connection
waitQueueMultiple=500 # when the pool is fully used 200 threads can wait
)
# Get the database object
databaseObject = mongoClient.sample
insertThreads = []
# Create insert threads
for threadNum in range(THREAD_COUNT):
insertThread = DataInsertThread(databaseObject, threadNum)
insertThreads.append(insertThread)
# Start the insert thread
insertThread.start()
# Wait till all the insert threads are complete
for insertThread in insertThreads:
insertThread.join()Output
To see the record count after executing the above example use the MongoDB shell and issue the command count().
> db.test.count()
400