MongoDB, the database for modern applications. If you are not familiar with MongoDB, its a general purpose, document based, distributed database.
What are we building today?
Today, we will setup a 3 Node Replica Set MongoDB Cluster and go through a couple of scenarios to showcase data durability on how MongoDB can recover from node failures using replication.
What is a MongoDB Replica Set
MongoDB Replica Sets are Groups of mongod proceeses that maintain the same dataset. Replica Sets provide high availability and redundancy. Replication provides redundancy and increases data availability. Copies of the data are replicated across nodes (mongod processes), replication provides a level of fault tolerance against the loss of a single database server.
Replication can provide increased read capacity, as applications can be configured to read from the slave nodes and write to the primary nodes. With data locality, you can spread your nodes accross regions and let the applications read from the nodes closest to them.
In a replica set, a primary receives all the write operations and you can only have one primary capable of confirming writes. The primary records all the changes in its oplog.
Install MongoDB
I have provisioned 3 Ubuntu 18 nodes on Civo:
As you can see they have private networking, so I will be adding the private addresses in my hosts file for each node, as all my communucation between the servers will happen via private networking:
$ cat /etc/hosts
10.163.68.26 mongodb-1.pvt
10.163.68.12 mongodb-2.pvt
10.163.68.9 mongodb-3.pvt
Go ahead update the repositories, add the mongodb repostory, update and install mongodb:
$ sudo apt update
$ sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 9DA31620334BD75D9DCB49F368818C72E52529D4
$ echo "deb [ arch=amd64 ] https://repo.mongodb.org/apt/ubuntu bionic/mongodb-org/4.0 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-4.0.list
$ sudo apt update
$ sudo apt install mongodb-org -y
Verify that mongodb is installed:
$ mongod --version
db version v4.0.10
git version: c389e7f69f637f7a1ac3cc9fae843b635f20b766
OpenSSL version: OpenSSL 1.1.1 11 Sep 2018
allocator: tcmalloc
modules: none
build environment:
distmod: ubuntu1804
distarch: x86_64
target_arch: x86_64
Keyfile Authorization
All 3 nodes will use the same preshared key to autenticate the members that will contribute to the replca set. Let's create the directory where the file will be stored:
$ sudo mkdir /var/lib/mongodb-pki
Use openssl or anything similar to generate a random key:
$ openssl rand -base64 741 > keyfile
Copy the file over to the other nodes with scp:
$ scp ./keyfile mongodb-2.pvt:/tmp/keyfile
$ scp ./keyfile mongodb-3.pvt:/tmp/keyfile
Move the keyfile in place on all 3 nodes and change the permissions of the file:
$ sudo mv keyfile /var/lib/mongodb-pki/keyfile
$ sudo chmod 600 /var/lib/mongodb-pki/keyfile
$ sudo chown -R mongodb:mongodb /var/lib/mongodb-pki
Configure Mongod
Time to configure our mongod instances, we will configure a configuration for each node. So for the first node:
storage:
dbPath: /var/lib/mongodb
journal:
enabled: true
systemLog:
destination: file
logAppend: true
path: /var/log/mongodb/mongod.log
net:
port: 27017
bindIp: 127.0.0.1,10.163.68.26
processManagement:
timeZoneInfo: /usr/share/zoneinfo
operationProfiling:
mode: "slowOp"
slowOpThresholdMs: 50
security:
authorization: enabled
keyFile: /var/lib/mongodb-pki/keyfile
replication:
replSetName: civo-demo-replset
For the 2nd node:
storage:
dbPath: /var/lib/mongodb
journal:
enabled: true
systemLog:
destination: file
logAppend: true
path: /var/log/mongodb/mongod.log
net:
port: 27017
bindIp: 127.0.0.1,10.163.68.12
processManagement:
timeZoneInfo: /usr/share/zoneinfo
operationProfiling:
mode: "slowOp"
slowOpThresholdMs: 50
security:
authorization: enabled
keyFile: /var/lib/mongodb-pki/keyfile
replication:
replSetName: civo-demo-replset
And for the 3rd node:
storage:
dbPath: /var/lib/mongodb
journal:
enabled: true
systemLog:
destination: file
logAppend: true
path: /var/log/mongodb/mongod.log
net:
port: 27017
bindIp: 127.0.0.1,10.163.68.9
processManagement:
timeZoneInfo: /usr/share/zoneinfo
operationProfiling:
mode: "slowOp"
slowOpThresholdMs: 50
security:
authorization: enabled
keyFile: /var/lib/mongodb-pki/keyfile
replication:
replSetName: civo-demo-replset
Enable and Start Mongod
Enable the service and start the mongod service:
$ sudo systemctl enable mongod
$ sudo systemctl restart mongod
Verify that the process has started:
$ sudo systemctl status mongod
mongod.service - MongoDB Database Server
Loaded: loaded (/lib/systemd/system/mongod.service; enabled; vendor preset: enabled)
Active: active (running) since Tue 2019-07-02 21:37:52 BST; 26s ago
Docs: https://docs.mongodb.org/manual
Main PID: 2950 (mongod)
CGroup: /system.slice/mongod.service
└─2950 /usr/bin/mongod --config /etc/mongod.conf
Jul 02 21:37:52 mongodb-1 systemd[1]: Started MongoDB Database Server.
Verify that the port is listening:
$ sudo netstat -tulpn | grep -E '(State|2950)'
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 127.0.0.1:27017 0.0.0.0:* LISTEN 2950/mongod
tcp 0 0 10.163.68.26:27017 0.0.0.0:* LISTEN 2950/mongod
Go ahead and do the exact same on the other 2 nodes.
Initialize the MongoDB Replica Set
When mongodb starts for the first time it allows an exception that you can logon without authentication to create the root account, but only on localhost. The exception is only valid until you create the user:
$ mongo --host 127.0.0.1 --port 27017
MongoDB shell version v4.0.10
connecting to: mongodb://127.0.0.1:27017/?gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("14ac25ae-016a-4456-b451-60b5d7e866c7") }
MongoDB server version: 4.0.10
Welcome to the MongoDB shell.
>
Switch to the admin database and initialize the mongodb replicaset:
> use admin
switched to db admin
> rs.initiate()
{
"info2" : "no configuration specified. Using a default configuration for the set",
"me" : "10.163.68.26:27017",
"ok" : 1
}
Now that we have initialized our replicaset config, create the admin user and apply the root role:
civo-demo-replset:PRIMARY> db.createUser({user: "mongo-admin", pwd: "mongo-pass", roles: [{role: "root", db: "admin"}]})
Successfully added user: {
"user" : "mongo-admin",
"roles" : [
{
"role" : "root",
"db" : "admin"
}
]
}
civo-demo-replset:PRIMARY> exit
Now that we have exited the mongo shell, logon to mongodb with the created credentials also pointing at the replica set in the connection string:
$ mongo --host civo-demo-replset/mongodb-1.pvt:27017 --username mongo-admin --password mongo-pass --authenticationDatabase admin
MongoDB shell version v4.0.10
connecting to: mongodb://mongodb-1.pvt:27017/?authSource=admin&gssapiServiceName=mongodb&replicaSet=civo-demo-replset
2019-07-02T21:48:05.486+0100 I NETWORK [js] Successfully connected to 10.163.68.26:27017 (1 connections now open to 10.163.68.26:27017 with a 5 second timeout)
MongoDB server version: 4.0.10
civo-demo-replset:PRIMARY>
Have a look at the replica set status using rs.status(), we will see the members in our replica set. At this moment it will only be one node as we have only initiated the one node. This node will elected itself as a primary. We still need to add the other 2 nodes to the replica set.
Have a look at the status:
civo-demo-replset:PRIMARY> rs.status()
{
"set" : "civo-demo-replset",
"date" : ISODate("2019-07-02T20:49:19.596Z"),
"myState" : 1,
"term" : NumberLong(1),
"syncingTo" : "",
"syncSourceHost" : "",
"syncSourceId" : -1,
"heartbeatIntervalMillis" : NumberLong(2000),
"optimes" : {
"lastCommittedOpTime" : {
"ts" : Timestamp(1562100552, 1),
"t" : NumberLong(1)
},
"readConcernMajorityOpTime" : {
"ts" : Timestamp(1562100552, 1),
"t" : NumberLong(1)
},
"appliedOpTime" : {
"ts" : Timestamp(1562100552, 1),
"t" : NumberLong(1)
},
"durableOpTime" : {
"ts" : Timestamp(1562100552, 1),
"t" : NumberLong(1)
}
},
"lastStableCheckpointTimestamp" : Timestamp(1562100512, 1),
"members" : [
{
"_id" : 0,
"name" : "10.163.68.26:27017",
"health" : 1,
"state" : 1,
"stateStr" : "PRIMARY",
"uptime" : 687,
"optime" : {
"ts" : Timestamp(1562100552, 1),
"t" : NumberLong(1)
},
"optimeDate" : ISODate("2019-07-02T20:49:12Z"),
"syncingTo" : "",
"syncSourceHost" : "",
"syncSourceId" : -1,
"infoMessage" : "",
"electionTime" : Timestamp(1562100271, 2),
"electionDate" : ISODate("2019-07-02T20:44:31Z"),
"configVersion" : 1,
"self" : true,
"lastHeartbeatMessage" : ""
}
],
"ok" : 1,
"operationTime" : Timestamp(1562100552, 1),
"$clusterTime" : {
"clusterTime" : Timestamp(1562100552, 1),
"signature" : {
"hash" : BinData(0,"ftyRJpO6BJ5U/oSfP+LzGUeGJvo="),
"keyId" : NumberLong("6709169581312704514")
}
}
}
We can also as the node if its the primary node, using rs.isMaster():
civo-demo-replset:PRIMARY> rs.isMaster()
{
"hosts" : [
"10.163.68.26:27017"
],
"setName" : "civo-demo-replset",
"setVersion" : 1,
"ismaster" : true,
"secondary" : false,
"primary" : "10.163.68.26:27017",
"me" : "10.163.68.26:27017",
"electionId" : ObjectId("7fffffff0000000000000001"),
"lastWrite" : {
"opTime" : {
"ts" : Timestamp(1562100572, 1),
"t" : NumberLong(1)
},
"lastWriteDate" : ISODate("2019-07-02T20:49:32Z"),
"majorityOpTime" : {
"ts" : Timestamp(1562100572, 1),
"t" : NumberLong(1)
},
"majorityWriteDate" : ISODate("2019-07-02T20:49:32Z")
},
"maxBsonObjectSize" : 16777216,
"maxMessageSizeBytes" : 48000000,
"maxWriteBatchSize" : 100000,
"localTime" : ISODate("2019-07-02T20:49:41.764Z"),
"logicalSessionTimeoutMinutes" : 30,
"minWireVersion" : 0,
"maxWireVersion" : 7,
"readOnly" : false,
"ok" : 1,
"operationTime" : Timestamp(1562100572, 1),
"$clusterTime" : {
"clusterTime" : Timestamp(1562100572, 1),
"signature" : {
"hash" : BinData(0,"qn7JFMzIuaiK9sVhNkLAq7NqsMI="),
"keyId" : NumberLong("6709169581312704514")
}
}
}
Adding Nodes to the Replica Set
Let's add the second node, mongodb-2.pvt to the replica set:
civo-demo-replset:PRIMARY> rs.add("mongodb-2.pvt:27017")
{
"ok" : 1,
"operationTime" : Timestamp(1562100629, 1),
"$clusterTime" : {
"clusterTime" : Timestamp(1562100629, 1),
"signature" : {
"hash" : BinData(0,"wTNLWmROCfMKaBT4hjZGIyaiCTc="),
"keyId" : NumberLong("6709169581312704514")
}
}
}
And also adding the 3rd node, mongodb-3.pvt the the replica set:
civo-demo-replset:PRIMARY> rs.add("mongodb-3.pvt:27017")
{
"ok" : 1,
"operationTime" : Timestamp(1562100664, 1),
"$clusterTime" : {
"clusterTime" : Timestamp(1562100664, 1),
"signature" : {
"hash" : BinData(0,"WeAHspyluGFOOfyxnVS7pCKHPN8="),
"keyId" : NumberLong("6709169581312704514")
}
}
}
Now when we look at the replica set status, we should see that the 3 nodes will be part of the replica set:
civo-demo-replset:PRIMARY> rs.status()
{
"set" : "civo-demo-replset",
"date" : ISODate("2019-07-02T20:51:20.003Z"),
"myState" : 1,
"term" : NumberLong(1),
"syncingTo" : "",
"syncSourceHost" : "",
"syncSourceId" : -1,
"heartbeatIntervalMillis" : NumberLong(2000),
"optimes" : {
"lastCommittedOpTime" : {
"ts" : Timestamp(1562100664, 1),
"t" : NumberLong(1)
},
"readConcernMajorityOpTime" : {
"ts" : Timestamp(1562100664, 1),
"t" : NumberLong(1)
},
"appliedOpTime" : {
"ts" : Timestamp(1562100664, 1),
"t" : NumberLong(1)
},
"durableOpTime" : {
"ts" : Timestamp(1562100664, 1),
"t" : NumberLong(1)
}
},
"lastStableCheckpointTimestamp" : Timestamp(1562100629, 1),
"members" : [
{
"_id" : 0,
"name" : "10.163.68.26:27017",
"health" : 1,
"state" : 1,
"stateStr" : "PRIMARY",
"uptime" : 808,
"optime" : {
"ts" : Timestamp(1562100664, 1),
"t" : NumberLong(1)
},
"optimeDate" : ISODate("2019-07-02T20:51:04Z"),
"syncingTo" : "",
"syncSourceHost" : "",
"syncSourceId" : -1,
"infoMessage" : "",
"electionTime" : Timestamp(1562100271, 2),
"electionDate" : ISODate("2019-07-02T20:44:31Z"),
"configVersion" : 3,
"self" : true,
"lastHeartbeatMessage" : ""
},
{
"_id" : 1,
"name" : "mongodb-2.pvt:27017",
"health" : 1,
"state" : 2,
"stateStr" : "SECONDARY",
"uptime" : 50,
"optime" : {
"ts" : Timestamp(1562100664, 1),
"t" : NumberLong(1)
},
"optimeDurable" : {
"ts" : Timestamp(1562100664, 1),
"t" : NumberLong(1)
},
"optimeDate" : ISODate("2019-07-02T20:51:04Z"),
"optimeDurableDate" : ISODate("2019-07-02T20:51:04Z"),
"lastHeartbeat" : ISODate("2019-07-02T20:51:18.288Z"),
"lastHeartbeatRecv" : ISODate("2019-07-02T20:51:19.843Z"),
"pingMs" : NumberLong(1),
"lastHeartbeatMessage" : "",
"syncingTo" : "",
"syncSourceHost" : "",
"syncSourceId" : -1,
"infoMessage" : "",
"configVersion" : 3
},
{
"_id" : 2,
"name" : "mongodb-3.pvt:27017",
"health" : 1,
"state" : 2,
"stateStr" : "SECONDARY",
"uptime" : 15,
"optime" : {
"ts" : Timestamp(1562100664, 1),
"t" : NumberLong(1)
},
"optimeDurable" : {
"ts" : Timestamp(1562100664, 1),
"t" : NumberLong(1)
},
"optimeDate" : ISODate("2019-07-02T20:51:04Z"),
"optimeDurableDate" : ISODate("2019-07-02T20:51:04Z"),
"lastHeartbeat" : ISODate("2019-07-02T20:51:18.292Z"),
"lastHeartbeatRecv" : ISODate("2019-07-02T20:51:19.662Z"),
"pingMs" : NumberLong(1),
"lastHeartbeatMessage" : "",
"syncingTo" : "",
"syncSourceHost" : "",
"syncSourceId" : -1,
"infoMessage" : "",
"configVersion" : 3
}
],
"ok" : 1,
"operationTime" : Timestamp(1562100664, 1),
"$clusterTime" : {
"clusterTime" : Timestamp(1562100664, 1),
"signature" : {
"hash" : BinData(0,"WeAHspyluGFOOfyxnVS7pCKHPN8="),
"keyId" : NumberLong("6709169581312704514")
}
}
}
Connecting to the Replica set
Connecting to a replicaset will always route you to the primary:
$ mongo --host civo-demo-replset/mongodb-2.pvt:27017 --username mongo-admin --password mongo-pass --authenticationDatabase admin
MongoDB shell version v4.0.10
connecting to: mongodb://mongodb-2.pvt:27017/?authSource=admin&gssapiServiceName=mongodb&replicaSet=civo-demo-replset
2019-07-02T21:53:03.943+0100 I NETWORK [js] Starting new replica set monitor for civo-demo-replset/mongodb-2.pvt:27017
2019-07-02T21:53:03.945+0100 I NETWORK [js] Successfully connected to mongodb-2.pvt:27017 (1 connections now open to mongodb-2.pvt:27017 with a 5 second timeout)
2019-07-02T21:53:03.946+0100 I NETWORK [js] Successfully connected to 10.163.68.26:27017 (1 connections now open to 10.163.68.26:27017 with a 5 second timeout)
2019-07-02T21:53:03.946+0100 I NETWORK [js] changing hosts to civo-demo-replset/10.163.68.26:27017,mongodb-2.pvt:27017,mongodb-3.pvt:27017 from civo-demo-replset/mongodb-2.pvt:27017
civo-demo-replset:PRIMARY>
We can verify if we are on the primary by doing the following:
civo-demo-replset:PRIMARY> rs.isMaster()['ismaster']
true
civo-demo-replset:PRIMARY> rs.isMaster()['me']
10.163.68.26:27017
If you exit and connect without the replica set in the connection string, then it will direct you to that node:
$ mongo --host mongodb-2.pvt:27017 --username mongo-admin --password mongo-pass --authenticationDatabase admin
MongoDB shell version v4.0.10
connecting to: mongodb://mongodb-2.pvt:27017/?authSource=admin&gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("3c343871-8814-4136-859d-350c78c2a2a7") }
MongoDB server version: 4.0.10
civo-demo-replset:SECONDARY>
Reading from Slave Nodes
When we have a replica set we can write to our Primary (master) node and Read from our Secondary (slave) nodes. Before we can read from the secondary nodes, we need to tell mongodb that we want to read.
By default when you connect to a secondary node and want to read you will get this exception:
civo-demo-replset:SECONDARY> show databases;
2019-07-02T21:59:13.169+0100 E QUERY [js] Error: listDatabases failed:{
"operationTime" : Timestamp(1562101152, 1),
"ok" : 0,
"errmsg" : "not master and slaveOk=false",
"code" : 13435,
"codeName" : "NotMasterNoSlaveOk”,
...
We first need to instruct mongodb that we want to read:
civo-demo-replset:SECONDARY> rs.slaveOk()
Now that we have done that, we can read from the secondary:
civo-demo-replset:SECONDARY> show databases;
admin 0.000GB
config 0.000GB
local 0.000GB
Upgrading a MongoDB Replica Set
When you want to upgrade a mongodb node in a replica set, you can accomplish that with a rolling upgrade.
You will first shutdown a secondary, do the upgrade, start the node, then upgrade the other secondary. When all the secondaries are upgraded, then upgrade the primary. This involves initiating a stepdown on the primary which will trigger an election, where one of the secondary nodes will be promoted as a primary, once the primary is selected on another node, the old primary can be upgraded.
The correct way of shutting down a mongodb node is with using db.shutdownServer():
civo-demo-replset:PRIMARY> use admin
switched to db admin
civo-demo-replset:PRIMARY> db.shutdownServer()
And to stepdown a node from primary to secondary:
civo-demo-replset:PRIMARY> rs.stepDown()
civo-demo-replset:SECONDARY>
Thanks
Thanks for reading, next up we will look at how we shard our mongodb database by introducing vertical scaling.
Comments