MongoDB Single Node Upgrade to Replica Set High Availability Cluster

Project Background

Due to historical reasons, we have a business for data synchronization, and MongoDB is using single node in production environment. However, as the business grows, considering the importance of this synchronization business and avoiding business stoppage due to single node failure, we need to upgrade to a replica set to ensure high availability.

Replica set architecture

The following architecture diagram is the MongoDB replica set high availability architecture that needs to be implemented in this article.

Precautions before upgrading architecture

In the production environment, before doing single node upgrade to cluster, make sure to backup all the data of mongodb first to avoid data loss due to operation error.

And make sure that no program will connect to MongoDB for read and write operations during the upgrade, it is recommended to stop the service upgrade and operate in the early morning during the low business peak.

I. Original single-node MongoDB configuration information

IP: 192.168.30.207
Port: 27017

1.1 Original configuration file

systemLog:
  destination: file   
  logAppend: true  
  path: /home/server/mongodb/logs/mongodb.log

storage:
  journal:
    enabled: true
  dbPath: /home/server/mongodb/data
  directoryPerDB: true
  wiredTiger:
     engineConfig:
        cacheSizeGB: 1
        directoryForIndexes: true
     collectionConfig:
        blockCompressor: zlib
     indexConfig:
        prefixCompression: true

processManagement:
  fork: true
  pidFilePath: /home/server/mongodb/pid/mongod.pid

net:
  port: 27017
  bindIp: 127.0.0.1,192.168.30.207
  maxIncomingConnections: 5000

security:
  authorization: enabled

1.2 Add replica set configuration to the original configuration file

replication:
  oplogSizeMB: 4096
  replSetName: rs1

Note: Here you need to comment out the Authentication configuration first, and turn it on after the replica set configuration is complete.

II. New node information

角色	IP	Port
PRIMARY	192.168.30.207	27017	Original single-node MongoDB
SECONDARY	192.168.30.213	27017	New node 1
SECONDARY	192.168.30.214	27017	New node 2

2.1 Add a new node profile

For these two SECONDARY node configuration files, simply copy the PRIMARY node configuration file and modify the corresponding “bindIp“.

SECONDARY node 1 configuration file

systemLog:
  destination: file   
  logAppend: true  
  path: /home/server/mongodb/logs/mongodb.log

storage:
  journal:
    enabled: true
  dbPath: /home/server/mongodb/data
  directoryPerDB: true
  wiredTiger:
     engineConfig:
        cacheSizeGB: 1
        directoryForIndexes: true
     collectionConfig:
        blockCompressor: zlib
     indexConfig:
        prefixCompression: true

processManagement:
  fork: true
  pidFilePath: /home/server/mongodb/pid/mongod.pid

net:
  port: 27017
  bindIp: 127.0.0.1,192.168.30.213
  maxIncomingConnections: 5000

#security:
  #authorization: enabled

replication:
  oplogSizeMB: 4096
  replSetName: rs1

SECONDARY node 2 configuration file

systemLog:
  destination: file   
  logAppend: true  
  path: /home/server/mongodb/logs/mongodb.log

storage:
  journal:
    enabled: true
  dbPath: /home/server/mongodb/data
  directoryPerDB: true
  wiredTiger:
     engineConfig:
        cacheSizeGB: 1
        directoryForIndexes: true
     collectionConfig:
        blockCompressor: zlib
     indexConfig:
        prefixCompression: true

processManagement:
  fork: true
  pidFilePath: /home/server/mongodb/pid/mongod.pid

net:
  port: 27017
  bindIp: 127.0.0.1,192.168.30.214
  maxIncomingConnections: 5000

#security:
  #authorization: enabled

replication:
  oplogSizeMB: 4096
  replSetName: rs1

2.2 Start 3 nodes

The PRIMARY node needs to be restarted and the 2 SECONDARY nodes are started directly.

# Start command
$ mongo -f /home/server/mongodb/conf/mongo.conf

# Stop command
$ mongo -f /home/server/mongodb/conf/mongo.conf --shutdown

III. Initialize the replica set

Use the mongo shell to connect to one of the nodes and execute the initialization command

3.1 Initial Configuration

config = {
  _id : "rs1", 
  members : [
    {_id:0, host:"192.168.30.207:27017"},
    {_id:1, host:"192.168.30.213:27017"},
    {_id:2, host:"192.168.30.214:27017"},
  ]
}

3.2 Initialization of the replica set

> rs.initiate(config)				//Initialize copy set
{
	"ok" : 1,		//Return ok: 1 success, return ok: 0 failure
	"operationTime" : Timestamp(1611042829, 1),
	"$clusterTime" : {
		"clusterTime" : Timestamp(1611042829, 1),
		"signature" : {
			"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
			"keyId" : NumberLong(0)
		}
	}
}

3.3 Viewing Replica Set Status

To view cluster status information

rs1:PRIMARY> rs.status()
{
	"set" : "rs1",
	"date" : ISODate("2021-01-20T01:51:53.063Z"),
	"myState" : 1,
	"term" : NumberLong(2),
	"heartbeatIntervalMillis" : NumberLong(2000),
	"optimes" : {
		"lastCommittedOpTime" : {
			"ts" : Timestamp(1611107505, 1),
			"t" : NumberLong(2)
		},
		"readConcernMajorityOpTime" : {
			"ts" : Timestamp(1611107505, 1),
			"t" : NumberLong(2)
		},
		"appliedOpTime" : {
			"ts" : Timestamp(1611107505, 1),
			"t" : NumberLong(2)
		},
		"durableOpTime" : {
			"ts" : Timestamp(1611107505, 1),
			"t" : NumberLong(2)
		}
	},
	"members" : [
		{
			"_id" : 0,
			"name" : "192.168.30.207:27017",
			"health" : 1,
			"state" : 1,
			"stateStr" : "PRIMARY",
			"uptime" : 52343,
			"optime" : {
				"ts" : Timestamp(1611107505, 1),
				"t" : NumberLong(2)
			},
			"optimeDate" : ISODate("2021-01-20T01:51:45Z"),
			"electionTime" : Timestamp(1611055182, 1),
			"electionDate" : ISODate("2021-01-19T11:19:42Z"),
			"configVersion" : 1,
			"self" : true
		},
		{
			"_id" : 1,
			"name" : "192.168.30.213:27017",
			"health" : 1,
			"state" : 2,
			"stateStr" : "SECONDARY",
			"uptime" : 52334,
			"optime" : {
				"ts" : Timestamp(1611107505, 1),
				"t" : NumberLong(2)
			},
			"optimeDurable" : {
				"ts" : Timestamp(1611107505, 1),
				"t" : NumberLong(2)
			},
			"optimeDate" : ISODate("2021-01-20T01:51:45Z"),
			"optimeDurableDate" : ISODate("2021-01-20T01:51:45Z"),
			"lastHeartbeat" : ISODate("2021-01-20T01:51:52.487Z"),
			"lastHeartbeatRecv" : ISODate("2021-01-20T01:51:52.487Z"),
			"pingMs" : NumberLong(0),
			"syncingTo" : "192.168.30.207:27017",
			"configVersion" : 1
		},
		{
			"_id" : 2,
			"name" : "192.168.30.214:27017",
			"health" : 1,
			"state" : 2,
			"stateStr" : "SECONDARY",
			"uptime" : 52328,
			"optime" : {
				"ts" : Timestamp(1611107505, 1),
				"t" : NumberLong(2)
			},
			"optimeDurable" : {
				"ts" : Timestamp(1611107505, 1),
				"t" : NumberLong(2)
			},
			"optimeDate" : ISODate("2021-01-20T01:51:45Z"),
			"optimeDurableDate" : ISODate("2021-01-20T01:51:45Z"),
			"lastHeartbeat" : ISODate("2021-01-20T01:51:52.487Z"),
			"lastHeartbeatRecv" : ISODate("2021-01-20T01:51:52.487Z"),
			"pingMs" : NumberLong(0),
			"syncingTo" : "192.168.30.207:27017",
			"configVersion" : 1
		}
	],
	"ok" : 1,
	"operationTime" : Timestamp(1611107505, 1),
	"$clusterTime" : {
		"clusterTime" : Timestamp(1611107505, 1),
		"signature" : {
			"hash" : BinData(0,"9fl/3w/5Gwk4Qd7h3fkrQVQtPhM="),
			"keyId" : NumberLong("6919423507650576385")
		}
	}
}

查看延时从库信息

rs1:PRIMARY> rs.printSlaveReplicationInfo()
source: 192.168.30.213:27017
	syncedTo: Wed Jan 20 2021 09:52:05 GMT+0800 (CST)
	0 secs (0 hrs) behind the primary 
source: 192.168.30.214:27017
	syncedTo: Wed Jan 20 2021 09:52:05 GMT+0800 (CST)
	0 secs (0 hrs) behind the primary

View delayed slave information

rs1:PRIMARY> rs.isMaster()
{
	"hosts" : [
		"192.168.30.207:27017",
		"192.168.30.213:27017",
		"192.168.30.214:27017"
	],
	"setName" : "rs1",
	"setVersion" : 1,
	"ismaster" : true,
	"secondary" : false,
	"primary" : "192.168.30.207:27017",
	"me" : "192.168.30.207:27017",
	"electionId" : ObjectId("7fffffff0000000000000002"),
	"lastWrite" : {
		"opTime" : {
			"ts" : Timestamp(1611108985, 1),
			"t" : NumberLong(2)
		},
		"lastWriteDate" : ISODate("2021-01-20T02:16:25Z"),
		"majorityOpTime" : {
			"ts" : Timestamp(1611108985, 1),
			"t" : NumberLong(2)
		},
		"majorityWriteDate" : ISODate("2021-01-20T02:16:25Z")
	},
	"maxBsonObjectSize" : 16777216,
	"maxMessageSizeBytes" : 48000000,
	"maxWriteBatchSize" : 100000,
	"localTime" : ISODate("2021-01-20T02:16:26.713Z"),
	"logicalSessionTimeoutMinutes" : 30,
	"minWireVersion" : 0,
	"maxWireVersion" : 6,
	"readOnly" : false,
	"ok" : 1,
	"operationTime" : Timestamp(1611108985, 1),
	"$clusterTime" : {
		"clusterTime" : Timestamp(1611108985, 1),
		"signature" : {
			"hash" : BinData(0,"/B6MSETtY8MEoZwXKABCkb1AMY8="),
			"keyId" : NumberLong("6919423507650576385")
		}
	}
}

IV. Replica set open authentication

4.1 Add a super administrator account via the master node

Note: If the original single-node mongo already has a super administrator account, this step can be ignored.

Simply add a user to the master node and the replica set will automatically synchronize the data on the master node.

Note that the account creation step needs to be done before authentication is enabled.

Creating a hypervisor account

Super Admin User: mongouser Password: 123456 Authentication Library: admin

$ mongo --host 192.168.30.207 --port 27017
rs1:PRIMARY> use admin
rs1:PRIMARY> db.createUser({user: "mongouser",pwd: "123456",roles:[ { role: "root", db:"admin"}]})

View created accounts

rs1:PRIMARY> use admin
switched to db admin
rs1:PRIMARY> db.getUsers()
[
	{
		"_id" : "admin.mongouser",
		"user" : "mongouser",
		"db" : "admin",
		"roles" : [
			{
				"role" : "root",
				"db" : "admin"
			}
		]
	}
]

4.2 Creating a keyfile for replica set authentication

All replica set nodes must use the same keyfile, which is usually generated on one machine and then copied to other machines, and must have read access, otherwise it will report errors in the future.
Make sure the keyfile is consistent, and the file location is random. But to make it easy to find, it is recommended to put it in a fixed location on each machine, all in a directory with the configuration file.

Generate the mongo.keyfile file

$ openssl rand -base64 90 -out /home/server/mongodb/conf/mongo.keyfile

Copy the mongo.keyfile file to the same directory of the other 2 nodes

$ scp /home/server/mongodb/conf/mongo.keyfile root@192.168.30.213:/home/server/mongodb/conf/
$ scp /home/server/mongodb/conf/mongo.keyfile root@192.168.30.214:/home/server/mongodb/conf/

4.3 Modify the MongoDB configuration file to enable authentication

Add and modify the following configuration in the configuration file.

security:
  keyFile: /home/server/mongodb/conf/mongo.keyfile
  authorization: enabled

Restart all mongo nodes

4.4 Verifying Replica Set Authentication

$ mongo -u mongouser -p 123456 --host 192.168.30.207 --port 27017 -authenticationDatabase admin