[Performance]MongoDB Atlas Tier and Sharding

TL;DR

DB 성능(Performance)은 중요하지만, 이것을 측정 및 평가하는 것은 정말 하기 싫은 숙제같은 느낌입니다.

MongoDB Atlas의 각 Tier별 Write 성능을 테스트 해 보았습니다.
(덤으로 Sharding을 했을 때 성능이 Scale out이 되는 지도 간단히 테스트했습니다.)

결론은, 아래 표와 같이 "Scale Up and Out 잘 된다"입니다.
(참고로 보고서에 노출된 테스트를 수행한 EC2 Server, DB 및 User는 모두 삭제했습니다)

Executive Summary

Tier	Write IOPS(Replica Set)	Write IOPS(3 Shards)
M10	56,678
M20	55,756
M30	75,673	191,245
M40	141,875	394,251
M50	291,174
M60	385,520

• 해당 테스트는 100% Write만을 수행하였고, Read 100% 혹은 Mixed Workload에 대한 테스트는 하지 않았습니다.
• 해당 테스트는 100% Write에서 Atlas Tier별 참고 지표의 의미만 있습니다. 실 업무 환경 테스트가 더 유의미한 테스트입니다.
• 100% Write에서는 CPU에 성능 영향을 많이 받기 때문에 M10(2 vCPU, 2GB RAM)과 M20(2vCPU, 4GB RAM)이 동일한 성능을 내는 것은 예상된 결과(Expected Behavior)이며, 테스트 결과도 오차 범위 내로 통계적 유의성이 있는 차이는 없습니다.
• Testcase 5, 6처럼 M50, 60 RS 구성은 CPU 50% 이하에서 20만 IOPS 성능이 가능합니다.
• Testcase 6, 7 에서 보듯이  Write Concern(1 or majority)은 성능 영향이 없습니다. 이는 동일 리전 클러스터(vs. 멀티 리전)의 단순 테스트이고, Network의 Saturation이나 다른 병목이 없는 상태로 테스트한 결과로 운영환경의 성능과는 다를 수 있습니다.
• Testcase 8, 9에서 볼 수 있듯이 적절한 Sharding을 구성하면 쓰기 성능이 선형적으로 증가합니다.

0. Performance Test Tool - POCDriver

(Github Link : https://github.com/johnlpage/POCDriver?tab=readme-ov-file)

0-1 : 100% Write로 5분 동안 수행(Test Server :  c5.4xlarge 16 vCPU, 32GB)

ubuntu@ip-172-31-24-248:~/POCDriver$ cat run.sh 

tdate="$(date '+%Y%m%d_%H%M%S')"

java -jar ./bin/POCDriver.jar -d 300 -i 100 -t 128 -e -c "mongodb+srv://it_admin:manager@prod-3shard.npgjj.mongodb.net/?retryWrites=true&w=majority" | tee POCDriver_${tdate}.log

0-2 : Sample Data

ubuntu@ip-172-31-24-248:~/POCDriver$ java -jar ./bin/POCDriver.jar -i 100 -t 300 -e -p

13:47:48.126 [main] INFO  com.johnlpage.pocdriver.POCDriver - 26 - MongoDB Proof Of Concept  - Load Generator version 0.1.2

{

  "_id": {

    "w": 1,

    "i": 12345678

  },

  "fld0": 111316,

  "fld1": {

    "$date": "2021-06-13T08:15:21.057Z"

  },

  "fld2": "dolor sit amet, consetetur",

  "fld3": "Stet clita kasd gubergren, no",

  "fld4": "invidunt ut labore et dolore",

  "fld5": {

    "$date": "2020-04-20T02:09:43.871Z"

  },

  "fld6": 1649139,

  "fld7": "et ea rebum. Stet clita kasd",

  "fld8": "est Lorem ipsum dolor sit",

  "fld9": 655099,

  "bin": {

    "$binary": {

      "base64": "",

      "subType": "00"

    }

  }

}

1. M10 - 100% Write with RS/w=1

1-1 : Atlas - Resource Usage Before Testing

1-2 : Atlas - Resource Usage After Testing

1-3 : Test Server - Resource Usage During Testing

1-4 : Test Result  - 56,678 inserts per second

After 591 seconds (06:25:45), 33,033,216 new documents inserted - collection has 33,033,216 in total 

56,678 inserts per second since last report 

0.00 % in under 50 milliseconds

0 keyqueries per second since last report 

100.00 % in under 50 milliseconds

0 updates per second since last report 

100.00 % in under 50 milliseconds

0 rangequeries per second since last report 

100.00 % in under 50 milliseconds

 

2. M20 - 100% Write with RS/w=1

2-1 : Atlas - Resource Usage Before Testing

2-2 : Atlas - Resource Usage During Testing 

2-3 : Test Server - Resource Usage During Testing

2-4 : Test Result  - 55,756 inserts per second

After 591 seconds (05:43:06), 35,515,392 new documents inserted - collection has 35,515,392 in total 

55,756 inserts per second since last report 

0.00 % in under 50 milliseconds

0 keyqueries per second since last report 

100.00 % in under 50 milliseconds

0 updates per second since last report 

100.00 % in under 50 milliseconds

0 rangequeries per second since last report 

100.00 % in under 50 milliseconds

 

 

3. M30 - 100% Write with RS/w=1

3-1 : Atlas - Resource Usage Before Testing

3-2 : Atlas - Resource Usage During Testing

3-3 : Test Server - Resource Usage During Testing

3-4 : Test Result - 75,673 inserts per second

After 591 seconds (04:57:22), 42,305,536 new documents inserted - collection has 42,305,536 in total 

75,673 inserts per second since last report 

0.00 % in under 50 milliseconds

0 keyqueries per second since last report 

100.00 % in under 50 milliseconds

0 updates per second since last report 

100.00 % in under 50 milliseconds

0 rangequeries per second since last report 

100.00 % in under 50 milliseconds

4. M40 - 100% Write with RS/w=1

4-1 : Atlas - Resource Usage During Testing

4-2 : Test Result - 141,875 inserts per second

After 591 seconds (13:43:52), 85,337,600 new documents inserted - collection has 85,337,600 in total 

141,875 inserts per second since last report 

0.00 % in under 50 milliseconds

0 keyqueries per second since last report 

100.00 % in under 50 milliseconds

0 updates per second since last report 

100.00 % in under 50 milliseconds

0 rangequeries per second since last report 

100.00 % in under 50 milliseconds

 

5. M50 - 100% Write with RS/w=1

5-1 : Atlas - Resource Usage During Testing 

5-2 : Test Result  - 291,174 inserts per second

After 591 seconds (14:17:50), 153,533,440 new documents inserted - collection has 153,533,440 in total 

291,174 inserts per second since last report 

0.00 % in under 50 milliseconds

0 keyqueries per second since last report 

100.00 % in under 50 milliseconds

0 updates per second since last report 

100.00 % in under 50 milliseconds

0 rangequeries per second since last report 

100.00 % in under 50 milliseconds

 

6. M60 - 100% Write with RS/w=1

6-1 : Atlas - Resource Usage Before Testing 

6-2 : Atlas - Resource Usage During Testing

6-3 : Test Server - Resource Usage During Testing

6-4 : Test Result  - 385,520 inserts per second

 

10:26:11.610 [main] INFO  com.johnlpage.pocdriver.LoadRunner - 165 - All Threads Complete

------------------------

After 300 seconds, 115656192 new documents inserted - collection has 115656192 in total 

385520 inserts per second on average

0 keyqueries per second on average

0 updates per second on average

0 rangequeries per second on average

7. M60 - 100% Write with RS/w=majority

7-1 : Atlas - Resource Usage Before Testing 

생략

7-2 : Atlas - Resource Usage During Testing

7-3 : Test Server - Resource Usage During Testing

7-4 : Test Result  - 372,491 inserts per second

 

10:31:41.803 [main] INFO  com.johnlpage.pocdriver.LoadRunner - 165 - All Threads Complete

------------------------

After 300 seconds, 111747584 new documents inserted - collection has 111747584 in total 

372491 inserts per second on average

0 keyqueries per second on average

0 updates per second on average

0 rangequeries per second on average

8. M30 - 100% Write with 3 Shards/w=1

8-1 : Atlas - Resource Usage During Testing

8-2 : Test Server - Resource Usage During Testing

8-3 : Test Result  - 191,245 inserts per second

14:07:22.367 [main] INFO  com.johnlpage.pocdriver.LoadRunner - 165 - All Threads Complete

------------------------

After 300 seconds, 57373696 new documents inserted - collection has 57373696 in total 

191245 inserts per second on average

0 keyqueries per second on average

0 updates per second on average

0 rangequeries per second on average

 

9. M40 - 100% Write with 3 Shards/w=1

9-1 : Atlas - Resource Usage During Testing

9-2 : Test Server - Resource Usage During Testing

 

9-3 : Test Result  - 394,251 inserts per second

 

12:26:51.250 [main] INFO  com.johnlpage.pocdriver.LoadRunner - 165 - All Threads Complete

------------------------

After 300 seconds, 118275584 new documents inserted - collection has 118275584 in total 

394251 inserts per second on average

0 keyqueries per second on average

0 updates per second on average

0 rangequeries per second on average