본문 바로가기
T./MongoDB

[Performance]Dump and Restore

by IT Journeyman 2024. 6. 6.

 

 

 

Dump and Restore Performance

 

Executive Summary

  • 성능 요약 - Dump
Atlas M60 Default Compression Parallel
c4.4xlarge 16m34.809s 8m58.539s 4m53.595s
c4.8xlarge     4m22.919s
c4.xlarge     21m38.932s
  • 성능 요약 - Restore
EC2 Tier Atlas Tier Default Zipped Parallel Write
Concern
c4.4xlarge Atlas M60 23m25.136s 21m19.640s 9m52.070s 9m50.477s
Parallel=8 8m53.361s 6m50.413s
Parallel=2 14m57.753s
c4.xlarge Atlas M60 Parallel=4 20m4.827s
Atlas M30 Parallel=4 37m36.827
c4.8xlarge Atlas M60 Parallel=16 7m19.881s
Parallel=8 7m22.263s
Parallel=4 7m4.918s

 

  • Recommended Parameters

 

0. Performance Test Environment

0-1 : Mongotools Version

wget https://fastdl.mongodb.org/tools/db/mongodb-database-tools-ubuntu2004-x86_64-100.9.4.tgz

tar -zxvf mongodb-database-tools*.tgz

0-2 : Sample Big Collection

아래  collection을 8개로 복제해서 테스트 수행 

0-3 : MongoDB Atlas Clusters for Test

MongoDB Atlas DB Size : 36.7GB

Dump size : unzip 127GB, zipped 6GB

 

0-4 : EC2 Servers Configuration

Region : us-east-1(Same Region as MongoDB Atlas)

Test Servers :  

  1. c4.4xlarge(16 vCPUs and 30 GiB memory)
  2. c4.8xlarge(36 vCPUs and 60 GiB memory)
  3. c4.xlarge(4 vCPUs and 7.5 GiB memory)

 

Dump@c4.4xlarge  w/ default options from M60

time ./mongodump \

--uri "mongodb+srv://it_admin:manager@dumptestcluster.lit28.mongodb.net/" \

--out ./dump_34GB

 

real    16m34.809s

user    5m28.577s

sys     5m38.051s

 

ubuntu@ip-172-31-50-176:~/mongodb-database-tools-ubuntu2004-x86_64-100.9.4/bin$ du -k

127054032       ./dump_34GB



 

Dump@c4.4xlarge w/ zip from M60

time ./mongodump --uri "mongodb+srv://it_admin:manager@restoretarget.npgjj.mongodb.net/" \

--gzip \

--out ./dump_34GB

 

real    8m58.539s

user    25m44.675s

sys     4m14.866s

 

ubuntu@ip-172-31-50-176:~/mongodb-database-tools-ubuntu2004-x86_64-100.9.4/bin$ du -k ./dump_34GB/

6171160 ./dump_34GB/







 

Dump@c4.4xlarge w/ P=8 from M60

time ./mongodump --uri "mongodb+srv://it_admin:manager@restoretarget.npgjj.mongodb.net/" \

--gzip --numParallelCollections=16 \

--out ./dump_34GB_p16

 

real    4m53.595s

user    29m51.929s

sys     4m57.832s

 

ubuntu@ip-172-31-50-176:~/mongodb-database-tools-ubuntu2004-x86_64-100.9.4/bin$ du -k ./dump_34GB_p16

6171160 ./dump_34GB_p16

 












 

Restore@c4.4xlarge w/ P=4 to M60

time ./mongorestore --uri "mongodb+srv://it_admin:manager@dumptestcluster.lit28.mongodb.net/" \

--numInsertionWorkersPerCollection=4 --numParallelCollections=4 \

--writeConcern="{w:'1'}" --gzip --drop \

./dump_34GB/

 

2024-05-22T02:37:34.164+0000    [#.......................]  loan_project.loans1  38.5MB/743MB  (5.2%)

2024-05-22T02:37:34.164+0000    [#.......................]  loan_project.loans2  38.4MB/737MB  (5.2%)

2024-05-22T02:37:34.164+0000    [#.......................]  loan_project.loans4  38.0MB/737MB  (5.2%)

2024-05-22T02:37:34.164+0000    [#.......................]  loan_project.loans5  38.4MB/737MB  (5.2%)

2024-05-22T02:37:34.164+0000

2024-05-22T02:37:37.162+0000    [#.......................]  loan_project.loans1  51.4MB/743MB  (6.9%)

2024-05-22T02:37:37.162+0000    [#.......................]  loan_project.loans2  51.3MB/737MB  (7.0%)

2024-05-22T02:37:37.162+0000    [#.......................]  loan_project.loans4  50.6MB/737MB  (6.9%)

2024-05-22T02:37:37.162+0000    [#.......................]  loan_project.loans5  51.2MB/737MB  (6.9%)

 

real    6m50.413s

user    31m9.085s

sys     2m23.181s

 

 

 

Restore@c4.4xlarge w/ P=8 to M60

time ./mongorestore --uri "mongodb+srv://it_admin:manager@dumptestcluster.lit28.mongodb.net/" \

--numInsertionWorkersPerCollection=8 --numParallelCollections=8 \

--writeConcern="{w:'1'}" --gzip --drop \

./dump_34GB/

2024-05-22T02:16:42.121+0000

2024-05-22T02:16:45.122+0000    [###################.....]  loan_project.loans1  597MB/743MB  (80.4%)

2024-05-22T02:16:45.122+0000    [###################.....]  loan_project.loans4  594MB/737MB  (80.7%)

2024-05-22T02:16:45.122+0000    [###################.....]  loan_project.loans2  593MB/737MB  (80.4%)

2024-05-22T02:16:45.122+0000    [###################.....]  loan_project.loans5  597MB/737MB  (81.0%)

2024-05-22T02:16:45.122+0000    [###################.....]  loan_project.loans6  595MB/737MB  (80.8%)

2024-05-22T02:16:45.122+0000    [###################.....]  loan_project.loans8  596MB/737MB  (80.9%)

2024-05-22T02:16:45.122+0000    [###################.....]  loan_project.loans7  593MB/737MB  (80.5%)

2024-05-22T02:16:45.122+0000    [###################.....]  loan_project.loans3  594MB/737MB  (80.6%)

2024-05-22T02:16:45.122+0000

2024-05-22T02:16:48.122+0000    [###################.....]  loan_project.loans1  601MB/743MB  (81.0%)

2024-05-22T02:16:48.122+0000    [###################.....]  loan_project.loans4  599MB/737MB  (81.2%)

2024-05-22T02:16:48.122+0000    [###################.....]  loan_project.loans2  596MB/737MB  (81.0%)

2024-05-22T02:16:48.122+0000    [###################.....]  loan_project.loans5  601MB/737MB  (81.5%)

2024-05-22T02:16:48.122+0000    [###################.....]  loan_project.loans6  599MB/737MB  (81.3%)

2024-05-22T02:16:48.122+0000    [###################.....]  loan_project.loans8  600MB/737MB  (81.4%)

2024-05-22T02:16:48.122+0000    [###################.....]  loan_project.loans7  597MB/737MB  (81.0%)

2024-05-22T02:16:48.122+0000    [###################.....]  loan_project.loans3  598MB/737MB  (81.1%)

 

real    8m53.361s

user    25m52.599s

sys     1m35.314s

 



Restore@c4.4xlarge w/ P=2 to M60

time ./mongorestore \

--uri "mongodb+srv://it_admin:manager@dumptestcluster.lit28.mongodb.net/" \

--numInsertionWorkersPerCollection=2 --numParallelCollections=2 \

--writeConcern="{w:'1'}" --gzip --drop \

./dump_34GB/

 

real    14m57.753s

user    21m56.867s

sys     1m52.477s

 

 

Restore@c4.4xlarge w/ P=4 W=majority to M60

time ./mongorestore \

--uri "mongodb+srv://it_admin:manager@dumptestcluster.lit28.mongodb.net/" \

--numInsertionWorkersPerCollection=4 --numParallelCollections=4 \

--gzip --drop \

./dump_34GB/

 

real    9m50.477s

user    27m55.147s

sys     2m4.680s

 




 

Restore@c4.4xlarge w/ P=4 to M60

time ./mongorestore \

--uri "mongodb+srv://it_admin:manager@dumptestcluster.lit28.mongodb.net/" \

--numInsertionWorkersPerCollection=4 \

--gzip --drop \

./dump_34GB/

 

real    9m52.070s

user    28m4.311s

sys     2m6.595s

 

 

 

Restore@c4.4xlarge w/ zip to M60

time ./mongorestore \

--uri "mongodb+srv://it_admin:manager@dumptestcluster.lit28.mongodb.net/" \

--gzip --drop \

./dump_34GB/

 

2024-05-22T03:15:46.064+0000

2024-05-22T03:15:49.064+0000    [##......................]  loan_project.loans1  63.3MB/743MB  (8.5%)

2024-05-22T03:15:49.064+0000    [##......................]  loan_project.loans4  62.6MB/737MB  (8.5%)

2024-05-22T03:15:49.064+0000    [##......................]  loan_project.loans2  62.7MB/737MB  (8.5%)

2024-05-22T03:15:49.064+0000    [##......................]  loan_project.loans5  62.4MB/737MB  (8.5%)

2024-05-22T03:15:49.064+0000

2024-05-22T03:15:52.064+0000    [##......................]  loan_project.loans1  66.8MB/743MB  (9.0%)

2024-05-22T03:15:52.064+0000    [##......................]  loan_project.loans4  66.1MB/737MB  (9.0%)

2024-05-22T03:15:52.064+0000    [##......................]  loan_project.loans2  66.3MB/737MB  (9.0%)

2024-05-22T03:15:52.064+0000    [##......................]  loan_project.loans5  65.8MB/737MB  (8.9%)

 

real    21m19.640s

user    22m9.486s

sys     1m32.323s

 





 

Restore@c4.4xlarge w/ unzip to M60

time ./mongorestore \

--uri "mongodb+srv://it_admin:manager@dumptestcluster.lit28.mongodb.net/" \

--drop \

./dump_34GB/

 

024-05-22T06:08:20.417+0000

2024-05-22T06:08:23.417+0000    [###############.........]  loan_project.loans2  9.83GB/15.1GB  (65.1%)

2024-05-22T06:08:23.417+0000    [###############.........]  loan_project.loans3  9.84GB/15.1GB  (65.1%)

2024-05-22T06:08:23.417+0000    [###############.........]  loan_project.loans4  9.83GB/15.1GB  (65.1%)

2024-05-22T06:08:23.417+0000    [###############.........]  loan_project.loans5  9.86GB/15.1GB  (65.3%)

2024-05-22T06:08:23.417+0000

2024-05-22T06:08:26.418+0000    [###############.........]  loan_project.loans2  9.88GB/15.1GB  (65.5%)

2024-05-22T06:08:26.418+0000    [###############.........]  loan_project.loans3  9.90GB/15.1GB  (65.6%)

2024-05-22T06:08:26.418+0000    [###############.........]  loan_project.loans4  9.89GB/15.1GB  (65.5%)

2024-05-22T06:08:26.418+0000    [###############.........]  loan_project.loans5  9.92GB/15.1GB  (65.7%)

 

real    23m25.136s

user    22m21.358s

sys     6m42.570s

 



 

Dump@c4.xlarge from M60

 

time ./mongodump --uri "mongodb+srv://it_admin:manager@dumptestcluster.lit28.mongodb.net/" \

--gzip \

--out ./dump_zipped

 

real    21m38.932s

user    42m27.024s

sys     18m29.375s

 



 

Restore@c4.xlarge to M60

 

time ./mongorestore --uri "mongodb+srv://it_admin:manager@dumptestcluster.lit28.mongodb.net/" \

--numInsertionWorkersPerCollection=4 --numParallelCollections=4 \

--writeConcern="{w:'1'}" --gzip --drop \

./dump_zipped/

 

2024-05-22T08:25:52.814+0000

2024-05-22T08:25:55.820+0000    [#######################.]  loan_project.loans8  717MB/737MB  (97.3%)

2024-05-22T08:25:55.820+0000    [#######################.]  loan_project.loans6  715MB/737MB  (97.0%)

2024-05-22T08:25:55.820+0000    [#######################.]  loan_project.loans7  714MB/737MB  (96.9%)

2024-05-22T08:25:55.820+0000    [#######################.]  loan_project.loans3  711MB/737MB  (96.5%)

2024-05-22T08:25:55.820+0000

2024-05-22T08:25:58.814+0000    [#######################.]  loan_project.loans8  721MB/737MB  (97.8%)

2024-05-22T08:25:58.814+0000    [#######################.]  loan_project.loans6  718MB/737MB  (97.5%)

2024-05-22T08:25:58.816+0000    [#######################.]  loan_project.loans7  717MB/737MB  (97.3%)

2024-05-22T08:25:58.817+0000    [#######################.]  loan_project.loans3  715MB/737MB  (97.0%)

 

real    20m4.827s

user    59m3.190s

sys     14m40.612s

 

 

 

Restore@c4.xlarge to M30

 

time ./mongorestore --uri "mongodb+srv://it_admin:manager@restoretest.lit28.mongodb.net/" \

--numInsertionWorkersPerCollection=4 --numParallelCollections=4 \

--writeConcern="{w:'1'}" --gzip --drop \

./dump_zipped/

 

2024-05-22T09:17:27.645+0000

2024-05-22T09:17:30.628+0000    [#####################...]  loan_project.loans8  661MB/737MB  (89.7%)

2024-05-22T09:17:30.628+0000    [#####################...]  loan_project.loans6  658MB/737MB  (89.2%)

2024-05-22T09:17:30.628+0000    [#####################...]  loan_project.loans7  657MB/737MB  (89.2%)

2024-05-22T09:17:30.628+0000    [#####################...]  loan_project.loans3  658MB/737MB  (89.3%)

2024-05-22T09:17:30.628+0000

2024-05-22T09:17:33.629+0000    [#####################...]  loan_project.loans8  663MB/737MB  (90.0%)

2024-05-22T09:17:33.629+0000    [#####################...]  loan_project.loans6  660MB/737MB  (89.6%)

2024-05-22T09:17:33.629+0000    [#####################...]  loan_project.loans7  659MB/737MB  (89.5%)

2024-05-22T09:17:33.629+0000    [#####################...]  loan_project.loans3  660MB/737MB  (89.6%)

 

real    37m36.827s

user    33m33.015s

sys     3m15.192s



 

 

Dump@c4.8xlarge from M60

time ./mongodump --uri "mongodb+srv://it_admin:manager@dumptestcluster.lit28.mongodb.net/" \

--gzip --numParallelCollections=32 \

--out ./dump

 

2024-05-22T09:37:37.448+0000

2024-05-22T09:37:40.448+0000    [##......................]  loan_project.loans4  2056411/18334644  (11.2%)

2024-05-22T09:37:40.448+0000    [##......................]  loan_project.loans3  2054834/18334644  (11.2%)

2024-05-22T09:37:40.448+0000    [##......................]  loan_project.loans5  2016255/18334644  (11.0%)

2024-05-22T09:37:40.448+0000    [##......................]  loan_project.loans1  2045562/18334644  (11.2%)

2024-05-22T09:37:40.448+0000    [##......................]  loan_project.loans2  2030998/18334644  (11.1%)

2024-05-22T09:37:40.448+0000    [##......................]  loan_project.loans7  2068551/18334644  (11.3%)

2024-05-22T09:37:40.448+0000    [##......................]  loan_project.loans8  2060291/18334644  (11.2%)

2024-05-22T09:37:40.448+0000    [##......................]  loan_project.loans6  2075344/18334644  (11.3%)

2024-05-22T09:37:40.448+0000

2024-05-22T09:37:43.448+0000    [##......................]  loan_project.loans4  2267873/18334644  (12.4%)

2024-05-22T09:37:43.448+0000    [##......................]  loan_project.loans3  2266871/18334644  (12.4%)

2024-05-22T09:37:43.448+0000    [##......................]  loan_project.loans5  2226932/18334644  (12.1%)

2024-05-22T09:37:43.448+0000    [##......................]  loan_project.loans1  2264800/18334644  (12.4%)

2024-05-22T09:37:43.448+0000    [##......................]  loan_project.loans2  2236546/18334644  (12.2%)

2024-05-22T09:37:43.448+0000    [##......................]  loan_project.loans7  2264793/18334644  (12.4%)

2024-05-22T09:37:43.448+0000    [##......................]  loan_project.loans8  2274482/18334644  (12.4%)

2024-05-22T09:37:43.448+0000    [###.....................]  loan_project.loans6  2297200/18334644  (12.5%)

 

real    4m22.919s

user    25m37.175s

sys     3m23.439s



 

 

Restore@c4.8xlarge to M60

time ./mongorestore --uri "mongodb+srv://it_admin:manager@dumptestcluster.lit28.mongodb.net/" \

--numInsertionWorkersPerCollection=16 --numParallelCollections=16 \

--writeConcern="{w:'1'}" --gzip --drop \

./dump/

 

real    7m19.881s

user    27m24.662s

sys     1m42.621s

 

 

Restore@c4.8xlarge to M60

time ./mongorestore --uri "mongodb+srv://it_admin:manager@dumptestcluster.lit28.mongodb.net/" \

--numInsertionWorkersPerCollection=8 --numParallelCollections=8 \

--writeConcern="{w:'1'}" --gzip --drop \

./dump/

 

real    7m22.263s

user    26m32.588s

sys     1m23.481s

 

 

Restore@c4.8xlarge to M60

time ./mongorestore --uri "mongodb+srv://it_admin:manager@dumptestcluster.lit28.mongodb.net/" \

--numInsertionWorkersPerCollection=4 --numParallelCollections=4 \

--writeConcern="{w:'1'}" --gzip --drop \

./dump/

 

real    7m4.918s

user    24m6.707s

sys     1m36.516s







 

Restore@c4.8xlarge to M60

time ./mongorestore --uri "mongodb+srv://it_admin:manager@dumptestcluster.lit28.mongodb.net/" \

--numInsertionWorkersPerCollection=8 --numParallelCollections=4 \

--writeConcern="{w:'1'}" --gzip --drop \

./dump/



real    7m9.112s

user    26m12.428s

sys     1m38.779s







저작자표시 비영리 변경금지 (새창열림)

'T. > MongoDB' 카테고리의 다른 글

Atlas Chart Demo w/ Lookup(Join)  (0) 2024.07.22
Atlas Chart Demo  (0) 2024.07.22
[Backup & Recovery]DB 복구, 그 아찔한 기억(Table Level Recovery)  (0) 2024.06.06
[Performance]MongoDB Atlas Tier and Sharding  (1) 2024.06.01
[HA]MongoDB Atlas SPOF(Single Point of Failure)  (0) 2024.01.16

관련글

티스토리툴바