Mongo on AWS

Here are some best practices when running Mongo on AWS:

• Always use XFS or Ext4 filesystems when running Mongo.  Both support consistent snapshots for backups.
• Use SSD volumes – General Purpose SSD (GP2) or Provisioned IOPS SSD (IO1).  Magnetic volumes provide much less IOPs than SSDs. 
• For best performance, use I2 – High I/O (I2) instances or D2 – Dense-storage (D2) instances.  The more RAM the merrier, which would keep your working set in memory and not hit disks.
• Use EBS-Optimized instances.  These instances use an “optimized configuration stack and provides additional, dedicated capacity for Amazon EBS I/O” (http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSOptimized.html).
• Mount drives used by Mongo without atime and diratime options.  You can mount the disk manually or write to the /etc/fstab file to keep mount options persistent after restarts:

  • sudo mount -o remount,noatime,noexec  /var/lib/mongo

  • /etc/fstab: /dev/xvde1 /var/lib/mongo/ ext4 noatime,noexec 0 2

• Raise the ulimits for Mongo. The defaults are usually low and not very performant for a production Mongo server. You can view ulimits with ulimit -a. Recommend settings are (https://docs.mongodb.org/v2.4/reference/ulimit/)
  • -f (file size): unlimited
  • -t (cpu time): unlimited
  • -v (virtual memory): unlimited
  • -n (open files): 64000
  • -m (memory size): unlimited
  • -u (processes/threads): 64000
• Set block read ahead to 32.  Example (assuming /dev/xvdfe is your block device for mount – run lsblk to confirm):

  • sudo blockdev --setra 32 /dev/xvdfe

• Pre-warm snaphosts.  This only applies if your volume was created from a snapshot.  When you create an AWS EBS volume which is restored from a snapshot it’s not initialized (http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-initialize.html).   This causes “an increase in the latency of an I/O operation the first time each block is accessed”.  You can access all blocks on a fresh EBS snapshot volumen by running:

  • sudo fio --filename=/dev/xvde --rw=randread --bs=128k --iodepth=32 --ioengine=libaio --direct=1 --name=volume-initialize

• Pre-warm documents into memory.  If you are promoting a SECONDARY into a PRIMARY, the working set data should be pre-warmed.  If not, you will see I/O degradation with faults increasing in mongostat until all data is read from disk.  Ideally you would want to have all documents in memory without touching disk (disk I/O access is much slower then RAM memory access).  You can pre-warm Mongo documents by running live production queries to load them into memory.  Also, you can leave a fresh instance as a SECONDARY for a day or two before promoting to PRIMARY.
• Run each replica-set node in a different Availability Zone (AZ).

Above: Image from “Mongo on AWS Whitepaper“

Sources:

• Amazon EBS- Optimized Instances – http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSOptimized.html
• MongoDB ulimits – https://docs.mongodb.org/v2.4/reference/ulimit/
• Mongo on AWS Whitepaper – https://d0.awsstatic.com/whitepapers/AWS_NoSQL_MongoDB.pdf