Function _compute_defaults

docker/test-with-docker.py:224–269

Compute default config options based on memory. The goal is to work reasonably on machines with about 60GB of memory, like Amazon's c4.8xlarge (36 CPUs, 60GB) or c5.9xlarge (36 CPUs, 72GB) or m4.4xlarge (16 CPUs, 64 GB). Based on some experiments, we set up defaults for different machine

()

def _compute_defaults():
  """Compute default config options based on memory.

  The goal is to work reasonably on machines with
  about 60GB of memory, like Amazon's c4.8xlarge (36 CPUs, 60GB)
  or c5.9xlarge (36 CPUs, 72GB) or m4.4xlarge (16 CPUs, 64 GB).

  Based on some experiments, we set up defaults for different
  machine sizes based on memory, with an eye towards
  having reasonable runtimes as well.

  Experiments on memory usage:

  suite               parallelism usage
                    Xmx    memlimit
  ee-test-parallel  4GB  8  5GB   33GB
  ee-test-parallel  4GB 16  7GB   37GB
  ee-test-serial    4GB  -  5GB   18GB
  cluster-test      4GB  -    -   13GB
  be-test           4GB  - 10GB   19GB
  fe-test           4GB  - 10GB    9GB
  """
  total_memory_gb = monitor.total_memory()
  cpus = multiprocessing.cpu_count()
  logging.info("CPUs: %s Memory (GB): %s", cpus, total_memory_gb)

  parallel_test_concurrency = min(cpus, 8)
  memlimit_gb = 8

  if total_memory_gb >= 140:
    suite_concurrency = 6
    memlimit_gb = 11
    parallel_test_concurrency = min(cpus, 12)
  elif total_memory_gb >= 95:
    suite_concurrency = 5
    memlimit_gb = 11
    parallel_test_concurrency = min(cpus, 12)
  elif total_memory_gb >= 65:
    suite_concurrency = 3
  elif total_memory_gb >= 35:
    suite_concurrency = 2
  else:
    logging.warning("This tool should be run on a machine with more memory.")
    suite_concurrency = 1

  return parallel_test_concurrency, suite_concurrency, memlimit_gb * 1024 * 1024 * 1024

class Suite(object):
  """Encapsulates a test suite.