API Reference

class dask_chtc.CHTCCluster(*, worker_image=None, gpu_lab=False, gpus=None, batch_name=None, python='./entrypoint.sh python3', **kwargs)[source]

A customized dask_jobqueue.HTCondorCluster subclass for spawning Dask workers in the CHTC HTCondor pool.

It provides a variety of custom arguments designed around the CHTC pool, and forwards any remaining arguments to dask_jobqueue.HTCondorCluster.

Parameters

  • worker_image (Optional[str]) – The Docker image to run the Dask workers inside. Defaults to daskdev/dask:latest (Dockerfile). See this page for advice on building Docker images for use with Dask-CHTC.

  • gpu_lab (bool) – If True, workers will be allowed to run on GPULab nodes. If this is True, the default value of gpus becomes 1. Defaults to False.

  • gpus (Optional[int]) – The number of GPUs to request. Defaults to 0 unless gpu_lab = True, in which case the default is 1.

  • batch_name (Optional[str]) – The HTCondor JobBatchName to assign to the worker jobs. This can be helpful for more sensible output for condor_q. Defaults to "dask-worker".

  • python (str) – The command to execute to start Python inside the worker job. Only modify this if you know what you’re doing!

  • kwargs (Any) – Additional keyword arguments, like cores or memory, are passed to dask_jobqueue.HTCondorCluster.

adapt(*args, minimum_jobs=None, maximum_jobs=None, **kwargs)

Scale Dask cluster automatically based on scheduler activity.

Parameters

  • minimum (int) – Minimum number of workers to keep around for the cluster

  • maximum (int) – Maximum number of workers to keep around for the cluster

  • minimum_memory (str) – Minimum amount of memory for the cluster

  • maximum_memory (str) – Maximum amount of memory for the cluster

  • minimum_jobs (int) – Minimum number of jobs

  • maximum_jobs (int) – Maximum number of jobs

  • **kwargs – Extra parameters to pass to dask.distributed.Adaptive

See also

dask.distributed.Adaptive

for more keyword arguments

scale(n=None, jobs=0, memory=None, cores=None)

Scale cluster to specified configurations.

Parameters

  • n (int) – Target number of workers

  • jobs (int) – Target number of jobs

  • memory (str) – Target amount of memory

  • cores (int) – Target number of cores

classmethod security()[source]

Return the Dask Security object used by Dask-CHTC. Can also be used to create a new Dask Client with the correct security settings for connecting to your workers, e.g. if you started your CHTCCluster via the Dask JupyterLab extension.