Dask-CHTC¶

Dask-CHTC builds on top of Dask-Jobqueue to spawn Dask workers in the CHTC HTCondor pool. It also provides tools for running Jupyter notebook servers in a controlled way on CHTC submit nodes, which you may find helpful for providing an interactive development environment to use Dask in.

Note

If you’re interested in using Dask at CHTC but have never used CHTC resources before, please fill out the CHTC contact form to get in touch with our Research Computing Facilitators. If you’ve already had an engagement meeting, send an email to chtc@cs.wisc.edu and let them know you’re interested in using Dask.

Attention

We currently only support the Dask-CHTC workflow on the submit3.chtc.wisc.edu submit node. If you do not have an account on submit3.chtc.wisc.edu, you will need to request one.

Attention

Dask-CHTC is prototype software! If you notice any issues or have any suggestions for improvements, please write up a GitHub issue detailing the problem or proposal. We also recommend “watching” the GitHub repository to keep track of new releases, and upgrading promptly when they occur.

These pages will get you started with Dask-CHTC:

Installing Dask-CHTC: How to install Python and Dask-CHTC on a CHTC submit node.
Running Jupyter through Dask-CHTC: How to use Dask-CHTC to run a Jupyter notebook server on a CHTC submit node.
Networking and Port Forwarding: Information on CHTC networking and how to forward ports over SSH, which will allow you to connect to Jupyter notebooks and Dask dashboards running on CHTC submit nodes.
Dask Cluster Creation: A brief example Jupyter notebook, showing how to start up a CHTCCluster and use it to perform some calculations.

These pages have information for troubleshooting problems and handling specific use cases:

Troubleshooting: Solutions and advice for tackling specific problems that might arise while using Dask-CHTC.
Configuration: Information on how to configure Dask-CHTC and the Dask JupyterLab extension.
Building Docker Images for Dask-CHTC: Information on how to build Docker images for use with Dask-CHTC.

Detailed information on the Python API and the associated command line tool can be found on these pages:

API Reference: API documentation for dask_chtc.
CLI Reference: Documentation for the dask-chtc CLI tool.