Troubleshooting pipInstall plugin

[Sid Ravinutala (sidravi1)]

Hi - I'm trying to use PipInstall plugin to install a private repo but something seems to be killing the workers. Is there anyway to see what/why?

Here's my code (i've remove my GH token)

from dask_gateway import Gateway
from dask.distributed import PipInstall

gateway = Gateway()
cluster = gateway.new_cluster()
client = cluster.get_client()

pkg_link = "git+https://<GH_TOKEN>@github.com/IDinsight/ds_nudge_up@as-package"
plugin = PipInstall(packages=[pkg_link], restart=True)
client.register_worker_plugin(plugin)

cluster.scale(2)

def check_pandas():
    import pandas
    return pandas.__version__

o = client.submit(check_pandas)

and blocks forever when running

o.result()

Looking at the dashboard, it seems the worker are killed (gracefully shutdown) and then restarted.

Here are the worker logs:

2023-01-24 01:53:57,294 - distributed.worker - INFO - Start worker at: tls://10.244.149.45:36245
2023-01-24 01:53:57,294 - distributed.worker - INFO - Listening to: tls://10.244.149.45:36245
2023-01-24 01:53:57,294 - distributed.worker - INFO - dashboard at: 10.244.149.45:8787
2023-01-24 01:53:57,294 - distributed.worker - INFO - Waiting to connect to: tls://dask-b340b140724e4f8084295aab3f0dc768.prod:8786
2023-01-24 01:53:57,295 - distributed.worker - INFO - -------------------------------------------------
2023-01-24 01:53:57,295 - distributed.worker - INFO - Threads: 1
2023-01-24 01:53:57,295 - distributed.worker - INFO - Memory: 8.00 GiB
2023-01-24 01:53:57,295 - distributed.worker - INFO - Local Directory: /home/jovyan/dask-worker-space/worker-9ej4nyrw
2023-01-24 01:53:57,295 - distributed.worker - INFO - -------------------------------------------------
2023-01-24 01:53:57,343 - distributed.worker - INFO - Starting Worker plugin pip
2023-01-24 01:54:22,997 - distributed.worker - INFO - Closing worker gracefully: tls://10.244.149.45:36245
2023-01-24 01:54:23,006 - distributed.worker - INFO - Registered to: tls://dask-b340b140724e4f8084295aab3f0dc768.prod:8786
2023-01-24 01:54:23,007 - distributed.worker - INFO - -------------------------------------------------

Things I have tried

1. I have tried installing the package locally and it installs fine
2. Running a LocalCluster on MPC gives me the following when registering the plugin

2023-01-24 02:13:21,043 - tornado.application - ERROR - Uncaught exception in write_error
Traceback (most recent call last):
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/tornado/web.py", line 1681, in _execute
    result = self.prepare()
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/jupyter_server/base/handlers.py", line 530, in prepare
    raise web.HTTPError(403)
tornado.web.HTTPError: HTTP 403: Forbidden

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/tornado/web.py", line 1217, in send_error
    self.write_error(status_code, **kwargs)
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/jupyter_server/base/handlers.py", line 615, in write_error
    html = self.render_template("%s.html" % status_code, **ns)
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/jupyter_server/base/handlers.py", line 543, in render_template
    template = self.get_template(name)
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/jupyter_server/base/handlers.py", line 539, in get_template
    return self.settings["jinja2_env"].get_template(name)
KeyError: 'jinja2_env'
2023-01-24 02:13:24,065 - tornado.application - ERROR - Exception in callback functools.partial(<bound method IOLoop._discard_future_result of <tornado.platform.asyncio.AsyncIOLoop object at 0x7fd1728597c0>>, <Task finished name='Task-958' coro=<Worker.close() done, defined at /srv/conda/envs/notebook/lib/python3.9/site-packages/distributed/worker.py:1385> exception=CommClosedError('ConnectionPool not running. Status: Status.closed')>)
Traceback (most recent call last):
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/distributed/comm/core.py", line 289, in connect
    comm = await asyncio.wait_for(
  File "/srv/conda/envs/notebook/lib/python3.9/asyncio/tasks.py", line 466, in wait_for
    await waiter
asyncio.exceptions.CancelledError

[Tom Augspurger (TomAugspurger)]

Hmm, I've never tried installing from a private GitHub repository. The main part of PipInstall is at https://github.com/dask/distributed/blob/99d4112d929ffc925ed26dfb4f063c4097c78043/distributed/diagnostics/plugin.py#L481-L492, if you want to try running that code locally.

And just to confirm, you've successfully used the PipInstall plugin on a public repository / from PyPI?

[Sid Ravinutala (sidravi1)]

Thanks! I'll try that out.

Yes - i successfully installed a different library from PyPI and it worked fine.

[Sid Ravinutala (sidravi1)]

I think i have it working now. Looks like it's restart=True in this line that was causing the problem:

plugin = PipInstall(packages=[pkg_link], restart=True)

Would that make sense?

[Tom Augspurger (TomAugspurger)]

Hmm I haven't personally used restart so I'm not sure off hand.

If you're installing a brand new package (rather than a new version of a package that was previously installed and perhaps imported), you should be OK with restart=False.