New --debugger flag does not propagate `imports=...` or `venv_site_packages`

[shayanhoshyari]

🐞 bug report

py_binary, py_test

a94bd0f recently added support for injecting dependencies for easier use of debugger. It allows injecting deps via

--@rules_python//python/config_settings:debugger=<target>

Is this a regression?

Not exactly as it is a new feature right off the press.

Description

I tried this feature for starting targets in vscode. I do this via this strategy

• Create a target name //debugpy that has

# BUILD file
py_library(
    name = "debugpy",
    # Rough sitecustomize content
    #
    # Detect a env var named bzl_debugpy_config that has the debugpy
    # port, and connect secret. If found use it to connect. This will both make
    # the process show up in vscode debugger, and patches os. process creation
    # files so new subprocesses also connect.
   srcs = ["sitecustomize.py"],
   imports = ["."],
    visibility = ["//:__subpackages__"],
)

• Then in vscode I start a python target with a script launch.py that
  • imports debugpy, finds the port and secret
  • Adds them as env var bzl_debugpy_config json encoded (for tests via --test_env)
  • Runs the target with --@rules_python//python/config_settings:debugger=//:debugpy

The issue I face is when I put my debugpy in a non root folder and add imports = [.] to it, when using rules_python//python/config_settings:debugger this imports don't get propagated to py_binary/test and sitecustomize does not get executed.

Alternatively, I tried using venv_site_packages and a .pth file and faced similar issue.

🔬 Minimal Reproduction

https://github.com/shayanhoshyari/issue-reports/tree/5d47835baf270e96f30cefd5a10843e389cfba7d/rules_python/vscode_debugger

🔥 Exception or Error

Using rules_python//python/config_settings:debugger is not propagating imports or venv_site_packages (while using deps=... does).

#	Summary	Command	The debugger connection hook is on path
1	--debugger + put in root	./bazel run //:app --@rules_python//python/config_settings:debugger=//:debugpy	✅
2	--debugger + put in venv site-packages	./bazel run //:app --@rules_python//python/config_settings:debugger=//debugpy:venv	❌
3	--debugger + use imports	./bazel run //:app --@rules_python//python/config_settings:debugger=//debugpy:imports	❌
4	explicit dep + put in root	Add //:debugpy to //:app.deps + ./bazel run //:app	✅
5	explicit dep + put in venv site-packages	Add //debugpy:venv to //:app.deps + ./bazel run //:app	✅
6	explicit dep + use imports	Add //debugpy:imports to //:app.deps + ./bazel run //:app	✅

🌍 Your Environment

Operating System: Macos 25
Output of bazel version: 8.5
Rules_python version: a94bd0f

[rickeylev]

Ah, this is probably an easy fix. In py_executable(), it calls collect_imports(), but doesn't pass along the extra debuger dep to it (as is done with collect_cc_info elsewhere).

[rickeylev]

@shayanhoshyari is pretty awesome you're trying to get debugpy working! This is probably one of the more promising debuggers for a nicer Just Works experience with bazel. I once had a remote debugger integrated with bazel+java+vim, and it was super slick being able to run bazel test and have the editor and process-under-test auto connect to each other and drop right into where the breakpoints were set.

LMK what other problems you run into. The two cases I'm thinking might come up are:

1. Having some environment vars automatically inherited?
2. Having a "start under debugger" hook? There is already the ADDITIONAL_INTERPRETER_ARGS env var, but idk if that's sufficient.

[shayanhoshyari]

@rickeylev seems to be working well now! 🎉

I have put the example, how it works and demo here: https://github.com/shayanhoshyari/issue-reports/tree/main/rules_python/vscode_debugger

Would you be interested in having this example directly in rules_python with a link here?

---

Here is a copy for convenience

Demo

• Install vscode recommended extensions: Cmd + P -> Show recommended extensions (honestly just the python extension matters)
• Select the target Bazel: run py run or Bazel: test py test in vscode. Only a python interpreter is needed so vscode can startup with (system install works).
  • py run: for tests and binaries. No sandboxing. You can type //:app or //:app_test in the input field.
  • py tests: for tests only. With sandboxing. You can type //:app_test in the input field.
• See breakpoints working in both main process (mattress) and subprocess (mattress/command.py).

How does it work?

flowchart TD
    vscode["vscode (debugpy DAP client)"]-->|configured in launch.json| launch_py["launch.py"]
   launch_py-->|"set secret and port in env var <br>+ include sitecustomize with --debugger"| bazel["bazel run/test"]
    bazel --> py_binary["py_binary / py_test"]
    py_binary --> |debugpy attach via sitecustomize + env var| vscode
    py_binary --> |subprocess launch| subprocess
    subprocess --> |"debugpy attach via patched entrypoint<br>(debugpy default behavior)"| vscode

Loading

In addition we pass down PYDEVD_RESOLVE_SYMLINKS to each process to make sure that the debugger hits breakpoints in original source and not the symlinks in bazel output.

In launch.json on can further set or unset IDE_PROJECT_ROOTS in lieu of justMyCode.

[shayanhoshyari]

Before this --debugger I had a much less trustworthy solution.

I used to run a cquery in launch.py to find bootstap stage 2 python file, then fish the python path out of it. Then run an aspect that wrote a json file with the providers fished out to know any env vars needed to be injected. Finally call bootstrap stage 2 with the python manually 😅

But now everything works with actual bazel without any reverse engineering even in sandboxed test

[shayanhoshyari]

ADDITIONAL_INTERPRETER_ARGS

Thanks with the pointer, I was able to use debugpy with this even without --debug :)

https://github.com/shayanhoshyari/issue-reports/tree/main/rules_python/vscode_debugger_v2

It only needs to things

1. A launch.json roughly like this

   {
       "version": "0.2.0",
       "configurations": [
           
           {
               "name": "Python: Bazel py run",
               "type": "debugpy",
               "request": "launch",
               "program": "./.vscode/debugpy/launch.py",
               "args": ["run", "${input:BazelArgs}"],
               "console": "integratedTerminal",
               "justMyCode": false,
               "env": {
                   // Due to how Bazel sets things up justMyCode: true won't work, instead set this env var
                   // Comment it out to get justMyCode: false
                   "IDE_PROJECT_ROOTS": "${workspaceFolder}",
               },
           },
           {
               "name": "Python: Bazel py test",
               "type": "debugpy",
               "request": "launch",
               "program": "./.vscode/debugpy/launch.py",
               "args": ["test", "${input:BazelArgs}"],
               "console": "integratedTerminal",
               "justMyCode": false,
               "env": {
                   // Due to how Bazel sets things up justMyCode: true won't work, instead set this env var
                   // Comment it out to get justMyCode: false
                   "IDE_PROJECT_ROOTS": "${workspaceFolder}",
               },
           },
   
       ],
       "inputs": [
           {
               "id": "BazelArgs",
               "type": "promptString",
               "description": "Optional args to pass to executable as single string",
           },
       ]
   }

2. And a launch.py ran by the launch.json (can be ran with any python, similar to system_python bootstrap situation)

   """
   Launcher script for .vscode.
   """
   import os
   import shlex
   import argparse
   import sys
   from typing import cast
   
   def main() -> None:
       parser = argparse.ArgumentParser(description="Launch bazel debugpy with test or run.")
       parser.add_argument('mode', choices=['test', 'run'], help='Choose whether to run a bazel test or run.')
       parser.add_argument('args', help='The bazel target to test or run (e.g., //foo:bar) and any additional args')
       args = parser.parse_args()
   
       # Import debugpy, provided by vscode
       try:
           # This is needed because it runs force_pydevd that does some path manipulation
           # See https://github.com/microsoft/debugpy/blob/main/src/debugpy/server/__init__.py
           import debugpy._vendored # type: ignore[import-not-found]
   
           from _pydev_bundle import pydev_monkey # type: ignore[import-not-found]
       except ImportError as exc:
           print(f"You must run launch.py as vscode debug target, err: {exc}")
           sys.exit(-1)
   
   
       # is_exec means if we are using os.exec API to replace this process.
       # I tried it to get rid of the top level process shown in vscode, but no luck.
       # when is_exec is used, one also needs to use pydev_monkey.send_process_about_to_be_replaced()
       patched_args = cast(list[str], pydev_monkey.patch_args(["python", "dummy.py"], is_exec=False))
       pydev_monkey.send_process_created_message()
   
       # Drop first and last one (python and dummy.py)
       patched_args = patched_args[1:-1]
       # Now ready to pass to rules_python. Shlex.join() does not work given how
       # rules python processes things in entrypoint. Not sure how bulletproof this
       # solution is.
       rules_python_interpretter_args = " ".join(patched_args)
   
       bzl_args = shlex.split(args.args)
       assert len(bzl_args) >= 1, "at least one arg should be given, which is the target to run"
   
       # Command line arguments
       cmd = [
           "./bazel",
           args.mode,
           # This is necessary so that the the debugger hits breakpoints in original
           # source and not the symlinks in bazel output.
           "--test_env=PYDEVD_RESOLVE_SYMLINKS=1",
           # This is necessary so tests can access the connect back env var.
           f"--test_env=RULES_PYTHON_ADDITIONAL_INTERPRETER_ARGS={rules_python_interpretter_args}",
           bzl_args[0],
       ]
       if bzl_args[1:]:
           # Extra arguments, only useful for mode == test
           cmd.append("--")
           cmd.extend(bzl_args[1:])
   
       # Env vars
       env = {
           **os.environ.copy(),
           # Similar to test_env but for run.
           "RULES_PYTHON_ADDITIONAL_INTERPRETER_ARGS" : rules_python_interpretter_args,
           "PYDEVD_RESOLVE_SYMLINKS" : "1",
       }
   
       # Run Bazel.
       os.execvpe(cmd[0], cmd, env)
    
   if __name__ == "__main__":
       main()

[shayanhoshyari]

Let me make a proper discussion to see if you are interested in adding this to rules_python in some way (doc, example, ...).

#3485