Fixes#3371
It seems like uv doesn't proactively enforce 3.8+ and in most cases just
issues a warning. This PR keeps that property, only adding the new check
when it is known to fail. I checked the imports in this file and the
other ones seem fine.
## Summary
We were writing the build dependencies into the `--target` directory,
which both made builds fail and led to them leaking into the user's
directory.
Closes https://github.com/astral-sh/uv/issues/3349.
In *some* places in our crates, `serde` (and `rkyv`) are optional
dependencies. I believe this was done out of reasons of "good sense,"
that is, it follows a Rust ecosystem pattern where serde integration
tends to be an opt-in crate feature. (And similarly for `rkyv`.)
However, ultimately, `uv` itself requires `serde` and `rkyv` to
function. Since our crates are strictly internal, there are limited
consumers for our crates without `serde` (and `rkyv`) enabled. I think
one possibility is that optional `serde` (and `rkyv`) integration means
that someone can do this:
cargo test -p pep440_rs
And this will run tests _without_ `serde` or `rkyv` enabled. That in
turn could lead to faster iteration time by reducing compile times. But,
I'm not sure this is worth supporting. The iterative compilation times
of
individual crates are probably fast enough in debug mode, even with
`serde` and `rkyv` enabled. Namely, `serde` and `rkyv` themselves
shouldn't need to be re-compiled in most cases. On `main`:
```
from-scratch: `cargo test -p pep440_rs --lib` 0.685
incremental: `cargo test -p pep440_rs --lib` 0.278s
from-scratch: `cargo test -p pep440_rs --features serde,rkyv --lib` 3.948s
incremental: `cargo test -p pep440_rs --features serde,rkyv --lib` 0.321s
```
So while a from-scratch build does take significantly longer, an
incremental build is about the same.
The benefit of doing this change is two-fold:
1. It brings out crates into alignment with "reality." In particular,
some crates were _implicitly_ relying on `serde` being enabled
without explicitly declaring it. This technically means that our
`Cargo.toml`s were wrong in some cases, but it is hard to observe it
because of feature unification in a Cargo workspace.
2. We no longer need to deal with the cognitive burden of writing
`#[cfg_attr(feature = "serde", ...)]` everywhere.
Split out of #3266
The "selector" concept doesn't seem well enough defined as-is. For
example, `PythonVersion` belongs there but isn't present. Going for
smaller modules instead.
Split out of https://github.com/astral-sh/uv/pull/3266
If `UV_BOOTSTRAP_DIR` and `CARGO_MANIFEST_DIR` are both unset, we
currently panic. This isn't good once we start to use managed toolchains
in production. We'll need to change this more later once the toolchain
directory is more user-facing.
Moves all of `uv-toolchain` into `uv-interpreter`. We may split these
out in the future, but the refactoring I want to do for interpreter
discovery is easier if I don't have to deal with entanglement. Includes
some restructuring of `uv-interpreter`.
Part of #2386
## Summary
The approach taken here is to model `--target` as an install scheme in
which all the directories are just subdirectories of the `--target`.
From there, everything else... just works? Like, upgrade, uninstalls,
editables, etc. all "just work".
Closes#1517.
I rebased https://github.com/astral-sh/uv/pull/2757 then realized that
we want to implement this for more than `uv venv`.
Closes https://github.com/astral-sh/uv/issues/2587
Closes https://github.com/astral-sh/uv/issues/2757
```
❯ cargo run -q -- pip install -p /Users/mz/bin/python3.7 anyio
warning: uv is only compatible with Python 3.8+, found Python 3.7.17.
Audited 1 package in 84ms
❯ cargo run -q -- venv -p /Users/mz/bin/python3.7
warning: uv is only compatible with Python 3.8+, found Python 3.7.17.
Using Python 3.7.17 interpreter at: /Users/mz/bin/python3.7
Creating virtualenv at: .venv
Activate with: source .venv/bin/activate
```
---------
Co-authored-by: Stevie Gayet <stegayet@users.noreply.github.com>
## Summary
I think these are useful to have for consistency, though the `--system`
variant requires some new threading.
Closes: https://github.com/astral-sh/uv/issues/2242.
If a virtual environment does not exist, we will create one for the
duration of the invocation.
Adds an `--isolated` flag to force this behavior (ignoring an existing
virtual environment).
Fixes https://github.com/astral-sh/uv/issues/3060
## Summary
Allows passing a virtual environment (the path to the directory, rather
than the path to the Python interpreter within the directory) to the
`--python` option of the `uv pip` command.
## Test Plan
Tested manually to confirm that the expected new functionality works.
The test suite still passes after this change.
I don't know how to add tests for a new feature like this. I would be
happy to do so if someone can give me some pointers on how to do it.
## Summary
This PR adds system install tests to verify the behavior described in
#2798. It turns out this behavior _also_ affects Fedora and Amazon
Linux, we just didn't have the right conditions enabled (specifically,
you need to create the virtualenv with `python -m venv` to get these
symlinks), so the test suite was expanded to capture that.
The issue itself is also fixed by way of deduplicating the
`site-packages` entries.
Closes: https://github.com/astral-sh/uv/issues/2798
## Summary
It turns out that `normalize_path` (sourced from Cargo) has a subtle
bug. If you pass it a relative path that traverses beyond the root, it
silently drops components. So, e.g., passing `../foo/bar`, it will just
drop the leading `..` and return `foo/bar`.
This PR encodes that behavior as a `Result` and avoids using it in such
cases.
Closes https://github.com/astral-sh/uv/issues/3012.
freethreaded python reintroduces abiflags since it is incompatible with
regular native modules and abi3.
Tests: None yet! We're lacking cpython 3.13 no-gil builds we can use in
ci.
My test setup:
```
PYTHON_CONFIGURE_OPTS="--enable-shared --disable-gil" pyenv install 3.13.0a5
cargo run -q -- venv -q -p python3.13 .venv3.13 --no-cache-dir && cargo run -q -- pip install -v psutil --no-cache-dir && .venv3.13/bin/python -c "import psutil"
```
Fixes#2429
See https://github.com/astral-sh/uv/issues/2617
Note this also includes:
- #2918
- #2931 (pending)
A first step towards Python toolchain management in Rust.
First, we add a new crate to manage Python download metadata:
- Adds a new `uv-toolchain` crate
- Adds Rust structs for Python version download metadata
- Duplicates the script which downloads Python version metadata
- Adds a script to generate Rust code from the JSON metadata
- Adds a utility to download and extract the Python version
I explored some alternatives like a build script using things like
`serde` and `uneval` to automatically construct the code from our
structs but deemed it to heavy. Unlike Rye, I don't generate the Rust
directly from the web requests and have an intermediate JSON layer to
speed up iteration on the Rust types.
Next, we add add a `uv-dev` command `fetch-python` to download Python
versions per the bootstrapping script.
- Downloads a requested version or reads from `.python-versions`
- Extracts to `UV_BOOTSTRAP_DIR`
- Links executables for path extension
This command is not really intended to be user facing, but it's a good
PoC for the `uv-toolchain` API. Hash checking (via the sha256) isn't
implemented yet, we can do that in a follow-up.
Finally, we remove the `scripts/bootstrap` directory, update CI to use
the new command, and update the CONTRIBUTING docs.
<img width="1023" alt="Screenshot 2024-04-08 at 17 12 15"
src="57bd3cf1-7477-4bb8-a8e9-802a00d772cb">
Scott schafer got me the idea: We can avoid repeating the path for
workspaces dependencies everywhere if we declare them in the virtual
package once and treat them as workspace dependencies from there on.
It is a common pattern to have an active conda base env (that sets
`CONDA_PREFIX`) and then create a venv on top of that (setting
`VIRTUAL_ENV`).
Previously, we would error when both `VIRTUAL_ENV` and `CONDA_PREFIX`
were set, now `VIRTUAL_ENV` takes precedence over `CONDA_PREFIX`.
Fixes#2028
## Summary
This PR changes our user-facing representation for paths to use relative
paths, when the path is within the current working directory. This
mirrors what we do in Ruff. (If the path is _outside_ the current
working directory, we print an absolute path.)
Before:
```shell
❯ uv venv .venv2
Using Python 3.12.2 interpreter at: /Users/crmarsh/workspace/uv/.venv/bin/python3
Creating virtualenv at: .venv2
Activate with: source .venv2/bin/activate
```
After:
```shell
❯ cargo run venv .venv2
Finished dev [unoptimized + debuginfo] target(s) in 0.15s
Running `target/debug/uv venv .venv2`
Using Python 3.12.2 interpreter at: .venv/bin/python3
Creating virtualenv at: .venv2
Activate with: source .venv2/bin/activate
```
Note that we still want to use the existing `.simplified_display()`
anywhere that the path is being simplified, but _still_ intended for
machine consumption (e.g., when passing to `.current_dir()`).
## Summary
If you have a file `typing.py` in the current working directory, `python
-m` doesn't work in some Python versions:
```sh
❯ python -m foo
Could not import runpy module
Traceback (most recent call last):
File "/Users/crmarsh/.local/share/rtx/installs/python/3.9.18/lib/python3.9/runpy.py", line 15, in <module>
import importlib.util
File "/Users/crmarsh/.local/share/rtx/installs/python/3.9.18/lib/python3.9/importlib/util.py", line 2, in <module>
from . import abc
File "/Users/crmarsh/.local/share/rtx/installs/python/3.9.18/lib/python3.9/importlib/abc.py", line 17, in <module>
from typing import Protocol, runtime_checkable
ImportError: cannot import name 'Protocol' from 'typing' (/Users/crmarsh/workspace/uv/typing.py)
```
This did _not_ cause problems for us on Python 3.11 or later, because we
set `PYTHONSAFEPATH`, which avoids adding the current working directory
to `sys.path`. However, on earlier versions, we _were_ failing with the
above. (It's important that we run interpreter discovery in the current
working directory, since doing otherwise breaks pyenv shims.)
The fix implemented here uses `-I` to run Python in isolated mode, which
is even stricter. The downside of isolated mode is that we currently
rely on setting `PYTHONPATH` to find the "fake module" that we create on
disk, and `-I` means `PYTHONPATH` is totally ignored. So, instead, we
run a script directly, and that _script_ injects the path we care about
into `PYTHONSAFEPATH`.
Closes https://github.com/astral-sh/uv/issues/2547.
## Summary
I tried out `cargo shear` to see if there are any unused dependencies
that `cargo udeps` isn't reporting. It turned out, there are a few. This
PR removes those dependencies.
## Test Plan
`cargo build`
## Summary
In reality, there's no such thing as the `site-packages` directory for a
given virtualenv. Rather, Python defines both `purelib` and `platlib`,
where the former is for pure-Python packages and the latter is for
packages that contain native code. These are almost always set to the
same thing... but they don't _have_ to be, and in fact of Fedora they
are not.
This PR changes the `site_packages` method to return an iterator of
directories.
Closes https://github.com/astral-sh/uv/issues/2527.
## Summary
By running `get_interpreter_info.py` outside of the current working
directory, we seem to have broken pyenv shims.
Closes https://github.com/astral-sh/uv/issues/2488.
## Test Plan
Without this change (resolving to the Homebrew Python, even though we
start with a shim):
```
DEBUG Starting interpreter discovery for Python @ `python3.11`
DEBUG Probing interpreter info for: /Users/crmarsh/.pyenv/shims/python3.11
DEBUG Found Python 3.11.7 for: /Users/crmarsh/.pyenv/shims/python3.11
Using Python 3.11.7 interpreter at: /opt/homebrew/opt/python@3.11/bin/python3.11
Creating virtualenv at: .venv
INFO Removing existing directory
Activate with: source .venv/bin/activate
```
With this change:
```
DEBUG Starting interpreter discovery for Python @ `python3.11`
DEBUG Probing interpreter info for: /Users/crmarsh/.pyenv/shims/python3.11
DEBUG Found Python 3.11.1 for: /Users/crmarsh/.pyenv/shims/python3.11
Using Python 3.11.1 interpreter at: /Users/crmarsh/.pyenv/versions/3.11.1/bin/python3.11
Creating virtualenv at: .venv
INFO Removing existing directory
Activate with: source .venv/bin/activate
```
The architecture of uv does not necessarily match that of the python
interpreter (#2326). In cross compiling/testing scenarios the operating
system can also mismatch. To solve this, we move arch and os detection
to python, vendoring the relevant pypa/packaging code, preventing
mismatches between what the python interpreter was compiled for and what
uv was compiled for.
To make the scripts more manageable, they are now a directory in a
tempdir and we run them with `python -m` . I've simplified the
pypa/packaging code since we're still building the tags in rust. A
`Platform` is now instantiated by querying the python interpreter for
its platform. The pypa/packaging files are copied verbatim for easier
updates except a `lru_cache()` python 3.7 backport.
Error handling is done by a `"result": "success|error"` field that allow
passing error details to rust:
```console
$ uv venv --no-cache
× Can't use Python at `/home/konsti/projects/uv/.venv/bin/python3`
╰─▶ Unknown operation system `linux`
```
I've used the [maturin sysconfig
collection](855f6d2cb1/sysconfig)
as reference. I'm unsure how to test these changes across the wide
variety of platforms.
Fixes#2326
## Summary
Per [PEP 508](https://peps.python.org/pep-0508/), `python_version` is
just major and minor:
Right now, we're using the provided version directly, so if it's, e.g.,
`-p 3.11.8`, we'll inject the wrong marker. This was causing `pandas` to
omit `numpy` when `-p 3.11.8` was provided, since its markers look like:
```
Requires-Dist: numpy<2,>=1.22.4; python_version < "3.11"
Requires-Dist: numpy<2,>=1.23.2; python_version == "3.11"
Requires-Dist: numpy<2,>=1.26.0; python_version >= "3.12"
```
Closes https://github.com/astral-sh/uv/issues/2392.
Behind error messages, the debug log is the second most important
resource to finding out what and why went wrong when there was a problem
with uv. It is important to see which paths it has found and how the
decisions in the resolver were made. I'm trying to improve the
experience interacting with the debug log.
The hierarchical layer is verbose and hard to follow, so it's moved to
the `-vv` extra verbose setting, while `-v` works like
`RUST_LOG=uv=debug`.
For installing jupyter with a warm cache:
* Default:
https://gist.github.com/konstin/4de6e466127311c5a5fc2f99c56a8e11
* `-v`: https://gist.github.com/konstin/e7bafe0ec7d07e47ba98a3865ae2ef3e
* `-vv`:
https://gist.github.com/konstin/3ee1aaff37f91cceb6275dd5525f180e
Ideally, we would have `-v`, `-vv` and `-vvv`, but we're lacking the the
`info!` layer for `-v`, so there's only two layers for now.
The `tracing_subcriber` formatter always print the current span, so i
replaced it with a custom formatter.
Best read commit-by-commit.
Preparing for #2058, i found it hard to follow where which discovery
function gets called. I moved all the discovery functions to a
`find_python` module (some exposed through `PythonEnvironment`) and
documented which subcommand uses which python discovery strategy.
No functional changes.
## Summary
It turns out that setuptools includes a shim to patch distutils. I'll
admit that I don't fully understand why or how it's different, but this
is the trick `pip` uses to ensure that it gets the "original" distutils.
We actually use distutils in two places: once for the system Python
scheme, and once for virtual environments. In virtualenv, they _do_ use
the patched distutils, so this could deviate in ways I don't understand.
Closes#2302.
## Summary
This PR enables use of the Windows Store Pythons even with `py` is not
installed. Specifically, we need to ensure that the `python.exe` and
`python3.exe` executables installed into the
`C:\Users\crmar\AppData\Local\Microsoft\WindowsApp` directory _are_ used
when they're not "App execution aliases" (which merely open the Windows
Store, to help you install Python).
When `py` is installed, this isn't strictly necessary, since the
"resolved" executables are discovered via `py`. These look like
`C:\Users\crmar\AppData\Local\Microsoft\WindowsApps\PythonSoftwareFoundation.Python.3.11_qbs5n2kfra8p0\python.exe`.
Closes https://github.com/astral-sh/uv/issues/2264.
## Test Plan
- Removed all Python installations from my Windows machine.
- Uninstalled `py`.
- Enabled "App execution aliases".
- Verified that for both `cargo run venv --python python.exe` and `cargo
run venv --python python3.exe`, `uv` exited with a failure that no
Python could be found.
- Installed Python 3.10 via the Windows Store.
- Verified that the above commands succeeded without error.
- Verified that `cargo run venv --python python3.10.exe` _also_
succeeded.
## Summary
In #2102, I did some refactor, and changed a method to return the Python
executable path rather than the parent directory path. But I missed this
one codepath for Conda on Windows.
Closes https://github.com/astral-sh/uv/issues/2269.
## Test Plan
- Installed micromamba on my Windows machine.
- Reproduced the failure in the linked issue.
- Verified that `python.exe` exists at `${CONDA_PREFIX}\python.exe`.
- Ran with this change; installed successfully.
## Summary
This PR adds support for pip's `--no-build-isolation`. When enabled,
build requirements won't be installed during PEP 517-style builds, but
the source environment _will_ be used when executing the build steps
themselves.
Closes https://github.com/astral-sh/uv/issues/1715.
`uv pip install mysqlclient==2.1.1` on python 3.12 on windows, where the
are no binary wheels:
Part of #2052.
## Summary
`pip` uses `sysconfig` for Python 3.10 and later by default; however, it
falls back to `distutils` for earlier Python versions, and distros can
actually tell `pip` to continue falling back to `distutils` via the
`_PIP_USE_SYSCONFIG` variable.
By _always_ using `sysconfig`, we're doing the wrong then when
installing into some system Pythons, e.g., on Debian prior to Python
3.10.
This PR modifies our logic to mirror `pip` exactly, which is what's been
recommended to me as the right thing to do.
Closes https://github.com/astral-sh/uv/issues/2113.
## Test Plan
Most notably, the new Debian tests pass here (which fail on main:
https://github.com/astral-sh/uv/pull/2144).
I also added Pyston as a second stress-test.
## Summary
This PR migrates our virtualenv creation from a setup that assumes prior
knowledge of the correct paths, to a technique borrowed from
`virtualenv` whereby we use `sysconfig` and `distutils` to determine the
paths. The general trick is to grab the expected paths with `sysconfig`,
then make them all relative, then make them absolute for a given
directory.
Closes#2095.
Closes#2153.
## Summary
This makes `--python python3` and `--python 3.10` more consistent on
Windows.
Closes https://github.com/astral-sh/uv/issues/2213.
## Test Plan
Ran `cargo run venv --python python3.12` with the Windows Store Python.
## Summary
We have logic in `python_query.rs` to filter out Windows Store shims
when you use invocations like `-p 3.10`, but not `--python python3`,
which is uncommon but allowed on Windows.
Closes#2211.
## Summary
Our Windows shim detection wasn't catching shims like `python3.12.exe`.
Closes#2208.
## Test Plan
Installed Python 3.12 via the Windows Store; verified that `cargo run
venv --python 3.12` failed before but passes after this change.
