## Summary
This PR migrates our virtualenv creation from a setup that assumes prior
knowledge of the correct paths, to a technique borrowed from
`virtualenv` whereby we use `sysconfig` and `distutils` to determine the
paths. The general trick is to grab the expected paths with `sysconfig`,
then make them all relative, then make them absolute for a given
directory.
Closes#2095.
Closes#2153.
## Summary
This makes `--python python3` and `--python 3.10` more consistent on
Windows.
Closes https://github.com/astral-sh/uv/issues/2213.
## Test Plan
Ran `cargo run venv --python python3.12` with the Windows Store Python.
## Summary
We have logic in `python_query.rs` to filter out Windows Store shims
when you use invocations like `-p 3.10`, but not `--python python3`,
which is uncommon but allowed on Windows.
Closes#2211.
## Summary
Our Windows shim detection wasn't catching shims like `python3.12.exe`.
Closes#2208.
## Test Plan
Installed Python 3.12 via the Windows Store; verified that `cargo run
venv --python 3.12` failed before but passes after this change.
## Summary
When determining "direct" dependencies, we need to ensure that we
respect markers. In the linked issue, the user had an optional
dependency like:
```toml
[project.optional-dependencies]
dev = [
"setuptools>=64",
"setuptools_scm>=8"
]
```
By not respecting markers, we tried to resolve `setuptools` to the
lowest-available version. However, since `setuptools>=64` _isn't_
enabled (since it's optional), we won't respect _that_ constraint.
To be consistent, we need to omit optional dependencies just as we will
at resolution time.
Closes https://github.com/astral-sh/uv/issues/2203.
## Test Plan
`cargo test`
Follow-up to #2086: Don't use timeouts for the entire workers, but only
for the section that's about communicating with the (potentially broken)
`python` subprocess. I've also raised the timeout to 60s.
Add a `--compile` option to `pip install` and `pip sync`.
I chose to implement this as a separate pass over the entire venv. If we
wanted to compile during installation, we'd have to make sure that
writing is exclusive, to avoid concurrent processes writing broken
`.pyc` files. Additionally, this ensures that the entire site-packages
are bytecode compiled, even if there are packages that aren't from this
`uv` invocation. The disadvantage is that we do not update RECORD and
rely on this comment from [PEP 491](https://peps.python.org/pep-0491/):
> Uninstallers should be smart enough to remove .pyc even if it is not
mentioned in RECORD.
If this is a problem we can change it to run during installation and
write RECORD entries.
Internally, this is implemented as an async work-stealing subprocess
worker pool. The producer is a directory traversal over site-packages,
sending each `.py` file to a bounded async FIFO queue/channel. Each
worker has a long-running python process. It pops the queue to get a
single path (or exists if the channel is closed), then sends it to
stdin, waits until it's informed that the compilation is done through a
line on stdout, and repeat. This is fast, e.g. installing `jupyter
plotly` on Python 3.12 it processes 15876 files in 319ms with 32 threads
(vs. 3.8s with a single core). The python processes internally calls
`compileall.compile_file`, the same as pip.
Like pip, we ignore and silence all compilation errors
(https://github.com/astral-sh/uv/issues/1559). There is a 10s timeout to
handle the case when the workers got stuck. For the reviewers, please
check if i missed any spots where we could deadlock, this is the hardest
part of this PR.
I've added `uv-dev compile <dir>` and `uv-dev clear-compile <dir>`
commands, mainly for my own benchmarking. I don't want to expose them in
`uv`, they almost certainly not the correct workflow and we don't want
to support them.
Fixes#1788Closes#1559Closes#1928
## Summary
We have at least one reported case of this happening. It's preferable
IMO to move on rather than fail hard despite sub-pbar registry behavior.
Closes https://github.com/astral-sh/uv/issues/2099.
## Summary
This will make it easier to use the paths returned by `distutils.py`
(for some cases). No code or behavior changes; just removing some fields
we don't need.
## Summary
Closes#1977
This allows us to send uv's version in the `uv-client` User Agent
header.
Here's how request headers look like to a server now:
```
...
Accept: application/vnd.pypi.simple.v1+json, application/vnd.pypi.simple.v1+html;q=0.2, text/html;q=0.01
User-Agent: uv/0.1.13
...
```
~~I went for a mix of Option 1 and 2 from #1977.~~ Open to alternative
naming as well, not tied too strongly here to the names picked.
~~Another possibility for this new crate is that we can use it to
consolidate metadata that exists across crates to ultimately be able to
create linehaul information described in #1958, but I haven't looked
into what those changes might look like.~~
<!-- What's the purpose of the change? What does it do, and why? -->
## Test Plan
<!-- How was it tested? -->
Added initial tests in the new crate to exercise its public API and
added a new test to uv-client to validate the headers using a 1-time
disposable server.
## Summary
Ensures that local dependencies function similarly to editables, in that
if they're `uv pip install`ed, we invalidate them.
Closes https://github.com/astral-sh/uv/issues/1651.
## Summary
Internal-only refactor to consolidate multiple codepaths we have for
checking whether a cached or installed entry is up-to-date with a local
requirement.
Error for `uv pip compile scripts/requirements/jupyter.in` without
internet:
**Before**
```
error: error sending request for url (https://pypi.org/simple/jupyter/): error trying to connect: dns error: failed to lookup address information: No such host is known. (os error 11001)
Caused by: error trying to connect: dns error: failed to lookup address information: No such host is known. (os error 11001)
Caused by: dns error: failed to lookup address information: No such host is known. (os error 11001)
Caused by: failed to lookup address information: No such host is known. (os error 11001)
```
**After**
```
error: Could not connect, are you offline?
Caused by: error sending request for url (https://pypi.org/simple/django/): error trying to connect: dns error: failed to lookup address information: Temporary failure in name resolution
Caused by: error trying to connect: dns error: failed to lookup address information: Temporary failure in name resolution
Caused by: dns error: failed to lookup address information: Temporary failure in name resolution
Caused by: failed to lookup address information: Temporary failure in name resolution
```
On linux, it would be "Temporary failure in name resolution" instead of
"No such host is known. (os error 11001)".
The implementation checks for "dne error" stringly as hyper errors are
opaque. The danger is that this breaks with a hyper update. We still get
the complete error trace since reqwest eagerly inlines errors
(https://github.com/seanmonstar/reqwest/issues/2147).
No test since i wouldn't know how to simulate this in cargo test.
Fixes#1971
## Summary
This PR expands environment variables in `-r` and `-c` paths _within_
requirements files. We already do this for `@` URL references and
others.
Closes https://github.com/astral-sh/uv/issues/1473.
## Summary
After https://github.com/astral-sh/uv/pull/2121, the only remaining
issue is that calling `canonicalize` on these Pythons returns an error.
Closes https://github.com/astral-sh/uv/issues/2105.
## Test Plan
Uninstalled all python.org Pythons on my Windows machine, then created a
virtualenv. The resulting config file:
```
Using Python 3.11.8 interpreter at: C:\Users\crmar\AppData\Local\Microsoft\WindowsApps\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\python.exe
Creating virtualenv at: .venv
Activate with: .venv\Scripts\activate
PS C:\Users\crmar\workspace\puffin> cat .\.venv\pyvenv.cfg
home = C:\Users\crmar\AppData\Local\Microsoft\WindowsApps\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0
implementation = CPython
version_info = 3.11.8
include-system-site-packages = false
uv = 0.1.13
prompt = puffin
```
Prior to this PR, it would fail with a canonicalization error.
Prior to #2121, it would leave a "bad" Python in the config file:
```
Using Python 3.11.8 interpreter at: C:\Users\crmar\AppData\Local\Microsoft\WindowsApps\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\python.exe
Creating virtualenv at: .venv
Activate with: .venv\Scripts\activate
PS C:\Users\crmar\workspace\puffin> cat .\.venv\pyvenv.cfg
home = C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.2288.0_x64__qbz5n2kfra8p0
implementation = CPython
version_info = 3.11.8
include-system-site-packages = false
uv = 0.1.13
prompt = puffin
```
Which, once activated, would fail with:
```
(venv) PS C:\Users\crmar\workspace\puffin> python
No Python at '"C:\Users\crmar\AppData\Local\Programs\Python\Python312\python.exe'
```
## Summary
When I install via the Windows Store, `interpreter.base_prefix` contains
a bunch of resolved information that leads to a broken environment.
Instead, we now use `sys._base_executable` on Windows by default,
falling back to `sys.base_prefix` if it doesn't exist. (There are some
issues with `sys.base_executable` that lead to complexity in
`virtualenv`, but they only affect POSIX.) Admittedly, I don't know when
`sys._base_executable` wouldn't exist. It exists in all the environments
I've tested.
Additionally, we use the system interpreter directly if we're outside of
a virtualenv.
## Summary
Closes#2012
This changes `RequirementsTxtParserError::Parser` to take a line, column
instead of cursor location to improve reporting of parser errors. A new
function was added to compute the line and column based on the content
and cursor location when a parser error occurs for simplicity.
Given `uv pip compile .\requirements.txt` of below
```
numpy>=1,<2
--borken
tqdm
```
Before:
```
error: Unexpected '-', expected '-c', '-e', '-r' or the start of a requirement in `.\requirements.txt` at position 14
```
After:
```
error: Unexpected '-', expected '-c', '-e', '-r' or the start of a requirement in `.\requirements.txt` at position 2:3
```
Open Question: Do we want to support `line:column` for other types of
errors? I didn't look dig other potential error types where this might
be desired.
## Test Plan
New test was added to `requirements-txt` crate with this example.
## Summary
It looks like these have been included since the very first gourgeist
commit, but `virtualenv` and `venv` don't include them, and they only
add complexity AFAICT.
## Summary
I think Camino is nice but it makes it much harder to work in
`uv-virtualenv`, since it's the _only_ crate that uses it. If we want to
use Camino, we should use it everywhere IMO.
## Summary
Right now, we have virtualenv construction encoded in a few different
places. Namely, it happens in both `gourgeist` and
`virtualenv_layout.rs` -- _and_ `interpreter.rs` also encodes some
knowledge about how they work, by way of reconstructing the
`SysconfigPaths`.
Instead, `gourgeist` now returns the complete layout, enumerating all
the directories it created. So, rather than returning a root directory,
and re-creating all those paths in `uv-interpreter`, we pass the data
directly back to it.
## Summary
Adds support for `--system-site-packages`. Unlike `pip`, we won't take
the system site packages into account in subsequent commands. I think
this is ok.
Closes https://github.com/astral-sh/uv/issues/1483.
Update #1918 to handle https://github.com/pypa/pip/pull/12536, where pip
removed their python minor entrypoint. The pip test is semi-functional
since it builds pip from source instead of using a wheel with the wrong
entrypoint, we have to update it when this pip version has a release.
Closes#1593.
## Summary
This is based on Pradyun's installer branch
(d01624e5f2/src/installer/scripts.py (L54)),
which is itself based on pip
(0ad4c94be7/src/pip/_vendor/distlib/scripts.py (L136)).
The gist of it is: on Posix platforms, if a path contains a space (or is
too long), we wrap the shebang in a `/bin/sh` invocation.
Closes https://github.com/astral-sh/uv/issues/2076.
## Test Plan
```
❯ cargo run venv "foo"
Finished dev [unoptimized + debuginfo] target(s) in 0.14s
Running `target/debug/uv venv foo`
Using Python 3.12.0 interpreter at: /Users/crmarsh/.local/share/rtx/installs/python/3.12.0/bin/python3
Creating virtualenv at: foo
Activate with: source foo/bin/activate
❯ source "foo bar/bin/activate"
❯ which black
black not found
❯ cargo run pip install black
Resolved 6 packages in 177ms
Installed 6 packages in 17ms
+ black==24.2.0
+ click==8.1.7
+ mypy-extensions==1.0.0
+ packaging==23.2
+ pathspec==0.12.1
+ platformdirs==4.2.0
❯ which black
/Users/crmarsh/workspace/uv/foo bar/bin/black
❯ black
Usage: black [OPTIONS] SRC ...
One of 'SRC' or 'code' is required.
❯ cat "foo bar/bin/black"
#!/bin/sh
'''exec' '/Users/crmarsh/workspace/uv/foo bar/bin/python' "$0" "$@"
' '''
# -*- coding: utf-8 -*-
import re
import sys
from black import patched_main
if __name__ == "__main__":
sys.argv[0] = re.sub(r"(-script\.pyw|\.exe)?$", "", sys.argv[0])
sys.exit(patched_main())
```
I'm not at all sure whether this is a correct fix or not, but it does
seem to make `pypy` work in at least some cases with `uv`. Previously,
I couldn't get it to work at all. Namely the virtualenv was created
with a `lib/python3.10/site-packages`, but whenever I did a `uv
pip install` in that virtualenv, it was looking for a non-existent
`lib/pypy3.10/site-packages` directory.
With this PR, the workflow reported as not working in #1488 now works
for me:
```
$ pypy3 --version
Python 3.10.13 (fc59e61cfbff, Jan 17 2024, 05:35:45)
[PyPy 7.3.15 with GCC 13.2.1 20230801]
$ uv venv --python $(which pypy3) --seed
Using Python 3.10.13 interpreter at: /usr/bin/pypy3
Creating virtualenv at: .venv
+ pip==24.0
+ setuptools==69.1.1
+ wheel==0.42.0
Activate with: source .venv/bin/activate
$ uv pip install 'alembic==1.0.11'
Resolved 9 packages in 8ms
Installed 9 packages in 14ms
+ alembic==1.0.11
+ greenlet==3.0.3
+ mako==1.3.2
+ markupsafe==2.1.5
+ python-dateutil==2.8.2
+ python-editor==1.0.4
+ six==1.16.0
+ sqlalchemy==2.0.27
+ typing-extensions==4.10.0
```
Where as previously (current `main`), I was hitting this error:
```
$ uv venv --python $(which pypy3) --seed
Using Python 3.10.13 interpreter at: /usr/bin/pypy3
Creating virtualenv at: .venv
+ pip==24.0
+ setuptools==69.1.1
+ wheel==0.42.0
Activate with: source .venv/bin/activate
$ uv pip install 'alembic==1.0.11'
error: Failed to list installed packages
Caused by: failed to read directory `/home/andrew/astral/issues/uv/i1488/.venv/lib/pypy3.10/site-packages`
Caused by: No such file or directory (os error 2)
```
Notice though that neither outcome above matches the error reported in #1488,
so this is likely not a complete fix. There are perhaps other lurking
issues.
Ref #1488
## Summary
This was a missed find-and-replace. We shouldn't assume `layout.platlib`
here, since `RECORD` will be written to `site_packages` (which could be
`layout.purelib`).
This is hard to reproduce. You need a _fresh_ environment where
`purelib` and `platlib` differ (which isn't the case for virtualenvs, at
least typically), and you need to be installing a new package that is a
purelib. I tested it by manually changing `platlib` to point to a
different path.
Closes https://github.com/astral-sh/uv/issues/2064.
Previously, `uv` would always prioritize the index given by
`--index-url`. It would then try any indexes after that given by zero
or more `--extra-index-url` flags. This differed from `pip` in that any
priority was given at all, where `pip` doesn't guarantee any priority
ordering of indexes.
We could go in the direction of mimicing `pip`'s behavior here, but it
at present has issues with dependency confusion attacks where packages
may get installed from indexes you don't control. More specifically,
there is an issue of different trust levels. See discussion in #171 and
[PEP-0708] for more on the security impact.
In contrast, `uv` will only select versions for a package from a single
index. That is, even if `foo` is in indexes `a` and `b`, it will
only consider the versions from the index that it checks first. This
probably helps with respect to dependency confusion attacks, but also
means that `uv` doesn't quite cover all of the same use cases as `pip`.
In this PR, we retain the notion of prioritizing indexes, but
tweak it so that PyPI is preferred last as opposed to first. Or
more precisely, the `--index-url` flag specifies a fallback index,
not the primary index, and is deprioritized beneath every index
specified by `--extra-index-url`. The ordering among indexes given by
`--extra-index-url` remains the same: earlier indexes are prioritized
over later indexes.
While this tweak likely won't hit all use cases, I believe it will
resolve some of the most common pain points without exacerbating
dependency confusion problems.
Ref #171, Fixes#1377, Fixes#1451, Fixes#1600
[PEP-0708]: https://peps.python.org/pep-0708/
