## Summary Follow up to https://github.com/astral-sh/uv/pull/15563 Closes https://github.com/astral-sh/uv/issues/13485 This is a first-pass at adding support for conditional support for Git LFS between git sources, initial feedback welcome. e.g. ``` [tool.uv.sources] test-lfs-repo = { git = "https://github.com/zanieb/test-lfs-repo.git", lfs = true } ``` For context previously a user had to set `UV_GIT_LFS` to have uv fetch lfs objects on git sources. This env var was all or nothing, meaning you must always have it set to get consistent behavior and it applied to all git sources. If you fetched lfs objects at a revision and then turned off lfs (or vice versa), the git db, corresponding checkout lfs artifacts would not be updated properly. Similarly, when git source distributions were built, there would be no distinction between sources with lfs and without lfs. Hence, it could corrupt the git, sdist, and archive caches. In order to support some sources being LFS enabled and other not, this PR adds a stateful layer roughly similar to how `subdirectory` works but for `lfs` since the git database, the checkouts and the corresponding caching layers needed to be LFS aware (requested vs installed). The caches also had to isolated and treated entirely separate when handling LFS sources. Summary * Adds `lfs = true` or `lfs = false` to git sources in pyproject.toml * Added `lfs=true` query param / fragments to most relevant url structs (not parsed as user input) * In the case of uv add / uv tool, `--lfs` is supported instead * `UV_GIT_LFS` environment variable support is still functional for non-project entrypoints (e.g. uv pip) * `direct-url.json` now has an custom `git_lfs` entry under VcsInfo (note, this is not in the spec currently -- see caveats). * git database and checkouts have an different cache key as the sources should be treated effectively different for the same rev. * sdists cache also differ in the cache key of a built distribution if it was built using LFS enabled revisions to distinguish between non-LFS same revisions. This ensures the strong assumption for archive-v0 that an unpacked revision "doesn't change sources" stays valid. Caveats * `pylock.toml` import support has not been added via git_lfs=true, going through the spec it wasn't clear to me it's something we'd support outside of the env var (for now). * direct-url struct was modified by adding a non-standard `git_lfs` field under VcsInfo which may be undersirable although the PEP 610 does say `Additional fields that would be necessary to support such VCS SHOULD be prefixed with the VCS command name` which could be interpret this change as ok. * There will be a slight lockfile and cache churn for users that use `UV_GIT_LFS` as all git lockfile entries will get a `lfs=true` fragment. The cache version does not need an update, but LFS sources will get their own namespace under git-v0 and sdist-v9/git hence a cache-miss will occur once but this can be sufficient to label this as breaking for workflows always setting `UV_GIT_LFS`. ## Test Plan Some initial tests were added. More tests likely to follow as we reach consensus on a final approach. For IT test, we may want to move to use a repo under astral namespace in order to test lfs functionality. Manual testing was done for common pathological cases like killing LFS fetch mid-way, uninstalling LFS after installing an sdist with it and reinstalling, fetching LFS artifacts in different commits, etc. PSA: Please ignore the docker build failures as its related to depot OIDC issues. --------- Co-authored-by: Zanie Blue <contact@zanie.dev> Co-authored-by: konstin <konstin@mailbox.org>
8.2 KiB
Contributing
Finding ways to help
We label issues that would be good for a first time contributor as
good first issue.
These usually do not require significant experience with Rust or the uv code base.
We label issues that we think are a good opportunity for subsequent contributions as
help wanted.
These require varying levels of experience with Rust and uv. Often, we want to accomplish these
tasks but do not have the resources to do so ourselves.
You don't need our permission to start on an issue we have labeled as appropriate for community contribution as described above. However, it's a good idea to indicate that you are going to work on an issue to avoid concurrent attempts to solve the same problem.
Please check in with us before starting work on an issue that has not been labeled as appropriate for community contribution. We're happy to receive contributions for other issues, but it's important to make sure we have consensus on the solution to the problem first.
Outside of issues with the labels above, issues labeled as
bug are the
best candidates for contribution. In contrast, issues labeled with needs-decision or
needs-design are not good candidates for contribution. Please do not open pull requests for
issues with these labels.
Please do not open pull requests for new features without prior discussion. While we appreciate exploration of new features, we will almost always close these pull requests immediately. Adding a new feature to uv creates a long-term maintenance burden and requires strong consensus from the uv team before it is appropriate to begin work on an implementation.
Setup
Rust (and a C compiler) are required to build uv.
On Ubuntu and other Debian-based distributions, you can install a C compiler with:
sudo apt install build-essential
On Fedora-based distributions, you can install a C compiler with:
sudo dnf install gcc
Testing
For running tests, we recommend nextest.
If tests fail due to a mismatch in the JSON Schema, run: cargo dev generate-json-schema.
Python
Testing uv requires multiple specific Python versions; they can be installed with:
cargo run python install
The storage directory can be configured with UV_PYTHON_INSTALL_DIR. (It must be an absolute path.)
Snapshot testing
uv uses insta for snapshot testing. It's recommended (but not necessary) to use
cargo-insta for a better snapshot review experience. See the
installation guide for more information.
In tests, you can use uv_snapshot! macro to simplify creating snapshots for uv commands. For
example:
#[test]
fn test_add() {
let context = TestContext::new("3.12");
uv_snapshot!(context.filters(), context.add().arg("requests"), @"");
}
To run and review a specific snapshot test:
cargo test --package <package> --test <test> -- <test_name> -- --exact
cargo insta review
Git and Git LFS
A subset of uv tests require both Git and Git LFS to execute properly.
These tests can be disabled by turning off either git or git-lfs uv features.
Local testing
You can invoke your development version of uv with cargo run -- <args>. For example:
cargo run -- venv
cargo run -- pip install requests
Running inside a Docker container
Source distributions can run arbitrary code on build and can make unwanted modifications to your system ("Someone's Been Messing With My Subnormals!" on Blogspot, "nvidia-pyindex" on PyPI), which can even occur when just resolving requirements. To prevent this, there's a Docker container you can run commands in:
$ docker build -t uv-builder -f crates/uv-dev/builder.dockerfile --load .
# Build for musl to avoid glibc errors, might not be required with your OS version
cargo build --target x86_64-unknown-linux-musl --profile profiling
docker run --rm -it -v $(pwd):/app uv-builder /app/target/x86_64-unknown-linux-musl/profiling/uv-dev resolve-many --cache-dir /app/cache-docker /app/scripts/popular_packages/pypi_10k_most_dependents.txt
We recommend using this container if you don't trust the dependency tree of the package(s) you are trying to resolve or install.
Profiling and Benchmarking
Please refer to Ruff's Profiling Guide, it applies to uv, too.
We provide diverse sets of requirements for testing and benchmarking the resolver in
scripts/requirements and for the installer in scripts/requirements/compiled.
You can use scripts/benchmark to benchmark predefined workloads between uv versions and with other
tools, e.g., from the scripts/benchmark directory:
uv run resolver \
--uv-pip \
--poetry \
--benchmark \
resolve-cold \
../scripts/requirements/trio.in
Analyzing concurrency
You can use tracing-durations-export to
visualize parallel requests and find any spots where uv is CPU-bound. Example usage, with uv and
uv-dev respectively:
RUST_LOG=uv=info TRACING_DURATIONS_FILE=target/traces/jupyter.ndjson cargo run --features tracing-durations-export --profile profiling -- pip compile scripts/requirements/jupyter.in
RUST_LOG=uv=info TRACING_DURATIONS_FILE=target/traces/jupyter.ndjson cargo run --features tracing-durations-export --bin uv-dev --profile profiling -- resolve jupyter
Trace-level logging
You can enable trace level logging using the RUST_LOG environment variable, i.e.
RUST_LOG=trace uv
Documentation
To preview any changes to the documentation locally:
-
Install the Rust toolchain.
-
Run
cargo dev generate-all, to update any auto-generated documentation. -
Run the development server with:
# For contributors. uvx --with-requirements docs/requirements.txt -- mkdocs serve -f mkdocs.public.yml # For members of the Astral org, which has access to MkDocs Insiders via sponsorship. uvx --with-requirements docs/requirements-insiders.txt -- mkdocs serve -f mkdocs.insiders.yml
The documentation should then be available locally at http://127.0.0.1:8000/uv/.
To update the documentation dependencies, edit docs/requirements.in and
docs/requirements-insiders.in, then run:
uv pip compile docs/requirements.in -o docs/requirements.txt --universal -p 3.12
uv pip compile docs/requirements-insiders.in -o docs/requirements-insiders.txt --universal -p 3.12
Documentation is deployed automatically on release by publishing to the Astral documentation repository, which itself deploys via Cloudflare Pages.
After making changes to the documentation, format the markdown files with:
npx prettier --prose-wrap always --write "**/*.md"
Note that the command above requires Node.js and npm to be installed on your system. As an alternative, you can run this command using Docker:
$ docker run --rm -v .:/src/ -w /src/ node:alpine npx prettier --prose-wrap always --write "**/*.md"
Releases
Releases can only be performed by Astral team members.
Changelog entries and version bumps are automated. First, run:
./scripts/release.sh
Then, editorialize the CHANGELOG.md file to ensure entries are consistently styled.
Then, open a pull request, e.g., Bump version to ....
Binary builds will automatically be tested for the release.
After merging the pull request, run the
release workflow with the version
tag. Do not include a leading v. The release will automatically be created on GitHub after
everything else publishes.