Support --find-links-style "flat" indexes in [[tool.uv.index]] (#12407)
Some checks are pending
CI / cargo dev generate-all (push) Blocked by required conditions
CI / cargo shear (push) Waiting to run
CI / Determine changes (push) Waiting to run
CI / lint (push) Waiting to run
CI / cargo clippy | ubuntu (push) Blocked by required conditions
CI / cargo clippy | windows (push) Blocked by required conditions
CI / cargo test | ubuntu (push) Blocked by required conditions
CI / cargo test | macos (push) Blocked by required conditions
CI / cargo test | windows (push) Blocked by required conditions
CI / check windows trampoline | aarch64 (push) Blocked by required conditions
CI / smoke test | windows x86_64 (push) Blocked by required conditions
CI / check windows trampoline | i686 (push) Blocked by required conditions
CI / check windows trampoline | x86_64 (push) Blocked by required conditions
CI / test windows trampoline | i686 (push) Blocked by required conditions
CI / test windows trampoline | x86_64 (push) Blocked by required conditions
CI / typos (push) Waiting to run
CI / check system | python3.12 via chocolatey (push) Blocked by required conditions
CI / mkdocs (push) Waiting to run
CI / build binary | linux libc (push) Blocked by required conditions
CI / build binary | linux musl (push) Blocked by required conditions
CI / build binary | macos aarch64 (push) Blocked by required conditions
CI / build binary | macos x86_64 (push) Blocked by required conditions
CI / build binary | windows x86_64 (push) Blocked by required conditions
CI / build binary | windows aarch64 (push) Blocked by required conditions
CI / cargo build (msrv) (push) Blocked by required conditions
CI / build binary | freebsd (push) Blocked by required conditions
CI / ecosystem test | pydantic/pydantic-core (push) Blocked by required conditions
CI / ecosystem test | prefecthq/prefect (push) Blocked by required conditions
CI / ecosystem test | pallets/flask (push) Blocked by required conditions
CI / smoke test | linux (push) Blocked by required conditions
CI / check system | alpine (push) Blocked by required conditions
CI / smoke test | macos (push) Blocked by required conditions
CI / smoke test | windows aarch64 (push) Blocked by required conditions
CI / integration test | conda on ubuntu (push) Blocked by required conditions
CI / integration test | deadsnakes python3.9 on ubuntu (push) Blocked by required conditions
CI / integration test | free-threaded on linux (push) Blocked by required conditions
CI / integration test | free-threaded on windows (push) Blocked by required conditions
CI / integration test | pypy on ubuntu (push) Blocked by required conditions
CI / integration test | pypy on windows (push) Blocked by required conditions
CI / integration test | graalpy on ubuntu (push) Blocked by required conditions
CI / integration test | graalpy on windows (push) Blocked by required conditions
CI / integration test | github actions (push) Blocked by required conditions
CI / integration test | determine publish changes (push) Blocked by required conditions
CI / integration test | uv publish (push) Blocked by required conditions
CI / integration test | uv_build (push) Blocked by required conditions
CI / check cache | ubuntu (push) Blocked by required conditions
CI / check cache | macos aarch64 (push) Blocked by required conditions
CI / check system | python on debian (push) Blocked by required conditions
CI / check system | python on fedora (push) Blocked by required conditions
CI / check system | python on ubuntu (push) Blocked by required conditions
CI / check system | python on opensuse (push) Blocked by required conditions
CI / check system | python on rocky linux 8 (push) Blocked by required conditions
CI / check system | python on rocky linux 9 (push) Blocked by required conditions
CI / check system | pypy on ubuntu (push) Blocked by required conditions
CI / check system | pyston (push) Blocked by required conditions
CI / check system | python on macos aarch64 (push) Blocked by required conditions
CI / check system | homebrew python on macos aarch64 (push) Blocked by required conditions
CI / check system | python on macos x86-64 (push) Blocked by required conditions
CI / check system | python3.10 on windows x86-64 (push) Blocked by required conditions
CI / check system | python3.10 on windows x86 (push) Blocked by required conditions
CI / check system | python3.13 on windows x86-64 (push) Blocked by required conditions
CI / check system | x86-64 python3.13 on windows aarch64 (push) Blocked by required conditions
CI / check system | windows registry (push) Blocked by required conditions
CI / check system | python3.9 via pyenv (push) Blocked by required conditions
CI / check system | python3.13 (push) Blocked by required conditions
CI / check system | conda3.11 on macos aarch64 (push) Blocked by required conditions
CI / check system | conda3.8 on macos aarch64 (push) Blocked by required conditions
CI / check system | conda3.11 on linux x86-64 (push) Blocked by required conditions
CI / check system | conda3.8 on linux x86-64 (push) Blocked by required conditions
CI / check system | conda3.11 on windows x86-64 (push) Blocked by required conditions
CI / check system | conda3.8 on windows x86-64 (push) Blocked by required conditions
CI / check system | amazonlinux (push) Blocked by required conditions
CI / check system | embedded python3.10 on windows x86-64 (push) Blocked by required conditions
CI / benchmarks (push) Blocked by required conditions

## Summary

This PR extends `[[tool.uv.index]]` to support `--find-links`-style
"flat" indexes, so that users can point to such indexes without using
`--find-links` _and_ get access to the full functionality of
`[[tool.uv.index]]` (e.g., they can now pin packages to
`--find-links`-style indexes).

Note that, at present, `--find-links` indexes actually have some quirky
behavior, in that we combine them into a single entity and then merge
the discovered distributions into each Simple API-style index. The
motivation here, IIRC, was to match pip's behavior quite closely. I'm
interested in _removing_ that behavior, but it'd be breaking (and may
also be inconvenient for some use-cases). So, the behavior for indexes
passed in via `--find-links` remains completely unchanged. However,
`[[tool.uv.index]]` entries with `format = "flat"` are now treated
identically to those defined with `format = "simple"` (the default), in
that we stop after we find the first-matching index, etc.

Closes https://github.com/astral-sh/uv/issues/11634.
This commit is contained in:
Charlie Marsh 2025-03-25 21:14:44 -04:00 committed by GitHub
parent f2a2d982b8
commit bd9c365b92
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
19 changed files with 826 additions and 122 deletions

View file

@ -4,7 +4,7 @@ use std::collections::BTreeMap;
use rustc_hash::FxHashMap;
use tracing::instrument;
use uv_client::FlatIndexEntries;
use uv_client::{FlatIndexEntries, FlatIndexEntry};
use uv_configuration::BuildOptions;
use uv_distribution_filename::{DistFilename, SourceDistFilename, WheelFilename};
use uv_distribution_types::{
@ -39,11 +39,10 @@ impl FlatIndex {
build_options: &BuildOptions,
) -> Self {
// Collect compatible distributions.
let mut index = FxHashMap::default();
let mut index = FxHashMap::<PackageName, FlatDistributions>::default();
for entry in entries.entries {
let distributions = index.entry(entry.filename.name().clone()).or_default();
Self::add_file(
distributions,
distributions.add_file(
entry.file,
entry.filename,
tags,
@ -59,8 +58,59 @@ impl FlatIndex {
Self { index, offline }
}
/// Get the [`FlatDistributions`] for the given package name.
pub fn get(&self, package_name: &PackageName) -> Option<&FlatDistributions> {
self.index.get(package_name)
}
/// Whether any `--find-links` entries could not be resolved due to a lack of network
/// connectivity.
pub fn offline(&self) -> bool {
self.offline
}
}
/// A set of [`PrioritizedDist`] from a `--find-links` entry for a single package, indexed
/// by [`Version`].
#[derive(Debug, Clone, Default)]
pub struct FlatDistributions(BTreeMap<Version, PrioritizedDist>);
impl FlatDistributions {
/// Collect all files from a `--find-links` target into a [`FlatIndex`].
#[instrument(skip_all)]
pub fn from_entries(
entries: Vec<FlatIndexEntry>,
tags: Option<&Tags>,
hasher: &HashStrategy,
build_options: &BuildOptions,
) -> Self {
let mut distributions = Self::default();
for entry in entries {
distributions.add_file(
entry.file,
entry.filename,
tags,
hasher,
build_options,
entry.index,
);
}
distributions
}
/// Returns an [`Iterator`] over the distributions.
pub fn iter(&self) -> impl Iterator<Item = (&Version, &PrioritizedDist)> {
self.0.iter()
}
/// Removes the [`PrioritizedDist`] for the given version.
pub fn remove(&mut self, version: &Version) -> Option<PrioritizedDist> {
self.0.remove(version)
}
/// Add the given [`File`] to the [`FlatDistributions`] for the given package.
fn add_file(
distributions: &mut FlatDistributions,
&mut self,
file: File,
filename: DistFilename,
tags: Option<&Tags>,
@ -86,7 +136,7 @@ impl FlatIndex {
file: Box::new(file),
index,
};
match distributions.0.entry(version) {
match self.0.entry(version) {
Entry::Occupied(mut entry) => {
entry.get_mut().insert_built(dist, vec![], compatibility);
}
@ -110,7 +160,7 @@ impl FlatIndex {
index,
wheels: vec![],
};
match distributions.0.entry(filename.version) {
match self.0.entry(filename.version) {
Entry::Occupied(mut entry) => {
entry.get_mut().insert_source(dist, vec![], compatibility);
}
@ -194,31 +244,6 @@ impl FlatIndex {
WheelCompatibility::Compatible(hash, priority, build_tag)
}
/// Get the [`FlatDistributions`] for the given package name.
pub fn get(&self, package_name: &PackageName) -> Option<&FlatDistributions> {
self.index.get(package_name)
}
/// Returns `true` if there are any offline `--find-links` entries.
pub fn offline(&self) -> bool {
self.offline
}
}
/// A set of [`PrioritizedDist`] from a `--find-links` entry for a single package, indexed
/// by [`Version`].
#[derive(Debug, Clone, Default)]
pub struct FlatDistributions(BTreeMap<Version, PrioritizedDist>);
impl FlatDistributions {
pub fn iter(&self) -> impl Iterator<Item = (&Version, &PrioritizedDist)> {
self.0.iter()
}
pub fn remove(&mut self, version: &Version) -> Option<PrioritizedDist> {
self.0.remove(version)
}
}
impl IntoIterator for FlatDistributions {

View file

@ -45,9 +45,7 @@ impl Indexes {
else {
continue;
};
let index = IndexMetadata {
url: index.url.clone(),
};
let index = index.clone();
let conflict = conflict.clone();
indexes.add(&requirement, Entry { index, conflict });
}

View file

@ -1,6 +1,6 @@
use std::future::Future;
use std::sync::Arc;
use uv_client::MetadataFormat;
use uv_configuration::BuildOptions;
use uv_distribution::{ArchiveMetadata, DistributionDatabase, Reporter};
use uv_distribution_types::{
@ -158,7 +158,7 @@ impl<Context: BuildContext> ResolverProvider for DefaultResolverProvider<'_, Con
.fetcher
.client()
.manual(|client, semaphore| {
client.simple(
client.package_metadata(
package_name,
index.map(IndexMetadataRef::from),
self.capabilities,
@ -174,11 +174,11 @@ impl<Context: BuildContext> ResolverProvider for DefaultResolverProvider<'_, Con
Ok(results) => Ok(VersionsResponse::Found(
results
.into_iter()
.map(|(index, metadata)| {
VersionMap::from_metadata(
.map(|(index, metadata)| match metadata {
MetadataFormat::Simple(metadata) => VersionMap::from_simple_metadata(
metadata,
package_name,
index.url(),
index,
self.tags.as_ref(),
&self.requires_python,
&self.allowed_yanks,
@ -188,7 +188,13 @@ impl<Context: BuildContext> ResolverProvider for DefaultResolverProvider<'_, Con
.and_then(|flat_index| flat_index.get(package_name))
.cloned(),
self.build_options,
)
),
MetadataFormat::Flat(metadata) => VersionMap::from_flat_metadata(
metadata,
self.tags.as_ref(),
&self.hasher,
self.build_options,
),
})
.collect(),
)),

View file

@ -6,7 +6,7 @@ use std::sync::OnceLock;
use pubgrub::Ranges;
use tracing::instrument;
use uv_client::{OwnedArchive, SimpleMetadata, VersionFiles};
use uv_client::{FlatIndexEntry, OwnedArchive, SimpleMetadata, VersionFiles};
use uv_configuration::BuildOptions;
use uv_distribution_filename::{DistFilename, WheelFilename};
use uv_distribution_types::{
@ -41,7 +41,7 @@ impl VersionMap {
///
/// PEP 592: <https://peps.python.org/pep-0592/#warehouse-pypi-implementation-notes>
#[instrument(skip_all, fields(package_name))]
pub(crate) fn from_metadata(
pub(crate) fn from_simple_metadata(
simple_metadata: OwnedArchive<SimpleMetadata>,
package_name: &PackageName,
index: &IndexUrl,
@ -116,6 +116,30 @@ impl VersionMap {
}
}
#[instrument(skip_all, fields(package_name))]
pub(crate) fn from_flat_metadata(
flat_metadata: Vec<FlatIndexEntry>,
tags: Option<&Tags>,
hasher: &HashStrategy,
build_options: &BuildOptions,
) -> Self {
let mut stable = false;
let mut local = false;
let mut map = BTreeMap::new();
for (version, prioritized_dist) in
FlatDistributions::from_entries(flat_metadata, tags, hasher, build_options)
{
stable |= version.is_stable();
local |= version.is_local();
map.insert(version, prioritized_dist);
}
Self {
inner: VersionMapInner::Eager(VersionMapEager { map, stable, local }),
}
}
/// Return the [`DistFile`] for the given version, if any.
pub(crate) fn get(&self, version: &Version) -> Option<&PrioritizedDist> {
match self.inner {