Add S3 request signing (#15925)
Some checks are pending
CI / integration test | uv publish (push) Blocked by required conditions
CI / check system | python on ubuntu (push) Blocked by required conditions
CI / cargo dev generate-all (push) Blocked by required conditions
CI / cargo shear (push) Waiting to run
CI / typos (push) Waiting to run
CI / check system | windows registry (push) Blocked by required conditions
CI / lint (push) Waiting to run
CI / cargo clippy | ubuntu (push) Blocked by required conditions
CI / cargo test | ubuntu (push) Blocked by required conditions
CI / cargo test | macos (push) Blocked by required conditions
CI / cargo test | windows (push) Blocked by required conditions
CI / Determine changes (push) Waiting to run
CI / cargo clippy | windows (push) Blocked by required conditions
CI / check windows trampoline | aarch64 (push) Blocked by required conditions
CI / check windows trampoline | i686 (push) Blocked by required conditions
CI / check windows trampoline | x86_64 (push) Blocked by required conditions
CI / test windows trampoline | aarch64 (push) Blocked by required conditions
CI / test windows trampoline | i686 (push) Blocked by required conditions
CI / test windows trampoline | x86_64 (push) Blocked by required conditions
CI / mkdocs (push) Waiting to run
CI / build binary | linux libc (push) Blocked by required conditions
CI / build binary | linux aarch64 (push) Blocked by required conditions
CI / build binary | linux musl (push) Blocked by required conditions
CI / build binary | freebsd (push) Blocked by required conditions
CI / build binary | macos x86_64 (push) Blocked by required conditions
CI / build binary | windows aarch64 (push) Blocked by required conditions
CI / ecosystem test | pydantic/pydantic-core (push) Blocked by required conditions
CI / ecosystem test | prefecthq/prefect (push) Blocked by required conditions
CI / smoke test | linux (push) Blocked by required conditions
CI / build binary | macos aarch64 (push) Blocked by required conditions
CI / build binary | windows x86_64 (push) Blocked by required conditions
CI / build binary | msrv (push) Blocked by required conditions
CI / ecosystem test | pallets/flask (push) Blocked by required conditions
CI / check system | python3.12 via chocolatey (push) Blocked by required conditions
CI / smoke test | linux aarch64 (push) Blocked by required conditions
CI / check system | alpine (push) Blocked by required conditions
CI / smoke test | macos (push) Blocked by required conditions
CI / smoke test | windows x86_64 (push) Blocked by required conditions
CI / smoke test | windows aarch64 (push) Blocked by required conditions
CI / integration test | activate nushell venv (push) Blocked by required conditions
CI / integration test | conda on ubuntu (push) Blocked by required conditions
CI / integration test | deadsnakes python3.9 on ubuntu (push) Blocked by required conditions
CI / integration test | free-threaded on windows (push) Blocked by required conditions
CI / integration test | graalpy on ubuntu (push) Blocked by required conditions
CI / integration test | aarch64 windows implicit (push) Blocked by required conditions
CI / integration test | aarch64 windows explicit (push) Blocked by required conditions
CI / integration test | pypy on ubuntu (push) Blocked by required conditions
CI / integration test | pypy on windows (push) Blocked by required conditions
CI / integration test | graalpy on windows (push) Blocked by required conditions
CI / integration test | pyodide on ubuntu (push) Blocked by required conditions
CI / integration test | github actions (push) Blocked by required conditions
CI / integration test | free-threaded python on github actions (push) Blocked by required conditions
CI / integration test | pyenv on wsl x86-64 (push) Blocked by required conditions
CI / integration test | determine publish changes (push) Blocked by required conditions
CI / integration test | registries (push) Blocked by required conditions
CI / integration test | uv_build (push) Blocked by required conditions
CI / check cache | ubuntu (push) Blocked by required conditions
CI / check cache | macos aarch64 (push) Blocked by required conditions
CI / check system | python on debian (push) Blocked by required conditions
CI / check system | python on fedora (push) Blocked by required conditions
CI / check system | python on rocky linux 8 (push) Blocked by required conditions
CI / check system | python on rocky linux 9 (push) Blocked by required conditions
CI / check system | graalpy on ubuntu (push) Blocked by required conditions
CI / check system | pypy on ubuntu (push) Blocked by required conditions
CI / check system | pyston (push) Blocked by required conditions
CI / check system | python on macos aarch64 (push) Blocked by required conditions
CI / check system | homebrew python on macos aarch64 (push) Blocked by required conditions
CI / check system | x86-64 python on macos aarch64 (push) Blocked by required conditions
CI / check system | python on macos x86-64 (push) Blocked by required conditions
CI / check system | python3.10 on windows x86-64 (push) Blocked by required conditions
CI / check system | python3.10 on windows x86 (push) Blocked by required conditions
CI / check system | python3.13 on windows x86-64 (push) Blocked by required conditions
CI / check system | x86-64 python3.13 on windows aarch64 (push) Blocked by required conditions
CI / check system | aarch64 python3.13 on windows aarch64 (push) Blocked by required conditions
CI / check system | python3.9 via pyenv (push) Blocked by required conditions
CI / check system | python3.13 (push) Blocked by required conditions
CI / check system | conda3.11 on macos aarch64 (push) Blocked by required conditions
CI / check system | conda3.8 on macos aarch64 (push) Blocked by required conditions
CI / check system | conda3.11 on linux x86-64 (push) Blocked by required conditions
CI / check system | conda3.8 on linux x86-64 (push) Blocked by required conditions
CI / check system | conda3.11 on windows x86-64 (push) Blocked by required conditions
CI / check system | conda3.8 on windows x86-64 (push) Blocked by required conditions
CI / check system | amazonlinux (push) Blocked by required conditions
CI / benchmarks | walltime aarch64 linux (push) Blocked by required conditions
CI / check system | embedded python3.10 on windows x86-64 (push) Blocked by required conditions
CI / benchmarks | instrumented (push) Blocked by required conditions
zizmor / Run zizmor (push) Waiting to run

## Summary

This PR enables users to mark a URL as an S3 endpoint, at which point uv
will sign requests to that URL by detecting credentials from the
standard AWS environment variables, configuration files, etc.

Signing is handled by the
[reqsign](https://docs.rs/reqsign/latest/reqsign/) crate, which we can
also use in the future to sign requests for other providers.
This commit is contained in:
Charlie Marsh 2025-09-22 19:59:52 -04:00 committed by GitHub
parent 3e6fd0b775
commit 7f7fac812c
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
17 changed files with 689 additions and 96 deletions

231
Cargo.lock generated
View file

@ -57,6 +57,15 @@ dependencies = [
"thiserror 2.0.16", "thiserror 2.0.16",
] ]
[[package]]
name = "android_system_properties"
version = "0.1.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "819e7219dbd41043ac279b19830f2efc897156490d7fd6ea916720117ee66311"
dependencies = [
"libc",
]
[[package]] [[package]]
name = "anes" name = "anes"
version = "0.1.6" version = "0.1.6"
@ -644,6 +653,19 @@ dependencies = [
"encoding_rs", "encoding_rs",
] ]
[[package]]
name = "chrono"
version = "0.4.42"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "145052bdd345b87320e369255277e3fb5152762ad123a901ef5c262dd38fe8d2"
dependencies = [
"iana-time-zone",
"js-sys",
"num-traits",
"wasm-bindgen",
"windows-link 0.2.0",
]
[[package]] [[package]]
name = "ciborium" name = "ciborium"
version = "0.2.2" version = "0.2.2"
@ -868,6 +890,32 @@ dependencies = [
"windows-sys 0.61.0", "windows-sys 0.61.0",
] ]
[[package]]
name = "const-oid"
version = "0.9.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c2459377285ad874054d797f3ccebf984978aa39129f6eafde5cdc8315b612f8"
[[package]]
name = "const-random"
version = "0.1.18"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "87e00182fe74b066627d63b85fd550ac2998d4b0bd86bfed477a0ae4c7c71359"
dependencies = [
"const-random-macro",
]
[[package]]
name = "const-random-macro"
version = "0.1.16"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f9d839f2a20b0aee515dc581a6172f2321f96cab76c1a38a4c584a194955390e"
dependencies = [
"getrandom 0.2.16",
"once_cell",
"tiny-keccak",
]
[[package]] [[package]]
name = "core-foundation" name = "core-foundation"
version = "0.9.4" version = "0.9.4"
@ -1116,6 +1164,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9ed9a281f7bc9b7576e61468ba615a66a5c8cfdff42420a70aa82701a3b1e292" checksum = "9ed9a281f7bc9b7576e61468ba615a66a5c8cfdff42420a70aa82701a3b1e292"
dependencies = [ dependencies = [
"block-buffer", "block-buffer",
"const-oid",
"crypto-common", "crypto-common",
"subtle", "subtle",
] ]
@ -1158,6 +1207,15 @@ dependencies = [
"syn", "syn",
] ]
[[package]]
name = "dlv-list"
version = "0.5.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "442039f5147480ba31067cb00ada1adae6892028e40e45fc5de7b7df6dcc1b5f"
dependencies = [
"const-random",
]
[[package]] [[package]]
name = "doc-comment" name = "doc-comment"
version = "0.3.3" version = "0.3.3"
@ -1903,6 +1961,30 @@ dependencies = [
"windows-registry", "windows-registry",
] ]
[[package]]
name = "iana-time-zone"
version = "0.1.64"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "33e57f83510bb73707521ebaffa789ec8caf86f9657cad665b092b581d40e9fb"
dependencies = [
"android_system_properties",
"core-foundation-sys",
"iana-time-zone-haiku",
"js-sys",
"log",
"wasm-bindgen",
"windows-core 0.61.2",
]
[[package]]
name = "iana-time-zone-haiku"
version = "0.1.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f31827a206f56af32e590ba56d5d2d085f558508192593743f16b2306495269f"
dependencies = [
"cc",
]
[[package]] [[package]]
name = "icu_collections" name = "icu_collections"
version = "2.0.0" version = "2.0.0"
@ -2755,6 +2837,16 @@ version = "0.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "04744f49eae99ab78e0d5c0b603ab218f515ea8cfe5a456d7629ad883a3b6e7d" checksum = "04744f49eae99ab78e0d5c0b603ab218f515ea8cfe5a456d7629ad883a3b6e7d"
[[package]]
name = "ordered-multimap"
version = "0.7.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "49203cdcae0030493bad186b28da2fa25645fa276a51b6fec8010d281e02ef79"
dependencies = [
"dlv-list",
"hashbrown 0.14.5",
]
[[package]] [[package]]
name = "ordered-stream" name = "ordered-stream"
version = "0.2.0" version = "0.2.0"
@ -3119,6 +3211,16 @@ dependencies = [
"version-ranges", "version-ranges",
] ]
[[package]]
name = "quick-xml"
version = "0.38.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "42a232e7487fc2ef313d96dde7948e7a3c05101870d8985e4fd8d26aedd27b89"
dependencies = [
"memchr",
"serde",
]
[[package]] [[package]]
name = "quinn" name = "quinn"
version = "0.11.8" version = "0.11.8"
@ -3379,6 +3481,104 @@ dependencies = [
"bytecheck", "bytecheck",
] ]
[[package]]
name = "reqsign"
version = "0.17.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "be18806fe2251c9924d875549573c9bf0e43b51d7efcf32a19ec31bb32196987"
dependencies = [
"reqsign-aws-v4",
"reqsign-command-execute-tokio",
"reqsign-core",
"reqsign-file-read-tokio",
"reqsign-http-send-reqwest",
]
[[package]]
name = "reqsign-aws-v4"
version = "1.0.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9b0fb0ac0a8222efdb0746d83c5ec36c6bdb0f5310b4b92147e3de7c45ef6657"
dependencies = [
"anyhow",
"async-trait",
"bytes",
"chrono",
"form_urlencoded",
"http",
"log",
"percent-encoding",
"quick-xml",
"reqsign-core",
"reqwest",
"rust-ini",
"serde",
"serde_json",
"serde_urlencoded",
"sha1",
]
[[package]]
name = "reqsign-command-execute-tokio"
version = "1.0.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8e145d2d409f3db11aa3094b8905d69b084d4060771c12e6522dde55924bfecd"
dependencies = [
"async-trait",
"reqsign-core",
"tokio",
]
[[package]]
name = "reqsign-core"
version = "1.0.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "35fd7359352a79b293168785b9cbb239e7017b4fa0c934351518feee394f92bc"
dependencies = [
"anyhow",
"async-trait",
"base64 0.22.1",
"bytes",
"chrono",
"form_urlencoded",
"hex",
"hmac",
"http",
"log",
"percent-encoding",
"sha1",
"sha2",
"thiserror 2.0.16",
"windows-sys 0.60.2",
]
[[package]]
name = "reqsign-file-read-tokio"
version = "1.0.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "857154204885ec746f5f221393aec7dc47cbde9216b18774eb0a0c6e966f1ee0"
dependencies = [
"anyhow",
"async-trait",
"reqsign-core",
"tokio",
]
[[package]]
name = "reqsign-http-send-reqwest"
version = "1.0.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f388d66e254d67e570eba0a4a3fe39427ff7f1df26c73a8b7330e8ca1e639f80"
dependencies = [
"anyhow",
"async-trait",
"bytes",
"http",
"http-body-util",
"reqsign-core",
"reqwest",
]
[[package]] [[package]]
name = "reqwest" name = "reqwest"
version = "0.12.22" version = "0.12.22"
@ -3591,6 +3791,16 @@ version = "0.20.0"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6c20b6793b5c2fa6553b250154b78d6d0db37e72700ae35fad9387a46f487c97" checksum = "6c20b6793b5c2fa6553b250154b78d6d0db37e72700ae35fad9387a46f487c97"
[[package]]
name = "rust-ini"
version = "0.21.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "796e8d2b6696392a43bea58116b667fb4c29727dc5abd27d6acf338bb4f688c7"
dependencies = [
"cfg-if",
"ordered-multimap",
]
[[package]] [[package]]
name = "rust-netrc" name = "rust-netrc"
version = "0.1.2" version = "0.1.2"
@ -3969,6 +4179,17 @@ dependencies = [
"unsafe-libyaml", "unsafe-libyaml",
] ]
[[package]]
name = "sha1"
version = "0.10.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e3bf829a2d51ab4a5ddf1352d8470c140cadc8301b2ae1789db023f01cedd6ba"
dependencies = [
"cfg-if",
"cpufeatures",
"digest",
]
[[package]] [[package]]
name = "sha2" name = "sha2"
version = "0.10.9" version = "0.10.9"
@ -4458,6 +4679,15 @@ dependencies = [
"tikv-jemalloc-sys", "tikv-jemalloc-sys",
] ]
[[package]]
name = "tiny-keccak"
version = "2.0.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2c9d3793400a45f954c52e73d068316d76b6f4e36977e3fcebb13a2721e80237"
dependencies = [
"crunchy",
]
[[package]] [[package]]
name = "tiny-skia" name = "tiny-skia"
version = "0.8.4" version = "0.8.4"
@ -5177,6 +5407,7 @@ dependencies = [
"insta", "insta",
"jiff", "jiff",
"percent-encoding", "percent-encoding",
"reqsign",
"reqwest", "reqwest",
"reqwest-middleware", "reqwest-middleware",
"rust-netrc", "rust-netrc",

View file

@ -151,6 +151,7 @@ ref-cast = { version = "1.0.24" }
reflink-copy = { version = "0.1.19" } reflink-copy = { version = "0.1.19" }
regex = { version = "1.10.6" } regex = { version = "1.10.6" }
regex-automata = { version = "0.4.8", default-features = false, features = ["dfa-build", "dfa-search", "perf", "std", "syntax"] } regex-automata = { version = "0.4.8", default-features = false, features = ["dfa-build", "dfa-search", "perf", "std", "syntax"] }
reqsign = { version = "0.17.0", features = ["aws", "default-context"], default-features = false }
reqwest = { version = "0.12.22", default-features = false, features = ["json", "gzip", "deflate", "zstd", "stream", "system-proxy", "rustls-tls", "rustls-tls-native-roots", "socks", "multipart", "http2", "blocking"] } reqwest = { version = "0.12.22", default-features = false, features = ["json", "gzip", "deflate", "zstd", "stream", "system-proxy", "rustls-tls", "rustls-tls-native-roots", "socks", "multipart", "http2", "blocking"] }
reqwest-middleware = { git = "https://github.com/astral-sh/reqwest-middleware", rev = "7650ed76215a962a96d94a79be71c27bffde7ab2", features = ["multipart"] } reqwest-middleware = { git = "https://github.com/astral-sh/reqwest-middleware", rev = "7650ed76215a962a96d94a79be71c27bffde7ab2", features = ["multipart"] }
reqwest-retry = { git = "https://github.com/astral-sh/reqwest-middleware", rev = "7650ed76215a962a96d94a79be71c27bffde7ab2" } reqwest-retry = { git = "https://github.com/astral-sh/reqwest-middleware", rev = "7650ed76215a962a96d94a79be71c27bffde7ab2" }

View file

@ -31,6 +31,7 @@ futures = { workspace = true }
http = { workspace = true } http = { workspace = true }
jiff = { workspace = true } jiff = { workspace = true }
percent-encoding = { workspace = true } percent-encoding = { workspace = true }
reqsign = { workspace = true }
reqwest = { workspace = true } reqwest = { workspace = true }
reqwest-middleware = { workspace = true } reqwest-middleware = { workspace = true }
rust-netrc = { workspace = true } rust-netrc = { workspace = true }

View file

@ -12,7 +12,7 @@ use uv_once_map::OnceMap;
use uv_redacted::DisplaySafeUrl; use uv_redacted::DisplaySafeUrl;
use crate::Realm; use crate::Realm;
use crate::credentials::{Credentials, Username}; use crate::credentials::{Authentication, Username};
type FxOnceMap<K, V> = OnceMap<K, V, BuildHasherDefault<FxHasher>>; type FxOnceMap<K, V> = OnceMap<K, V, BuildHasherDefault<FxHasher>>;
@ -35,11 +35,11 @@ impl Display for FetchUrl {
pub struct CredentialsCache { pub struct CredentialsCache {
/// A cache per realm and username /// A cache per realm and username
realms: RwLock<FxHashMap<(Realm, Username), Arc<Credentials>>>, realms: RwLock<FxHashMap<(Realm, Username), Arc<Authentication>>>,
/// A cache tracking the result of realm or index URL fetches from external services /// A cache tracking the result of realm or index URL fetches from external services
pub(crate) fetches: FxOnceMap<(FetchUrl, Username), Option<Arc<Credentials>>>, pub(crate) fetches: FxOnceMap<(FetchUrl, Username), Option<Arc<Authentication>>>,
/// A cache per URL, uses a trie for efficient prefix queries. /// A cache per URL, uses a trie for efficient prefix queries.
urls: RwLock<UrlTrie>, urls: RwLock<UrlTrie<Arc<Authentication>>>,
} }
impl Default for CredentialsCache { impl Default for CredentialsCache {
@ -59,7 +59,11 @@ impl CredentialsCache {
} }
/// Return the credentials that should be used for a realm and username, if any. /// Return the credentials that should be used for a realm and username, if any.
pub(crate) fn get_realm(&self, realm: Realm, username: Username) -> Option<Arc<Credentials>> { pub(crate) fn get_realm(
&self,
realm: Realm,
username: Username,
) -> Option<Arc<Authentication>> {
let realms = self.realms.read().unwrap(); let realms = self.realms.read().unwrap();
let given_username = username.is_some(); let given_username = username.is_some();
let key = (realm, username); let key = (realm, username);
@ -93,7 +97,7 @@ impl CredentialsCache {
/// Note we do not cache per username, but if a username is passed we will confirm that the /// Note we do not cache per username, but if a username is passed we will confirm that the
/// cached credentials have a username equal to the provided one — otherwise `None` is returned. /// cached credentials have a username equal to the provided one — otherwise `None` is returned.
/// If multiple usernames are used per URL, the realm cache should be queried instead. /// If multiple usernames are used per URL, the realm cache should be queried instead.
pub(crate) fn get_url(&self, url: &Url, username: &Username) -> Option<Arc<Credentials>> { pub(crate) fn get_url(&self, url: &Url, username: &Username) -> Option<Arc<Authentication>> {
let urls = self.urls.read().unwrap(); let urls = self.urls.read().unwrap();
let credentials = urls.get(url); let credentials = urls.get(url);
if let Some(credentials) = credentials { if let Some(credentials) = credentials {
@ -112,7 +116,7 @@ impl CredentialsCache {
} }
/// Update the cache with the given credentials. /// Update the cache with the given credentials.
pub(crate) fn insert(&self, url: &Url, credentials: Arc<Credentials>) { pub(crate) fn insert(&self, url: &Url, credentials: Arc<Authentication>) {
// Do not cache empty credentials // Do not cache empty credentials
if credentials.is_empty() { if credentials.is_empty() {
return; return;
@ -139,8 +143,8 @@ impl CredentialsCache {
fn insert_realm( fn insert_realm(
&self, &self,
key: (Realm, Username), key: (Realm, Username),
credentials: &Arc<Credentials>, credentials: &Arc<Authentication>,
) -> Option<Arc<Credentials>> { ) -> Option<Arc<Authentication>> {
// Do not cache empty credentials // Do not cache empty credentials
if credentials.is_empty() { if credentials.is_empty() {
return None; return None;
@ -166,24 +170,33 @@ impl CredentialsCache {
} }
#[derive(Debug)] #[derive(Debug)]
struct UrlTrie { struct UrlTrie<T> {
states: Vec<TrieState>, states: Vec<TrieState<T>>,
} }
#[derive(Debug, Default)] #[derive(Debug)]
struct TrieState { struct TrieState<T> {
children: Vec<(String, usize)>, children: Vec<(String, usize)>,
value: Option<Arc<Credentials>>, value: Option<T>,
} }
impl UrlTrie { impl<T> Default for TrieState<T> {
fn default() -> Self {
Self {
children: vec![],
value: None,
}
}
}
impl<T> UrlTrie<T> {
fn new() -> Self { fn new() -> Self {
let mut trie = Self { states: vec![] }; let mut trie = Self { states: vec![] };
trie.alloc(); trie.alloc();
trie trie
} }
fn get(&self, url: &Url) -> Option<&Arc<Credentials>> { fn get(&self, url: &Url) -> Option<&T> {
let mut state = 0; let mut state = 0;
let realm = Realm::from(url).to_string(); let realm = Realm::from(url).to_string();
for component in [realm.as_str()] for component in [realm.as_str()]
@ -198,7 +211,7 @@ impl UrlTrie {
self.states[state].value.as_ref() self.states[state].value.as_ref()
} }
fn insert(&mut self, url: &Url, value: Arc<Credentials>) { fn insert(&mut self, url: &Url, value: T) {
let mut state = 0; let mut state = 0;
let realm = Realm::from(url).to_string(); let realm = Realm::from(url).to_string();
for component in [realm.as_str()] for component in [realm.as_str()]
@ -226,7 +239,7 @@ impl UrlTrie {
} }
} }
impl TrieState { impl<T> TrieState<T> {
fn get(&self, component: &str) -> Option<usize> { fn get(&self, component: &str) -> Option<usize> {
let i = self.index(component).ok()?; let i = self.index(component).ok()?;
Some(self.children[i].1) Some(self.children[i].1)
@ -260,28 +273,21 @@ impl From<(Realm, Username)> for RealmUsername {
#[cfg(test)] #[cfg(test)]
mod tests { mod tests {
use crate::Credentials;
use crate::credentials::Password; use crate::credentials::Password;
use super::*; use super::*;
#[test] #[test]
fn test_trie() { fn test_trie() {
let credentials1 = Arc::new(Credentials::basic( let credentials1 =
Some("username1".to_string()), Credentials::basic(Some("username1".to_string()), Some("password1".to_string()));
Some("password1".to_string()), let credentials2 =
)); Credentials::basic(Some("username2".to_string()), Some("password2".to_string()));
let credentials2 = Arc::new(Credentials::basic( let credentials3 =
Some("username2".to_string()), Credentials::basic(Some("username3".to_string()), Some("password3".to_string()));
Some("password2".to_string()), let credentials4 =
)); Credentials::basic(Some("username4".to_string()), Some("password4".to_string()));
let credentials3 = Arc::new(Credentials::basic(
Some("username3".to_string()),
Some("password3".to_string()),
));
let credentials4 = Arc::new(Credentials::basic(
Some("username4".to_string()),
Some("password4".to_string()),
));
let mut trie = UrlTrie::new(); let mut trie = UrlTrie::new();
trie.insert( trie.insert(
@ -339,10 +345,10 @@ mod tests {
fn test_url_with_credentials() { fn test_url_with_credentials() {
let username = Username::new(Some(String::from("username"))); let username = Username::new(Some(String::from("username")));
let password = Password::new(String::from("password")); let password = Password::new(String::from("password"));
let credentials = Arc::new(Credentials::Basic { let credentials = Arc::new(Authentication::from(Credentials::Basic {
username: username.clone(), username: username.clone(),
password: Some(password), password: Some(password),
}); }));
let cache = CredentialsCache::default(); let cache = CredentialsCache::default();
// Insert with URL with credentials and get with redacted URL. // Insert with URL with credentials and get with redacted URL.
let url = Url::parse("https://username:password@example.com/foobar").unwrap(); let url = Url::parse("https://username:password@example.com/foobar").unwrap();

View file

@ -1,21 +1,24 @@
use std::borrow::Cow;
use std::fmt;
use std::io::Read;
use std::io::Write;
use std::str::FromStr;
use base64::prelude::BASE64_STANDARD; use base64::prelude::BASE64_STANDARD;
use base64::read::DecoderReader; use base64::read::DecoderReader;
use base64::write::EncoderWriter; use base64::write::EncoderWriter;
use serde::{Deserialize, Serialize}; use http::Uri;
use std::borrow::Cow;
use std::fmt;
use uv_redacted::DisplaySafeUrl;
use netrc::Netrc; use netrc::Netrc;
use reqsign::aws::DefaultSigner;
use reqwest::Request; use reqwest::Request;
use reqwest::header::HeaderValue; use reqwest::header::HeaderValue;
use std::io::Read; use serde::{Deserialize, Serialize};
use std::io::Write;
use url::Url; use url::Url;
use uv_redacted::DisplaySafeUrl;
use uv_static::EnvVars; use uv_static::EnvVars;
#[derive(Clone, Debug, PartialEq)] #[derive(Clone, Debug, PartialEq, Eq)]
pub enum Credentials { pub enum Credentials {
Basic { Basic {
/// The username to use for authentication. /// The username to use for authentication.
@ -345,6 +348,127 @@ impl Credentials {
} }
} }
#[derive(Clone, Debug)]
pub(crate) enum Authentication {
/// HTTP Basic or Bearer Authentication credentials.
Credentials(Credentials),
/// AWS Signature Version 4 signing.
Signer(DefaultSigner),
}
impl PartialEq for Authentication {
fn eq(&self, other: &Self) -> bool {
match (self, other) {
(Self::Credentials(a), Self::Credentials(b)) => a == b,
(Self::Signer(..), Self::Signer(..)) => true,
_ => false,
}
}
}
impl Eq for Authentication {}
impl From<Credentials> for Authentication {
fn from(credentials: Credentials) -> Self {
Self::Credentials(credentials)
}
}
impl From<DefaultSigner> for Authentication {
fn from(signer: DefaultSigner) -> Self {
Self::Signer(signer)
}
}
impl Authentication {
/// Return the password used for authentication, if any.
pub(crate) fn password(&self) -> Option<&str> {
match self {
Self::Credentials(credentials) => credentials.password(),
Self::Signer(..) => None,
}
}
/// Return the username used for authentication, if any.
pub(crate) fn username(&self) -> Option<&str> {
match self {
Self::Credentials(credentials) => credentials.username(),
Self::Signer(..) => None,
}
}
/// Return the username used for authentication, if any.
pub(crate) fn as_username(&self) -> Cow<'_, Username> {
match self {
Self::Credentials(credentials) => credentials.as_username(),
Self::Signer(..) => Cow::Owned(Username::none()),
}
}
/// Return the username used for authentication, if any.
pub(crate) fn to_username(&self) -> Username {
match self {
Self::Credentials(credentials) => credentials.to_username(),
Self::Signer(..) => Username::none(),
}
}
/// Return `true` if the object contains a means of authenticating.
pub(crate) fn is_authenticated(&self) -> bool {
match self {
Self::Credentials(credentials) => credentials.is_authenticated(),
Self::Signer(..) => true,
}
}
/// Return `true` if the object contains no credentials.
pub(crate) fn is_empty(&self) -> bool {
match self {
Self::Credentials(credentials) => credentials.is_empty(),
Self::Signer(..) => false,
}
}
/// Apply the authentication to the given request.
///
/// Any existing credentials will be overridden.
#[must_use]
pub(crate) async fn authenticate(&self, mut request: Request) -> Request {
match self {
Self::Credentials(credentials) => credentials.authenticate(request),
Self::Signer(signer) => {
// Build an `http::Request` from the `reqwest::Request`.
// SAFETY: If we have a valid `reqwest::Request`, we expect (e.g.) the URL to be valid.
let uri = Uri::from_str(request.url().as_str()).unwrap();
let mut http_req = http::Request::builder()
.method(request.method().clone())
.uri(uri)
.body(())
.unwrap();
*http_req.headers_mut() = request.headers().clone();
// Sign the parts.
let (mut parts, ()) = http_req.into_parts();
signer
.sign(&mut parts, None)
.await
.expect("AWS signing should succeed");
// Copy over the signed headers.
request.headers_mut().extend(parts.headers);
// Copy over the signed path and query, if any.
if let Some(path_and_query) = parts.uri.path_and_query() {
request.url_mut().set_path(path_and_query.path());
request.url_mut().set_query(path_and_query.query());
}
request
}
}
}
}
#[cfg(test)] #[cfg(test)]
mod tests { mod tests {
use insta::assert_debug_snapshot; use insta::assert_debug_snapshot;

View file

@ -4,6 +4,7 @@ use tracing::trace;
use uv_redacted::DisplaySafeUrl; use uv_redacted::DisplaySafeUrl;
use crate::credentials::Authentication;
pub use access_token::AccessToken; pub use access_token::AccessToken;
use cache::CredentialsCache; use cache::CredentialsCache;
pub use credentials::{Credentials, Username}; pub use credentials::{Credentials, Username};
@ -43,7 +44,7 @@ pub(crate) static CREDENTIALS_CACHE: LazyLock<CredentialsCache> =
pub fn store_credentials_from_url(url: &DisplaySafeUrl) -> bool { pub fn store_credentials_from_url(url: &DisplaySafeUrl) -> bool {
if let Some(credentials) = Credentials::from_url(url) { if let Some(credentials) = Credentials::from_url(url) {
trace!("Caching credentials for {url}"); trace!("Caching credentials for {url}");
CREDENTIALS_CACHE.insert(url, Arc::new(credentials)); CREDENTIALS_CACHE.insert(url, Arc::new(Authentication::from(credentials)));
true true
} else { } else {
false false
@ -53,7 +54,7 @@ pub fn store_credentials_from_url(url: &DisplaySafeUrl) -> bool {
/// Populate the global authentication store with credentials on a URL, if there are any. /// Populate the global authentication store with credentials on a URL, if there are any.
/// ///
/// Returns `true` if the store was updated. /// Returns `true` if the store was updated.
pub fn store_credentials(url: &DisplaySafeUrl, credentials: Arc<Credentials>) { pub fn store_credentials(url: &DisplaySafeUrl, credentials: Credentials) {
trace!("Caching credentials for {url}"); trace!("Caching credentials for {url}");
CREDENTIALS_CACHE.insert(url, credentials); CREDENTIALS_CACHE.insert(url, Arc::new(Authentication::from(credentials)));
} }

View file

@ -12,7 +12,8 @@ use uv_preview::{Preview, PreviewFeatures};
use uv_redacted::DisplaySafeUrl; use uv_redacted::DisplaySafeUrl;
use uv_warnings::owo_colors::OwoColorize; use uv_warnings::owo_colors::OwoColorize;
use crate::providers::HuggingFaceProvider; use crate::credentials::Authentication;
use crate::providers::{HuggingFaceProvider, S3EndpointProvider};
use crate::pyx::{DEFAULT_TOLERANCE_SECS, PyxTokenStore}; use crate::pyx::{DEFAULT_TOLERANCE_SECS, PyxTokenStore};
use crate::{ use crate::{
AccessToken, CREDENTIALS_CACHE, CredentialsCache, KeyringProvider, AccessToken, CREDENTIALS_CACHE, CredentialsCache, KeyringProvider,
@ -21,7 +22,6 @@ use crate::{
index::{AuthPolicy, Indexes}, index::{AuthPolicy, Indexes},
realm::Realm, realm::Realm,
}; };
use crate::{Index, TextCredentialStore, TomlCredentialError}; use crate::{Index, TextCredentialStore, TomlCredentialError};
/// Strategy for loading netrc files. /// Strategy for loading netrc files.
@ -292,7 +292,7 @@ impl Middleware for AuthMiddleware {
next: Next<'_>, next: Next<'_>,
) -> reqwest_middleware::Result<Response> { ) -> reqwest_middleware::Result<Response> {
// Check for credentials attached to the request already // Check for credentials attached to the request already
let request_credentials = Credentials::from_request(&request); let request_credentials = Credentials::from_request(&request).map(Authentication::from);
// In the middleware, existing credentials are already moved from the URL // In the middleware, existing credentials are already moved from the URL
// to the headers so for display purposes we restore some information // to the headers so for display purposes we restore some information
@ -301,7 +301,7 @@ impl Middleware for AuthMiddleware {
let auth_policy = self.indexes.auth_policy_for(request.url()); let auth_policy = self.indexes.auth_policy_for(request.url());
trace!("Handling request for {url} with authentication policy {auth_policy}"); trace!("Handling request for {url} with authentication policy {auth_policy}");
let credentials: Option<Arc<Credentials>> = if matches!(auth_policy, AuthPolicy::Never) { let credentials: Option<Arc<Authentication>> = if matches!(auth_policy, AuthPolicy::Never) {
None None
} else { } else {
if let Some(request_credentials) = request_credentials { if let Some(request_credentials) = request_credentials {
@ -325,7 +325,7 @@ impl Middleware for AuthMiddleware {
// making a failing request // making a failing request
let credentials = self.cache().get_url(request.url(), &Username::none()); let credentials = self.cache().get_url(request.url(), &Username::none());
if let Some(credentials) = credentials.as_ref() { if let Some(credentials) = credentials.as_ref() {
request = credentials.authenticate(request); request = credentials.authenticate(request).await;
// If it's fully authenticated, finish the request // If it's fully authenticated, finish the request
if credentials.is_authenticated() { if credentials.is_authenticated() {
@ -422,7 +422,7 @@ impl Middleware for AuthMiddleware {
if let Some(credentials) = credentials.as_ref() { if let Some(credentials) = credentials.as_ref() {
if credentials.is_authenticated() { if credentials.is_authenticated() {
trace!("Retrying request for {url} with credentials from cache {credentials:?}"); trace!("Retrying request for {url} with credentials from cache {credentials:?}");
retry_request = credentials.authenticate(retry_request); retry_request = credentials.authenticate(retry_request).await;
return self return self
.complete_request(None, retry_request, extensions, next, auth_policy) .complete_request(None, retry_request, extensions, next, auth_policy)
.await; .await;
@ -440,7 +440,7 @@ impl Middleware for AuthMiddleware {
) )
.await .await
{ {
retry_request = credentials.authenticate(retry_request); retry_request = credentials.authenticate(retry_request).await;
trace!("Retrying request for {url} with {credentials:?}"); trace!("Retrying request for {url} with {credentials:?}");
return self return self
.complete_request( .complete_request(
@ -456,7 +456,7 @@ impl Middleware for AuthMiddleware {
if let Some(credentials) = credentials.as_ref() { if let Some(credentials) = credentials.as_ref() {
if !attempt_has_username { if !attempt_has_username {
trace!("Retrying request for {url} with username from cache {credentials:?}"); trace!("Retrying request for {url} with username from cache {credentials:?}");
retry_request = credentials.authenticate(retry_request); retry_request = credentials.authenticate(retry_request).await;
return self return self
.complete_request(None, retry_request, extensions, next, auth_policy) .complete_request(None, retry_request, extensions, next, auth_policy)
.await; .await;
@ -492,7 +492,7 @@ impl AuthMiddleware {
/// If credentials are present, insert them into the cache on success. /// If credentials are present, insert them into the cache on success.
async fn complete_request( async fn complete_request(
&self, &self,
credentials: Option<Arc<Credentials>>, credentials: Option<Arc<Authentication>>,
request: Request, request: Request,
extensions: &mut Extensions, extensions: &mut Extensions,
next: Next<'_>, next: Next<'_>,
@ -524,7 +524,7 @@ impl AuthMiddleware {
/// Use known request credentials to complete the request. /// Use known request credentials to complete the request.
async fn complete_request_with_request_credentials( async fn complete_request_with_request_credentials(
&self, &self,
credentials: Credentials, credentials: Authentication,
mut request: Request, mut request: Request,
extensions: &mut Extensions, extensions: &mut Extensions,
next: Next<'_>, next: Next<'_>,
@ -559,7 +559,7 @@ impl AuthMiddleware {
.get_realm(Realm::from(request.url()), credentials.to_username()) .get_realm(Realm::from(request.url()), credentials.to_username())
}; };
if let Some(credentials) = maybe_cached_credentials { if let Some(credentials) = maybe_cached_credentials {
request = credentials.authenticate(request); request = credentials.authenticate(request).await;
// Do not insert already-cached credentials // Do not insert already-cached credentials
let credentials = None; let credentials = None;
return self return self
@ -571,7 +571,7 @@ impl AuthMiddleware {
.cache() .cache()
.get_url(request.url(), credentials.as_username().as_ref()) .get_url(request.url(), credentials.as_username().as_ref())
{ {
request = credentials.authenticate(request); request = credentials.authenticate(request).await;
// Do not insert already-cached credentials // Do not insert already-cached credentials
None None
} else if let Some(credentials) = self } else if let Some(credentials) = self
@ -583,7 +583,7 @@ impl AuthMiddleware {
) )
.await .await
{ {
request = credentials.authenticate(request); request = credentials.authenticate(request).await;
Some(credentials) Some(credentials)
} else if index.is_some() { } else if index.is_some() {
// If this is a known index, we fall back to checking for the realm. // If this is a known index, we fall back to checking for the realm.
@ -591,7 +591,7 @@ impl AuthMiddleware {
.cache() .cache()
.get_realm(Realm::from(request.url()), credentials.to_username()) .get_realm(Realm::from(request.url()), credentials.to_username())
{ {
request = credentials.authenticate(request); request = credentials.authenticate(request).await;
Some(credentials) Some(credentials)
} else { } else {
Some(credentials) Some(credentials)
@ -610,11 +610,11 @@ impl AuthMiddleware {
/// Supports netrc file and keyring lookups. /// Supports netrc file and keyring lookups.
async fn fetch_credentials( async fn fetch_credentials(
&self, &self,
credentials: Option<&Credentials>, credentials: Option<&Authentication>,
url: &DisplaySafeUrl, url: &DisplaySafeUrl,
index: Option<&Index>, index: Option<&Index>,
auth_policy: AuthPolicy, auth_policy: AuthPolicy,
) -> Option<Arc<Credentials>> { ) -> Option<Arc<Authentication>> {
let username = Username::from( let username = Username::from(
credentials.map(|credentials| credentials.username().unwrap_or_default().to_string()), credentials.map(|credentials| credentials.username().unwrap_or_default().to_string()),
); );
@ -646,13 +646,25 @@ impl AuthMiddleware {
return credentials; return credentials;
} }
// Support for known providers, like Hugging Face. // Support for known providers, like Hugging Face and S3.
if let Some(credentials) = HuggingFaceProvider::credentials_for(url).map(Arc::new) { if let Some(credentials) = HuggingFaceProvider::credentials_for(url)
.map(Authentication::from)
.map(Arc::new)
{
debug!("Found Hugging Face credentials for {url}"); debug!("Found Hugging Face credentials for {url}");
self.cache().fetches.done(key, Some(credentials.clone())); self.cache().fetches.done(key, Some(credentials.clone()));
return Some(credentials); return Some(credentials);
} }
if let Some(credentials) = S3EndpointProvider::credentials_for(url, self.preview)
.map(Authentication::from)
.map(Arc::new)
{
debug!("Found S3 credentials for {url}");
self.cache().fetches.done(key, Some(credentials.clone()));
return Some(credentials);
}
// If this is a known URL, authenticate it via the token store. // If this is a known URL, authenticate it via the token store.
if let Some(base_client) = self.base_client.as_ref() { if let Some(base_client) = self.base_client.as_ref() {
if let Some(token_store) = self.pyx_token_store.as_ref() { if let Some(token_store) = self.pyx_token_store.as_ref() {
@ -682,7 +694,7 @@ impl AuthMiddleware {
let credentials = token.map(|token| { let credentials = token.map(|token| {
trace!("Using credentials from token store for {url}"); trace!("Using credentials from token store for {url}");
Arc::new(Credentials::from(token)) Arc::new(Authentication::from(Credentials::from(token)))
}); });
// Register the fetch for this key // Register the fetch for this key
@ -778,6 +790,7 @@ impl AuthMiddleware {
} else { } else {
None None
} }
.map(Authentication::from)
.map(Arc::new); .map(Arc::new);
// Register the fetch for this key // Register the fetch for this key
@ -787,9 +800,9 @@ impl AuthMiddleware {
} }
} }
fn tracing_url(request: &Request, credentials: Option<&Credentials>) -> DisplaySafeUrl { fn tracing_url(request: &Request, credentials: Option<&Authentication>) -> DisplaySafeUrl {
let mut url = DisplaySafeUrl::from(request.url().clone()); let mut url = DisplaySafeUrl::from(request.url().clone());
if let Some(creds) = credentials { if let Some(Authentication::Credentials(creds)) = credentials {
if let Some(username) = creds.username() { if let Some(username) = creds.username() {
let _ = url.set_username(username); let _ = url.set_username(username);
} }
@ -930,10 +943,10 @@ mod tests {
let cache = CredentialsCache::new(); let cache = CredentialsCache::new();
cache.insert( cache.insert(
&base_url, &base_url,
Arc::new(Credentials::basic( Arc::new(Authentication::from(Credentials::basic(
Some(username.to_string()), Some(username.to_string()),
Some(password.to_string()), Some(password.to_string()),
)), ))),
); );
let client = test_client_builder() let client = test_client_builder()
@ -984,7 +997,10 @@ mod tests {
let cache = CredentialsCache::new(); let cache = CredentialsCache::new();
cache.insert( cache.insert(
&base_url, &base_url,
Arc::new(Credentials::basic(Some(username.to_string()), None)), Arc::new(Authentication::from(Credentials::basic(
Some(username.to_string()),
None,
))),
); );
let client = test_client_builder() let client = test_client_builder()
@ -1377,7 +1393,10 @@ mod tests {
// URL. // URL.
cache.insert( cache.insert(
&base_url, &base_url,
Arc::new(Credentials::basic(Some(username.to_string()), None)), Arc::new(Authentication::from(Credentials::basic(
Some(username.to_string()),
None,
))),
); );
let client = test_client_builder() let client = test_client_builder()
.with(AuthMiddleware::new().with_cache(cache).with_keyring(Some( .with(AuthMiddleware::new().with_cache(cache).with_keyring(Some(
@ -1426,17 +1445,17 @@ mod tests {
// Seed the cache with our credentials // Seed the cache with our credentials
cache.insert( cache.insert(
&base_url_1, &base_url_1,
Arc::new(Credentials::basic( Arc::new(Authentication::from(Credentials::basic(
Some(username_1.to_string()), Some(username_1.to_string()),
Some(password_1.to_string()), Some(password_1.to_string()),
)), ))),
); );
cache.insert( cache.insert(
&base_url_2, &base_url_2,
Arc::new(Credentials::basic( Arc::new(Authentication::from(Credentials::basic(
Some(username_2.to_string()), Some(username_2.to_string()),
Some(password_2.to_string()), Some(password_2.to_string()),
)), ))),
); );
let client = test_client_builder() let client = test_client_builder()
@ -1621,17 +1640,17 @@ mod tests {
// Seed the cache with our credentials // Seed the cache with our credentials
cache.insert( cache.insert(
&base_url_1, &base_url_1,
Arc::new(Credentials::basic( Arc::new(Authentication::from(Credentials::basic(
Some(username_1.to_string()), Some(username_1.to_string()),
Some(password_1.to_string()), Some(password_1.to_string()),
)), ))),
); );
cache.insert( cache.insert(
&base_url_2, &base_url_2,
Arc::new(Credentials::basic( Arc::new(Authentication::from(Credentials::basic(
Some(username_2.to_string()), Some(username_2.to_string()),
Some(password_2.to_string()), Some(password_2.to_string()),
)), ))),
); );
let client = test_client_builder() let client = test_client_builder()
@ -2334,20 +2353,20 @@ mod tests {
DisplaySafeUrl::parse("https://pypi-proxy.fly.dev/basic-auth/simple").unwrap() DisplaySafeUrl::parse("https://pypi-proxy.fly.dev/basic-auth/simple").unwrap()
); );
let creds = Credentials::Basic { let creds = Authentication::from(Credentials::Basic {
username: Username::new(Some(String::from("user"))), username: Username::new(Some(String::from("user"))),
password: None, password: None,
}; });
let req = create_request("https://pypi-proxy.fly.dev/basic-auth/simple"); let req = create_request("https://pypi-proxy.fly.dev/basic-auth/simple");
assert_eq!( assert_eq!(
tracing_url(&req, Some(&creds)), tracing_url(&req, Some(&creds)),
DisplaySafeUrl::parse("https://user@pypi-proxy.fly.dev/basic-auth/simple").unwrap() DisplaySafeUrl::parse("https://user@pypi-proxy.fly.dev/basic-auth/simple").unwrap()
); );
let creds = Credentials::Basic { let creds = Authentication::from(Credentials::Basic {
username: Username::new(Some(String::from("user"))), username: Username::new(Some(String::from("user"))),
password: Some(Password::new(String::from("password"))), password: Some(Password::new(String::from("password"))),
}; });
let req = create_request("https://pypi-proxy.fly.dev/basic-auth/simple"); let req = create_request("https://pypi-proxy.fly.dev/basic-auth/simple");
assert_eq!( assert_eq!(
tracing_url(&req, Some(&creds)), tracing_url(&req, Some(&creds)),
@ -2368,7 +2387,7 @@ mod tests {
let mut store = TextCredentialStore::default(); let mut store = TextCredentialStore::default();
let service = crate::Service::try_from(base_url.to_string()).unwrap(); let service = crate::Service::try_from(base_url.to_string()).unwrap();
let credentials = let credentials =
crate::Credentials::basic(Some(username.to_string()), Some(password.to_string())); Credentials::basic(Some(username.to_string()), Some(password.to_string()));
store.insert(service.clone(), credentials); store.insert(service.clone(), credentials);
let client = test_client_builder() let client = test_client_builder()

View file

@ -1,8 +1,13 @@
use std::borrow::Cow;
use std::sync::LazyLock; use std::sync::LazyLock;
use reqsign::aws::DefaultSigner;
use tracing::debug; use tracing::debug;
use url::Url; use url::Url;
use uv_preview::{Preview, PreviewFeatures};
use uv_static::EnvVars; use uv_static::EnvVars;
use uv_warnings::warn_user_once;
use crate::Credentials; use crate::Credentials;
use crate::realm::{Realm, RealmRef}; use crate::realm::{Realm, RealmRef};
@ -47,3 +52,45 @@ impl HuggingFaceProvider {
None None
} }
} }
/// The [`Url`] for the S3 endpoint, if set.
static S3_ENDPOINT_REALM: LazyLock<Option<Realm>> = LazyLock::new(|| {
let s3_endpoint_url = std::env::var(EnvVars::UV_S3_ENDPOINT_URL).ok()?;
let url = Url::parse(&s3_endpoint_url).expect("Failed to parse S3 endpoint URL");
Some(Realm::from(&url))
});
/// A provider for authentication credentials for S3 endpoints.
#[derive(Debug, Clone, PartialEq, Eq)]
pub(crate) struct S3EndpointProvider;
impl S3EndpointProvider {
/// Returns the credentials for the S3 endpoint, if available.
pub(crate) fn credentials_for(url: &Url, preview: Preview) -> Option<DefaultSigner> {
if let Some(s3_endpoint_realm) = S3_ENDPOINT_REALM.as_ref().map(RealmRef::from) {
if !preview.is_enabled(PreviewFeatures::S3_ENDPOINT) {
warn_user_once!(
"The `s3-endpoint` option is experimental and may change without warning. Pass `--preview-features {}` to disable this warning.",
PreviewFeatures::S3_ENDPOINT
);
}
// Treat any URL on the same domain or subdomain as available for S3 signing.
let realm = RealmRef::from(url);
if realm == s3_endpoint_realm || realm.is_subdomain_of(s3_endpoint_realm) {
// TODO(charlie): Can `reqsign` infer the region for us? Profiles, for example,
// often have a region set already.
let region = std::env::var(EnvVars::AWS_REGION)
.map(Cow::Owned)
.unwrap_or_else(|_| {
std::env::var(EnvVars::AWS_DEFAULT_REGION)
.map(Cow::Owned)
.unwrap_or_else(|_| Cow::Borrowed("us-east-1"))
});
let signer = reqsign::aws::default_signer("s3", &region);
return Some(signer);
}
}
None
}
}

View file

@ -81,6 +81,21 @@ pub(crate) struct RealmRef<'a> {
port: Option<u16>, port: Option<u16>,
} }
impl RealmRef<'_> {
/// Returns true if this realm is a subdomain of the other realm.
pub(crate) fn is_subdomain_of(&self, other: Self) -> bool {
other.scheme == self.scheme
&& other.port == self.port
&& other.host.is_some_and(|other_host| {
self.host.is_some_and(|self_host| {
self_host
.strip_suffix(other_host)
.is_some_and(|prefix| prefix.ends_with('.'))
})
})
}
}
impl<'a> From<&'a Url> for RealmRef<'a> { impl<'a> From<&'a Url> for RealmRef<'a> {
fn from(url: &'a Url) -> Self { fn from(url: &'a Url) -> Self {
Self { Self {
@ -215,4 +230,87 @@ mod tests {
Ok(()) Ok(())
} }
#[test]
fn test_is_subdomain_of() -> Result<(), ParseError> {
use crate::realm::RealmRef;
// Subdomain relationship: sub.example.com is a subdomain of example.com
let subdomain_url = Url::parse("https://sub.example.com")?;
let domain_url = Url::parse("https://example.com")?;
let subdomain = RealmRef::from(&subdomain_url);
let domain = RealmRef::from(&domain_url);
assert!(subdomain.is_subdomain_of(domain));
// Deeper subdomain: foo.bar.example.com is a subdomain of example.com
let deep_subdomain_url = Url::parse("https://foo.bar.example.com")?;
let deep_subdomain = RealmRef::from(&deep_subdomain_url);
assert!(deep_subdomain.is_subdomain_of(domain));
// Deeper subdomain: foo.bar.example.com is also a subdomain of bar.example.com
let parent_subdomain_url = Url::parse("https://bar.example.com")?;
let parent_subdomain = RealmRef::from(&parent_subdomain_url);
assert!(deep_subdomain.is_subdomain_of(parent_subdomain));
// Not a subdomain: example.com is not a subdomain of sub.example.com
assert!(!domain.is_subdomain_of(subdomain));
// Same domain is not a subdomain of itself
assert!(!domain.is_subdomain_of(domain));
// Different TLD: example.org is not a subdomain of example.com
let different_tld_url = Url::parse("https://example.org")?;
let different_tld = RealmRef::from(&different_tld_url);
assert!(!different_tld.is_subdomain_of(domain));
// Partial match but not a subdomain: notexample.com is not a subdomain of example.com
let partial_match_url = Url::parse("https://notexample.com")?;
let partial_match = RealmRef::from(&partial_match_url);
assert!(!partial_match.is_subdomain_of(domain));
// Different scheme: http subdomain is not a subdomain of https domain
let http_subdomain_url = Url::parse("http://sub.example.com")?;
let https_domain_url = Url::parse("https://example.com")?;
let http_subdomain = RealmRef::from(&http_subdomain_url);
let https_domain = RealmRef::from(&https_domain_url);
assert!(!http_subdomain.is_subdomain_of(https_domain));
// Different port: same subdomain with different port is not a subdomain
let subdomain_port_8080_url = Url::parse("https://sub.example.com:8080")?;
let domain_port_9090_url = Url::parse("https://example.com:9090")?;
let subdomain_port_8080 = RealmRef::from(&subdomain_port_8080_url);
let domain_port_9090 = RealmRef::from(&domain_port_9090_url);
assert!(!subdomain_port_8080.is_subdomain_of(domain_port_9090));
// Same port: subdomain with same explicit port is a subdomain
let subdomain_with_port_url = Url::parse("https://sub.example.com:8080")?;
let domain_with_port_url = Url::parse("https://example.com:8080")?;
let subdomain_with_port = RealmRef::from(&subdomain_with_port_url);
let domain_with_port = RealmRef::from(&domain_with_port_url);
assert!(subdomain_with_port.is_subdomain_of(domain_with_port));
// Default port handling: subdomain with implicit port is a subdomain
let subdomain_default_url = Url::parse("https://sub.example.com")?;
let domain_explicit_443_url = Url::parse("https://example.com:443")?;
let subdomain_default = RealmRef::from(&subdomain_default_url);
let domain_explicit_443 = RealmRef::from(&domain_explicit_443_url);
assert!(subdomain_default.is_subdomain_of(domain_explicit_443));
// Edge case: empty host (shouldn't happen with valid URLs but testing defensive code)
let file_url = Url::parse("file:///path/to/file")?;
let https_url = Url::parse("https://example.com")?;
let file_realm = RealmRef::from(&file_url);
let https_realm = RealmRef::from(&https_url);
assert!(!file_realm.is_subdomain_of(https_realm));
assert!(!https_realm.is_subdomain_of(file_realm));
// Subdomain with path (path should be ignored)
let subdomain_with_path_url = Url::parse("https://sub.example.com/path")?;
let domain_with_path_url = Url::parse("https://example.com/other")?;
let subdomain_with_path = RealmRef::from(&subdomain_with_path_url);
let domain_with_path = RealmRef::from(&domain_with_path_url);
assert!(subdomain_with_path.is_subdomain_of(domain_with_path));
Ok(())
}
} }

View file

@ -445,11 +445,10 @@ impl<'a> IndexLocations {
.map(ToString::to_string) .map(ToString::to_string)
.unwrap_or_else(|| index.url.to_string()) .unwrap_or_else(|| index.url.to_string())
); );
let credentials = Arc::new(credentials);
uv_auth::store_credentials(index.raw_url(), credentials.clone());
if let Some(root_url) = index.root_url() { if let Some(root_url) = index.root_url() {
uv_auth::store_credentials(&root_url, credentials.clone()); uv_auth::store_credentials(&root_url, credentials.clone());
} }
uv_auth::store_credentials(index.raw_url(), credentials);
} }
} }
} }

View file

@ -1,7 +1,6 @@
use std::collections::BTreeMap; use std::collections::BTreeMap;
use std::io; use std::io;
use std::path::{Path, PathBuf}; use std::path::{Path, PathBuf};
use std::sync::Arc;
use either::Either; use either::Either;
use thiserror::Error; use thiserror::Error;
@ -230,7 +229,6 @@ impl LoweredRequirement {
)); ));
}; };
if let Some(credentials) = index.credentials() { if let Some(credentials) = index.credentials() {
let credentials = Arc::new(credentials);
uv_auth::store_credentials(index.raw_url(), credentials); uv_auth::store_credentials(index.raw_url(), credentials);
} }
let index = IndexMetadata { let index = IndexMetadata {
@ -464,7 +462,6 @@ impl LoweredRequirement {
)); ));
}; };
if let Some(credentials) = index.credentials() { if let Some(credentials) = index.credentials() {
let credentials = Arc::new(credentials);
uv_auth::store_credentials(index.raw_url(), credentials); uv_auth::store_credentials(index.raw_url(), credentials);
} }
let index = IndexMetadata { let index = IndexMetadata {

View file

@ -19,6 +19,7 @@ bitflags::bitflags! {
const DETECT_MODULE_CONFLICTS = 1 << 7; const DETECT_MODULE_CONFLICTS = 1 << 7;
const FORMAT = 1 << 8; const FORMAT = 1 << 8;
const NATIVE_AUTH = 1 << 9; const NATIVE_AUTH = 1 << 9;
const S3_ENDPOINT = 1 << 10;
} }
} }
@ -38,6 +39,7 @@ impl PreviewFeatures {
Self::DETECT_MODULE_CONFLICTS => "detect-module-conflicts", Self::DETECT_MODULE_CONFLICTS => "detect-module-conflicts",
Self::FORMAT => "format", Self::FORMAT => "format",
Self::NATIVE_AUTH => "native-auth", Self::NATIVE_AUTH => "native-auth",
Self::S3_ENDPOINT => "s3-endpoint",
_ => panic!("`flag_as_str` can only be used for exactly one feature flag"), _ => panic!("`flag_as_str` can only be used for exactly one feature flag"),
} }
} }
@ -85,6 +87,7 @@ impl FromStr for PreviewFeatures {
"detect-module-conflicts" => Self::DETECT_MODULE_CONFLICTS, "detect-module-conflicts" => Self::DETECT_MODULE_CONFLICTS,
"format" => Self::FORMAT, "format" => Self::FORMAT,
"native-auth" => Self::NATIVE_AUTH, "native-auth" => Self::NATIVE_AUTH,
"s3-endpoint" => Self::S3_ENDPOINT,
_ => { _ => {
warn_user_once!("Unknown preview feature: `{part}`"); warn_user_once!("Unknown preview feature: `{part}`");
continue; continue;
@ -260,6 +263,7 @@ mod tests {
"detect-module-conflicts" "detect-module-conflicts"
); );
assert_eq!(PreviewFeatures::FORMAT.flag_as_str(), "format"); assert_eq!(PreviewFeatures::FORMAT.flag_as_str(), "format");
assert_eq!(PreviewFeatures::S3_ENDPOINT.flag_as_str(), "s3-endpoint");
} }
#[test] #[test]

View file

@ -886,6 +886,11 @@ impl EnvVars {
/// Disable Hugging Face authentication, even if `HF_TOKEN` is set. /// Disable Hugging Face authentication, even if `HF_TOKEN` is set.
pub const UV_NO_HF_TOKEN: &'static str = "UV_NO_HF_TOKEN"; pub const UV_NO_HF_TOKEN: &'static str = "UV_NO_HF_TOKEN";
/// The URL to treat as an S3-compatible storage endpoint. Requests to this endpoint
/// will be signed using AWS Signature Version 4 based on the `AWS_ACCESS_KEY_ID`,
/// `AWS_SECRET_ACCESS_KEY`, `AWS_PROFILE`, and `AWS_CONFIG_FILE` environment variables.
pub const UV_S3_ENDPOINT_URL: &'static str = "UV_S3_ENDPOINT_URL";
/// The URL of the pyx Simple API server. /// The URL of the pyx Simple API server.
pub const PYX_API_URL: &'static str = "PYX_API_URL"; pub const PYX_API_URL: &'static str = "PYX_API_URL";
@ -908,4 +913,28 @@ impl EnvVars {
/// Specifies the directory where uv stores pyx credentials. /// Specifies the directory where uv stores pyx credentials.
pub const PYX_CREDENTIALS_DIR: &'static str = "PYX_CREDENTIALS_DIR"; pub const PYX_CREDENTIALS_DIR: &'static str = "PYX_CREDENTIALS_DIR";
/// The AWS region to use when signing S3 requests.
pub const AWS_REGION: &'static str = "AWS_REGION";
/// The default AWS region to use when signing S3 requests, if `AWS_REGION` is not set.
pub const AWS_DEFAULT_REGION: &'static str = "AWS_DEFAULT_REGION";
/// The AWS access key ID to use when signing S3 requests.
pub const AWS_ACCESS_KEY_ID: &'static str = "AWS_ACCESS_KEY_ID";
/// The AWS secret access key to use when signing S3 requests.
pub const AWS_SECRET_ACCESS_KEY: &'static str = "AWS_SECRET_ACCESS_KEY";
/// The AWS session token to use when signing S3 requests.
pub const AWS_SESSION_TOKEN: &'static str = "AWS_SESSION_TOKEN";
/// The AWS profile to use when signing S3 requests.
pub const AWS_PROFILE: &'static str = "AWS_PROFILE";
/// The AWS config file to use when signing S3 requests.
pub const AWS_CONFIG_FILE: &'static str = "AWS_CONFIG_FILE";
/// The AWS shared credentials file to use when signing S3 requests.
pub const AWS_SHARED_CREDENTIALS_FILE: &'static str = "AWS_SHARED_CREDENTIALS_FILE";
} }

View file

@ -621,11 +621,10 @@ async fn do_lock(
for index in target.indexes() { for index in target.indexes() {
if let Some(credentials) = index.credentials() { if let Some(credentials) = index.credentials() {
let credentials = Arc::new(credentials);
uv_auth::store_credentials(index.raw_url(), credentials.clone());
if let Some(root_url) = index.root_url() { if let Some(root_url) = index.root_url() {
uv_auth::store_credentials(&root_url, credentials.clone()); uv_auth::store_credentials(&root_url, credentials.clone());
} }
uv_auth::store_credentials(index.raw_url(), credentials);
} }
} }

View file

@ -900,11 +900,10 @@ fn store_credentials_from_target(target: InstallTarget<'_>) {
// Iterate over any indexes in the target. // Iterate over any indexes in the target.
for index in target.indexes() { for index in target.indexes() {
if let Some(credentials) = index.credentials() { if let Some(credentials) = index.credentials() {
let credentials = Arc::new(credentials);
uv_auth::store_credentials(index.raw_url(), credentials.clone());
if let Some(root_url) = index.root_url() { if let Some(root_url) = index.root_url() {
uv_auth::store_credentials(&root_url, credentials.clone()); uv_auth::store_credentials(&root_url, credentials.clone());
} }
uv_auth::store_credentials(index.raw_url(), credentials);
} }
} }

View file

@ -7686,7 +7686,7 @@ fn preview_features() {
show_settings: true, show_settings: true,
preview: Preview { preview: Preview {
flags: PreviewFeatures( flags: PreviewFeatures(
PYTHON_INSTALL_DEFAULT | PYTHON_UPGRADE | JSON_OUTPUT | PYLOCK | ADD_BOUNDS | PACKAGE_CONFLICTS | EXTRA_BUILD_DEPENDENCIES | DETECT_MODULE_CONFLICTS | FORMAT | NATIVE_AUTH, PYTHON_INSTALL_DEFAULT | PYTHON_UPGRADE | JSON_OUTPUT | PYLOCK | ADD_BOUNDS | PACKAGE_CONFLICTS | EXTRA_BUILD_DEPENDENCIES | DETECT_MODULE_CONFLICTS | FORMAT | NATIVE_AUTH | S3_ENDPOINT,
), ),
}, },
python_preference: Managed, python_preference: Managed,
@ -7910,7 +7910,7 @@ fn preview_features() {
show_settings: true, show_settings: true,
preview: Preview { preview: Preview {
flags: PreviewFeatures( flags: PreviewFeatures(
PYTHON_INSTALL_DEFAULT | PYTHON_UPGRADE | JSON_OUTPUT | PYLOCK | ADD_BOUNDS | PACKAGE_CONFLICTS | EXTRA_BUILD_DEPENDENCIES | DETECT_MODULE_CONFLICTS | FORMAT | NATIVE_AUTH, PYTHON_INSTALL_DEFAULT | PYTHON_UPGRADE | JSON_OUTPUT | PYLOCK | ADD_BOUNDS | PACKAGE_CONFLICTS | EXTRA_BUILD_DEPENDENCIES | DETECT_MODULE_CONFLICTS | FORMAT | NATIVE_AUTH | S3_ENDPOINT,
), ),
}, },
python_preference: Managed, python_preference: Managed,

View file

@ -493,6 +493,12 @@ uv will require that all dependencies have a hash specified in the requirements
Equivalent to the `--resolution` command-line argument. For example, if set to Equivalent to the `--resolution` command-line argument. For example, if set to
`lowest-direct`, uv will install the lowest compatible versions of all direct dependencies. `lowest-direct`, uv will install the lowest compatible versions of all direct dependencies.
### `UV_S3_ENDPOINT_URL`
The URL to treat as an S3-compatible storage endpoint. Requests to this endpoint
will be signed using AWS Signature Version 4 based on the `AWS_ACCESS_KEY_ID`,
`AWS_SECRET_ACCESS_KEY`, `AWS_PROFILE`, and `AWS_CONFIG_FILE` environment variables.
### `UV_STACK_SIZE` ### `UV_STACK_SIZE`
Use to set the stack size used by uv. Use to set the stack size used by uv.
@ -568,6 +574,38 @@ Defaults to `24`.
Path to user-level configuration directory on Windows systems. Path to user-level configuration directory on Windows systems.
### `AWS_ACCESS_KEY_ID`
The AWS access key ID to use when signing S3 requests.
### `AWS_CONFIG_FILE`
The AWS config file to use when signing S3 requests.
### `AWS_DEFAULT_REGION`
The default AWS region to use when signing S3 requests, if `AWS_REGION` is not set.
### `AWS_PROFILE`
The AWS profile to use when signing S3 requests.
### `AWS_REGION`
The AWS region to use when signing S3 requests.
### `AWS_SECRET_ACCESS_KEY`
The AWS secret access key to use when signing S3 requests.
### `AWS_SESSION_TOKEN`
The AWS session token to use when signing S3 requests.
### `AWS_SHARED_CREDENTIALS_FILE`
The AWS shared credentials file to use when signing S3 requests.
### `BASH_VERSION` ### `BASH_VERSION`
Used to detect Bash shell usage. Used to detect Bash shell usage.