mirrors/uv
1
0
Fork 0
mirror of https://github.com/astral-sh/uv.git synced 2025-11-20 03:49:54 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 3

Add DisplaySafeUrl newtype to prevent leaking of credentials by default (#13560)

Browse source 
Prior to this PR, there were numerous places where uv would leak
credentials in logs. We had a way to mask credentials by calling methods
or a recently-added `redact_url` function, but this was not secure by
default. There were a number of other types (like `GitUrl`) that would
leak credentials on display.

This PR adds a `DisplaySafeUrl` newtype to prevent leaking credentials
when logging by default. It takes a maximalist approach, replacing the
use of `Url` almost everywhere. This includes when first parsing config
files, when storing URLs in types like `GitUrl`, and also when storing
URLs in types that in practice will never contain credentials (like
`DirectorySourceUrl`). The idea is to make it easy for developers to do
the right thing and for the compiler to support this (and to minimize
ever having to manually convert back and forth). Displaying credentials
now requires an active step. Note that despite this maximalist approach,
the use of the newtype should be zero cost.

One conspicuous place this PR does not use `DisplaySafeUrl` is in the
`uv-auth` crate. That would require new clones since there are calls to
`request.url()` that return a `&Url`. One option would have been to make
`DisplaySafeUrl` wrap a `Cow`, but this would lead to lifetime
annotations all over the codebase. I've created a separate PR based on
this one (#13576) that updates `uv-auth` to use `DisplaySafeUrl` with
one new clone. We can discuss the tradeoffs there.

Most of this PR just replaces `Url` with `DisplaySafeUrl`. The core is
`uv_redacted/lib.rs`, where the newtype is implemented. To make it
easier to review the rest, here are some points of note:

* `DisplaySafeUrl` has a `Display` implementation that masks
credentials. Currently, it will still display the username when there is
both a username and password. If we think is the wrong choice, it can
now be changed in one place.
* `DisplaySafeUrl` has a `remove_credentials()` method and also a
`.to_string_with_credentials()` method. This allows us to use it in a
variety of scenarios.
* `IndexUrl::redacted()` was renamed to
`IndexUrl::removed_credentials()` to make it clearer that we are not
masking.
* We convert from a `DisplaySafeUrl` to a `Url` when calling `reqwest`
methods like `.get()` and `.head()`.
* We convert from a `DisplaySafeUrl` to a `Url` when creating a
`uv_auth::Index`. That is because, as mentioned above, I will be
updating the `uv_auth` crate to use this newtype in a separate PR.
* A number of tests (e.g., in `pip_install.rs`) that formerly used
filters to mask tokens in the test output no longer need those filters
since tokens in URLs are now masked automatically.
* The one place we are still knowingly writing credentials to
`pyproject.toml` is when a URL with credentials is passed to `uv add`
with `--raw`. Since displaying credentials is no longer automatic, I
have added a `to_string_with_credentials()` method to the `Pep508Url`
trait. This is used when `--raw` is passed. Adding it to that trait is a
bit weird, but it's the simplest way to achieve the goal. I'm open to
suggestions on how to improve this, but note that because of the way
we're using generic bounds, it's not as simple as just creating a
separate trait for that method.
This commit is contained in:
John Mumm 2025-05-27 00:05:30 +02:00 • committed by GitHub
parent b80cafd5e8
commit c19a294a48
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
100 changed files with 1266 additions and 2249 deletions
Download patch file Download diff file Expand all files Collapse all files

5
crates/uv-client/src/base_client.rs
View file

@ -21,6 +21,7 @@ use uv_configuration::{KeyringProviderType, TrustedHost};
use uv_fs::Simplified;
use uv_pep508::MarkerEnvironment;
use uv_platform_tags::Platform;
use uv_redacted::DisplaySafeUrl;
use uv_static::EnvVars;
use uv_version::version;
use uv_warnings::warn_user_once;
@ -407,7 +408,7 @@ enum Security {
impl BaseClient {
/// Selects the appropriate client based on the host's trustworthiness.
pub fn for_host(&self, url: &Url) -> &ClientWithMiddleware {
pub fn for_host(&self, url: &DisplaySafeUrl) -> &ClientWithMiddleware {
if self.disable_ssl(url) {
&self.dangerous_client
} else {
@ -416,7 +417,7 @@ impl BaseClient {
}
/// Returns `true` if the host is trusted to use the insecure client.
pub fn disable_ssl(&self, url: &Url) -> bool {
pub fn disable_ssl(&self, url: &DisplaySafeUrl) -> bool {
self.allow_insecure_host
.iter()
.any(|allow_insecure_host| allow_insecure_host.matches(url))

7
crates/uv-client/src/cached_client.rs
View file

@ -12,6 +12,7 @@ use tracing::{Instrument, debug, info_span, instrument, trace, warn};
use uv_cache::{CacheEntry, Freshness};
use uv_fs::write_atomic;
use uv_redacted::DisplaySafeUrl;
use crate::BaseClient;
use crate::base_client::is_extended_transient_error;
@ -481,11 +482,11 @@ impl CachedClient {
cached: DataWithCachePolicy,
new_cache_policy_builder: CachePolicyBuilder,
) -> Result<CachedResponse, Error> {
let url = req.url().clone();
let url = DisplaySafeUrl::from(req.url().clone());
debug!("Sending revalidation request for: {url}");
let response = self
.0
.for_host(req.url())
.for_host(&url)
.execute(req)
.instrument(info_span!("revalidation_request", url = url.as_str()))
.await
@ -521,7 +522,7 @@ impl CachedClient {
&self,
req: Request,
) -> Result<(Response, Option<Box<CachePolicy>>), Error> {
let url = req.url().clone();
let url = DisplaySafeUrl::from(req.url().clone());
trace!("Sending fresh {} request for {}", req.method(), url);
let cache_policy_builder = CachePolicyBuilder::new(&req);
let response = self

42
crates/uv-client/src/error.rs
View file

@ -3,11 +3,10 @@ use std::ops::Deref;
use async_http_range_reader::AsyncHttpRangeReaderError;
use async_zip::error::ZipError;
use url::Url;
use uv_distribution_filename::{WheelFilename, WheelFilenameError};
use uv_normalize::PackageName;
use uv_redacted::redacted_url;
use uv_redacted::DisplaySafeUrl;
use crate::middleware::OfflineError;
use crate::{FlatIndexError, html};
@ -30,12 +29,12 @@ impl Error {
}
/// Create a new error from a JSON parsing error.
pub(crate) fn from_json_err(err: serde_json::Error, url: Url) -> Self {
pub(crate) fn from_json_err(err: serde_json::Error, url: DisplaySafeUrl) -> Self {
ErrorKind::BadJson { source: err, url }.into()
}
/// Create a new error from an HTML parsing error.
pub(crate) fn from_html_err(err: html::Error, url: Url) -> Self {
pub(crate) fn from_html_err(err: html::Error, url: DisplaySafeUrl) -> Self {
ErrorKind::BadHtml { source: err, url }.into()
}
@ -160,10 +159,10 @@ pub enum ErrorKind {
Flat(#[from] FlatIndexError),
#[error("Expected a file URL, but received: {0}")]
NonFileUrl(Url),
NonFileUrl(DisplaySafeUrl),
#[error("Expected an index URL, but received non-base URL: {0}")]
CannotBeABase(Url),
CannotBeABase(DisplaySafeUrl),
#[error("Failed to read metadata: `{0}`")]
Metadata(String, #[source] uv_metadata::Error),
@ -196,16 +195,22 @@ pub enum ErrorKind {
/// An error that happened while making a request or in a reqwest middleware.
#[error("Failed to fetch: `{0}`")]
WrappedReqwestError(Url, #[source] WrappedReqwestError),
WrappedReqwestError(DisplaySafeUrl, #[source] WrappedReqwestError),
#[error("Received some unexpected JSON from {}", redacted_url(url))]
BadJson { source: serde_json::Error, url: Url },
#[error("Received some unexpected JSON from {}", url)]
BadJson {
source: serde_json::Error,
url: DisplaySafeUrl,
},
#[error("Received some unexpected HTML from {}", redacted_url(url))]
BadHtml { source: html::Error, url: Url },
#[error("Received some unexpected HTML from {}", url)]
BadHtml {
source: html::Error,
url: DisplaySafeUrl,
},
#[error("Failed to read zip with range requests: `{0}`")]
AsyncHttpRangeReader(Url, #[source] AsyncHttpRangeReaderError),
AsyncHttpRangeReader(DisplaySafeUrl, #[source] AsyncHttpRangeReaderError),
#[error("{0} is not a valid wheel filename")]
WheelFilename(#[source] WheelFilenameError),
@ -232,13 +237,13 @@ pub enum ErrorKind {
Encode(#[source] rmp_serde::encode::Error),
#[error("Missing `Content-Type` header for {0}")]
MissingContentType(Url),
MissingContentType(DisplaySafeUrl),
#[error("Invalid `Content-Type` header for {0}")]
InvalidContentTypeHeader(Url, #[source] http::header::ToStrError),
InvalidContentTypeHeader(DisplaySafeUrl, #[source] http::header::ToStrError),
#[error("Unsupported `Content-Type` \"{1}\" for {0}. Expected JSON or HTML.")]
UnsupportedMediaType(Url, String),
UnsupportedMediaType(DisplaySafeUrl, String),
#[error("Reading from cache archive failed: {0}")]
ArchiveRead(String),
@ -253,11 +258,14 @@ pub enum ErrorKind {
}
impl ErrorKind {
pub(crate) fn from_reqwest(url: Url, error: reqwest::Error) -> Self {
pub(crate) fn from_reqwest(url: DisplaySafeUrl, error: reqwest::Error) -> Self {
Self::WrappedReqwestError(url, WrappedReqwestError::from(error))
}
pub(crate) fn from_reqwest_middleware(url: Url, err: reqwest_middleware::Error) -> Self {
pub(crate) fn from_reqwest_middleware(
url: DisplaySafeUrl,
err: reqwest_middleware::Error,
) -> Self {
if let reqwest_middleware::Error::Middleware(ref underlying) = err {
if let Some(err) = underlying.downcast_ref::<OfflineError>() {
return Self::Offline(err.url().to_string());

18
crates/uv-client/src/flat_index.rs
View file

@ -10,7 +10,7 @@ use uv_cache_key::cache_digest;
use uv_distribution_filename::DistFilename;
use uv_distribution_types::{File, FileLocation, IndexUrl, UrlString};
use uv_pypi_types::HashDigests;
use uv_redacted::redacted_url;
use uv_redacted::DisplaySafeUrl;
use uv_small_str::SmallString;
use crate::cached_client::{CacheControl, CachedClientError};
@ -20,13 +20,13 @@ use crate::{CachedClient, Connectivity, Error, ErrorKind, OwnedArchive};
#[derive(Debug, thiserror::Error)]
pub enum FlatIndexError {
#[error("Expected a file URL, but received: {0}")]
NonFileUrl(Url),
NonFileUrl(DisplaySafeUrl),
#[error("Failed to read `--find-links` directory: {0}")]
FindLinksDirectory(PathBuf, #[source] FindLinksDirectoryError),
#[error("Failed to read `--find-links` URL: {0}")]
FindLinksUrl(Url, #[source] Error),
FindLinksUrl(DisplaySafeUrl, #[source] Error),
}
#[derive(Debug, thiserror::Error)]
@ -159,7 +159,7 @@ impl<'a> FlatIndexClient<'a> {
/// Read a flat remote index from a `--find-links` URL.
async fn read_from_url(
&self,
url: &Url,
url: &DisplaySafeUrl,
flat_index: &IndexUrl,
) -> Result<FlatIndexEntries, Error> {
let cache_entry = self.cache.entry(
@ -180,7 +180,7 @@ impl<'a> FlatIndexClient<'a> {
.client
.uncached()
.for_host(url)
.get(url.clone())
.get(Url::from(url.clone()))
.header("Accept-Encoding", "gzip")
.header("Accept", "text/html")
.build()
@ -189,7 +189,7 @@ impl<'a> FlatIndexClient<'a> {
async {
// Use the response URL, rather than the request URL, as the base for relative URLs.
// This ensures that we handle redirects and other URL transformations correctly.
let url = response.url().clone();
let url = DisplaySafeUrl::from(response.url().clone());
let text = response
.text()
@ -208,7 +208,7 @@ impl<'a> FlatIndexClient<'a> {
Ok(file) => Some(file),
Err(err) => {
// Ignore files with unparsable version specifiers.
warn!("Skipping file in {}: {err}", redacted_url(&url));
warn!("Skipping file in {}: {err}", &url);
None
}
}
@ -294,7 +294,7 @@ impl<'a> FlatIndexClient<'a> {
};
// SAFETY: The index path is itself constructed from a URL.
let url = Url::from_file_path(entry.path()).unwrap();
let url = DisplaySafeUrl::from_file_path(entry.path()).unwrap();
let file = File {
dist_info_metadata: false,
@ -303,7 +303,7 @@ impl<'a> FlatIndexClient<'a> {
requires_python: None,
size: None,
upload_time_utc_ms: None,
url: FileLocation::AbsoluteUrl(UrlString::from(&url)),
url: FileLocation::AbsoluteUrl(UrlString::from(url)),
yanked: None,
};

285
crates/uv-client/src/html.rs
View file

@ -8,6 +8,7 @@ use url::Url;
use uv_pep440::VersionSpecifiers;
use uv_pypi_types::{BaseUrl, CoreMetadata, File, Hashes, Yanked};
use uv_pypi_types::{HashError, LenientVersionSpecifiers};
use uv_redacted::DisplaySafeUrl;
/// A parsed structure from PyPI "HTML" index format for a single package.
#[derive(Debug, Clone)]
@ -27,7 +28,7 @@ impl SimpleHtml {
// Parse the first `<base>` tag, if any, to determine the base URL to which all
// relative URLs should be resolved. The HTML spec requires that the `<base>` tag
// appear before other tags with attribute values of URLs.
let base = BaseUrl::from(
let base = BaseUrl::from(DisplaySafeUrl::from(
dom.nodes()
.iter()
.filter_map(|node| node.as_tag())
@ -37,7 +38,7 @@ impl SimpleHtml {
.transpose()?
.flatten()
.unwrap_or_else(|| url.clone()),
);
));
// Parse each `<a>` tag, to extract the filename, hash, and URL.
let mut files: Vec<File> = dom
@ -278,21 +279,7 @@ mod tests {
insta::assert_debug_snapshot!(result, @r#"
SimpleHtml {
base: BaseUrl(
Url {
scheme: "https",
cannot_be_a_base: false,
username: "",
password: None,
host: Some(
Domain(
"download.pytorch.org",
),
),
port: None,
path: "/whl/jinja2/",
query: None,
fragment: None,
},
https://download.pytorch.org/whl/jinja2/,
),
files: [
File {
@ -335,21 +322,7 @@ mod tests {
insta::assert_debug_snapshot!(result, @r#"
SimpleHtml {
base: BaseUrl(
Url {
scheme: "https",
cannot_be_a_base: false,
username: "",
password: None,
host: Some(
Domain(
"download.pytorch.org",
),
),
port: None,
path: "/whl/jinja2/",
query: None,
fragment: None,
},
https://download.pytorch.org/whl/jinja2/,
),
files: [
File {
@ -395,21 +368,7 @@ mod tests {
insta::assert_debug_snapshot!(result, @r#"
SimpleHtml {
base: BaseUrl(
Url {
scheme: "https",
cannot_be_a_base: false,
username: "",
password: None,
host: Some(
Domain(
"index.python.org",
),
),
port: None,
path: "/",
query: None,
fragment: None,
},
https://index.python.org/,
),
files: [
File {
@ -452,21 +411,7 @@ mod tests {
insta::assert_debug_snapshot!(result, @r#"
SimpleHtml {
base: BaseUrl(
Url {
scheme: "https",
cannot_be_a_base: false,
username: "",
password: None,
host: Some(
Domain(
"download.pytorch.org",
),
),
port: None,
path: "/whl/jinja2/",
query: None,
fragment: None,
},
https://download.pytorch.org/whl/jinja2/,
),
files: [
File {
@ -509,21 +454,7 @@ mod tests {
insta::assert_debug_snapshot!(result, @r#"
SimpleHtml {
base: BaseUrl(
Url {
scheme: "https",
cannot_be_a_base: false,
username: "",
password: None,
host: Some(
Domain(
"download.pytorch.org",
),
),
port: None,
path: "/whl/jinja2/",
query: None,
fragment: None,
},
https://download.pytorch.org/whl/jinja2/,
),
files: [
File {
@ -566,21 +497,7 @@ mod tests {
insta::assert_debug_snapshot!(result, @r#"
SimpleHtml {
base: BaseUrl(
Url {
scheme: "https",
cannot_be_a_base: false,
username: "",
password: None,
host: Some(
Domain(
"download.pytorch.org",
),
),
port: None,
path: "/whl/jinja2/",
query: None,
fragment: None,
},
https://download.pytorch.org/whl/jinja2/,
),
files: [
File {
@ -621,21 +538,7 @@ mod tests {
insta::assert_debug_snapshot!(result, @r#"
SimpleHtml {
base: BaseUrl(
Url {
scheme: "https",
cannot_be_a_base: false,
username: "",
password: None,
host: Some(
Domain(
"download.pytorch.org",
),
),
port: None,
path: "/whl/jinja2/",
query: None,
fragment: None,
},
https://download.pytorch.org/whl/jinja2/,
),
files: [
File {
@ -673,28 +576,14 @@ mod tests {
";
let base = Url::parse("https://download.pytorch.org/whl/jinja2/").unwrap();
let result = SimpleHtml::parse(text, &base).unwrap();
insta::assert_debug_snapshot!(result, @r###"
insta::assert_debug_snapshot!(result, @r"
SimpleHtml {
base: BaseUrl(
Url {
scheme: "https",
cannot_be_a_base: false,
username: "",
password: None,
host: Some(
Domain(
"download.pytorch.org",
),
),
port: None,
path: "/whl/jinja2/",
query: None,
fragment: None,
},
https://download.pytorch.org/whl/jinja2/,
),
files: [],
}
"###);
");
}
#[test]
@ -711,28 +600,14 @@ mod tests {
"#;
let base = Url::parse("https://download.pytorch.org/whl/jinja2/").unwrap();
let result = SimpleHtml::parse(text, &base).unwrap();
insta::assert_debug_snapshot!(result, @r###"
insta::assert_debug_snapshot!(result, @r"
SimpleHtml {
base: BaseUrl(
Url {
scheme: "https",
cannot_be_a_base: false,
username: "",
password: None,
host: Some(
Domain(
"download.pytorch.org",
),
),
port: None,
path: "/whl/jinja2/",
query: None,
fragment: None,
},
https://download.pytorch.org/whl/jinja2/,
),
files: [],
}
"###);
");
}
#[test]
@ -752,21 +627,7 @@ mod tests {
insta::assert_debug_snapshot!(result, @r#"
SimpleHtml {
base: BaseUrl(
Url {
scheme: "https",
cannot_be_a_base: false,
username: "",
password: None,
host: Some(
Domain(
"download.pytorch.org",
),
),
port: None,
path: "/whl/jinja2/",
query: None,
fragment: None,
},
https://download.pytorch.org/whl/jinja2/,
),
files: [
File {
@ -807,21 +668,7 @@ mod tests {
insta::assert_debug_snapshot!(result, @r#"
SimpleHtml {
base: BaseUrl(
Url {
scheme: "https",
cannot_be_a_base: false,
username: "",
password: None,
host: Some(
Domain(
"download.pytorch.org",
),
),
port: None,
path: "/whl/jinja2/",
query: None,
fragment: None,
},
https://download.pytorch.org/whl/jinja2/,
),
files: [
File {
@ -863,21 +710,7 @@ mod tests {
Ok(
SimpleHtml {
base: BaseUrl(
Url {
scheme: "https",
cannot_be_a_base: false,
username: "",
password: None,
host: Some(
Domain(
"download.pytorch.org",
),
),
port: None,
path: "/whl/jinja2/",
query: None,
fragment: None,
},
https://download.pytorch.org/whl/jinja2/,
),
files: [
File {
@ -920,21 +753,7 @@ mod tests {
Ok(
SimpleHtml {
base: BaseUrl(
Url {
scheme: "https",
cannot_be_a_base: false,
username: "",
password: None,
host: Some(
Domain(
"download.pytorch.org",
),
),
port: None,
path: "/whl/jinja2/",
query: None,
fragment: None,
},
https://download.pytorch.org/whl/jinja2/,
),
files: [
File {
@ -994,21 +813,7 @@ mod tests {
insta::assert_debug_snapshot!(result, @r#"
SimpleHtml {
base: BaseUrl(
Url {
scheme: "https",
cannot_be_a_base: false,
username: "",
password: None,
host: Some(
Domain(
"storage.googleapis.com",
),
),
port: None,
path: "/jax-releases/jax_cuda_releases.html",
query: None,
fragment: None,
},
https://storage.googleapis.com/jax-releases/jax_cuda_releases.html,
),
files: [
File {
@ -1076,21 +881,7 @@ mod tests {
insta::assert_debug_snapshot!(result, @r#"
SimpleHtml {
base: BaseUrl(
Url {
scheme: "https",
cannot_be_a_base: false,
username: "",
password: None,
host: Some(
Domain(
"account.d.codeartifact.us-west-2.amazonaws.com",
),
),
port: None,
path: "/pypi/shared-packages-pypi/simple/flask/",
query: None,
fragment: None,
},
https://account.d.codeartifact.us-west-2.amazonaws.com/pypi/shared-packages-pypi/simple/flask/,
),
files: [
File {
@ -1179,21 +970,7 @@ mod tests {
insta::assert_debug_snapshot!(result, @r#"
SimpleHtml {
base: BaseUrl(
Url {
scheme: "https",
cannot_be_a_base: false,
username: "",
password: None,
host: Some(
Domain(
"download.pytorch.org",
),
),
port: None,
path: "/whl/jinja2/",
query: None,
fragment: None,
},
https://download.pytorch.org/whl/jinja2/,
),
files: [
File {
@ -1252,21 +1029,7 @@ mod tests {
insta::assert_debug_snapshot!(result, @r#"
SimpleHtml {
base: BaseUrl(
Url {
scheme: "https",
cannot_be_a_base: false,
username: "",
password: None,
host: Some(
Domain(
"account.d.codeartifact.us-west-2.amazonaws.com",
),
),
port: None,
path: "/pypi/shared-packages-pypi/simple/flask/",
query: None,
fragment: None,
},
https://account.d.codeartifact.us-west-2.amazonaws.com/pypi/shared-packages-pypi/simple/flask/,
),
files: [
File {

8
crates/uv-client/src/middleware.rs
View file

@ -1,19 +1,19 @@
use http::Extensions;
use std::fmt::Debug;
use uv_redacted::DisplaySafeUrl;
use reqwest::{Request, Response};
use reqwest_middleware::{Middleware, Next};
use url::Url;
/// A custom error type for the offline middleware.
#[derive(Debug, Clone, PartialEq, Eq)]
pub(crate) struct OfflineError {
url: Url,
url: DisplaySafeUrl,
}
impl OfflineError {
/// Returns the URL that caused the error.
pub(crate) fn url(&self) -> &Url {
pub(crate) fn url(&self) -> &DisplaySafeUrl {
&self.url
}
}
@ -43,7 +43,7 @@ impl Middleware for OfflineMiddleware {
) -> reqwest_middleware::Result<Response> {
Err(reqwest_middleware::Error::Middleware(
OfflineError {
url: req.url().clone(),
url: DisplaySafeUrl::from(req.url().clone()),
}
.into(),
))

54
crates/uv-client/src/registry_client.rs
View file

@ -31,7 +31,7 @@ use uv_pep440::Version;
use uv_pep508::MarkerEnvironment;
use uv_platform_tags::Platform;
use uv_pypi_types::{ResolutionMetadata, SimpleJson};
use uv_redacted::redacted_url;
use uv_redacted::DisplaySafeUrl;
use uv_small_str::SmallString;
use uv_torch::TorchStrategy;
@ -251,12 +251,12 @@ impl RegistryClient {
}
/// Return the [`BaseClient`] used by this client.
pub fn uncached_client(&self, url: &Url) -> &ClientWithMiddleware {
pub fn uncached_client(&self, url: &DisplaySafeUrl) -> &ClientWithMiddleware {
self.client.uncached().for_host(url)
}
/// Returns `true` if SSL verification is disabled for the given URL.
pub fn disable_ssl(&self, url: &Url) -> bool {
pub fn disable_ssl(&self, url: &DisplaySafeUrl) -> bool {
self.client.uncached().disable_ssl(url)
}
@ -485,10 +485,7 @@ impl RegistryClient {
// ref https://github.com/servo/rust-url/issues/333
.push("");
trace!(
"Fetching metadata for {package_name} from {}",
redacted_url(&url)
);
trace!("Fetching metadata for {package_name} from {url}");
let cache_entry = self.cache.entry(
CacheBucket::Simple,
@ -554,13 +551,13 @@ impl RegistryClient {
async fn fetch_remote_index(
&self,
package_name: &PackageName,
url: &Url,
url: &DisplaySafeUrl,
cache_entry: &CacheEntry,
cache_control: CacheControl,
) -> Result<OwnedArchive<SimpleMetadata>, Error> {
let simple_request = self
.uncached_client(url)
.get(url.clone())
.get(Url::from(url.clone()))
.header("Accept-Encoding", "gzip")
.header("Accept", MediaType::accepts())
.build()
@ -569,7 +566,7 @@ impl RegistryClient {
async {
// Use the response URL, rather than the request URL, as the base for relative URLs.
// This ensures that we handle redirects and other URL transformations correctly.
let url = response.url().clone();
let url = DisplaySafeUrl::from(response.url().clone());
let content_type = response
.headers()
@ -629,7 +626,7 @@ impl RegistryClient {
async fn fetch_local_index(
&self,
package_name: &PackageName,
url: &Url,
url: &DisplaySafeUrl,
) -> Result<OwnedArchive<SimpleMetadata>, Error> {
let path = url
.to_file_path()
@ -669,7 +666,7 @@ impl RegistryClient {
/// A local file path.
Path(PathBuf),
/// A remote URL.
Url(Url),
Url(DisplaySafeUrl),
}
let wheel = wheels.best_wheel();
@ -770,14 +767,15 @@ impl RegistryClient {
&self,
index: &IndexUrl,
file: &File,
url: &Url,
url: &DisplaySafeUrl,
capabilities: &IndexCapabilities,
) -> Result<ResolutionMetadata, Error> {
// If the metadata file is available at its own url (PEP 658), download it from there.
let filename = WheelFilename::from_str(&file.filename).map_err(ErrorKind::WheelFilename)?;
if file.dist_info_metadata {
let mut url = url.clone();
url.set_path(&format!("{}.metadata", url.path()));
let path = format!("{}.metadata", url.path());
url.set_path(&path);
let cache_entry = self.cache.entry(
CacheBucket::Wheels,
@ -818,7 +816,7 @@ impl RegistryClient {
};
let req = self
.uncached_client(&url)
.get(url.clone())
.get(Url::from(url.clone()))
.build()
.map_err(|err| ErrorKind::from_reqwest(url.clone(), err))?;
Ok(self
@ -844,7 +842,7 @@ impl RegistryClient {
async fn wheel_metadata_no_pep658<'data>(
&self,
filename: &'data WheelFilename,
url: &'data Url,
url: &'data DisplaySafeUrl,
index: Option<&'data IndexUrl>,
cache_shard: WheelCache<'data>,
capabilities: &'data IndexCapabilities,
@ -874,7 +872,7 @@ impl RegistryClient {
if index.is_none_or(|index| capabilities.supports_range_requests(index)) {
let req = self
.uncached_client(url)
.head(url.clone())
.head(Url::from(url.clone()))
.header(
"accept-encoding",
http::HeaderValue::from_static("identity"),
@ -895,7 +893,7 @@ impl RegistryClient {
let mut reader = AsyncHttpRangeReader::from_head_response(
self.uncached_client(url).clone(),
response,
url.clone(),
Url::from(url.clone()),
headers.clone(),
)
.await
@ -949,7 +947,7 @@ impl RegistryClient {
// Create a request to stream the file.
let req = self
.uncached_client(url)
.get(url.clone())
.get(Url::from(url.clone()))
.header(
// `reqwest` defaults to accepting compressed responses.
// Specify identity encoding to get consistent .whl downloading
@ -1141,7 +1139,11 @@ impl SimpleMetadata {
}
/// Read the [`SimpleMetadata`] from an HTML index.
fn from_html(text: &str, package_name: &PackageName, url: &Url) -> Result<Self, Error> {
fn from_html(
text: &str,
package_name: &PackageName,
url: &DisplaySafeUrl,
) -> Result<Self, Error> {
let SimpleHtml { base, files } =
SimpleHtml::parse(text, url).map_err(|err| Error::from_html_err(err, url.clone()))?;
@ -1220,10 +1222,9 @@ impl Connectivity {
mod tests {
use std::str::FromStr;
use url::Url;
use uv_normalize::PackageName;
use uv_pypi_types::{JoinRelativeError, SimpleJson};
use uv_redacted::DisplaySafeUrl;
use crate::{SimpleMetadata, SimpleMetadatum, html::SimpleHtml};
@ -1263,7 +1264,7 @@ mod tests {
}
"#;
let data: SimpleJson = serde_json::from_str(response).unwrap();
let base = Url::parse("https://pypi.org/simple/pyflyby/").unwrap();
let base = DisplaySafeUrl::parse("https://pypi.org/simple/pyflyby/").unwrap();
let simple_metadata = SimpleMetadata::from_files(
data.files,
&PackageName::from_str("pyflyby").unwrap(),
@ -1300,7 +1301,7 @@ mod tests {
"#;
// Note the lack of a trailing `/` here is important for coverage of url-join behavior
let base = Url::parse("https://account.d.codeartifact.us-west-2.amazonaws.com/pypi/shared-packages-pypi/simple/flask")
let base = DisplaySafeUrl::parse("https://account.d.codeartifact.us-west-2.amazonaws.com/pypi/shared-packages-pypi/simple/flask")
.unwrap();
let SimpleHtml { base, files } = SimpleHtml::parse(text, &base).unwrap();
@ -1309,7 +1310,10 @@ mod tests {
.iter()
.map(|file| uv_pypi_types::base_url_join_relative(base.as_url().as_str(), &file.url))
.collect::<Result<Vec<_>, JoinRelativeError>>()?;
let urls = urls.iter().map(Url::as_str).collect::<Vec<_>>();
let urls = urls
.iter()
.map(DisplaySafeUrl::to_string)
.collect::<Vec<_>>();
insta::assert_debug_snapshot!(urls, @r#"
[
"https://account.d.codeartifact.us-west-2.amazonaws.com/pypi/shared-packages-pypi/simple/0.1/Flask-0.1.tar.gz",