mirror of
https://github.com/astral-sh/uv.git
synced 2025-11-03 21:23:54 +00:00
5219d37250
This PR adds initial support for [rkyv] to puffin. In particular, the main aim here is to make puffin-client's `SimpleMetadata` type possible to deserialize from a `&[u8]` without doing any copies. This PR **stops short of actuallying doing that zero-copy deserialization**. Instead, this PR is about adding the necessary trait impls to a variety of types, along with a smattering of small refactorings to make rkyv possible to use. For those unfamiliar, rkyv works via the interplay of three traits: `Archive`, `Serialize` and `Deserialize`. The usual flow of things is this: * Make a type `T` implement `Archive`, `Serialize` and `Deserialize`. rkyv helpfully provides `derive` macros to make this pretty painless in most cases. * The process of implementing `Archive` for `T` *usually* creates an entirely new distinct type within the same namespace. One can refer to this type without naming it explicitly via `Archived<T>` (where `Archived` is a clever type alias defined by rkyv). * Serialization happens from `T` to (conceptually) a `Vec<u8>`. The serialization format is specifically designed to reflect the in-memory layout of `Archived<T>`. Notably, *not* `T`. But `Archived<T>`. * One can then get an `Archived<T>` with no copying (albeit, we will likely need to incur some cost for validation) from the previously created `&[u8]`. This is quite literally [implemented as a pointer cast][rkyv-ptr-cast]. * The problem with an `Archived<T>` is that it isn't your `T`. It's something else. And while there is limited interoperability between a `T` and an `Archived<T>`, the main issue is that the surrounding code generally demands a `T` and not an `Archived<T>`. **This is at the heart of the tension for introducing zero-copy deserialization, and this is mostly an intrinsic problem to the technique and not an rkyv-specific issue.** For this reason, given an `Archived<T>`, one can get a `T` back via an explicit deserialization step. This step is like any other kind of deserialization, although generally faster since no real "parsing" is required. But it will allocate and create all necessary objects. This PR largely proceeds by deriving the three aforementioned traits for `SimpleMetadata`. And, of course, all of its type dependencies. But we stop there for now. The main issue with carrying this work forward so that rkyv is actually used to deserialize a `SimpleMetadata` is figuring out how to deal with `DataWithCachePolicy` inside of the cached client. Ideally, this type would itself have rkyv support, but adding it is difficult. The main difficulty lay in the fact that its `CachePolicy` type is opaque, not easily constructable and is internally the tip of the iceberg of a rat's nest of types found in more crates such as `http`. While one "dumb"-but-annoying approach would be to fork both of those crates and add rkyv trait impls to all necessary types, it is my belief that this is the wrong approach. What we'd *like* to do is not just use rkyv to deserialize a `DataWithCachePolicy`, but we'd actually like to get an `Archived<DataWithCachePolicy>` and make actual decisions used the archived type directly. Doing that will require some work to make `Archived<DataWithCachePolicy>` directly useful. My suspicion is that, after doing the above, we may want to mush forward with a similar approach for `SimpleMetadata`. That is, we want `Archived<SimpleMetadata>` to be as useful as possible. But right now, the structure of the code demands an eager conversion (and thus deserialization) into a `SimpleMetadata` and then into a `VersionMap`. Getting rid of that eagerness is, I think, the next step after dealing with `DataWithCachePolicy` to unlock bigger wins here. There are many commits in this PR, but most are tiny. I still encourage review to happen commit-by-commit. [rkyv]: https://rkyv.org/ [rkyv-ptr-cast]: https://docs.rs/rkyv/latest/src/rkyv/util/mod.rs.html#63-68 |
||
|---|---|---|
| .. | ||
| src | ||
| Cargo.lock | ||
| Cargo.toml | ||
| License-Apache | ||
| License-BSD | ||
| Readme.md | ||
Dependency specifiers (PEP 508) in Rust
A library for python dependency specifiers, better known as PEP 508.
Usage
In Rust
use std::str::FromStr;
use pep508_rs::Requirement;
let marker = r#"requests [security,tests] >= 2.8.1, == 2.8.* ; python_version > "3.8""#;
let dependency_specification = Requirement::from_str(marker).unwrap();
assert_eq!(dependency_specification.name, "requests");
assert_eq!(dependency_specification.extras, Some(vec!["security".to_string(), "tests".to_string()]));
In Python
from pep508_rs import Requirement
requests = Requirement(
'requests [security,tests] >= 2.8.1, == 2.8.* ; python_version > "3.8"'
)
assert requests.name == "requests"
assert requests.extras == ["security", "tests"]
assert [str(i) for i in requests.version_or_url] == [">= 2.8.1", "== 2.8.*"]
Python bindings are built with maturin, but you can also use the normal pip install .
Version and VersionSpecifier from pep440_rs are reexported to avoid type mismatches.
Markers
Markers allow you to install dependencies only in specific environments (python version, operating system, architecture, etc.) or when a specific feature is activated. E.g. you can say importlib-metadata ; python_version < "3.8" or itsdangerous (>=1.1.0) ; extra == 'security'. Unfortunately, the marker grammar has some oversights (e.g. https://github.com/pypa/packaging.python.org/pull/1181) and the design of comparisons (PEP 440 comparisons with lexicographic fallback) leads to confusing outcomes. This implementation tries to carefully validate everything and emit warnings whenever bogus comparisons with unintended semantics are made.
In python, warnings are by default sent to the normal python logging infrastructure:
from pep508_rs import Requirement, MarkerEnvironment
env = MarkerEnvironment.current()
assert not Requirement("numpy; extra == 'science'").evaluate_markers(env, [])
assert Requirement("numpy; extra == 'science'").evaluate_markers(env, ["science"])
assert not Requirement(
"numpy; extra == 'science' and extra == 'arrays'"
).evaluate_markers(env, ["science"])
assert Requirement(
"numpy; extra == 'science' or extra == 'arrays'"
).evaluate_markers(env, ["science"])
from pep508_rs import Requirement, MarkerEnvironment
env = MarkerEnvironment.current()
Requirement("numpy; python_version >= '3.9.'").evaluate_markers(env, [])
# This will log:
# "Expected PEP 440 version to compare with python_version, found '3.9.', "
# "evaluating to false: Version `3.9.` doesn't match PEP 440 rules"