The original request to add a concurrency variable in uv was requested by me because, among other things, I wanted to measure the difference between uv and pip: https://github.com/astral-sh/uv/issues/3311
I'm not really following this performance discussion as I think it's gone off the rails.
It didn't really become that popular. Not as popular as poetry at least. I think the poetry guy famous, but I don't really remember. He was very opinionated, and pdm addressed many of the issues people had with poetry.
> The only advantage of uv is to have support for parallel async extraction.
This isn't true, the biggest speed ups are for the warm cases where we've already unpacked the files into the cache, as notatallshaw mentions above. We can also make low-level optimizations during resolution, e.g., in version parsing, that are not possible in pure Python code.
> If pip extracted multiple files/wheels in parallel without being blocked by the GIL, pip could easily match or outcompete uv.
I'm a bit confused by these claims about the GIL? The expensive IO operations release the GIL.
> When the user edits one file, all files are modified simultaneously
This is why we default to reflinks or copy-on-write semantics when creating environments, not all file systems support it but it's becoming more common.
> Hypothetically, a simple deduplication of binary files (.dll .so) should achieve 50% of the savings without significant drawbacks
> I'm a bit confused by these claims about the GIL? The expensive IO operations release the GIL.
FYI: The extraction of files is largely python code that doesn't release the GIL. (cf. the zipfile class from the python interpreter has large layers of abstraction with a massive overhead in python code).
Regardless, python packages are thousands of tiny files, so pip never gets to release the GIL for any meaningful duration.
If you were writing an app that only extracted large GB files, you could take advantage of some I/O operations and some zlib operations freeing the GIL for a bit. Unfortunately pip is the opposite use case, lots of tiny files.
> Interesting, I would be very surprised if this made a significant difference? but I'll take a look.
Optimizing empty files was actually quite worthwhile for pip, because about 10% of python packages are empty init files.
This might not give the same result for uv though. pip is fully linear, every single open/read/write/stat operation we removed was a direct performance gain. uv does parallel async IO, you could very well remove 10% of filesystem calls and barely affect the overall duration. :D
> FYI: The extraction of files is largely python code that doesn't release the GIL. (cf. the zipfile class from the python interpreter has large layers of abstraction with a massive overhead in python code).
TIL. I ran some benchmarks and confirmed this is the case for many small files as you'd see in wheels — the GIL is released in a meaningful way for larger files though. Thanks!
> This might not give the same result for uv though.
Yeah, I built a prototype and ran some benchmarks. It makes a big difference if the entire wheel is empty files but for any real world examples it's within noise of the baseline.
> We can also make low-level optimizations during resolution, e.g., in version parsing, that are not possible in pure Python code.
As a complete aside, uv can and does do this, but for this particular optimization I'm not sure how much absolute time it ends up saving compared to pure Python in real world resolution scenarios.
uv's total memory usage isn't that much leaner than pip's, and for parsing speed it turned out that the library pip uses, packaging, was just very unoptimized at the time uv launched. This has been significantly addressed since then:
At this point large dependency resolves in pip are spending very little of their time doing things in packaging, like version parsing. The main non-IO time spent in large resolves is now in the core resolver, resolvelib, which I hope to one day replace with my experimental resolver nab: https://github.com/notatallshaw/nab. Nab scales to large resolves much more efficiently than resolvelib (in fact I've cross-ported some of the algorithmic efficiency gains to uv already ;o)).
We didn't compare to pip at the time, but it saved a lot of absolute time for us as reported in the benchmarks from the pull request (https://github.com/astral-sh/uv/pull/789) it improved a boto3 case by 3x (30s to 10s) and our "standard" solve benchmark by 2x. It's plausible some of those gains have been reduced by other optimizations in our solver since then though.
But this was just one example optimization, we do other low-level things, like zero-copy deserialization from our cache. The point is not that we do specific things, but that we have more levers to pull to improve performance. It's great to see all the improvements happening in pip performance regardless :)
> it improved a boto3 case by 3x (30s to 10s) and our "standard" solve benchmark by 2x. It's plausible some of those gains have been reduced by other optimizations in our solver since then though.
This makes a lot of sense because pubgrub makes heavy use of version comparison, compared to simple DFS algorithms like the one resolvelib uses. A lot of Pubgrub optimizations come from finding clever ways to not need to keep comparing versions.
> But this was just one example optimization, we do other low-level things, like zero-copy deserialization from our cache. The point is not that we do specific things, but that we have more levers to pull to improve performance.
Oh yes, I agree with the general point, I was just picking on the specific example for a fun exploration of performance optimizations.
Also, FWIW, I have in my professional career, not OSS work, implemented zero-copy deserialization from cache in pure Python, there are many surprising levers in Python when you are willing to explore the weird corners of the standard library.
If you want to open an issue with verbose logs (`-vv`) I'd be happy to look into it. You can set `UV_LOG_CONTEXT=1` to emit timing information.
If you're running Python scripts, I'd recommend using `uv lock --script <path>` to generate a lockfile — we don't do that by default for scripts yet but that will avoid unexpected upgrades.
We want to support that workflow / experience but we want to design it differently because there are a lot of nuances to pip's interface and the pip maintainers have encouraged us to rethink it.
It's also non-trivial to implement because of how our cache design differs from pip's.
Yes! It's in progress — it's already there, just hidden as we iterate on it.
(but as a note, the initial `uv lock` will already attempt to choose the highest possible versions and `uv lock --upgrade` will do so again, requests for `uv upgrade` are generally centered around changing the constraints in the `pyproject.toml`)
Unfortunately dynamic metadata is not efficient for package resolution, so we don't want to encourage it. Instead, we want to design a better workflow for using source control for versioning, but haven't had the time to do so.
Interesting, a better way of doing versioning would be nice. The tool I use requires a git repo setup with at least 1 commit, annoying for brand new projects. I'll be eagerly waiting to see it eventually.
At what point is that the case? When a package is built, it's version is determined at that moment from whatever dynamic source, and is then written into the metadata in a static way (or surely could be if for some weird reason it isn't).
reply