Also as a pip contributor. The only advantage of uv is to have support for parallel async extraction. If pip extracted multiple files/wheels in parallel without being blocked by the GIL, pip could easily match or outcompete uv.
I'm personally not looking forward to any deduplication/hardlink in pip. Hardlinks are very dangerous and system dependent.
It's very dangerous for empty files (init.py, empty.log yet not written). When the user edits one file, all files are modified simultaneously, all venv ever created by the user can be broken by editing one file, which is quite catastrophic.
It's also dangerous for small files with repeated content, for example random settings files that would contain a "1" or "true". Again, when the user edits one file, all files are edited and they were supposed to be different!
Hypothetically, a simple deduplication of binary files (.dll .so) should achieve 50% of the savings without significant drawbacks
I'd venture to say that pip extraction is more optimized than uv in at least one way. We have optimization for empty files (0 bytes) because there is nothing to write and checksum. uv doesn't seem to have the same optimizations, though I could be wrong, I just had a cursory look and my rust is not great. uv should probably review their treatment of empty files, it's counter productive to do any file system operation open/read/write because there is no content, it might be counterproductive to use any cache/comparison/hardlink if it takes more operations than doing nothing.
> The only advantage of uv is to have support for parallel async extraction.
This isn't true, the biggest speed ups are for the warm cases where we've already unpacked the files into the cache, as notatallshaw mentions above. We can also make low-level optimizations during resolution, e.g., in version parsing, that are not possible in pure Python code.
> If pip extracted multiple files/wheels in parallel without being blocked by the GIL, pip could easily match or outcompete uv.
I'm a bit confused by these claims about the GIL? The expensive IO operations release the GIL.
> When the user edits one file, all files are modified simultaneously
This is why we default to reflinks or copy-on-write semantics when creating environments, not all file systems support it but it's becoming more common.
> Hypothetically, a simple deduplication of binary files (.dll .so) should achieve 50% of the savings without significant drawbacks
> I'm a bit confused by these claims about the GIL? The expensive IO operations release the GIL.
FYI: The extraction of files is largely python code that doesn't release the GIL. (cf. the zipfile class from the python interpreter has large layers of abstraction with a massive overhead in python code).
Regardless, python packages are thousands of tiny files, so pip never gets to release the GIL for any meaningful duration.
If you were writing an app that only extracted large GB files, you could take advantage of some I/O operations and some zlib operations freeing the GIL for a bit. Unfortunately pip is the opposite use case, lots of tiny files.
> Interesting, I would be very surprised if this made a significant difference? but I'll take a look.
Optimizing empty files was actually quite worthwhile for pip, because about 10% of python packages are empty init files.
This might not give the same result for uv though. pip is fully linear, every single open/read/write/stat operation we removed was a direct performance gain. uv does parallel async IO, you could very well remove 10% of filesystem calls and barely affect the overall duration. :D
> FYI: The extraction of files is largely python code that doesn't release the GIL. (cf. the zipfile class from the python interpreter has large layers of abstraction with a massive overhead in python code).
TIL. I ran some benchmarks and confirmed this is the case for many small files as you'd see in wheels — the GIL is released in a meaningful way for larger files though. Thanks!
> This might not give the same result for uv though.
Yeah, I built a prototype and ran some benchmarks. It makes a big difference if the entire wheel is empty files but for any real world examples it's within noise of the baseline.
> We can also make low-level optimizations during resolution, e.g., in version parsing, that are not possible in pure Python code.
As a complete aside, uv can and does do this, but for this particular optimization I'm not sure how much absolute time it ends up saving compared to pure Python in real world resolution scenarios.
uv's total memory usage isn't that much leaner than pip's, and for parsing speed it turned out that the library pip uses, packaging, was just very unoptimized at the time uv launched. This has been significantly addressed since then:
At this point large dependency resolves in pip are spending very little of their time doing things in packaging, like version parsing. The main non-IO time spent in large resolves is now in the core resolver, resolvelib, which I hope to one day replace with my experimental resolver nab: https://github.com/notatallshaw/nab. Nab scales to large resolves much more efficiently than resolvelib (in fact I've cross-ported some of the algorithmic efficiency gains to uv already ;o)).
We didn't compare to pip at the time, but it saved a lot of absolute time for us as reported in the benchmarks from the pull request (https://github.com/astral-sh/uv/pull/789) it improved a boto3 case by 3x (30s to 10s) and our "standard" solve benchmark by 2x. It's plausible some of those gains have been reduced by other optimizations in our solver since then though.
But this was just one example optimization, we do other low-level things, like zero-copy deserialization from our cache. The point is not that we do specific things, but that we have more levers to pull to improve performance. It's great to see all the improvements happening in pip performance regardless :)
> it improved a boto3 case by 3x (30s to 10s) and our "standard" solve benchmark by 2x. It's plausible some of those gains have been reduced by other optimizations in our solver since then though.
This makes a lot of sense because pubgrub makes heavy use of version comparison, compared to simple DFS algorithms like the one resolvelib uses. A lot of Pubgrub optimizations come from finding clever ways to not need to keep comparing versions.
> But this was just one example optimization, we do other low-level things, like zero-copy deserialization from our cache. The point is not that we do specific things, but that we have more levers to pull to improve performance.
Oh yes, I agree with the general point, I was just picking on the specific example for a fun exploration of performance optimizations.
Also, FWIW, I have in my professional career, not OSS work, implemented zero-copy deserialization from cache in pure Python, there are many surprising levers in Python when you are willing to explore the weird corners of the standard library.
There are a ton of advantages to using uv over pip. I don't use UV because it's faster, although I do appreciate that. I use it because it's smart enough to manage virtual environments for each environment and tool, it can isolate to different python versions trivially, and it handles locking in a way that is actual sane.
> The only advantage of uv is to have support for parallel async extraction
Maybe the only advantage in a particular area (installing)? Because there are many other advantages.
The rich lock file for instance allows for much better cross-platform tooling. I can build a linux Docker container from a macos build host, for example, without VMs or any other emulation - simply using the cross-platform details in uv.lock and the correct tooling. I can even cross-compile numpy and other native wheels (linux -> macos, macos -> linux).
Other locker tools provide similar cross-platform information (Poetry, PDM), but pip is still lacking.
it's very bizarre for a python developer like me to read comments like this
uv has been absolutely black magic for everyone in the way it just works and works much faster, but here you are telling us that actually no it's somehow not that good
Imho deduplication belongs at the filesystem level, so the user won't (directly) see it or even know about it. Modern file systems like btrfs have the api for it.
The dedup functionality in something like zfs or btrfs isn't all that great. It tends to be extremely memory hungry and to slow things down significantly. E.g. ZFS needs around 1-5GB of ram per TB of storage and writes need to be compared to a hash table to dedup properly.
Using hints or knowledge at the app level is a much better experience if the app can tell the FS that two files are identical. The FS doesn't have to worry about hashing blocks within the file, correcting alignments, etc.
Yes, but package managers are not that great either. Better keep them as simple as possible. And you don't have to do deduplication in an online fashion; you can do it overnight, if you want, as just a simple example.
Do you specific issues with uv that makes you distrust the implementation? Otherwise that reasoning is pretty weird - it's possible to write good software, even when the existing options aren't good. Why would we ask them to limit themselves to what might make sense for worse code?
You're missing the point. No normal user program can ever do what a filesystem can do: deduplicate in a way that is hidden for the user of the filesystem. Unless you want to change everything into a black box managed by the package manager, making everything confusing for users and also maintainers.
Why would the deduplication need to be hidden from the user? Why would everything need to become a blackbox?
Hardlinks have long been a well-documented feature of many file systems. I don't see how their use in a context users largely do not meddle with is too confusing or complicated.
Because why would you make everything more complicated if the file system can handle it just fine, plus you can automatically deduplicate other files that have nothing to do with package management?
Because (going by scheme271's numbers) I might not want to waste lots of expensive RAM on something or mess around with offline deduplication, when a less resource-intensive alternative is entirely sufficient for whatever my use case is.
Maybe it's because I've been recently playing around with deduplicating subtrees in a different domain, but I don't see what makes this feature too complex to be worth the cost. Sure, symlinks and hardlinks require some additional care compared to plain files, but it's easy enough to add tests for those cases.
Well uv runs on many systems and most don't have filesystem level dedupe configured, unfortunately.
Anyways it can be both? It could let the filesystem handle it if it detected the FS has file or block dedupe and fallback to hardlinks otherwise. It doesn't but that could be done if it showed it was worth it.
...per unique TB written to datasets with deduplication enabled, of course, not the pool as a whole (even though that ram cost is indeed at the pool level).
there are plenty of useful things to be done with zfs dedupe at practically no cost if you design for it ahead of time. unfortunately for some reason people parachute into every thread where zfs comes up to say you can never ever turn it on, so people are taught not to even try.
mainly i just mean dataset configuration and arrangement within the filesystem in general as it pertains to whatever you're actually doing. if you can reliably send writes with different storage characteristics (deduplicability, compressibility, etc.) to different datasets configured accordingly, you don't have to deal with the drawbacks of trying to dedupe service logs or installed packages (which will just slowly inflate the dedupe tables for no real benefit) or compress media/encrypted files (admittedly you can usually just bail out early compressing so this side doesn't matter much but imagine the next really good compression algorithm can't fail fast for some reason). so as long as you only enable dedupe where it will actually help you because you designated a dataset for some highly duplicated purpose (and a bunch of transient writes won't harm you by growing the tables needlessly), there's really no reason not to go for it unless tight write latency constraints are also in play.
for instance, i have a pool with plenty of datasets that have dedupe off and only a crucial few with it enabled: the ones that are nfs-mounted media storage volumes for fediverse instance containers that would otherwise all duplicate all media they see in common, which is most media posted to any of them and a good bit of the remote media received by any of them. the size of the dedupe tables in memory is purely a function of the number of blocks written to those few datasets and i expect to either sunset this laptop from 2012 or migrate the pool to a fresh one long before i would be worried about the memory cost of deduplication swelling a few orders of magnitude to where i'd have to care.
> The only advantage of uv is to have support for parallel async extraction. If pip extracted multiple files/wheels in parallel without being blocked by the GIL, pip could easily match or outcompete uv.
That sounds like it would be relatively easy to demonstrate: You could tweak uv to perform downloads/extractions sequentially and then compare its runtime with pip. Has any such comparison been done?
The original request to add a concurrency variable in uv was requested by me because, among other things, I wanted to measure the difference between uv and pip: https://github.com/astral-sh/uv/issues/3311
I'm not really following this performance discussion as I think it's gone off the rails.
> I wonder whether the next bottleneck becomes software scheduling rather than silicon
Yep, the scheduling has been a problem for a while. There was an amazing article few years ago about how the Linux kernel was accidentally hardcoded to 8 cores, you can probably google and find it.
IMO the most interesting problem right now is the cache, you get a cache miss every time a task is moving core. Problem, with thousands of threads switching between hundreds of cores every few milliseconds, we're dangerously approaching the point where all the time is spent trashing and reloading the CPU cache.
That's the one. Funny thing, it's not actually clickbait.
The bug made it to the kernel mailing list where some Intel people looked into it and confirmed there is a bug. There is a problem where is the kernel allocation logic was capped to 8 cores, which leaves a few percent of performance off the table as the number of cores increase and the allocation is less and less optimal.
It's classic tragedy of the commons. CPU have got so complicated, there may only be a handful of people in the world who could work and comprehend a bug like this.
> These sorts of core-density increases are how I win cloud debates in an org.
The core density is bullshit when each core is so slow that it can't do any meaningful work. The reality is that Intel is 3 times behind AMD/TSMC on performance vs power consumption ratio.
People would be better off having a look at the high frequency models (9xx5F models like the 9575F), that was the first generation of CPU server to reach ~5 GHz and sustain it on 32+ cores.
Intel seem to be deliberately hiding the clock frequency of this thing, the xeon-6-plus-product-deck.pdf has no mention of clock frequency or how LLC is shared.
Not competitive at all. It's easily visible on the laptop lines, where the same GPU manufactured on TSMC has 3 times the power/performance ratio compared to the Intel one.
Putting more cores is just another desperate move to play the benchmark. Power is roughly quadratic with frequency, every time you fall behind competition, you can double the number of cores and reduce the frequency by 1.414 to compensate.
Repeat a few times and you get CPU with hundreds of cores, but each core is so slow it can hardly do any work.
??? GPU vs CPU workloads are completely different. Comparing Panther Lake iGPU vs Ryzen iGPU is not going to tell you much about how high density server CPU performance will work out.
The Panther Lake vs Ryzen laptop performance comparisons show that Pather Lake does well, basically trading against top end Ryzen AI laptop chips in both absolute performance, and performance per watt.
If you're not aware, Intel has released a lineup of laptops, with some models having the GPU made by them and some having the same GPU made by TSMC. That makes the comparison very direct. TSMC can deliver nearly 3 times the power/performance.
GPU and CPU manufacturing is the same thing, same node, same result. GPU is always maximizing perf/power ratio because it's embarrassingly parallel, leaving no room to game the benchmark. CPU can be gamed by having a single fast core, that drops performance in half as soon as you use another core.
> Does anyone actually know why they don't offer a symmetric product like the niche fibre ISPs?
Short version: The UK regulator OFCOM defines ultrafast internet as 30 Mbps download speed. That's why UK internet providers (openreach and related) have deals starting as low as 30 Mbps and they can't be arsed to provide a faster speed (unless you pay £££).
> I believe that I have noticed that smaller games (~a few hundred MB or maybe a GB or two) will download quite a bit slower than large games, but I'm not very confident in that observation.
You can see that on the HellDivers screenshot, it takes 20 seconds to reach 500 Mbps, because TCP takes a while to adjust the bandwidth and is very conservative. TCP and home computers are not designed to make use of gigabit connections.
> ...it takes 20 seconds to reach 500 Mbps, because TCP takes a while to adjust the bandwidth and is very conservative. TCP and home computers are not designed to make use of gigabit connections.
I very much doubt that that is an artifact of TCP. I can go from nothing to 10gbit/s symmetric in 100->200ms when running Iperf3 over TCP against another one of my LAN hosts.
And, back when I had a 1.5gbit/s Internet downlink, it took far, far less than 20 seconds to reach > 500mbps for big Steam downloads and other such well-provisioned things.
Old houses should have 1 extra socket in the master bedroom at the very least, because the master of the house was expected to plug a phone in there, back in the days. (my parents and grandparents all have one).
Incidentally, this is likely to be the furthest room on the furthest floor, so it can be a good place to add a wifi access point for coverage.
Depends on what you class as “old”. Remember that a great many British homes are 50+ years old. You certainly wouldn’t have considered having multiple phones in a house when they were built. So the extra socket was added after it was built.
Adding extension sockets was a very easy job. So easy that many homeowners did it themselves.
So it’s very likely your parents and grandparents bedroom phone wasn’t part of the original wiring.
> the cables don't have a little chip or anything saying "I'm not suitable for high speed" the card will figure out whether this looks plausible and just do it.
You're actually wrong on all of that ^^
The cables actually have a rating to say what they are suitable to. See the markings on the cable: category Cat5/Cat5e/Cat6 + frequency range 100/250 Mhz + insulation UTP/FTP/STP/mix.
Ethernet cards don't negotiate, they typically only check whether the pairs can transmit any signal. You could end up in a situation where they go for gigabit and it doesn't work well.
Fortunately, the main issue for signal transmission is loss over distance. Ethernet is designed to work over 100m every time in a noisy industrial environment. You've got a pretty good chance for it to work on a short run, even with poor cables.
The alternatives being discussed ADSL/VDSL/G.hn actually detect the capability of the medium and adjust the transmission rates and frequency to give the maximum possible speed. IMO they are much more advanced technologically and much more interesting. (Ethernet is doing exactly 250 Mbps on one pair, G.hn can do up to 1700 Mbps on the same pair, automatically adjusted, the article is getting 1300 Mbps which is insane!)
Worthwhile to point out: The Cat5 cable required for gigabit Ethernet is merely twisted pairs with no insulation, which is pretty much a dumb basic cable (with 8 wires). That's why any cable can work in practice.
I don't know how possible it is to find a really bad cable (untwisted) and it might work on a short length anyway. (Your 1980s office cabling must have been 8 wires if you were able to get gigabit later, so it was far beyond basic phone wires or Cat1 from the time).
Sure, they will have been bundles of 4 pairs and I suppose we could say that is a matter of luck, it will have been installed from the outset in anticipation of networking - there's a period in the late 1980s when everybody is iterating on what will soon become 10baseT and the people in that building would have known all about it - but there's no reason back then to know 4 pairs will be an auspicious choice rather than 3 or 6.
So yes, those cables though they weren't Cat 5e because it didn't exist when they were manufactured, also were not basic phone cables, and I believe when the building was formally opened it had "ground breaking" 10Mbit Ethernet to every laboratory.
It's a setup seen in a lot of new builds flat from the 2000s and 2010s, which is a very large amount of the housing stock in London for flats (There has been so many constructions!).
It might have detected the wrong country/city for you. Check Settings -> Downloads -> Region
Otherwise it's just your WiFi being patchy. I think Steam is doing "friendly" bulk download, it slows down before the connection is saturated, to avoid disconnecting your wife/mum/siblings watching Youtube or on a videoconference.
I'm personally not looking forward to any deduplication/hardlink in pip. Hardlinks are very dangerous and system dependent.
It's very dangerous for empty files (init.py, empty.log yet not written). When the user edits one file, all files are modified simultaneously, all venv ever created by the user can be broken by editing one file, which is quite catastrophic.
It's also dangerous for small files with repeated content, for example random settings files that would contain a "1" or "true". Again, when the user edits one file, all files are edited and they were supposed to be different!
Hypothetically, a simple deduplication of binary files (.dll .so) should achieve 50% of the savings without significant drawbacks
I'd venture to say that pip extraction is more optimized than uv in at least one way. We have optimization for empty files (0 bytes) because there is nothing to write and checksum. uv doesn't seem to have the same optimizations, though I could be wrong, I just had a cursory look and my rust is not great. uv should probably review their treatment of empty files, it's counter productive to do any file system operation open/read/write because there is no content, it might be counterproductive to use any cache/comparison/hardlink if it takes more operations than doing nothing.
reply