Docker the company bet big on Swarm being the de facto container orchestration platform for businesses. It just got completely overshadowed by k8s. Swarm continues to exist and be actively developed, but it’s doomed to fade into obscurity.
Async and Django don't mix well and I honestly see the whole Django Async as wasted resources, all those "a" prefixed functions etc.
To be honest, I never liked the way async is done in python at all.
However, I love Django and Python in general. When I need "async" in a http cycle flow, I use celery and run it in background.
If client side needs to be updated about the state of the background task, the best is to send the data to a websocket channel known to the client side. Either it's Chat response with LLM or importing a huge CSV file.
Simple rule for me is, "don't waste HTTP time, process quick and return quick".
> If client side needs to be updated about the state of the background task, the best is to send the data to a websocket channel known to the client side.
Django should just not be used period. Fast API + Uvicorn is all you need these days. It does the async for you.
With LLMS, you shit out working production ready web apps in 2 days now that are quite performant, as long as you don't care about code maintainability long term.
FastAPI+Unicorn is good but it is barebones compared to what django already has. I can see why someone with experience in latter would not choose a new stack.
Also performance wise FastAPI + uvicorn have many pitfalls as well, most of them being of asyncio.
Ooooh that's a neat one. I really like the hard links.
On my machine, there are like 100s of not thousands of venvs.
I simply have all of them under ~/.python_venvs/<project_name>/
Does that mean, no matter how many projects I install pytorch and tensoflow and huggingface and all the heavy machinery, they'll be counted only once as long as they're unique?
If that's the case, then I can leave my habit of pip and move to uv.
This is something that always bugged my mind about virtual environments in almost all the package managers.
"Does that mean, no matter how many projects I install pytorch and tensoflow and huggingface and all the heavy machinery, they'll be counted only once as long as they're unique?"
I think so, based on my understanding of how this all works. You may end up with different copies for different Python versions, but it should still save you a ton of space.
Just an update for whoever ends up on this comment.
This feature works as long as your venv that uv creates and the uv cache (in user home directory, or anywhere else that is configured to keep the cache folder) are both on the same filesystem.
The drive where the venv was created was NTFS and the user home directory where uv cache exists was ext4 file system.
Therefore the caching didn't work and a warning was shown by uv hinting to the problem.
1. Write code that crosses a certain complexity treshold. Let's say tou also need compiled wheels for a performance critical section of a library that was written in Rust, have some non-public dependencies on a company-internal got server
2. Try deploying said code on a fleet of servers whose version and exact operating system versions (and python versions!) are totally out of your control. Bonus points for when
your users need to install it themselves
3. Wait for the people to contact you
4. Now do monthly updates on their servers while updating dependencies for your python program
If that was never your situation, congrats on your luck, but that just means you really weren't in a situation where the strengths of uv had played out. I had to wrestle with this for years.
This is where uv shines. Install uv, run with uv. Everything else just works, including getting the correct python binary, downloading the correct wheel, downloading dependencies from the non-public git repo (provided the access has been given), ensuring the updates go fine, etc.
This explains a lot for me. On the server side, all my for-pay stuff is deployed using Docker. We have a single Python environment and complete control over it. We do multistage for compilation.
Client side, we don't get the privilege of deploying code: we need to build installers, which means again we have complete control over the environment because we package Python and all associated dependencies.
I'm sure there are marginal benefits to uv even with the above scenarios (better dependency management for example), but it seems that there's a middle ground here which I have largely avoided which is where uv really shines.
Yeah makes sense, with docker in the mix the things uv brings are less interesting, although using uv for small one-off scripts is also an interesting application (there is a way of making uv your shebang, declaring sependencies within the python file and essentially getting a uv-ran python script that will auto-download the needed dependencies).
Over the years I encountered many situations where other solutions (pip+pyenv, poetry, easy_install) lead to hour long stops in dependency hell. Meanwhile uv just works. Even the most complicated projects I transfered over since I decided to make the switch worked first try.
I am not the person who has to go for the newest shiniest thing just because, but if that new shiny thing does the job instead of wasting my time sign me up.
TBH I feel the same. And for development on my laptop, that seems fine. For the Python package I'm working on how, a single run of pytest takes less than five seconds.
Where things get annoying is when I push to GitHub and Tox runs through GitHub Actions. I've set up parallel runs for each Python version, but the "Prepare Tox" step (which is where Python packages are downloaded & installed) can take up to 3 minutes, where the "Run Tox" step (which is where pytest runs) takes 1½ minutes.
GitHub Actions has a much better network connection than me, but the free worker VMs are much slower. That is where I would look at making a change, continuing to use pip locally but using uv in GitHub Actions.
Run a program should never ever require more than a single simple run command.
If your project requires creating an env and switching to shit and then running it’s a bad program and you should feel bad.
Quite frankly the fact that Python requires explaining and understanding a virtual environment is an embarrassing failure.
uv run foo.py
I never ever want running any python program to ever require more than that. And it better work first time 100%. No missing dependencies errors are ever permitted.
Also, Conda can fucking die in a fire. I wil never ever ever install conda or mini-conda onto my system ever again. Keep those abominations away.
I didn’t downvote you, but the “this tool is bad and if you take the time to argue with me you’re a Rust cultist” line is a bit tiresome. Damned if you do, damned if you don’t.
It’s a bit like if anyone who said you should switch to desktop Linux got yelled at for being in the pocket of Big Systemd.
“if you like it, use it” is well and good, but haranguing people who explain why they like/use what they use is just as lame as the purported cult defense of uv or whatever tool is popular.
I dunno man, fads and stupid fixations happen in software sometimes, but most of the time hyped tools are hyped because they’re better.
If it sold itself on its merits I don’t think we’d see all these fawning posts about it. It’s a Rust fan thing. You can see how any criticism gets treated. I’m sure it works for some people and obviously if it does, then great. But it’s got this same weird cult following and pretend talk of speed that lots of Rust stuff has. It’s getting a little tiring. If you like it, use it, evangelizing is obnoxious.
I'm not a Rustacean, but I'll tell you what merited me installing uv (via its MacPorts package) last week.
I had decided to do something via a one-off Python script. I wanted to use some Python packages for the script (like `progressbar2`). I decided to use Inline Script Metadata[0], so I could include the package dependencies at the top of the script.
I'm not using pipenv or poetry right now, and decided to give uv a try for this. So I did a `sudo port install uv`, followed by a `uv run myscript.py --arguments`. It worked fine, making & managing a venv somewhere. As I developed the script, adding & changing dependencies, the `uv run …` installed things as needed.
After everything was done, cleanup was via `uv cache clean`.
Will I immediately go and replace everything with uv? No. As I mentioned in another post, I'll probably next look at using uv in my CI runs. But I don't feel any need to rush.
Yeah, but it sold itself on its merrits. That is the point. Maybe venv and pip works fine for some toy projects that are deployed on the developer controlled OS without regular dependency updates, but let me assure you I had hours of fights with updating python services with complex needs on Debian boxes from various ages while ensuring whst I ran as a dev is the stuff that is guaranteed to run in production.
With uv it just works and that in a fraction of the time. Where before updates would mean to mentally prepare that a thing that should take 5 seconds in the best and 15 minutes in the worst case could occupy my whole day, it has now become very predictable.
I don't care what it is written in. It works. If you think people love it because it was written in some language it just means you never had a job where what uv brings was really needed and thus you can't really judge its usefulness.
Also one has to chuckle at the notion that re-writing package management in Rust is some kind of fanbois with hammers looking for nails activism. Rust is almost certainly the best option for this in the 2020s, especially for a package ecosystem as deranged as Python's.
By this point I feel reminded of a former collegue I ate lunch with, who would repeatedly make jokes about how "vegans constantly need to talk about their veganism". During our shared time he brought that topic up probably a hundred times, while the single time veganism was brought up by anybody was by a female intern after she was asked by him why she doesn't like to try the meat.
This is what reflexive criticism of Rust starts to feel like. I get that this somehow grinds some peoples gears, but come on. Who cares what it is written in if it is good software. And as someone who tried all major ways of dependency management in Python I have to say it is the best. Don't like that it is written in Rust for ideological reasons? Go ahead and write it better in C¹ or whatever.
¹: Nothing against C, I regularily use it for embedded programming, but it appears many of the loudest Rust allergics come from there
What’s more obnoxious is dismissing it as a rust fanboy conspiracy. Know what I like about it? `uv install .` with a few dozen top-level dependencies takes under a second on my machine. All the tools work as documented all the time. `uv run …` is nearly instant. Those are the reasons I like it.
I couldn’t care less that it’s written in rust. It could be conjured from malbolge for all I care. It works as advertised, whatever it’s written in.
I use golang, rust and c++ here and there, but majority of my time is spent working in Python projects. I'm not alien to the concept of speed and performance, especially the tooling around them.
While I like the idea of pip or uv to be insanely fast, I still don't see it revolutionize my development experience.
Installing and uninstalling package is not something I do every 1 to 10 minutes. It doesn't save me any much time. Also, activating a venv is once a session in terminal and sometime a week goes by without ever activating a venv, because the IDE does that automatically on whatever I do.
That's why, personally for me it really doesn't change much.
Where I like things being fast in my development time is pre-commit and linting, where ruff shines. Which that I also don't use, even though I work on a small-medium 600k LoC project, I only pass the changed files to isort, flake8 and black and it's all done in less than 5 seconds.
To me, the only advantage of uv is being fast, which is something I haven't been bothered with so far, where 99% of things happen in less than 1 or max couple of seconds.
Ever had customers deploy your project on 4 different debian versions without docker? Probably not, because there are problems lurking you didn't even know could exist. And 99% of them are gone with uv.
You don't have to believe the parts you made up. The comment you're replying to didn't actually state that a single customer was deploying to four different Debian versions. As written, the comment only requires you to believe that four Debian versions were in use collectively across the customer base.
Granted, but that is relevant to the point I made in which way?
In reality you will have people running different OS versions. Maybe not within one org, but across users? For sure. If you are not using containers for one reason or another uv has shown to be a very good, reliable and easy to use way of dealing with the issues like these.
Additionally it has some other benefits when it comes to dev dependecies etc. Not that you couldn't somehow manage without it, it just makes certain things so much less pain as they were.
The speed is nice. It’s not the only advantage, though. It’s so pleasant being able to `uv run [git-repo]` and having it work. The same design that makes it so fast makes it delightfully good at doing other complicated things.
As an occasional Python user that loves uv -- I do care that it's in Rust, because Rust enforces a separation between mutable and immutable state that consistently leads to higher-quality outcomes.
I don’t totally not care that it’s written in Rust. That means there are whole classes of bugs it won’t have, and it’s probably rigorous about data structure and state management.
Async or not in modules is a huge pain in the ass for web-dev, but thankfully a lot of Python still isn't web dev. As a Data Scientist you can live your life peacefully without ever worrying about this. But I can see why web devs like it, even though personally I really don't want to see anymore Javascript sneaking inty my Python. Especially when there is already support for IO bound concurrency elsewhere in the language. If I want to do JS syntax, I'll fucking use JS. And I really don't want to see Python go the C++ route where it just wants to be everything and do everything so that you end up with so many possible approaches to the same problem that two devs can't read each others code anymore.
I'm relatively new to Python - how does one do concurrent IO without async/await?
My main complaint, though, about Python async is - because it is opt-in I never know if I forgot a sync IO call somewhere that will block my worker. JS makes everything async by default and there is effectively no chance of blocking.
Async was a soft keyword for many, many years in order to maintain compat. And before that, asyncio used yield to be even more compatible.
They took a decade to solidify that. At some point, you have to balance evolution and stability. For a language as popular as Python, you can not break the world every week, but you can't stay put for 5 years.
I know/heard there are "some" (which I haven't seen by the way) libraries that depend on import side effects, but the advantage is much bigger.
First of all, the circular import problem will go away, especially on type hints. Although there was a PEP or recent addition to make the annotation not not cause such issue.
Second and most important of all, is the launch time of Python applications. A CLI that uses many different parts of the app has to wait for all the imports to be done.
The second point becomes a lot painful when you have a large application, like a Django project where the auto reload becomes several seconds. Not only auto reload crawls, the testing cycle is slow as well. Every time you want to run test command, it has to wait several seconds. Painful.
So far the solution has been to do the lazy import by importing inside the methods where it's required. That is something, I never got to like to be honest.
Maybe it will be fixed in Python 4, where the JIT uses the type hints as well /s
Top-level code should not be able to fail except in incredibly deterministic ways that are tested during development. Failing fast is not as good as not failing at all. Lazy imports mean the power to avoid importing things that don't need to be imported at all on this run. Good design also cares about performance to some extent. On my machine, asking pip to do literally nothing takes several times as long as creating a new virtual environment --without-pip .
Python really seems like a bad fit for that. So your imports succeed, what now? Do they have all the functions or fields your program needs? Those are still resolved at the last possible moment. If you want to be sure your program actually runs you will have to write and run tests with and without lazy imports.
venvs are namespace isolation, they are like containers.
Even in huge monorepos you can just use something like a Makefile to produce a local venv using PHONY and add it to clean too
This is how I actually test old versions of python, with versioned build targets, cython vs ...
You can set up almost any IDE to activate them automatically too.
The way to get you coworkers to quit complaining is to automate the build env setup, not fighting dependency hell, which is a battle you will never win.
It really is one of the most expensive types of coupling.
Would recommend you to install pyenv[1]. It was very useful when my team had to update a lot of projects using <=3.10 to 3.11
[1] https://github.com/pyenv/pyenv
I have multiple versions of Python built from source. If I want to test what my code will do on a given version, I spin up a new venv (near instantaneous using `--without-pip`, which I achieve via a small Bash wrapper) and try installing it (using the `--python` option to Pip, through another wrapper, allowing me to reuse a single global copy).
No matter what tooling you have, that kind of test is really the only way to be sure anyway.
Some libraries break across different versions of Python for a variety of reasons.
Pinning python version with asdf (in conjunction with a venv) gets you reasonably far in ensuring a certain project works across a lot of people in a team.
A venv does not actually install a different Python interpreter. It's bound to the Python version that created it. You cannot make a Python 2.7 venv using a Python 3 interpreter. You need Python 3.10 to create a Python 3.10 venv.
There are plenty of situations where the Python interpreter version matters. As a non-exhaustive list, you have libraries that compile code, non-Python languages that link to cpython, build scripts that do different things depending on wheel/setup/other-bundled-stuff, Python code that uses removed compat shims like importlib-metadata...
If you haven't run into one of those situations yet, congratulations. I've been through this already, and making a reproducible environment does require first installing a pinned version of the Python interpreter and THEN setting up a venv using that particular interpreter.
> You need Python 3.10 to create a Python 3.10 venv.
Yes, but this doesn't need to cause a problem for those of us using bare-bones tooling. Speaking for myself, I just... run venv with that version of Python. Because I have a global installation of everything from 3.5 to 3.13 inclusive, plus 2.7 (just so I can verify what I post online about how things used to work in 2.x). And I never install anything to my base Python versions, because my system Python is Apt's territory and the ones I built from source are only and specifically there to make venvs with.
This seems like a solved problem with pyenv which is very popular. You can also include a `.python-version` in your git repo root dir to automatically use the correct python interpreter version when in the scope of that repo.
How do you expect to support violating that contract, where the OS vendor, package maintainers issue updates and security fixes, while you maintain your code, with the realization that Python releases have two years of bug fixes and three additional years of security fixes. Are you actually forced into this model or are you just adding to code debt refers and adding to the accumulation of poorly written or unmaintainable code that makes future changes even harder than they are today?
Using OS package managers helps you avoid accumulating so much code debt, making moving to newer versions the easy path actually results in better written and maintainable code.
If you just listen to depreciation warnings, and prioritize trying to stay close to the current released versions, and insisting that running on unsupported versions is an incident and not hiding it under the rug, things get better over time.
That said, the Python direct download paths are very stable.
I would still recommend creating packages from those downloads, while there are possibly better options, fpm has treated me well for well over a decade.
Looking in homebrew, which is probably one of the bigger unknowns, they have targets for everything going back to 3.8, with all major Linux distros supporting farther back and obviously VM/containers are an option there too.
I have run into the situations above, but the use of unsupported and/or end-of-life software is not something you intentionally help an organization do.
Especially with the actions by several agencies to bust EULA's
While IANAL we will see what happens in the courts, when the CISA labels something as "This dangerous practice is especially egregious in technologies accessible from the Internet"
There is a very big difference between sustainment needs and active development, one is a reality, the other could possibly be framed as "Gross negligence", invalidating any EULA protections in many parts of the country.
Obviously if you are running other peoples software the calculus of trade-offs changes.
But I have done the Python 2.7 to 3.x migration for several open source and commercial projects in the past. I can promise you that the time you waste on using 2.7 with software you control is far more expensive than updating.
If you are stuck on 2.7 on a product under active development, which has been EOL for 5 years and was warned to be going away for almost a decade prior to that...the problem you have is not with any 3rd party dependancies, the problem is with your organization.
No 3rd party vendor solution will fix those problems for you.
This is more or less my experience, but I think in part it took a while for pip to actually get into a usable position, hence some of the proliferation of other options.
That might be fine in your context. People's problems are real, though. What they're almost always missing is separating the source code from the compiled output ("lock files"). Pick a tool to help with that, commit both files to your ("one's") project, problem solved.
People end up committing either one or the other, not both, but:
- You need the source code, else your project is hard to update ("why did they pick these versions exactly?" - the answer is the source code).
- You need the compiled pinned versions in the lock file, else if dependencies are complicated or fast-moving or a project goes unmaintained, installing it becomes a huge mindless boring timesink (hello machine learning, all three counts).
Whenever I see people complaining about python dependencies, most of the time it seems just that somebody lacked this concept, or didn't know how to do it with python, or are put off by too many choices? That plus that ML projects are moving quickly and may have heavy "system" dependencies (CUDA).
In the source code - e.g. requirements.in (in the case of pip-tools or uv's clone of that: uv pip compile + uv pip sync), one lists the names of the projects one's application depends on, with a few version constraints explained with comments (`someproject <= 5.3 # right now spamalyzer doesn't seem to work with 5.4`).
In the compiled output - i.e. the lock files (pip-tools or uv pip sync/compile use requirements.txt for this) one makes sure every version is pinned to one specific version, to form a set of versions that work together. A tool (like uv pip compile) will generate the lock files from the source code, picking versions that are declared (in PyPI metadata) should work together.
My advice: pip-tools (pip-compile + pip-sync) does this very nicely - even better, uv's clone of pip-tools (uv pip compile + uv pip sync), which runs faster. Goes nicely with:
- pyproject.toml (project config / metadata)
- plain old setuptools (works fine, doesn't change: great)
- requirements.in: the source for pip-tools (that's all pip-tools does: great! uv has a faster clone)
- pyenv to install python versions for you (that's all it does: great! again uv has a faster clone)
- virtualenv to make separate sandboxed sets of installed python libraries (that's all it does: great! again uv has a faster clone)
- maybe a few tiny bash scripts, maybe a Makefile or similar just as a way to list out some canned commands
- actually write down the commands you run in your README
PS: the point of `uv pip sync` over `uv pip install -r requirements.txt` is that the former will uninstall packages that aren't explicitly listed in requirements.txt.
uv also has a poetry-like do-everything 'managed' everything-is-glued-together framework (OK you can see my bias). Personally I don't understand the benefits of that over its nice re-implementations of existing unix-y tools, except I guess for popularizing python lockfiles - but can't we just market the idea "lock your versions"? The idea is the good part!
That's been my experience too. The main complaint I hear about this workflow is that venvs can't be moved without breaking. I just rebuild my venv in each new new location, but that rebuild time can add up for projects with many large scientific packages. Uv solved that pain point for me, since it provides a "pip install" implementation that runs in a fraction of the time.
Anyone remember the leftpad fiasco in the node ecosystem? That could happen in any dependency system that allows owners to unpublish dependencies and that's one risk users must weigh when adding them.
Yeah, I assume pinning the version is something everyone does? Or probably many just don't and will have those "python deps management is a mess drama".
TBH, I've seen tutorials or even some companies simply do `pip freeze > requirements.txt` :shrug: which is a mess.
I was reacting to a comment that said a dependencies.txt and a venv was enough, so in the model I critisized there is no pyproject.toml
> Regardless, majority of the times, deployment is done via Docker.
What, generally? In your peer circle? I'd say [citation needed] — docker has the problematic habit of inducing more moveable parts that can bite you, so I know many who — if given the choice — would rather deploy python projects without it. Having deployed many python application on actual bare metal and VMs alike I'd say the ratio of Docker VS just Python is more like 1:8.
