The GitHub README says it builds on the Python .ZIP App format (PEP441), which in turns says you can put your app in a .ZIP file and Python will run it as an app.
I think I went this route with PyInstaller's one-file output option. But I found it to be too slow because (1) my app was a CLI app and I needed it to start, run, and end quickly, (2) I imported something from Django because it was useful (which ballooned the size), and (3) the single-file .ZIP needed to be extracted into a temporary directory, run, and then deleted every time (!!!).
Does anyone know how Pex / PEP441 deals with this? Is the extract-run-delete thing normal/standard? Is there a way to cache the extracted app?
I did find https://github.com/google/mount-zip which might be useful, but you would likely have to still mount the zip manually. However, you don’t have to worry as much about cleaning up after yourself.
I wonder why isn't there an easy way to just package the interpreter inside an EXE for people that don't care about binary size just so that it would make it seamless to package up utilities written in python.
I've had automated builds running mostly untouched for years here https://github.com/Numerlor/Auto_Neutron/blob/master/pyinsta..., though it doesn't have any particularly troublesome packages other than PySide6
Fortunately, that was only the 32-bit version of Python 2.7. Using 64-bit versions or Python 3 was enough to not get flagged as malicious. I figured that out when I decided I didn't want to teach myself Go just then to deploy something that had worked the day before.
Would recommend!
The only time I really use python is when I need a quick and dirty script.
The packaging/deployment of python keeps me from using it for anything other than one-off, disposable programs.
In "standalone" mode, the executable really is just a normal program. It's actually the various fragments of the Python bytecode interpreter unrolled according to the Python bytecode of your program and the packages it uses. All the C extension modules are shared libraries in the same directory (this means you can use LGPL C extension modules like PyQt BTW) and any package data files are alongside in that directory too. This makes it seem a bit messy but in reality it's cleaner than what onefile does under the hood.
(This is what I was alluding to in my parent comment but I didn't explain myself.)
I'll certainly checkout PEX, I think that distribution of a binary is likely the largest one for Python right now. Java solved it with a JAR file, static-compiled binaries for most compiled languages.
At Google PAR files have been quite a useful way to handle this for at least 18 years now, hope this can get reasonably solved everywhere.
JARs still require a JRE to run, and the runtime needs to be invoked. The equivalent would probably be a Python zipapp (which are just Python files zipped into a .pyz the interpreter can run).
Static binaries are one advantage that languages like Go and Rust have though, yeah.
Also don’t forget that many compiled languages depend on a particular libc so you’re not totally in the clear there.
An ordinary compiled binary would be a fraction of that size, saving you time in provisioning and deployment.
Most programmers (HN is a bubble) hate learning new languages.
Docker should have never been necessary.
In your mind what would a good dynamic linker look like?
C/C++ build systems suck. But deploying a C++ program is much much easier than Python. And Linux is waaaay worse than Windows for both building and deploying imho.
I'm not saying it's the only reason to choose python now but it's definitely among the biggest reasons.
It is safe to say in 2024 that people do not want FFIs.
Why not just call the C code you've already written in C ? Because they would rather use python (or python like) syntax.
I don't think we actually disagree here. Even your point about the better C-API doesn't indicate that syntax wasn't a deciding factor, just that one of several options had better compatibility.
There are too many environemnt systems (conda, venv, I think I'm forgetting one more), not all of them build or ship on all systems, then you need to manage them, and sometimes to have the exact correct python version installed because god help you if you're on 3.12 and not 3.11; and set as PATH for pip to find the correct dependencies, but you need to do that before you set up your venv and run pip, otherwise you need to tear that down and star tover. Sometimes the dependencies build, and break, because some package is missing from your system.
It's kind of a miserable experience.
I already have a success story of helping a friend install ArchiveBox [2] with uv. ArchiveBox required an older version of Python than their system had. The quickest and easiest solution I could think of was:
pipx install uv
uv tool install --python 3.8 archivebox
These tools come and go in the Python world... It's basically like JavaScript frameworks. It seems that people recommend a different one every month
No, it's not solved! And probably will not be solved neither.
Obviously, things get better honestly, 2015 was far worst than now, but currently it's very far from perfect.
For instance, go static build are much superior. Or even Rust, with all the language issues, at least the tooling is good and the software distribution works great.
It is just a way to wrap your python code into a self bootstrapping executable: https://ofek.dev/pyapp/latest/runtime/
You can get it setup pretty easily with hatch: https://hatch.pypa.io/1.9/plugins/builder/app/
But yea I agree the problem with python is there isn’t a tool like deno that just does everything that has to do with packaging, locking, venvs, and building a cross platform binary.
You run one command that it generates a single executable that can run simultaneously on Mac Linux and windows. Pretty nice for just deploying simple Python scripts.
is go better at building trully multiplatform from linux, in practice?
It's been around for years now, is super battle tested, and user tooling continues to get easier. As a bonus, it works for not only python but other tooling as well.
Don't use pex, use shiv: https://shiv.readthedocs.io/
Pex doesn't work on windows, is slower than shiv and shiv solves the resource files problem that you will encounter with dependencies like django. To achieve this, shiv unzips the whole zipapp in a cache on disk and create a transparent proxy to it.
Still, I wish zipapps had better tooling honestly. Using either pex or shiv is more complex than it needs to be.
I hope uv will eventually give you something like "uv zipapp" and make it good and simple.
Right now, to bundle all deps for shiv you have to download first all wheels, then figure out the proper incantation to include them. And if you use WSGI, you probably want some kind of dedicated entry point.
The tech is cool, but it deserves maturing to be a true ".war for python".
Oh that’s such a weird limitation.
In the early 2000’s open source hubris, next year was always the year of the Linux desktop. Since then consumer windows matured into a totally ok OS and one with the best support for graphics and C++ development at that.
Non-windows support nowadays is a fairly strong signal of a non-serious software offering if there is no obvious reason for it. And that’s totally fine, hobby tools developed by enthusiasts rock - but they are not industrial in scope as such.
I think many people on HN would be surprised at how many orgs are using Windows servers heavily, because of their familiarity and comfort with Microsoft, or because some application requires it.
Of course they are non-tech companies using the servers internally for enterprise applications, not web servers, but there is absolutely a lot of windows server usage in corporate environments.
I think most HN readers are well aware that there are a lot of Windows servers out there, especially in the sorts of environments where it's "The Server".
That doesn't change the fact that there are orders of magnitude more Linux servers in the world, and as the post you replied to said Linux is the default assumption. Basically every container and the vast majority of VM guests are Linux. I'd be willing to bet that more Linux servers have been deployed in the time it took me to type this post than Windows servers will be deployed this week.
I'm simply trying to say that Linux is not the default assumption in many contexts where a lot of money gets spent on licenses and services for server software.
There are of course large contexts (tech companies and web servers) where Linux is the default assumption.
Twitter ran primarily OSX (for devs) and Linux (for servers).
I agree though that Windows support is an important feature for a tool like this, and it's not complete without it.
- it doesn't provide a single file for a full project with all dependencies.
- it requires internet access on the target machine to install things.
- it can't give you access to underlying API for things like wsgi.
- it requires pipx on the target machine.
- it requires a wheels made from a build, which many projects don't have (such as a lot of web projects).
I'm not a pipx expert (I'm a "I got it to work for my use case and then stopped" person :) ) but I was able to get my project working after installing a local wheel file (so i don't think you need access to the Internet to install things).
I thought the wheel file has all the dependencies since (my understanding of) pipx uses a venv for each individual wheel file/app. Certainly I'm using a bunch of packages that I needed to pip install into my development environment, but it's entirely possible that my pipx-managed wheels are using those installs, too.
I think the last two points are accurate, and I've never tried to use APIs like wsgi with it.
If you have a shiv working it stays working, assuming you have a solution to distribute the required interpreter version.
- It's slow.
- The underlying python code cannot use open() to access its own resources, but must use pkg_resources, and most packages on pypi don't do that because most dev don't even know it exists or what problem is soves.
- _ _ file _ _ is not available, breaking a lot of things. zip_safe is a build option, but actual compliance to it is hard work and rare.
Wonder what the FUSE equivalent for windows is.
A 2016 comparison by the developer of WinFsp says WinFsp is several times faster in performance tests [1]. According to the readme for Donaky [2], since then Dokany got faster.
[1] https://github.com/winfsp/winfsp/issues/19#issuecomment-2653...
[2] https://github.com/dokan-dev/dokany/blob/255e4a508b744996191...
Thanks for pointing it out.
It shouldn't be that hard to get a general 'jar/war', not just for python.
Perhaps jart's cosmolitan libc with a zip file appendage which it auto extracts and optionally caches on first use.
A heavyweight solution, but still lighter than Docker.
They don't apply to scripts, so Python is fine, but they do apply to executables.
Having to sign your zip every time you produce it would nullify the benefit of the concept. And because the checksum of the executable file change according to the content of the zip, you can't just sign the cosmopolitan part.
https://spark.apache.org/docs/latest/api/python/user_guide/p...
However EMR requires the same Docker image for the entire cluster - why not per job!?!?
But, of course, the market is has information that I don't.
this is a topic near to my heart; I have spent the last several years building my career around doing this sort of work (specifically writing tools to expand the capabilities of python code). the key idea is not that we "keep people employed as python developers", it is that python (you can also substitute ruby, javascript, php, or any other popular dynamic language) is a genuinely powerful and expressive language from the point of view of converting ideas to code, and having that code be readable, maintainable, and extensible by other humans.
where is falls down is on the "machine facing" end - you don't get the guarantees of static typing, the import system is a bit of a mess, it isn't as fast as lower level, statically-compiled-to-native-code languages, and (as the current discussion shows) packaging and distributing an end user binary is somewhere between a morass and a nightmare. however, none of these things are a problem with the language per se, and they can be fixed with some combination of improving the implementation, writing better tooling (cf valgrind from the c world), making it easier to write small bits of critical code in c/rust/zig/etc, and improving language features while not sacrificing what draws people to use python in the first place.
is python a local maximum? almost certainly; it would be foolish to think that nothing better could come along, and indeed pyret already looks distinctly better in terms of pure dynamic language design. but the answer to that is not to "level up" people who are happy and productive using python to some language like rust or c++ that imposes a heavier cognitive burden on the user to take care of low level details that their applications might not need, or one like golang that pushes the complexity out into the application where problems need to be solved via design patterns that people need to learn and copy, rather than just building them into the language or stdlib.
> none of these things are a problem with the language per se, and they can be fixed with some combination of improving the implementation, writing better tooling
As a JavaScript developer, I worked through a decade of this thinking, and it produced an oft-ridiculed morass from which we are still emerging. Or do I have that backwards? Were we too satisfied with our incrementally improved tooling all along the way?
> c++... imposes a heavier cognitive burden on the user to take care of low level details that their applications might not need; golang... pushes the complexity out into the application where problems need to be solved via design patterns that people need to learn and copy, rather than just building them into the language or stdlib.
These are intriguing critiques of the languages, and it follows from them that to package an executable, a Pythonista must either "take care of low level details," or "learn and copy... design patterns", all the while sacrificing the qualities of "readable, maintainable, and extensible by other humans". Steep costs, indeed! Arguably enough to justify the overhead of pretty much any Python portability hack.
I don't wholeheartedly agree, but I respect the logic.
Compared to PyInstaller, it doesn't dump any temporary files on disk.
Compared to Nuitka, it uses the real CPython, not a different implementation.
And unfortunately, PyOxidizer went into maintenance mode: https://gregoryszorc.com/blog/2024/03/17/my-shifting-open-so...
Everyone has their solution to the problem - so there is no solution.