Python Packaging: Why we can't have nice things – Part 2: Stupid Pipx Tricks
5 points by zahlman 2 days ago | 5 comments
  • zahlman 2 days ago |
    Hi HN,

    This is the second post in my series, which I wrapped up quite late last night and am finally getting around to submitting here, not-as-late tonight. In this one I highlight some of the benefits of using Pipx to install Python applications - relevant both for end users and for developers. I also discuss some of the limits that you'll run into with it (especially as a developer), and show some neat ways to work around issues and get more out of your Pipx installation.

    In particular, a few setup steps and wrapper scripts allow you to reuse Pipx's vendored copy of Pip for any virtual environment on your system (and create new virtual environments faster, by not bootstrapping a copy of Pip into them), and to get raw access to the virtual environments that Pipx creates - allowing you to explore and debug installed code, and treat Pipx like a primitive environment manager.

    From my point of view, the advice I offer here is a stopgap - one of the tools I'm developing, Paper, is intended to replace both Pip and Pipx (although the intended scope is still nowhere near "complete workflow tool"). But hopefully you find it useful right away.

    Later this month I'm planning a broader piece about my experience with Linux, and especially my experience of using Python on Linux, and the scripts I show here will be some useful background for that.

    My next planned post is about a long-standing (I mean over a decade) bug in Pip which I consider both surprising and embarrassing (and arguably an important security issue).

    • ggm 2 days ago |
      If you use homebrew on OSX you are recommended to use pipx by pip3, under the homebrew model (if you install python as a brew)

      I didn't for ages. Then, I did and I found it worked, and did not whine about breaking pip based brew based package management so I am happy(ier)

      • zahlman 2 days ago |
        Interesting. I probably should come back to this piece later and expand it to talk more about why you'd want to solve the problems that Pipx solves (i.e. the externally managed environment stuff - I think I only included a link for context).
  • rednafi 2 days ago |
    All of these issues are kind of solved by uv. It took the Python community a long time to realize that Python isn’t a great language for writing Python tooling. uv solves the bootstrapping problem, and its resolver is infinitely faster than poetry’s or pipx’s. It’s hard to recommend anything else these days.
    • zahlman 2 days ago |
      I've said many times that the problems solved by uv aren't really caused by using Python as an implementation language - they're primarily caused by the design of the existing tools. Resolvers in particular are constrained by many things that aren't the implementation language: the algorithm (is it making likely-to-succeed guesses first?), how thorough the resolver tries to be, the approach to caching.... (Of course your network connection also matters, but tools can't really do anything about that. Yes, I do count that as part of resolution because you can't actually get all the necessary metadata in one shot because of the way that the packaging standards and PyPI work.)

      But I don't have the kind of hard numbers I'd really like in order to make that argument properly. Hopefully I will by the end of the series, since I'll also be working on Paper.

      Bootstrapping isn't a real problem unless you actually want to get Python itself (not a virtual environment; those are trivial) and find the standard approaches insufficient. I have already worked out how Paper will bootstrap itself - and I expect it to end up being much smaller than Pip, never mind the full Pipx package or a compiled uv binary. I guess hardly anyone cares about saving a few megabytes of disk nowadays, but it's a real annoyance for me (and the planned subject of another post, not in this series).

      If you like having an opinionated, integrated workflow tool, uv comes highly praised. I personally am much more in favour of the Unix philosophy. If you use uv only to install applications and libraries then that seems like a waste to me.