Well, here it is. All none of you reading this will have eagerly awaited it: The final Python packaging rant.

Opinion

In the following discourse, I will be criticizing a specific person very heavily. I would like to make clear that what I say is my opinion, and whatever I get wrong, is wrong.

History

First, a bit of history. In the beginning, Python for Windows shipped as a single MSI package. This had the usual task-centered MSI user interface with a feature tree, a page to override the installation path, and so on. It was, on the whole, good.

Python has for a long time been a very popular language, so at some point, Microsoft started supporting Python development and, in particular, its convenient deployment on Windows. They had already done the community a great service by making sure WiX was steadily evolving and improving, and apparently decided to hire a central figure in the Python-on-Windows development to further improve things.

This person, either before or after, came up with what must surely be the worst thing that ever happened to Windows Installer: burn. I have a comprehensive rant on the subject somewhere on this site already. In any case, I believe this developer, more than anything, needed a flagship project to demonstrate the awesome power of his creation, and unfortunately for everyone, he seized on Python. The resulting abomination, the Python executable installer, initially replaced a single MSI per platform with 22, and it looks like that number has increased to 25 by now. (Some of these packages do not contain a single file.)

I cannot blame the larger CPython community for this; most developers are centered on *nix and need not care much for how a Python environment makes its way onto a Windows system. The switch from a single MSI to the burn bundle was foisted upon Windows users purely as a consequence of the “peak syllogism”: “We must do something. This is something. Therefore, we must do this.”

After the burn bundle was introduced, at first I tried avoiding it and installing the needed MSIs manually, but that approach was obviously not viable, so I arranged myself with the new regime and used the limited customization features of the burn bundle to set it up to meet my needs.

For some reason that I have not tried investigating because I like my stomach where it is, it appears the time has come to make an even bigger mistake, however. I only noticed this recently, but the burn bundle is now to be replaced by a monstrosity called the “Python Install Manager”, and this thing definitely takes the crown of “worst idea ever”.

I mention this in my burn rant, but it is worth repeating here: The purpose of an application installer is to put the application onto the target system as easily and with as little friction as possible. On Windows, the best implementation of this goal is … an MSI package.

The worst implementation must be something that has to be installed as a separate package first, then run, possibly repeatedly, to create the actual application. The “Python Install Manager” has the following features:

• It is distributed as both MSIX and MSI packages, with the MSI apparently deprecated ab initio. The MSIX has no customization capability, for example for the installation path, because it springs from Microsoft’s effort to make PCs into larger phones.

• It does not allow the user to select features to install or omit from the installed Python environment, except that a distribution can provide different feature sets as “tags” if they feel like it.

• It hijacks the py command for no identifiable reason, and it overlays its subcommands over files that might exist in the file system, i.e. py install installs something, it doesn’t run a file named “install”. Despite the absence of the customary file extension, this might actually be a Python script.

  For some reason it also includes a “pymanager” command, presumably because someone did have a clue, namely that a separate command would fix this problem.

• It relies (?) on “app execution aliases”, however, it was released with the knowledge that there is a bug in their management that prevents a newly installed application (in this case, the install manager) from overriding an existing alias. Before installing the thing, running python on the command line will take you to the Microsoft Store to download Python. Intrusive, but possibly helpful. After installing the install manager, the same thing happens.

Stable interfaces

Obviously the Python-on-Windows developers, under the questionable leadership of the aforementioned person, do not care about stable interfaces, in this case, maintaining a deployment method for longer than their attention span. In this effort, it has been announced that the burn bundle is going away with Python 3.16, leaving the install manager as the only way supported by CPython to install Python on Windows. While seeing burn burn should be great news, I’m sorry to say that the replacement is, if anything, worse.

Software management

Among the first bugs I ever reported in the burn bundle was that it creates its Uninstall key in HKCU of the installing user. This can have fatal consequences in an enterprise environment where software is managed centrally, because deployment tools usually do their work on the target system using a dedicated user account. Deleting this user profile means losing the most accessible source of information about installed Python environments.

While the install manager MSI itself, to my great surprise, actually manages to create its Uninstall key in HKLM, it absolutely insists on only creating strictly per-user installations. There is an option to redirect the installation to a different path, but if it is used, no Uninstall key is created at all. This means that a software management tool either has to scan all user profiles for Python environments in the prescribed path, or load all user Registry hives. With the burn bundle, it is at least enough to look in the Registry of the dedicated deployment user, even though depending on the tool this may involve loading that profile while the scan is happening.

Without a large amount of work on the part of whatever admin gets stuck with it, this will result in old to ancient Python versions accumulating in user profiles because nobody ever cleans up after themselves.

The fix

A long time ago, when I submitted yet another bug against the burn bundle that included my standard postscript about the advantages of a single MSI, this developer suggested that if I liked that so much more, I should just make my own. At the time I dismissed that idea because I did not want to do all the work necessary to build Python, gather all the bits together, and turn them into a sensible package. However, thanks to the install manager, I noticed that since Python 3.12 or so, there are Zip archives available that contain the entire environment. These are what the, er, monstrosity uses itself, so I presume they will be around until someone comes up with a yet worse idea – I fear that this is possible, unlikely as it may seem right now.

These Zip archives are trivially easy to repackage into MSI form, including the necessary Registry bits to make the existing Python launcher work (another victim to be of the install manager).

https://github.com/chrullrich/wix-cpython/

Ceterum censeo: All the bugs listed above will henceforth be avoided by using MSI as the distribution package format.

World-shattering Vim discovery of the day: %.

Turns out I just discovered how GitHub actions work when I started setting them up in Gitea. OH. MY. GOD.

I spent quite some time wondering about the sheer audacity of fetching action implementations from GitHub. OK, not a big deal, I can do that manually and put them into Gitea so at least it doesn’t try going out to GitHub each time.

But then.

The next error message from the test workflow in the Gitea docs came just after a successful checkout:

Cannot find: node in PATH

So simple. So … deceiving.

OK, something that should be in $PATH isn’t. A bit weird to call a path entry a file system node, but whatever.

To find out what it was that I was missing, I looked at the implementation of actions/checkout.

…

(Look away now, kids, this isn’t good for you.)

…

THE BL@@DY CHECKOUT ACTION THAT NEEDS TO RUN ONE, COUNT ‘EM, ONE, GIT COMMAND LINE IS WRITTEN AS SOME FIFTEEN TYPESCRIPT FILES THAT WANT TO BE RUN WITH NODE.JS?!?

ARE YOU F@@@ING KIDDING ME?!? TRUMP HIMSELF COULD NOT HAVE COME UP WITH ANYTHING EVEN APPROACHING THIS UTTER BENIGHTEDNESS!

No wonder everything takes ages these days despite our computers being immeasurably faster than Way Back When.

I’m … let’s call it flummoxed. That is certainly a nicer word than several others I have in mind.

Would the authors of actions/checkout please consider rewriting this … this THING in a saner, and more preserving of its user’s sanity, way?

Thank you. I’ll go cry in the corner now.

i’m scared to look at any others

In my experience the VMware Tools installers will work only when run on VMware VMs and crash immediately (with error 0xC0000096, STATUS_PRIVILEGED_INSTRUCTION) everywhere else. This is a problem when you move a VM to another platform and either forget to uninstall the tools first, or are reluctant to lose the VMXNET drivers.

There are various sets of instructions on the Internet on how to uninstall VMware Tools using other means than the regular uninstallation, but I have found they either do not work, or are hopelessly difficult to follow. (The latter is not something that should stop you, of course.)

So here is my own hopelessly difficult to follow set of instructions:

1. Get a current VMware Tools installer from VMware.
2. Create an AIP from it (VMware-tools-....exe /a).
3. Use Orca or another MSI editor to remove the VM_LogStart and VM_LogEnd_Def actions from the InstallExecuteSequence and InstallUISequence tables in the .msi file.
4. Run msiexec /l*v msi.log /x <.msi file>. It will exit immediately, and the log will show that the custom action you just removed was magically run nonetheless.
5. Look for “Package we’re running from” in the log file. This will show a path in C:\WINDOWS\Installer, and it means that because the product is already installed, msiexec used the cached package instead of the one you gave it.
6. Rename that cached package to get it out of the way.
7. Re-run the uninstallation.

This should work. If not, start over.

So Microsoft recently decided that Notepad, the simplest text editor ever that has been a wrapper around a multiline edit control and a few common dialogs for basically ever, needed tabs. Because everything needs tabs now, I suppose. The measure of a complete program is no longer “it can send e-mail”, but “it has tabs”. Or something.

Not only does it have tabs now, but it remembers the contents of those tabs so it can display them again next time you start it. It was not good enough to remember the files to load them again, no, it caches their content.

This means that one use case for Notepad is made massively useless by this highly not-sought-after misfeature: Looking at secrets. It goes like this:

1. Open a file that has a secret key in it or something.
2. Use it.
3. Close Notepad.
4. Remove the removable storage that the file is on, or “lock” the encrypted volume, or whatever, so the key is isolated from the outside world again.
5. Start Notepad again.
6. Stare in horror at the secret key you thought was safe, but which Notepad copied to insecure local storage without asking.
7. Frenziedly search for where that copy is (%LOCALAPPDATA%\Packages\Microsoft.WindowsNotepad....\LocalState\TabState) and delete it.
8. Work out that the abomination now called Notepad has a settings window (via the gear menu on the right of the menu bar) and that you can disable the caching by setting it to open a new window on startup (this also deletes the cache) and even get rid of the tabs by setting it to open files in a new window.

Have a nice day.

It’s time for my favorite thing. Let’s have a Python packaging rant.

Most people know that the Python world’s approach to packaging and distribution is best summarized, and probably directly inspired, by xkcd.

A long time ago, the decision was made, or accepted by mutual apathy, how installation tools like pip and the late, unlamented easy_install would find packages:

1. Predict the file name (possibly more than one).
2. Load an HTML page.
3. Check every <a> on that page.
4. Use the one that points at the most preferred file name variation.

The only thing in the page that is at all interesting in this model are the //a/@hrefs. Everything else is noise.

Enter pip 22.0, released ~2 days ago. It now uses a different parser for HTML pages than previous versions. Instead of the certainly entirely overpowered html5lib, whatever that is, it is now using something else, apparently called html.parser.

Remember, the approach to looking up download links for package files consists of looking at <a> elements. Finding them in HTML, however close to, or far from, the spec it may be, is not a big issue. (Anyone who allows user-generated content on a download page is beyond help anyway.)

Why, then, the switch to a parser that, in order to find <a> elements, requires that the page have a correct <!DOCTYPE html> declaration, i.e. the claim that it adheres to the HTML5 spec?

The HTML5 “spec” shall not be gone into here any further.

The result is pip 22.0.2, an emergency release so urgent that it could not even be tagged in the repo before it was put on PyPI (and the tag has still not been created as of press time). In the typical manner, whenever it encounters a page that does not have the canonical <!DOCTYPE> it will print a warning blaming the user for their audacity in installing packages from sources that do not wrap their unstructured lists of <a> elements in proper, (un)well-specified HTML5.

Hey, PyPA, you want a proper fix for that? Publish a file. One single file. Its name will end in “.xsd”, and it will describe how lists of package links are supposed to look.

Ceterum censeo: This bug would have been avoided by not suggesting to package publishers that you will accept any line noise as long as there are <a> elements in it, then suddenly deciding that the stuff surrounding the <a> elements really matters to you.

tl;dr: Shame on you. This took at least six minutes to write, go enjoy it.

The trials of testing

PyCharm is a great IDE for Python development. Its support for Django projects is really good (if only it applied environment variables from the Django project setup to run configurations …).

As with everything Python, the structure of Django projects is not set in stone and everyone has their own preferred style. Mine is to have applications in (below) a common subdirectory. Unfortunately this means that Django’s built-in test framework (or possibly just unittest) does not find the tests on its own and I had to invent some complicated workarounds to automate them to my satisfaction.

This morning I discovered that all it needs to work is for me to tell it that that common subdirectory is a module by giving it an __init__.py.

So the PHP people, in a display of cleverness akin to that generally displayed in the development of their product, have put their downloads behind an automated guess at the “human-ness” of the client.

This obviously broke all automated updating everywhere, including by the FreeBSD Ports Collection. Very helpful, that.

Their bug tracker is apparently on its Christmas holiday, too.

Edge updates (and also others that include something called the Edge WebView2) fail with 0x800951dd on some computers, but not others. Hm. 9 is FACILITY_SECURITY. The event log is full of “Windows Event Reporting”; dozens of events for each installation attempt. What is going on?

From the log (cleverly stored in %PROGRAMDATA%) is appears that the updater really wants to phone home, and if it cannot, it prefers to crash. Where the security thing comes from is unclear, particularly because 0x51dd is undocumented. Direct HTTP is blocked, so the installer prematurely departs this mortal coil. And tries again. And again. And again.

On computers where it works the log shows that the updater figures out where the proxy is, so the quick fix (until I figure out how to make the updater behave) is to tell the updater the same on the problematic systems.

From the poudriere logs:

Time saved thanks to ccache (your praise be sung in all homes!): 95.13 %, or in other words, a ~2000 % speedup.