How does Odin plan to avoid repeating C/Python's dependency management fiasco?

General
1

I apologize for somewhat clickbait-y title but the question nevertheless is worth asking.

Let me set the stage first.

For a long time python didn’t have a default dependency management tool. And then got one but it is still wasn’t good enough. This led to the present day when there are 100500 different package/project management tools in the python ecosystem (pip, pipfile, poetry, uv - just to name a few) - and all of them are competing, fragmenting the ecosystem and making regular developer’s life harder.

Very similar situation is observed in C and C++ word. Build tools aside - there are no standard dependency management tools. There is OS package manger, then there is Conan, VCPKG, Meson’s wrap then there are makeshift solutions which exploit build tool’s ability to fetch remote URL (git or http) to fetch something from the internet at build time. None of this helps developers, who have to deal with these tools. It is a known fact that build+ dependency management tool is a unique snowflake in every C or C++ project and switching from one project to another is not an easy task fro a regular Joe.

Both python and C, C++ languages are living examples of xkcd: Standards

Golang didn’t have a package manager in the early days which led to an explosion of 3rd party tools. Luckily golang’s leadership recognized the problem until it was too late and now Go has decent standard go.mod solution and previously fragmented ecosystem united and now every Go project uses the standard tool.

Now the question part:

In numerous interviews and podcasts @gingerBill said that he considers package managers harmful for various reasons (one of the main ones is that they make it too easy to create deep and unmanageable dependency chains, which leads to poor code quality, bloat etc). While I agree with his views I also can’t ignore the fact that absence of a standard tool will lead to ecosystem fragmentation. Today Odin is small enough to be immune to this problem, but it is only a matter of time.

  • How is Odin planning to address this situation (if it plans at all)?
  • Is there a way to avoid the fragmentation via 3rd party tooling in some other way (without providing a default tool?) - perhaps be more aggressive with vendor collection?
  • Somehow most of the golang ecosystem still strives for simplicity and minimal dependencies (don’t get me wrong, there are many exceptions) so this makes me think that there is a strong cultural aspect to this. And a default dependency management tool only gives a way out for those who desperately need it without fragmenting the ecosystem. This is only a side of the coin that I see. I wonder what perspective I might be missing…
3 Likes
2

Python still remains popular even with fragmented package manager ecosystem. BDFL already made his views clear on package managers and I tend to agree with his views. Is it really that difficult to download a folder from Github and put it in your project directory? :grinning:

Why the need to standardize package management?

6 Likes
3

I’m not sure I agree with Bill’s input on pkg mgrs. I agree with the objective part obviously (messes up compile times, there’s a bunch of ‘unknown’ code running etc) but there’s a reason they’re so predominant and ime making something difficult for people rarely discourages them from doing it. They just find a different way to do it, programmers are persistent that way. I’ve myself abused -collection: compiler feature into using it for conditional imports, as well as having ‘namespaces’ in Odin so pkg imports aren’t local dir-paths.

But then he has also said he’s the worst marketer of his own language so I don’t think he specifically loses sleep over Odin being used like this.

2 Likes
4

TLDR: I agree with Bill, and don’t think there’s a need for pkg manager. But I’d love to have a core package to replace custom scripts for slightly more complicated build/codegen setups.

And you technically don’t need one either. People could’ve just modify PYTHONPATH and dump the files where it points to. All the complications start from python needing C/Fortran libs to do anything CPU intensive work, and then you need to juggle C’s madness.

There is no standard dependency management tool, because there’s no standard build tool. And there’s no standard build tool because there’s no standard way to build C. And there’s no standard way to build C (IMHO) because compilers are not interchangeable and not all platform specific configuration can be done through the C preprocessor so you needed hacks to just even figure out if you are on 64bit or 32bit system in a semi cross-platform way.

See CMake’s recommended way of checking the size of void*… which compiles a program that prints sizeof(void*) because you can’t do #if 8 == sizeof(void*)… And of course, everyone thinks their hack is better than the rest.

Basically, the moment there’s a standard way to build things, package management is “download the package and build it like you build the rest of your codebase”.

Which could have been just a script to do git clone with a specific commit or tag. There’s 0 difference in updating a script or updating your go.mod file. The difference is that it’s now built into the compiler/package semantics and if you need anything more than just fetching a libary you still need custom build steps, at which point you could’ve just git clone in one of the make targets…

There is a proposal for core:build, here. I personally hope this would be added, as it would allow to get rid of shell/batch scripts but keep the freedom of “do your build however it works for you” as long as the starting point would be odin run build.odin -file or with compiler support to be able to just do odin build pkg/ and do the same if there’s a build.odin present.

People are just lazy… But then they end up working around the package manager to get reproducible builds or better control over package sources. Like running their own registry to be able to pull patched or strictly vetted crates with cargo because it’s still simpler then figuring out all the magic flags cargo passes to rustc based on Cargo.toml… In my experience package managers work on toy examples and then fall apart the moment you need more control than what average Joe needs.

2 Likes
5

Java for the longest time (over a decade) also didn’t have a dependency manager, instead if you wanted to use a library you downloaded a jar and added it to your classpath.

That jar could just include the .class files with bytecode, or it could include a copy of the javadoc and/or source code.

If creating a library was as simple as odin exportlib foo which emits an .odinlib file (actually a .zip with the sources) that you could point the compiler to that would in my opinion be great.

2 Likes
6

I agree with Bill’s input on pkg mgrs. Moreover I think languages lika Odin and JS/Python are in different niche of usage. Python is a very popular language no doubt but I think most of the time it is used to write small isolated scripts / calculation stuff. You want to get results fast and want to get some magic tools that will help you to do the job.

At opposite, I believe that if you choose Odin you are likely going to work on some sort of a “big” project or app and will be very picky about what dependecies to load and in the scope of this project the problem with “dependency management” is the least problem you will encounter and mostly devs would tend to keep dependencies in minimum.

Please, keep in mind that it’s just subjective point of view based on my experience and observations.

3 Likes
7

What would the benefit be compared to just downloading a repo or whatever and putting the source code package into your project?

One downside to a zip file like that is to change anything in it you have to first extract it to disk, THEN change it, and then what? Put it in a package folder?

8

Today downloading a repo and then building it is much easier because git is so widespread, if you are doing any programming you probably have git. However doing a git clone on that repo means you now have the entire commit history of that repo, history that you don’t care about. A standardized archive layout will mean that other people wanting to use your library doesn’t need to deal with having to unpack it before using it.

How often do you actually need to change something in a downloaded mature library?

Java’s jar ecosystem was amazing because it was stupid simple to distribute your library, create that singular jar file and put it up for download on any server that allows hosting of arbitrary files. There were command line tools to extract all the public classes and function signatures, there were ways to create a browsable version of an entire jar and ides knew how to deal with them.

1 Like
9

Most git hosts like Github allow you to download the repo as a zip directly for any commit. Also, you can clone without history. I think the command is just git clone --depth 1.

1 Like
10

As I said, the way to not have the “official” mess is to not even get into it. As I’ve discussed many times before elsewhere, package managers automate dependency hell, something which should not be automated. And the only way to prevent this hell is to well… not participate it in it.

That’s how you “avoid” that fiasco. It’s a cultural problem; not a tooling problem.

6 Likes
11

I am using Raylib in a project, and it takes cstrings for stuff like measuring the pixel length of a string, and drawing text on the screen.
They do not want to add length-based string procedures for these procs, as I understand.
Whether they do or not however, doesn’t matter - I was able to just quickly glance at the source and add two procs that take a text length parameter and it Just Worked.

12

The majority of arguments here against having a standard boil down to: “we did it for a decade this way”, which is a fallacious argument, and falls under Appeal to Tradition, and is not a sufficient argument. In this case, its ignoring the very real differences in how we develop now software vs 20 years ago: who is doing that development has changed (go look at the vairous stack overflow language surveys or related for this), and the risks associated with software development have changed (here is a good overview of APT actors note: the very first mentioned change is Supply Chain Attacks). Not having an answer doesn’t ‘solve the problem’, it just causes a very real different problem that has been observed in other languages and ecosystems. Python had the insane issue with setuptools and distutils (which i just learned is still a thing), Golang: there were (at least) 3 competing dependency management systems before the official one was released. I was an IC and team lead for both of these fiasco, and it was hell managing teams, migrating code, setting up infrastructure, setting up team members. My story is not unique: there are countless others like it, (which are documented in each community/ecosystem).

The fact is, most languages have either developed a formal standard, or a de facto standard to deal with this. There is a lot of documentation for the various languages in the non-exhaustive list below, with 1000s of real world examples on why this is a good idea. End of day, you can always ignore the standard and do something like use golangs vendor/ directory, or manually manage your classpath in java. there may even be good reasons to do this, but they are the exception.

Java:maven
Python: pypi
Golang: go mod
rust: cargo
lua: luarocks
ruby: bundle
node: npm
etc.

(this is not an appeal to the crowd: its a rational reading of the documented justifications for the expenditure of huge amounts of engineering effort, from the PoV of: engineering productivity, security, stability, engineering management, risk, and probably some other domains)

Professionally, not having a standard way of managing dependencies makes it harder for me to recommend a language professionally because it makes it significantly harder to, 1) train engineers, manage dependencies across teams (custom tooling), track CVEs (because I don’t have a standard way of managing dependencies or scanning them), ensure reproducible builds, prevent supply chain attacks, etc.

all of the ‘serious’ problems (supply chain, reproducible builds, etc) are actually less important in a profession setting than ‘i can easily train engineers to use the language’. because if I can’t get them easily shipping code, I can’t pay them, and I am not going to recommend such a language for adoption at scale.

this is why i’ve on-boarded/migrated 4 companies to Go, instead of, say, python. because even though python has a’standard’, the user story is horrible vs. go build, or if you’re feeling spicy: throw in a GONOSUMDB to completely remove that protection against supply chain attacks, or just dumb upstream developers changing what a tag points to because it worked on their laptop. Even with that protection removed: you’re in a significantly better state professionally (e.g shipping nice features) and risk wise (eg, not shipping broken features) than the alternative.

On the topic of ‘automating dependency hell’, there is a world of difference between where we started in the hellscape of early ANT files, requirement.txt, YOLOing an insane makefile with dynamic LD flags, vs where we are with golang has ended up with its cerca v1.22 go.mod and go.workspace, and modern rust cargos

which formalises a lot of the things you need to do manually while giving you control new user cannot put lots of urls in: pretend this is a link to the gomod reference is it perfect? no. golang doesn’t try to be. does it cover 95% of issues usecases? yes. does if remove whole catagories of attacks? yup. can you just straight up ignore it and do your won ENV hackery in the grand tradition of LD_ flags and Makefile nonsense? Go for it! Hell, i do that stuff too on occasion, and I like the go mod system.

on a purely pragmatic and pessimistic note: if there is no official solution, some idiot with no idea what they are doing is going to make a tool. It will be used because people are lazy, and engineers even lazier. We will then suffer through the worst of all worlds in that situation. You want an example? nodejs. the hell is that mess? better to make one that is sane from the get go. less trouble over all, easier to be lazy that way.

5 Likes
13

“Appeal to Tradition” is not what people are doing here nor is it a fallacious argument to do so. The corollary of this is Chronological Snobbery.

How people develop code now is WORSE than how it was developed 20 years ago, so trying to make it better for people who work with things now, is not a good argument in my opinion.

Also, I’ve discussed this elsewhere before but I absolutely hate package managers, and for very good reasons. And I NEVER want an official one.

You are complaining about not having a standard/official way to “manage dependencies” but you are not actually considering all of the problems that come with automating dependency hell.

Not everything has to be automated, dependency hell is one of those things I do want to encourage.

6 Likes
14

I get why you would bring up Chronological snobbery, because i would agree that there have been some horrible missteps in automating dependencies, and automated dependency management isnt even my primary concern. Again npm is the classic example of how bad it can be. I have survived, if barely, a ‘bespoke’ java dependency sysyem. I am not claiming these good because they are new. I believe they are bad solutions:

I think pythons package management is bad in general with occasional flashes of tollerable, nodejs is a joke, maven only works due to stockholm syndrome and an absurd level of systemic masochism.

But there are two parts to my argument: risks associated with modern development have change: 1 Advanced Persistent Threat (for example: supply chain attacks), 2: there have been some legitimate and valuable developments in the space (rust, golang, nix).

Even if you dont want automated dependency management (fair enough), the threats still exist. A reduced scope: package definition, would go a long way to supporting things like SBOMs and reproducable builds.

Then in some future where some demigod mad lad savant comes along with a proposal that you actually like for automation comes along, thats a future you problem.

But please dont reject all i said without at least considering the risks. Even if your only action is to look into APT threats, trusting build chains, reproducability, etc.

2 Likes