You must log in or register to comment.
I used to struggle with this, until I realized what’s really going on. To do conventional web development, you have to download a zillion node modules so you can:
- Build one or more “transpilers” (e.g. Typescript, Sass support, JSX)
- Build linters and other SAST/DAST tooling
- Build packaging tools, to bundle, tree-shake, and minify your code
- Use shims/glue to hold all that together
- Use libraries that support the end product (e.g. React)
- Furnish multiple versions of dependencies in order for each tool to have its own (stable) graph
All this dwarfs any code you’re going to write by multiple orders of magnitude. I once had a node_modules tree that clocked in at over 1.5GB of sourcecode. What I was writing would have fit on a floppy-disk.
That said, it’s kind of insane. The problem is that there’s no binary releases, nor fully-vendored/bundled packages. The entire toolchain source, except nodejs and npm, is downloaded in its entirety, on every such project you run.
In contrast, if you made C++ or Rust developers rebuild their entire toolchain from source on every project, they’d riot. Or, they would re-invent binary releases that weekend.
And if you made JavaScript developers use compatible versions for everything they’d riot. And also every build would fail for, like, at least a week
Wait until OP finds out about interpreters and compilers.
Why write code, when someone else already wrote it?
Why not import all code ever created by human kind just in case we might need some of it.
…and then we can grind all the code ever created by human kind into a fine paste, and write a clever algorithm to regurgitate it as a squishy code slurry in response to questions about problems that the standard libraries already solved.
I want to build a kick scooter. For that I need some wheels. So I import the well-known semi-truck framework. From that framework I take some huge wheels. They are too large and too many, but I guess I can make do with them.
But I need to attach the wheels to one another, so I import the bridge-building-library, because they have steel bars in there.
Lastly, to attach all of that together I import the NASA space ship framework because there’s a hand welder in there, that’s been deprecated years ago, but it’s still rotting away in there because some important products still require the hand welder class for some entirely unrelated use cases.
Except in the picture on the left, someone’s actually reading it.
Something’s gone wrong if you’re looking in the node_modules folder.
Sometimes you gotta monkey patch that library because they won’t accept your pull requests to fix that bug.
At least you can monkeypatch it.
Also C programmers using glibc
Feels like a lot of “not inventing the wheel” - which is good? There are plenty of good wheels out there.
Is this why pip packages are called wheels…?
“Yes, I’d like a wheel. I don’t want to invent it. Why, of course, give me the full package of wheel, axis, rotor, engine, fuel tank, windshield, mirrors, tire, front panel, brakes. This wheel will be great for me manually spinning cotton!”
And I have the option to spin cotton off-road on rugged terrain…so there’s that…
You say that, but I’ve watched the JS community move from one framework and tool suite to the next quite rapidly. By my recollection, I’ve seen a wholesale change in popular tooling at least four times in the last decade. Granted, that’s not every developer’s trajectory through all this, but (IMO) that’s still a lot.
But changing frameworks is not why node_modules is so large. You don’t import Angular and Vue.
I agree. Rather each one of those is rather substantial on its own. Plus the churn of going from framework to framework makes it less useful to compress and bundle all this stuff into fixed versions on a slower schedule (e.g. like Ubuntu packages do). I think that all contributes to bloat.
The problem is “I need function, library with 1000 functions has function, include.” Library’s 823rd function turns out to have a vulnerability.
Enter tree-shaking.
But I don’t NEED a wheel, I just need a tarp to put over this metal frame on my patio, and for some reason the tarp manufacturer attaches wheels and plane wings to it!?
The package comes with all the bells and whistles but the final build only contains the tarp, if you import it right and tree shake it.
This person nodes…
Until those wheels contain malware and spyware.
Or bugs that you only work out much later on.
Or someone deletes the axle repo and the wheel flies off.
Yes, but my moon rover’s wheels need to fulfill different requirements.
And this is why tree shaking exists.
I feel like the end result of tree shaking would be the image here https://fedia.io/m/programmer_humor@programming.dev/t/2547352/CSS-Gardening
Gotta fix that link
It’s great when it works, yeah.
What is that?
Dead code elimination but with a different name for some reason
Because we’re monkeys
If you import 1% of your module code, you only compile the actual used code. Tree shaking is removing dead code paths that aren’t used.
Ah ok gotcha
the one on the right is also packages in node_modules that you’re actually using and specifically requested.
Very true.
Python feels like that sometimes too. Except much more standard library which is much better than node modules.
Rust as well. Seems to just be a modern language thing.
At least Rust compiles down to what is used. I don’t know if js has any of that, but at least with rust the final program doesn’t ship tons of bloat.
Yes and no, the linker does nicely trim a lot of the fat, but rust binaries are still pretty chonky. Its good chonky (debug etc), and static compile doesnt help, but they are quite fat.
Also doesnt help compile times that you have to build all this extra stuff, only to throw most of it away.
I sort of have a suspicion that there is some mathematical proof that, as soon as it becomes quick and easy to import an arbitrary number of dependencies into your project along with their dependencies, the size of the average project’s dependencies starts to follow an exponential growth curve increasing every year, without limit.
I notice that this stuff didn’t happen with package managers + autoconf/automake. It was only once it became super-trivial to do from the programmer side, that the growth curve started. I’ve literally had trivial projects pull in thousands of dependencies recursively, because it’s easier to do that than to take literally one hour implementing a little modified-file watcher function or something.
Its certainly more painful to collect dependencies with cmake, so its not worth doing if you can hand roll your own easily enough.
The flip side is that by using a library, it theoretically means it should be fairly battle-tested code, and should be using appropriate APIs. File watching has a bunch of different OS specific APIs that could be used, in addition to the naive “read everything periodically” approach, so while you could knock something together in an hour, the library should be the correct approach. Sadly, at least in rust land, there are a ton of badly written libraries to wade through… 🤷
Yeah. I have no idea what the answer is, just describing the nature of the issue. I come from the days when you would maybe import like one library to do something special like .png reading or something, and you basically did all the rest yourself. The way programming gets done today is wild to me.
I’m not sure its a problem in of itself, but i agree it definitely enables a problem. Between “is-even” and vibe coding, modern software engineering is in a very sorry state.
Yeah. I feel like in a few years when literally nothing works or is maintainable, people are going to have a resurgent realization of the importance of reliability in software design, that just throwing bodies and lines of code at the problem builds up a shaky structure that just isn’t workable anymore once it grows beyond a certain size.
We used to know that, and somehow we forgot.
also applicable to “installing modern drupal via composer”.
Be the change you want to see in the world, people. Don’t use any Node (or Rust or Python or Java or whatever) modules that have more dependencies than they absolutely, positively, 100%, for real have to. It’s really not that hard. It doesn’t have to be this way.
Too late,
is_even_rsnow depends ontokiocries in legacy systems
This applies to developers, too.
External dependencies put end users at risk, so I avoid them as much as possible. If that means I have to rethink my design or write some boring modules myself, then so be it.
Depends on the use case, and what you mean by “external dependencies”.
Black box remote services you’re invoking over HTTP, or source files that are available for inspection and locked by their hash so their contents don’t change without explicit approval?
Cuz I’ll almost entirely agree on the former, but almost entirely disagree on the latter.
In my career:
I’ve seen multiple vulns introduced by devs hand-writing code that doesn’t follow best practices while there were packages available that did.
I have not yet seen a supply chain attack make it to prod.
The nice thing about supply chain attacks though: they get publicly disclosed. Your intern’s custom OAuth endpoint that leaks the secret? Nobody’s gonna tell you about that.
I didn’t think I would have to spell this out, but when I wrote “as much as possible”, I was acknowledging that some libraries are either too complex or too security-sensitive to be reasonably homebrewed by the unqualified. (Perhaps “as much as reasonably possible” would have been better phrasing.) Where the line lies will depend on the person/team, of course, but the vast majority of libraries do not fall into that category. I was generalizing.
And yes, some third-party libs might get so much public scrutiny as to be considered safer than what someone would create in-house, depending on their skills. But safety in numbers sometimes turns out to be a false assumption, and at the end of the day, choosing this approach still pushes external risks (attack surface) onto users. Good luck. It hardly matters to the general point, though, because most libs do not have this level of scrutiny.
Let’s also remember that pinning dependencies is not a silver bullet. If I didn’t trust someone to follow “best practices”, I don’t think I would trust their certification of a third-party library hash any more than I would trust their own code.
With all that said, let me re-state my approach for clarity:
- I minimize dependencies first. Standard libraries are great for this.
- When something more cannot reasonably be avoided, I choose very carefully, prioritizing the safety of my users over my own convenience.
- Sometimes that means changing my original design, or spending my time learning or building things that I hadn’t planned to. I find the results to be worth it.
Which sounds like great, practical advice in a theoretical perfect world!
But, the reality of the situation is that professionals are usually balancing a myriad of concerns and considerations using objective and subjective evaluations of what’s required of us and quite often inefficiency, whether in the form of programmatic complexity or in the form of disk storage or otherwise, has a relatively low precedent compared to everything else we need to achieve if we want happy clients and a pay check.
Lol yeah working in enterprise software for a long time, it’s more like:
- Import what you think you need, let the CI do a security audit, and your senior engineers to berate you if you import a huge unnecessary library where you only need one thing
- Tree shake everything during the CI build so really the only code that gets built for production is what is being used
- Consistently audit imports for security flaws and address them immediately (again, a CI tool)
- CI
Basically just have a really good set of teams working on CI in addition to the backend/frontend/ux/security/infrastructure/ whatever else teams you have
Saying “we can’t in practice reduce the complexity of our dependency tree because we need happy clients and a pay check” is like saying “we can’t in practice turn on the propeller because we need to get this airplane off the ground”.
Clients don’t care much about the dependency graph. They do care about delivering on time and sometimes not reinventing a bunch of wheels is crucial for that.
As the guy people come to when they’ve spent days banging their heads against a dependency conflict problem rather than delivering value for the business, I wish the folks on my team would take the proverb “a little copying is better than a little dependency” to heart a little more.
Amen.
I have sorted out so many JS dependency tangles for my team members, both front end AND back end, that I am loathe to import anything I don’t absolutely have no choice about.
I will rewrite some stuff before I import it…
,
Off topic but what’s the point of a book that thick other than novelty? Would make much more sense to just separate into volumes
You get books like that for voluminous stuff like parliament debate transcripts for an entire parliamentary term.
They’re generally one-off or only a handful printed and kept as archival records.
Almost noone would ever need the physical book, it exists as a physical tome to cite/reference.
Seems like it would break quickly with use
I should check Go’s pkg folder…
Rust is like nodejs
