permalink

34

Checking in front-end dependencies

checkingin

Package management is an important and evolving area in front-end development. It’s also a contentious one with some favoring Bower, others preferring npm and Browserify and a number remaining undecided about the true benefits of these tools. Regardless of the solution you opt for you will come to a point where you will need to decide whether or not you want to check-in the dependencies installed by your package manager into source control.

Note: For those new to Bower, it’s an unopinionated, generic package manager for the web platform that aims to help with front-end package management. Bower works over git, its packages can be made up of any type of asset HTML/CSS/JS and use any type of transport (CJS/AMD/ES6 modules). It’s currently used by Bootstrap, Normalize, AngularJS and many other open-source projects.

For those used to working in Node, this is not a new problem and has been regularly discussed in terms of checking-in node_modules into git. It is however a topic that requires some thought around newer solutions like Bower. The following advice is listed on the Bower homepage:

“If you aren’t authoring a package that is intended to be consumed by others (e.g., you’re building a web app), you should always check installed packages into source control.”

This takes a number of developers by surprise, as some feel you should only have to check-in your bower.json file and run bower install as a part of your tooling-chain instead of checking in all of your /bower_components. Let’s talk through the pros and cons of both approaches.

Check Them In?

Let’s say you’re deploying to production and perhaps have an install or pre-deploy step that runs bower install. GitHub is having network issues, so now you can’t deploy because your dependencies cannot be fetched. This is one of a number of reasons you might want to check in your /bower_components folder.

Those that like checking in their dependencies do so to avoid the remote being down (which may in fact prevent deployment) and to prevent bad dependencies from breaking their app. This could happen even with the presence of a build step as you probably don’t test on each build. Another concern people have is the longevity of package managers and their tooling – as one developer put it: “I worry that 5 years down the road when my app is still in use, these tools may not be around or be completely different”.

That said, the main reason proponents of checking dependencies in favor it is that GitHub may be down, a sub-sub dependency might be unpublished and you really just want to be able to keep the same environment most of the time. Sindre Sorhus, of the Yeoman and Bower teams, prefers this approach, as locking down dependency versions, even at a deep level, isn’t always sufficient as tags and versions may be overridden.

Checking in dependencies is considered a best-practice for deployable projects (not reusable modules) in Node and some feel it should be followed when using Bower too.

Keep Them Out?

The flip-side of the argument to check in your dependencies is that you should only check in your bower.json file, and not your /bower_components folder. The benefit of this approach is that this will keep your repo smaller.

As mentioned, those that prefer to just keep around their bower.json file might just be running bower install as a part of their continuous integration. Your CI might just pull in your dependencies, run all the tests and if it happens that everything works, deploy the exact copy generated onto your staging system.

The downside to this approach is that while you can pin down the dependencies your application directly uses, you can’t directly control the dependencies further down the graph until bower receives a shrinkwrap capability. Running bower install used to be quite slow, but with the recent release of 1.0 bower gained a dramatically improved cache which makes this less of an issue.

With a clear separation between the staging and production deployment processes, this approach can work really well. On a staging system, the application can be thoroughly tested and once it has been verified to work with its current dependencies, the whole project can be cloned over to the production system. This ensures that no further version changes occur between the staging and production deployment.

In Bower 1.0, the --offline flag can help if the remote isn’t available as it can fully fetch package dependencies if they have been fetched before and happen to be compatible. Bower will also be getting a shrinkwrap option in the future, which will allow you to lock a project’s dependencies to a specific version installed so that when someone installs a locked project it will get the exact same code (by checking out commits etc). A combination of the –offline flag, a persistent CI environment keeps the Bower cache between builds and Bower’s shrinkwrap might alleviate the need to commit dependencies.

Words from the Bower team

On this topic, Andre Cruz, one of the main maintainers of Bower feels that if the project you are working on is a medium-large in size where you can’t simply rely on the package manager, then commit your dependencies. On the other side, if you can’t live with huge repositories, where 3rd party code is also committed, then don’t do it. Note that committing deps is only a problem that exists at the top level (application level), not in reusable modules.

So…

Ultimately, the choice of whether or not to check-in all of your /bower_components directory is up to you but it’s important to be able to justify the option you choose for the right reasons. We hope that this post at least provides some food for thought during that decision process.

With thanks to Sindre Sorhus, Pascal Hartig, Stephen Sawchuck and Andre Cruz

34 Comments

  1. One problem we had recently with a .NET project is that certain build processes (not sure which ones – I’m just the front-end guy) will only process file paths up to a certain length so checking in all the node modules would lead to some of their files not being included and the build would break. So sometimes the app’s architecture precludes checking in the modules. Having said that, I’m convinced that checking them in is worth doing if possible – on two occasions minor patches to node modules broke our build and it took forever to find the culprit.

  2. Let’s begin by saying I have no experience with Bower. I’m a back-end developer, and I’ve used Git submodules, Composer (PHP) and Bundler (Ruby) to manage my dependencies.

    I understand that not being able to lock dependencies to a specific version is a huge downside for most projects, as you won’t be able to guarantee a consistent development environment between all people working on the software.

    However, when this will be resolved, I can’t really see any reasons to keep dependencies in your repository. I have only experienced an issue with GitHub’s availability once, but it was a DNS problem that couldn’t be resolved by the website’s maintainers.

    And do you seriously fear that a project as mature as Bower will suddenly disappear? Even if it was discontinued, which is very unlikely, you would surely have the time to migrate to an equivalent tool.

    Uploading all of your project’s dependencies for these reasons seems a bit “dirty” and overkill, unless you have a serious reason to do it.

    • I’m more concerned about the disappearance of my Bower-managed dependencies, since they’re wired directly to other people’s GH repos. A repo owner may get hacked and have their account purged, or decide or become legally obliged to take a repo down or make it private, breaking my next fresh `bower install`. Fixable and avoidable problems by other means, but not necessarily ones I want to discover during a deploy.

      I agree that the proposed shrinkwrap solution helps considerably with the other problems mentioned in this post (and 1.0′s cache handling is sweet), but Sindre’s advice remains the safest approach. For those who understand the risks and have other means to avoid or mitigate them, that’s great, but I think the right call was made when “check in your components” became Bower’s official advice.

      • Most package managers lock the version at the hash and not the tag.
        As Git commit hashes are based on everything that came before its not really possible to hack that.

        Depending on a 3rd party library/component always involves a risk of discontinuing.
        This is why you should try not to depend on any 3rd party library/component, always make it possible to being able to replace it with something else with minimum effort.

        • Yep, that’s one of the “other problems” shrinkwrap will solve. Addy didn’t explicitly say so in this post, but Bower’s proposed “shrinkwrap” solution involves locking to a hash/checksum.

          However, that won’t solve my concern of dependency obliteration, so I agree with the cautious spirit of your comment, and with Bower’s official recommendation to check in front-end dependencies rather than relying on them for deployments.

  3. How do you handle files (from dependency packages) that are not needed by your application? A lot of the bower packages contains files for test, documentation, etc. Do you recommend adding them to source control? How about their bower.json file?

    • Yeah, that’s currently the biggest issue with the Bower ecosystem. Package authors aren’t using the `ignore` property in bower.json to exclude unneeded files. You could however add some of the junk manually to your .gitignore. As for their bower.json, yes, as it’s used by Bower.

  4. I had a problem with checking in nom/grunt dependencies. Because of the recursive nature of some of the plugins. When cloning a repo, the file paths are too long for windows, and some files are not copied.

    Not a problem with an npm install.

  5. At my company, we opted to check-in our dependencies for many of the reasons listed above. But we ran into an issue with a package that installed architecture-specific binaries.

    Since we have developers that use OSX, Windows, and Linux, we wound up having to manually patch this package to include all the binaries, and some extra logic for choosing the right one. I logged an issue here: https://github.com/Obvious/phantomjs/issues/64

    Will be interesting to see if the community starts leaning one way or the other, with regards to expecting dependencies to always be installed locally, or anticipating the dependencies-under-source-control model.

    Great write-up. Thanks!

  6. Is there a recommended approach for using only small pieces of large packages? jQuery UI’s Widget Factory, for example, or jQuery Mobile’s touch events library.

    Clobbering parts of dependencies with .gitignore feels dirty; I think what I’m after is something more like a complementary “ignore” option to fine-tune dependencies whose bower.json doesn’t cut it for me.

  7. Does bower support the idea of a self-hosted repo similar to other package managers? And if so, is it easy to pull down a package from the normal public repo and then publish it to the internal one and have the deploy process be restricted to only using the internally self-hosted repo?

  8. One caveat not mentioned in the article that’s worth noting is that people setup bower dependencies as approximations (usually), which can cause unexpected version increases at ‘bower install’ time.

    There are several options with how you define your dependencies that allow you to set your tolerance for version increases:
    see the npm package.json spec: https://npmjs.org/doc/json.html#dependencies

    {
    “dependencies”: {
    “sass-bootstrap”: “2.3.0″, // must install exactly version 2.3.0
    “requirejs”: “~2.1.4″, // will install anywhere from 2.1.4 to =2.6.2″, // will upgrade to the latest, but at least 2.6.2
    “jquery”: “2.0.x” // similar to tilde style, >=2.0.0 < 2.1.0
    }
    }

    So it's worth noting that you need to make sure you aren't accidentally installing higher versions. The above example is more likely a setting for devDependencies, which allows for version upgrades and testing minor releases as they appear (for dev), but the actual production "dependencies" config is much better hard coded to precise versions of dependencies. As long as you do it that way, I don't think committing the deps is worth doing–but then again, I deploy using git anyway, so I already have a dependency on github being available at deployment time.

    • But that doesn’t solve the problem mentioned in the article: You currently cannot enforce that sub-dependencies follow a strict dependency requirement scheme. This means you may test against a different version than you eventually deploy.

      E.g. you require dependency A at version 1.0.0, which requires dependency B at version “~0.1.0″. Everything works fine in your environment so you deploy it and in the meantime B 0.1.1 is released with a devastating bug that brings down your whole application. You cannot guard against that problem unless you create a deployment setup which clones a components structure which is known to be working.

  9. So to summerize the point of view of those of us who don’t feel good about checking in front end dependenies: we want the benefits of a package/dependency manager, but we don’t like the idea of checking in somebody else’s whole project into our repo.

    Just thinking out loud here, but a good compromise might be to use an automated process to grab the actual files from the local components directory, copy them into the project, then commit those copied files.

    For example, a Grunt task that grabs the files from the Bower components directory that are actually needed (Ex: bootstrap.js) and copies them into directories in the project (something like app/js/lib/bootstrap.js).

  10. Pingback: Checking in front-end dependencies | Nuixen Technology

  11. I was tackling this problem just last week. One of the biggest downsides to committing the dependencies is that you end up with a ton of READMEs, tests, licenses & other junk you don’t really need in your repo.

    Usually you just want the `main` file, but unless the bower component has a super aggressive `ignore` property, or you bloat your own .gitignore with component-specific paths, its tricky to achieve.

    Bower-installer solves this nicely: https://github.com/blittle/bower-installer

  12. Interesting article.

    I’m using git submodules for my front end dependencies, which has the benefit of locking the dependency and keeping the code out of my repository.

    Of course, I still have the problem if GitHub is down. And there may still be projects not on GitHub, but I haven’t used any recently!

  13. I’m actually on the side of not checking them in. I feel that the deployment tool, or build process whatever that may be, should build things so that the deps are included into what gets deployed.

    That said, there should be a way to lock things at a version number, and also some way to simply install the files of a package that you need, and not entire repositories, with all the deps and test code, build tools etc required for a particular package. This creates a lot of unneeded files to be downloaded when installing a package. There are ways to trim the fat sure, but not built into the package managers in any straight-forward way, that I know of anyways.

    I definitely don’t feel that deps should be checkedin, cuz they could then become stale, and start to create a lot of clutter. The larger the repos grow, the more clutter there is, the more difficult it becomes to tell what is being used for what. and what is not used at all even. who cleans that stuff up. That takes time, and money that may be better spent elsewhere.

    I sometimes can be quite opinionated though.

  14. How big are these dependencies? We’re talking about web apps here. Who cares about putting a few MB into your source control?

  15. I recently checked in a new project. The preliminary mock consisted of four local files and 34 files in the bower_components sub folders. I was surely tempted to just check just those four files and .gitignore the bower_components folder, knowing all I have to do is issue a “bower install” to recover them. However…

    1) I did not constrain my bower.json entries to specific version, just latest ones and those dependencies were constrained by them to only minimal ones. Imagine trying to restore this thing in a few years…sheesh.

    2) Normally I have access to the internet. I can’t imagine I will every need to perform software work on this project where I don’t have access to the internet, especially when I need to recover a functional level of this version…which means…it will happen :)

    3) It’s beautiful to know that your commit is a relatively self contained representation of your project (on a target runtime).

  16. Thanks , I have just been looking for information approximately this
    subject for a while and yours is the greatest I have discovered so far.
    But, what about the conclusion? Are you sure about the source?

  17. The issue I have with committing these packages is the structure of the package them selves.

    For example the bootstrap project includes docs, tests, and other junk that has no place in my project.

    I wish these packages were ment as deployable packages with standards rather than the authors complete project.

Leave a Reply

Required fields are marked *.