dencold

Understanding npm package upgrades

Sat, 04 May 2019 13:16:08 -0700

I maintain a small task management app called Attainment that I used to teach myself more about front-end javascript development. It uses the Vue.js framework and I use npm to manage my upstream dependencies.

I don’t get much consistent time to hack on it, so it’s usually several months in between development cycles. When I come back to the project, I’ve invariably found myself asking “what’s changed and what should I update?” when it comes to my dependencies. I’ve frequently found myself wondering what the difference is between npm upgrade and npm update and how do I figure out what dependencies are out of date in my web projects. This post is my attempt to learn the best ways to manage this and hopefully it will help others as well.

What’s the status?

My first question is “how do I know where I stand”. I’d like to get a listing of things that are currently out of date. The command to do that is npm outdated. Here’s what it looks like against Attainment after a long period of neglect:

coldwd at thoreau in ~/src/github.com/dencold/attainment-web on master
$ npm outdated
Package                 Current  Wanted         Latest  Location
@vue/cli-plugin-babel     3.0.4   3.7.0          3.7.0  attainment-vue
@vue/cli-plugin-eslint    3.0.4   3.7.0          3.7.0  attainment-vue
@vue/cli-service          3.0.4   3.7.0          3.7.0  attainment-vue
bootstrap                 3.3.7   3.4.1          4.3.1  attainment-vue
chartist                 0.10.1  0.10.1         0.11.0  attainment-vue
firebase                  5.5.2  5.11.1         5.11.1  attainment-vue
fuse.js                   3.2.1   3.4.4          3.4.4  attainment-vue
moment                   2.22.2  2.24.0         2.24.0  attainment-vue
node-sass                 4.9.3  4.12.0         4.12.0  attainment-vue
vue                      2.5.17  2.6.10         2.6.10  attainment-vue
vue-datetime              0.7.1   0.7.1  1.0.0-beta.10  attainment-vue
vue-fuse                  1.5.2   1.5.2          2.0.2  attainment-vue
vue-js-modal             1.3.26  1.3.31         1.3.31  attainment-vue
vue-moment                3.2.0   3.2.0          4.0.0  attainment-vue
vue-router                3.0.1   3.0.6          3.0.6  attainment-vue
vue-template-compiler    2.5.17  2.6.10         2.6.10  attainment-vue
vuex                      3.0.1   3.1.0          3.1.0  attainment-vue

This gives us the dependency in the left most column, followed by the version currently installed, the one we want, and the latest available version upstream. Why is it that sometimes what we want is not the latest version listed? For example look at the details for bootstrap from that output:

Package                 Current  Wanted         Latest  Location
bootstrap                 3.3.7   3.4.1          4.3.1  attainment-vue

The latest version is 4.3.1, but we want 3.4.1. What’s up with that? Well, npm understands semver and we have the “dependencies” section of our package.json file defined as:

  "dependencies": {
    "bootstrap": "^3.4.1",
    "chartist": "^0.10.1",
    "firebase": "^5.11.1",
    "fuse.js": "^3.4.4",
    "moment": "^2.24.0",
    "vue": "^2.6.10",
    "vue-clickaway": "^2.2.2",
    "vue-datetime": "^0.7.1",
    "vue-fuse": "^1.5.2",
    "vue-js-modal": "^1.3.31",
    "vue-moment": "^3.2.0",
    "vue-router": "^3.0.6",
    "vuex": "^3.1.0"
  },

Notice that bootstrap is configured with version ^3.4.1 this tells npm that we can accept any patch or minor version updates (e.g. anything within the 3.0.0 release) safely. So in the output of npm outated, above, bootstrap’s latest version is 4.3.1, but we’ll only accept the latest in the 3.0.0 line, which is 3.4.1.

Let’s update!

Now that we know what is out of date, and we know that npm is (hopefully) protecting us from any breaking changes in a major version change, let’s update our dependencies. We do this using npm update:

coldwd at thoreau in ~/src/github.com/dencold/attainment-web on master
$ npm update
npm WARN deprecated joi@14.3.1: This module has moved and is now available at @hapi/joi. Please update your dependencies as this version is no longer maintained an may contain bugs and security issues.
npm WARN deprecated topo@3.0.3: This module has moved and is now available at @hapi/topo. Please update your dependencies as this version is no longer maintained an may contain bugs and security issues.
npm WARN deprecated hoek@6.1.3: This module has moved and is now available at @hapi/hoek. Please update your dependencies as this version is no longer maintained an may contain bugs and security issues.

> grpc@1.20.0 install /home/coldwd/src/github.com/dencold/attainment-web/node_modules/grpc
> node-pre-gyp install --fallback-to-build --library=static_library

node-pre-gyp WARN Using request for node-pre-gyp https download 
[grpc] Success: "/home/coldwd/src/github.com/dencold/attainment-web/node_modules/grpc/src/node/extension_binary/node-v67-linux-x64-glibc/grpc_node.node" is installed via remote

> node-sass@4.12.0 install /home/coldwd/src/github.com/dencold/attainment-web/node_modules/node-sass
> node scripts/install.js

Downloading binary from https://github.com/sass/node-sass/releases/download/v4.12.0/linux-x64-67_binding.node
Download complete..] - :
Binary saved to /home/coldwd/src/github.com/dencold/attainment-web/node_modules/node-sass/vendor/linux-x64-67/binding.node
Caching binary to /home/coldwd/.npm/node-sass/4.12.0/linux-x64-67_binding.node

> node-sass@4.12.0 postinstall /home/coldwd/src/github.com/dencold/attainment-web/node_modules/node-sass
> node scripts/build.js

Binary found at /home/coldwd/src/github.com/dencold/attainment-web/node_modules/node-sass/vendor/linux-x64-67/binding.node
Testing binary
Binary is fine
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@1.2.9 (node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@1.2.9: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})

+ fuse.js@3.4.4
+ bootstrap@3.4.1
+ moment@2.24.0
+ @vue/cli-plugin-babel@3.7.0
+ @vue/cli-plugin-eslint@3.7.0
+ node-sass@4.12.0
+ vue-js-modal@1.3.31
+ @vue/cli-service@3.7.0
+ vuex@3.1.0
+ vue-template-compiler@2.6.10
+ vue-router@3.0.6
+ vue@2.6.10
+ firebase@5.11.1
added 151 packages from 368 contributors, removed 280 packages, updated 380 packages, moved 33 packages and audited 24396 packages in 53.893s
found 1 vulnerability (1 high)
  run `npm audit fix` to fix them, or `npm audit` for details

That was…more output than I expected. Let’s break that down.

Deciphering update’s output

The first section of output:

npm WARN deprecated joi@14.3.1: This module has moved and is now available at @hapi/joi. Please update your dependencies as this version is no longer maintained an may contain bugs and security issues.
npm WARN deprecated topo@3.0.3: This module has moved and is now available at @hapi/topo. Please update your dependencies as this version is no longer maintained an may contain bugs and security issues.
npm WARN deprecated hoek@6.1.3: This module has moved and is now available at @hapi/hoek. Please update your dependencies as this version is no longer maintained an may contain bugs and security issues.

That’s telling me that I need to change the joi dependency to @hapi/joi, but the joi doesn’t appear anywhere in my package.json file. The answer is to use the npm ls command:

coldwd at thoreau in ~/src/github.com/dencold/attainment-web on master
$ npm ls joi
attainment-vue@0.1.0 /home/coldwd/src/github.com/dencold/attainment-web
└─┬ @vue/cli-plugin-babel@3.7.0
  └─┬ @vue/cli-shared-utils@3.7.0
    └── joi@14.3.1

So, the @vue/cli-shared-utils package depends on joi, and the @vue/cli-shared-utils itself is a dependency of @vue/cli-plugin-babel. We can see this captured in the package-lock.json file:

    "@vue/cli-shared-utils": {
      "version": "3.7.0",
      "resolved": "https://registry.npmjs.org/@vue/cli-shared-utils/-/cli-shared-utils-3.7.0.tgz",
      "integrity": "sha512-+LPDAQ1CE3ci1ADOvNqJMPdqyxgJxOq5HUgGDSKCHwviXF6GtynfljZXiSzgWh5ueMFxJphCfeMsTZqFWwsHVg==",
      "dev": true,
      "requires": {
        "chalk": "^2.4.1",
        "execa": "^1.0.0",
        "joi": "^14.3.0",
        "launch-editor": "^2.2.1",
        "lru-cache": "^5.1.1",
        "node-ipc": "^9.1.1",
        "opn": "^5.3.0",
        "ora": "^3.4.0",
        "request": "^2.87.0",
        "request-promise-native": "^1.0.7",
        "semver": "^6.0.0",
        "string.prototype.padstart": "^3.0.0"
      },
      "dependencies": {
        "semver": {
          "version": "6.0.0",
          "resolved": "https://registry.npmjs.org/semver/-/semver-6.0.0.tgz",
          "integrity": "sha512-0UewU+9rFapKFnlbirLi3byoOuhrSsli/z/ihNnvM24vgF+8sNBiI1LZPBSH9wJKUwaUbw+s3hToDLCXkrghrQ==",
          "dev": true
        }
      }
    },

Buuuuuttttt, where the hell does the root package @vue/cli-plugin-babel come from, you may be asking yourself. It also wasn’t listed in the package.json. That’s because I left out the devDependencies section of package.json:

  "devDependencies": {
    "@vue/cli-plugin-babel": "^3.7.0",
    "@vue/cli-plugin-eslint": "^3.7.0",
    "@vue/cli-service": "^3.7.0",
    "node-sass": "^4.12.0",
    "sass-loader": "^7.0.1",
    "vue-template-compiler": "^2.6.10"
  }

So, I’m getting a warning for a package that is depended by another package that itself is a dependency for the original package that was actually listed as a project dependency. This is a light version of what craziness in package management commonly known as dependency hell in js development.

So, now we’ve figured out where these warnings are coming from, what do we do about them? The answer is…nothing. According to the project maintainers, these are safe to ignore and are the result of transient dependencies that they don’t have control over. ¯\_(ツ)_/¯

The next bit of output that is actually helpful is this one:

+ fuse.js@3.4.4
+ bootstrap@3.4.1
+ moment@2.24.0
+ @vue/cli-plugin-babel@3.7.0
+ @vue/cli-plugin-eslint@3.7.0
+ node-sass@4.12.0
+ vue-js-modal@1.3.31
+ @vue/cli-service@3.7.0
+ vuex@3.1.0
+ vue-template-compiler@2.6.10
+ vue-router@3.0.6
+ vue@2.6.10
+ firebase@5.11.1

These are acutal updates that were performed on our dependencies, you’ll see that it matches up with what we were expecting, above. For example, that bootstrap dependency we made a fuss about has indeed been updated to 3.4.1. Awesome.

Audit checks

If you were paying close attention, you probably noticed a pretty important note at the end of the output:

found 1 vulnerability (1 high)
  run `npm audit fix` to fix them, or `npm audit` for details

This is important and highlights security vulnerabilities that should be taken care of ASAP. Let’s do as the update recommends and run npm audit and see what’s going on here:

coldwd at thoreau in ~/src/github.com/dencold/attainment-web on master
$ npm audit
                                                                                
                       === npm audit security report ===                        
                                                                                
┌──────────────────────────────────────────────────────────────────────────────┐
│                                Manual Review                                 │
│            Some vulnerabilities require your attention to resolve            │
│                                                                              │
│         Visit https://go.npm.me/audit-guide for additional guidance          │
└──────────────────────────────────────────────────────────────────────────────┘
┌───────────────┬──────────────────────────────────────────────────────────────┐
│ High          │ Arbitrary File Overwrite                                     │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Package       │ tar                                                          │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Patched in    │ >=4.4.2                                                      │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Dependency of │ node-sass [dev]                                              │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Path          │ node-sass > node-gyp > tar                                   │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ More info     │ https://npmjs.com/advisories/803                             │
└───────────────┴──────────────────────────────────────────────────────────────┘
found 1 high severity vulnerability in 24396 scanned packages
  1 vulnerability requires manual review. See the full report for details.

So we have a problem with the tar package, specifically any version before 4.4.2. The audit tool lets us know that tar is a dependency of node-sass which is in our devDependencies of our package.json file. Let’s confirm this using the npm ls command:

coldwd at thoreau in ~/src/github.com/dencold/attainment-web on master
$ npm ls tar
attainment-vue@0.1.0 /home/coldwd/src/github.com/dencold/attainment-web
├─┬ firebase@5.11.1
│ └─┬ @firebase/firestore@1.2.2
│   └─┬ grpc@1.20.0
│     └─┬ node-pre-gyp@0.12.0
│       └── tar@4.4.8 
└─┬ node-sass@4.12.0
  └─┬ node-gyp@3.8.0
    └── tar@2.2.1

So we have dependencies on tar in two places: firebase which is already on a version that includes the fix, and node-sass which mathes what we saw in the audit results. We can also use the npm view command to check out what the npm registry knows about the offending package:

coldwd at thoreau in ~/src/github.com/dencold/attainment-web on master
$ npm view node-sass

node-sass@4.12.0 | MIT | deps: 17 | versions: 135
Wrapper around libsass
https://github.com/sass/node-sass

keywords: css, libsass, preprocessor, sass, scss, style

bin: node-sass

dist
.tarball: https://registry.npmjs.org/node-sass/-/node-sass-4.12.0.tgz
.shasum: 0914f531932380114a30cc5fa4fa63233a25f017
.integrity: sha512-A1Iv4oN+Iel6EPv77/HddXErL2a+gZ4uBeZUy+a8O35CFYTXhgA8MgLCWBtwpGZdCvTvQ9d+bQxX/QC36GDPpQ==
.unpackedSize: 1.8 MB

dependencies:
async-foreach: ^0.1.3  glob: ^7.0.3           nan: ^2.13.2           stdout-stream: ^1.4.0  
chalk: ^1.1.1          in-publish: ^2.0.0     node-gyp: ^3.8.0       true-case-path: ^1.0.2 
cross-spawn: ^3.0.0    lodash: ^4.17.11       npmlog: ^4.0.0         
gaze: ^1.0.0           meow: ^3.7.0           request: ^2.88.0       
get-stdin: ^4.0.1      mkdirp: ^0.5.1         sass-graph: ^2.2.4     

maintainers:
- am11 <adeelbm@outlook.com>
- andrewnez <andrewnez@gmail.com>
- saperski <npm@saper.info>
- xzyfer <xzyfer@gmail.com>

dist-tags:
beta: 4.11.0    latest: 4.12.0  next: 4.8.3     

published a week ago by xzyfer <xzyfer@gmail.com>

From this, we can see that we are indeed on on the latest version 4.12.0. The output also lists the project’s homepage which is on github, checking out it’s issues, I found issue 2625 which breaks down what the package maintainers are doing to deal with this. They are waiting for their dependency, node-gyp, to fix it for them.

Dependency hell indeed. Ughhhh.

Checking up

Okay, after all those segues, it might be hard to remember that we did actually run a successful update. What do things look like now? If we run npm outdated again, here’s what the results look like:

coldwd at thoreau in ~/src/github.com/dencold/attainment-web on master*
$ npm outdated
Package       Current  Wanted         Latest  Location
bootstrap       3.4.1   3.4.1          4.3.1  attainment-vue
chartist       0.10.1  0.10.1         0.11.0  attainment-vue
vue-datetime    0.7.1   0.7.1  1.0.0-beta.10  attainment-vue
vue-fuse        1.5.2   1.5.2          2.0.2  attainment-vue
vue-moment      3.2.0   3.2.0          4.0.0  attainment-vue

Exactly as expected, all that’s left are the packages that have major updates left. Nice! Running a git status we see:

coldwd at thoreau in ~/src/github.com/dencold/attainment-web on master*
$ git status
On branch master
Your branch is up to date with 'origin/master'.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

	modified:   package-lock.json
	modified:   package.json

no changes added to commit (use "git add" and/or "git commit -a")

…and diffing package.json:

coldwd at thoreau in ~/src/github.com/dencold/attainment-web on master*
$ git d package.json

 package.json | 26 +++++++++++++-------------
 1 file changed, 13 insertions(+), 13 deletions(-)

diff --git a/package.json b/package.json
index 6bb946c..3382fba 100644
--- a/package.json
+++ b/package.json
@@ -9,26 +9,26 @@
     "lint": "vue-cli-service lint"
   },
   "dependencies": {
-    "bootstrap": "^3.3.7",
+    "bootstrap": "^3.4.1",
     "chartist": "^0.10.1",
-    "firebase": "^5.5.2",
-    "fuse.js": "^3.2.1",
-    "moment": "^2.22.2",
-    "vue": "^2.5.17",
+    "firebase": "^5.11.1",
+    "fuse.js": "^3.4.4",
+    "moment": "^2.24.0",
+    "vue": "^2.6.10",
     "vue-clickaway": "^2.2.2",
     "vue-datetime": "^0.7.1",
     "vue-fuse": "^1.5.2",
-    "vue-js-modal": "^1.3.16",
+    "vue-js-modal": "^1.3.31",
     "vue-moment": "^3.2.0",
-    "vue-router": "^3.0.1",
-    "vuex": "^3.0.1"
+    "vue-router": "^3.0.6",
+    "vuex": "^3.1.0"
   },
   "devDependencies": {
-    "@vue/cli-plugin-babel": "^3.0.4",
-    "@vue/cli-plugin-eslint": "^3.0.4",
-    "@vue/cli-service": "^3.0.4",
-    "node-sass": "^4.9.0",
+    "@vue/cli-plugin-babel": "^3.7.0",
+    "@vue/cli-plugin-eslint": "^3.7.0",
+    "@vue/cli-service": "^3.7.0",
+    "node-sass": "^4.12.0",
     "sass-loader": "^7.0.1",
-    "vue-template-compiler": "^2.5.17"
+    "vue-template-compiler": "^2.6.10"
   }
 }

We can see the changes correctly accounted for.

So…what’s the difference between update and upgrade?

Nothing! npm upgrade is an alias for npm update. As seen from the help output:

$ npm -h update
npm update [-g] [<pkg>...]

aliases: up, upgrade, udpate

However, there is, confusingly enough, an npm-upgrade which is an indpendent package that gives you an interactive mode, with changelog support.

Helpful resources

I found these articles/posts very helpful in writing up my findings:

Photo by Brandon Green on Unsplash

Introducing Hasper

Mon, 14 Nov 2016 08:27:49 -0800

Hasper is a theme for the static site generator hugo that I started hacking primarily for the purposes of serving up content on my own blog dencold.com. It sources heavily from the beautiful casper theme by the folks at Ghost, hence the name: “Hasper”. If you like the styling of casper and use hugo, Hasper may be just what you are looking for.

You can get a sense for what the Hasper theme looks like by browsing around this site. Hasper is responsible for the layout and presentation of everything you see. I also run the latest version of the code here. To give you a quick idea of some of the features in Hasper, here are some screen captures with some highlights:

Hasper Home Page

Hasper Post Page

Hasper is open source (MIT licensed) and can be found on github at: https://github.com/dencold/hasper. The README there goes into more detail on how you can install and use Hasper/Hugo for your own personal website.

I welcome comments and feedback via github issues or pull requests. Please let me know if you are using the theme and how it can be improved.

Features

Hapser has a couple of nice features out of the box:

multiple author support via standard hugo metadata
linking to individual author bio pages
beautiful cover/banner image support
configurable sharing buttons for post details
full support for highlightjs themes
splash image size user-configurable

I keep a CHANGELOG and use SemVer to help keep users abreast of any changes that may affect them.

Attribution

Hasper was originally a fork of the hugo-theme-casper. However, the original author was not responding to pull requests, and I had several changes I needed, so I decided to create Hasper instead.

Dennis Coldwell

Thu, 08 Sep 2016 21:53:28 -0800

Bio

Dennis is a software engineer with over 15 years of experience working in multiple industries. He loves living in California and feels lucky to live in a place surrounded by so much natural beauty. In the winter, you’ll find him most weekends in the Tahoe Basin skiing at Homewood Mountain, where he volunteers as a member of the National Ski Patrol. During the summer, he’s often on his bike exploring the bay area. He is also a volunteer DJ and producer at the mighty 90.7 KALX radio station. He lives in Berkeley with his wife, son, and their pet cat, Mochi.

Employment status

Dennis is currently available for hire. You can find out more about him through these resources:

Migrating to AWS Container Registry

Sat, 14 May 2016 11:53:28 -0800

I use Docker to manage deployments for my applications and, up until April of 2016, I was using Docker Hub as the registry for all of my images. If you have public images, I recommend this route. However, if you have images that need to be kept private (as I do for my client work) you’ll need to pay for the service and it can get quite pricey for many repositories. Thankfully, Docker’s registry is open-sourced and the Docker organization has done a fantastic job of documenting the process to host your own registry. But, if you deploy to AWS, there is an even easier way…

AWS Container Registry announced

In December of 2015 AWS announced that their Container Registery (ECR) was generally available (in AWS terms, “generally” means “us-east-1”). In March of 2016 they opened up the service to us-west-2. This is the region that I deploy my applications on, so I was finally able to make the switch to a managed registry. AWS’s pricing for this service is crazy cheap:

Obviously, YMMV, but I’ve never had these costs add up to more than a dollar, per-month.

Some terms…

AWS provisions a registry for you when you create your first ECR repository. If you are asking yourself, “wait, what’s the difference between a registry and a repository” like I was, this stackoverflow post is for you. So, the AWS registry is analagous to Docker Hub and repositories are how your images (and their tags) are organized within the registry. Here is the list of my public repositories on Docker Hub:

If I wanted to migrate all of those over to AWS, I’d need to explicitly create a repository for each before making a push to the registry. This is different behavior from Docker Hub, which will automatically create a repository for you the first time you push an image.

Step-by-step instructions

Okay, enough background. Let’s get your first repository setup. First, I prefer to do my AWS provisioning using their excellent aws-cli tool. If you haven’t done so already, make sure to install the tool via instructions on their site. Additionally, make sure you’ve run the aws configure command and provided an access key that has admin privilages.

For this example, I’m going to show you how I would take my public pgcli repository and bring it over to my private AWS instance.

Create the registry using the ecr create-repository subcommand (note that the repository name is set to “pgcli”, you should change this to be the name of your repository).

$ aws ecr create-repository --region us-west-2 --repository-name pgcli
{
    "repository": {
        "registryId": "000042290000",
        "repositoryName": "pgcli",
        "repositoryArn": "arn:aws:ecr:us-west-2: 000042290000:repository/pgcli",
        "repositoryUri": "000042290000.dkr.ecr.us-west-2.amazonaws.com/pgcli"
    }
}

Make special note of the repositoryUri in the json response. At this point I like to take the hostname and store it in an env variable because it’s pretty unwieldy (and sadly, AWS does not currently have a way to CNAME/Alias the host). Make sure to replace 000042290000 with your registryId.
```
$ export AWS_ECR_HOST=000042290000.dkr.ecr.us-west-2.amazonaws.com
```

Find out your registry login(note that <base64-encoded-auth> will be a very long byte sequence).

$ aws ecr get-login --region us-west-2
docker login -u AWS -p <base64-encoded-auth> -e none https://000042290000.dkr.ecr.us-west-2.amazonaws.com

If that looks okay, you can eval it directly to log yourself in via docker:

$ $(aws ecr get-login --region us-west-2)
WARNING: login credentials saved in /Users/coldwd/.docker/config.json
Login Succeeded

Tag the latest local version of your image with the full AWS repository name (docker requires this for pushes to private registries).
```
$ docker tag dencold/pgcli:latest $AWS_ECR_HOST/pgcli:latest
```

Finally, you should be able to now push the image up to the repository on your new registry:

$ docker push $AWS_ECR_HOST/pgcli:latest
The push refers to a repository [000042290000.dkr.ecr.us-west-2.amazonaws.com/pgcli]
5f70bf18a086: Layer already exists
15a88e76afed: Layer already exists
97315b41b490: Pushed
c57b1a499184: Layer already exists
cff209dfde52: Layer already exists
5439b1ec108e: Layer already exists
61a73490eb04: Layer already exists
a2e0f03e8793: Layer already exists
65f3b0435c42: Pushed
latest: digest: sha256:63216aaa1cd7aa940cc719ffa8b18cb249567906e7c6ba6baa7b1781d6a9d3fd size: 10312

You’re done! If you login to your AWS console and click on the “Repositories” section of Amazon ECS, you should see it listed:

If you drill into the detail page by clicking on the repository name you should see that the hash checksum matches the output from our push:

Pulling images

You can pull down images from your registry from any host. Only prequisites:

Docker is installed.
aws-cli is installed.
aws-cli is configured with your access key.
You’ve run docker login via the same $(aws ecr get-login --region us-west-2) command you ran above.

To pull down that image we pushed in the last step, you’d just need to run:

$ docker pull 000042290000.dkr.ecr.us-west-2.amazonaws.com/pgcli:latest

Closing thoughts

One disapointment is that AWS gives you no way to name your registry (e.g. 000042290000.dkr.ecr.us-west-2.amazonaws.com) into something a little friendlier, like registry.yourdomain.com. Since there is no way to associate your own TLS certificate with the registry, you’ll get certificate errors if you try to CNAME the host:

$ docker login -u AWS -p REDACTED -e none https://registry.mydomain.com
Error response from daemon: invalid registry endpoint https://registry.mydomain.com/v0/: unable to ping registry endpoint https://registry.mydomain.com/v0/
v2 ping attempt failed with error: Get https://registry.mydomain.com/v2/: x509: certificate is valid for *.dkr.ecr.us-west-2.amazonaws.com, not registry.mydomain.com
v1 ping attempt failed with error: Get https://registry.mydomain.com/v1/_ping: x509: certificate is valid for *.dkr.ecr.us-west-2.amazonaws.com, not registry.mydomain.com

This has been raised on the following AWS form post. Hopefully it is something AWS will support in the future.

Going Nuclear with Git Removal

Thu, 18 Feb 2016 21:53:28 -0800

We recently had a developer commit several large movie files into the git repository. Although this can be reverted with a simple call to git rm, it doesn’t entirely solve the problem. Since git is just tracking snapshots, the mp4 is still in the repository’s history. Every time you initiate a git clone, you will be pulling down that mp4 file. This is unnecessary bloat and should be removed. Here’s a guide to completely torch a file from git so that it is as if it never existed in the first place.

Prep Work

This is going to be a dangerous journey. You are rewriting history and irrevocably deleting data from your repository. Let’s make sure to have some appropriate backups.

First, ensure that you are up to date with your remote:

> cd ~/my_large_repo
> git pull
Already up-to-date.

Once you’re up to date take a full copy of the repo (this can also be achieved with a call to git clone, but a straight copy will make sure all your git remotes/hooks/etc. are preserved).

> cp -Rp ~/my_large_repo ~/my_large_repo.backup

The Simple Option

As mentioned earlier, the easy way to remove the files from your filesystem is via a quick call to git rm. For those who are newer to git, read on. For those who are experienced and just want to fully delete the files, jump to the next section.

Let’s tease this out why git rm doesn’t quite do the job. Here’s what the filesystem looks like:

$ ls -altr videos
total 93768
-rw-r--r--  1 coldwd  staff  20581288 Nov  4 18:08 griffith.webm
-rw-r--r--  1 coldwd  staff  22814007 Nov  4 18:08 griffith.mp4
-rw-r--r--  1 coldwd  staff   2498504 Nov  4 18:08 abstract.webm
-rw-r--r--  1 coldwd  staff   2110649 Nov  4 18:08 abstract.mp4
drwxr-xr-x  8 coldwd  staff       272 Nov  4 18:08 ..
drwxr-xr-x  6 coldwd  staff       204 Nov  4 18:08 .

Those four files come in at just under 48MB of space on the filesystem. Let’s also take note of our entire repository size:

$ du -sk my_large_repo/
251356  my_large_repo/

251MB. Let’s try to chop this down by removing the video files with a call to git rm and also commit it using git commit:

$ git rm -rf videos/*
rm 'videos/abstract.mp4'
rm 'videos/abstract.webm'
rm 'videos/griffith.mp4'
rm 'videos/griffith.webm'
$ git commit -m 'removing video assets'

How did this affect the repo’s size?

$ du -sk my_large_repo/
204468  my_large_repo/

Sweet, that’s what we were hoping for, the filesystem has dropped by same amount as the removed videos. We’re done here, right? Not quite. Git actually still has a record of these files. All we have to do is checkout the previous SHA to bring the deleted files back to life. Here’s what our git history looks like right now:

$ git log
commit a5cec74b28e5d0600e83360f3f4dad16df46d3cc
Author: Dennis Coldwell
Date:   Mon Nov 4 17:01:37 2013 -0800

    removing video assets

commit 046ebf0ddf90273090c193a8c18523ed5da1ca75
Author: Dennis Coldwell
Date:   Sun Nov 3 11:55:13 2013 -0800

    another day, another commit

Let’s checkout the previous commit and see what happens to our repo.

$ git checkout 046ebf0ddf90273090c193a8c18523ed5da1ca75
Previous HEAD position was a5cec74... removing video assets
HEAD is now at 046ebf0... another day, another commit

Filesize goes back to 251MB now:

$ du -sk my_large_repo/
251356  my_large_repo/

So those files are still taking up space, even though you removed them.

The Nuclear Option

There are a couple of ways to go about a full history deletion in git. The one that I prefer is filter-branch. The git book refers to this technique as the nuclear option for good reason:

There is another history-rewriting option that you can use if you need to rewrite a larger number of commits in some scriptable way — for instance, changing your e-mail address globally or removing a file from every commit. The command is filter-branch, and it can rewrite huge swaths of your history.¹

The command is powerful and is meant as a way to apply filters across the entire repository. Let’s take a look at the command option syntax:

git filter-branch [--env-filter <command>] [--tree-filter <command>]
        [--index-filter <command>] [--parent-filter <command>]
        [--msg-filter <command>] [--commit-filter <command>]
        [--tag-name-filter <command>] [--subdirectory-filter <directory>]
        [--prune-empty]
        [--original <namespace>] [-d <directory>] [-f | --force]
        [--] [<rev-list options>…]

That’s a lot of options. Here are the ones that we actually care about:

--index-filter: does most of the work. It will rewrite the git index and apply the given command.
--prune-empty: if there are any commits that become empty after applying the filter command, this option will prune the commit and make a cleaner history.
--tag-name-filter: this option will take care of any tags that exist on your repository. We’ll pass the identity command of cat to this option.
--: indicates the end of the filter-branch options.
--all: this will apply the changes to all refs.

Removing from history

Now here’s the command that will blow that sucker away from your git repo, forever!

git filter-branch \
    --index-filter "git rm --cached -f --ignore-unmatch videos/abstract.mp4 videos/abstract.webm videos/griffith.mp4 videos/griffith.webm" \
    --prune-empty --tag-name-filter cat -- --all

Important It’s really important to give the full path to the files that you want to delete. The first time I ran filter-branch, I just gave the filenames, but git won’t find a match and no changes will be made to your repository.

When you execute the command git will process every commit from the very first one made on your repository. You should see output like the following:

Rewrite aca02f66208ab310a30ac1bad747c44e8e8edbcf (277/403)rm 'videos/griffith.mp4'
rm 'videos/griffith.webm'
Rewrite ae0a6f00bcaa995946365486792ee5aa046f3293 (278/403)rm 'videos/griffith.mp4'
rm 'videos/griffith.webm'
Rewrite 2dde79b5bc88a0416a3d17c0879186993911ef86 (279/403)rm 'videos/abstract.mp4'
rm 'videos/abstract.webm'
rm 'videos/griffith.mp4'
rm 'videos/griffith.webm'

This tells us a couple of interesting things.

The first time one of these videos hit the repository was commit #277 (SHA: aca02f662).
The abstract videos didn’t come into being until two commits later.
Git figures this out and only rewrites the files that match for each commit.

Once everything is finished, let’s review the log again:

$ git log
commit 39c68f216b073e33220fdc6d658c16cad2999782
Author: Dennis Coldwell
Date:   Sun Nov 3 11:55:13 2013 -0800

    another day, another commit

commit 0b76a5e6081a75305f23675d04e561bc7bb31e85
Author: Dennis Coldwell
Date:   Sat Nov 2 17:27:53 2013 -0700

    work, work

Interesting! My original git rm commit a5cec74b28e5d0600e83360f3f4dad16df46d3cc is completely gone. Further, the prior commit 046ebf0ddf90273090c193a8c18523ed5da1ca75 has now been reset to 39c68f216b073e33220fdc6d658c16cad2999782. In fact every commit after the first time one of these videos hit the repository has been reindexed (commit #277/aca02f66208ab310a30ac1bad747c44e8e8edbcf).

This is exactly what we were hoping to have happen. But…the filesize hasn’t dropped as dramatically as we may have hoped:

du -sk my_large_repo/
207192  my_large_repo/

This is because the objects will still exist in the local repository until they’ve been dereferenced/garbage collected. Github has a handy reference on cleaning up and reclaiming space. Here’s a summary from that post:

$ rm -rf .git/refs/original/
$ git reflog expire --expire=now --all
$ git gc --prune=now
# Counting objects: 2437, done.
# Delta compression using up to 4 threads.
# Compressing objects: 100% (1378/1378), done.
# Writing objects: 100% (2437/2437), done.
# Total 2437 (delta 1461), reused 1802 (delta 1048)

$ git gc --aggressive --prune=now
# Counting objects: 2437, done.
# Delta compression using up to 4 threads.
# Compressing objects: 100% (2426/2426), done.
# Writing objects: 100% (2437/2437), done.
# Total 2437 (delta 1483), reused 0 (delta 0)

What does the filesystem look like now?

$ du -sk my_large_repo/
117004  my_large_repo/

MUCH better! We’ve dropped the overall repo size by more than half. We are in much better shape than just having run a git rm.

Some Helpful Links

I made use of help from the following posts:

https://git-scm.com/book/en/v2/Git-Tools-Rewriting-History#The-Nuclear-Option:-filter-branch ^[return]

Django Caching with Redis

Wed, 08 Jan 2014 16:18:29 -0800

Django has a pretty good documentation page for enabling caching on the framework. It gives an overview of why caches are a good idea, when to use them, and the various configuration options to make your backend play nice with Django. However, of the four backends mentioned in the docs (memcached, database, filesystem, local-memory), Redis is not listed as an option. This is surprising as Redis is a key-value store that works very well as a cache server and compares favorably to memcached.

Here’s a little guide to get up and running with Redis to remedy this omission on the django docs.

Redis Features

Redis also has a lot of features that memcached is lacking:

persistence: Redis has persistence configured by default. If your cache server gets restarted, you don’t lose all of your data.
robust data structures: memcached can only handle strings & serialized objects. Redis has strings, sets, lists, hashes, and sorted sets.
replication: it’s a one line configuration setting to enable master-slave replication in Redis.

Installing Redis

You’ll first want to install Redis on your local machine, I’m on a Mac, so I’ve included notes on how to setup your system with OS X. You can also find the installation instructions directly on the Redis website if you are on another OS.

If your machine is on Mac OS X 10.9 (Mavericks), make sure to install xcode developer tools so you have access to gcc. Luckily, Apple has made this easy and can be installed with a one-liner from the command line:

$ xcode-select --install

If you are on an older OS X version, refer to this Stack Overflow post to get developer tools installed. Once that’s squared away, here are the steps to install Redis 2.8.3 (latest stable release as of this post):

$ mkdir -p ~/sysinfra/redis
$ cd ~/sysinfra/redis
$ curl -O http://download.redis.io/releases/redis-2.8.3.tar.gz
$ tar xzf redis-2.8.3.tar.gz
$ cd redis-2.8.3
$ make

You’ll see a lot of compiler output. After it finishes, it recommends you to run make test to ensure that the install was successful. Let’s do that, but make sure to do this with sudo as you’ll get errors when Redis tries to check for memory leaks on separate processes. Here’s the command:

$ sudo make test

If everything worked, you’ll see something like this:

If so, you’re good to go! Let’s get the Redis server up and running:

$ src/redis-server

Which should look like this:

Props to antirez for the awesome use of ascii art. Also, a note that in a production environment, we’d want to use a process control system (like supervisor) and not run the server in the foreground. For the purposes of learning, it will be helpful to monitor what is going on with the server.

Django bindings

Redis is installed, running, and ready to cache stuff. But how do we get our django application to communicate with the Redis server? If you look into the official django docs, you’ll see a full writeup for memcached, here’s what that setup looks like in your settings.py:

CACHES = {
    'default': {
        'BACKEND': 'django.core.cache.backends.memcached.MemcachedCache',
        'LOCATION': '127.0.0.1:11211',
    }
}

This is super straightforward. We just choose the memchached backend that is included in django’s core package and specify a location that the memcached server is running. With Redis, we’ll need to first install the backend bindings and then configure the cache in a similar manner.

The paradox of choice

The two major projects that provide Redis bindings to django are:

It can be a little hard to distinguish between the two libraries. They both have similar github metrics:

vs.

Not sure which to choose, I decided to conduct a “scientific” poll on the #django IRC channel, here are the results:

Pretty inconclusive. Digging deeper, it turns out that django-redis actually began as a fork of django-redis-cache. If you dig into the source code you’ll see that a lot of the core classes are still very similar. Also, django-redis still installs to the same “redis_cache” namespace as django-redis-cache. This means if you were ever to install both packages, you’d end up with a namespace clash. This has been raised as an issue and the author has acknowledged the problem.

My recommendation is to go with django-redis-cache. It is the original library and has more community involvement. Additionally, there are contributions on the repository from jezdez and carljm, two devs I highly respect.

Installation

Use pip to install the package (preferably into a virtualenv):

$ pip install django-redis-cache

This will automatically pull and install the redis python package as it is a dependency.

While you’re at it, you should also install the hiredis parser. This is the most efficient library for parsing the data coming back from the Redis server.

$ pip install hiredis

Configuration

We’re now ready to configure django to use Redis. As a simple configuration you can add this to your settings.py:

CACHES = {
    'default': {
        'BACKEND': 'redis_cache.RedisCache',
        'LOCATION': '127.0.0.1:6379',
        'OPTIONS': {
            'DB': 0,
            'PASSWORD': '',
            'PARSER_CLASS': 'redis.connection.HiredisParser'
        },
    },
}

Let’s quickly break that down. The django-redis-cache package installs under the “redis_cache” namespace. Here, we are setting up our “default” cache to use our django-redis-cache backend, and the server is located on localhost at 127.0.0.1:6379. Redis allows for multiple databases per server and defines this with a zero-based numeric index. The Redis default database is always 0, and we are setting our configuration to match that with 'DB': 0. We didn’t set a password on the Redis server so we keep that configured as an empty string.

Testing the cache

Now that we have the Redis server running and django is configured to talk to it. Let’s test this out and set some cache values. To do this, let’s use the django shell management command.

$ cd to/your/django/project
$ python manage.py shell
Python 2.7.5 (default, Nov  6 2013, 00:00:21)
[GCC 4.2.1 Compatible Apple LLVM 5.0 (clang-500.2.79)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>>>

From here we should be able to set some items to cache:

>>> from django.core.cache import cache
>>> cache.set('life_universe_everything', '42')
True
>>> cache.get('life_universe_everything')
'42'

The default TTL for django cache items is 300 seconds. So if you wait five minutes and try to get our value from the cache again. Here’s what happens:

>>> cache.get('life_universe_everything')
>>>

The cache has expired and the backend returns None. You can change the TTL by passing an optional timeout to the set method like so:

>>> cache.set('life_universe_everything', '42', 3600)
True

That call would cache the “life_universe_everything” key for 3600 seconds – one hour.

Closing it out

That’s pretty much all you need to get started with Redis and django. You can now cache expensive calculations very easily in your view/model code. This can have a major impact to the response times on your site.

This post has focused mostly on getting a Redis server up and functional, but it has only scratched the surface of what you can do with the cache. The testing examples above use the low level cache API. However, you can also make use of full site caching, per-view caching, and cached sessions.

Also, if you are really curious on how the cache is implemented, you can read my post analyzing cache records.

Happy caching!

The End of Object Inheritance

Mon, 18 Mar 2013 21:53:28 -0800

Here are my notes from the PyCon2013 talk given by Augie Fackler & Nathaniel Manista titled The End Of Object Inheritance & The Beginning Of A New Modularity. That link will bring you to a youtube video of the talk.

Three Premises for Software

Augie & Nathaniel first discuss three basic concepts that are “uncontroversial” in software development.

Use types for nouns

When we are talking about data and components that we need in the system, we are using types to refer to them. Nathaniel brings up the differences between a static language like C where the type will be defined at compile-time vs. a dynamic language like Python where we express type in terms of attributes and methods that we rely on from that object.

Express ourselves structurally

Makes the distinction away from comments which you hardly ever read, or the docstrings which you may read half of the time. Instead, good code should document itself. For example, we could just throw all our code into a giant module, but instead we break it up into directory structures that make it much more maintainable.

Most programming is parametric programming

Meaning most programming you do is partial. You are not writing all of the logic in one giant function, rather you are relying on inputs to determine the behavior to take. For functions you have arguments that will change the behavior of the output, for a module you have dependencies specified in your import list.

Example

class MyAbstractBase(object):
    # pretend this is fully implemented
    def orange_method(self):
        ...

    def green_method(self):
        raise NotImplementedError()

    def blue_method(self):
        ...

So the client of this class would subclass the base and then provide an implementation for green_method. Something like:

class ClientSubclass(MyAbstractBase):
    # we've now got a concrete implementation of the green method
    def green_method(self):
        ...

Notice that the client has no reference to blue and orange methods, but it is free to call them. However, what happens if you don’t want the client class to call the orange or blue methods? There really isn’t a good solution to hide them from subclasses. People generally rely on comments or docstrings to explain and note that you cannot call certain methods which is kludgy. “If you are explaining, even if you’re right, you’re losing.” This is the case with the inheritance based implementation, above. It’s technically correct, but you are using that out-of-band documentation channel that many people don’t check. In closing, we are violating premises 1 & 2 in order to get to 3.

Composition

Everyone has heard of the phrase “favor composition over inheritance”, it means rather than inheriting you are going to pass one layer into another. It’s an explicitly one-way relationship. With inheritance you can have bi-directional communication between the layers, for example, maybe the blue and orange methods have some communication through attributes on self.

Inhertiance rewrite

Here’s how you could use composition to solve the problem above.

class Blue(object):
    def blue_method(self):
        raise NotImplementedError()

class Green(object):
    def green_method(self):
        raise NotImplementedError()

class Orange(object):
    def orange_method(self):
        raise NotImplementedError()

You now have 3 interfaces for the behaviors you are looking to implement. This means that we have given types to every noun we were talking about. Here’s how we implement:

class ImplementedBlue(Blue):
    def blue_method(self):
        ...

class ImplementedGreen(Green):
    def __init__(self, blue):
        self._blue = blue

    def green_method(self):
        ...

Note that now the green implementation has no idea about Orange. It doesn’t need it and doesn’t care. Instead we inject the blue depency on init and we can use it explicitly.

Overlapping interfaces

Augie makes the point that one of the nice things about inheritance is that you can share method definitions between things very easily. You can define one method that you’ll get “for free” to all derived subclasses. However, readability really suffers in this case. Say you have a method that is defined in the the AbstractBaseClass, then overrided in a subclass (but with a call to super to get the base class functionality) that you are using in your implementation. You have to look at that subclass’s implementation, then go up the MRO to figure out the base implementation. If you have multiple inheritance in effect, it’s a lot of work just to figure out what the full implementation looks like.

Delegation, on the other hand, is very simple. You define a one-line function that states that you need a resource “green” that access the “foo” attribute and return it, is much clearer. You don’t have to manually traverse what the interpretter is going to figure out for you at runtime as we do in the inheritance example.

Fault (In)Tolerance

In software architecture it is very important to have fault intolerance. We want things to fail as soon as possible and to let us know that we are doing wrong. Allowing an error to fail silently is a “bad thing”.

Object inhertiance fails at fault tolerance. What happens if an extending class calls a method that it’s not supposed to? You could have silent state corruption if the class stores mutable state. Nathaniel brought up a really good example of having perfect behavior for years if, for example, you have a base class that reserves the right to call a function, but doesn’t until a future version. Maybe you have an inherited method that finds all occurances of a word in self.file (initialized by the base class, of course) and then deletes the file. Your concrete implementation was calling because it wanted that behavior, but then a future version changes that behavior and the base class calls it directly. That file is now deleted and your implementation is broken, without you having changed any of your code! With dependency injection, you wouldn’t have that problem as the self.file wouldn’t be shared state that both methods are relying on.

Remember to “Make Illegal States Unrepresentable.”
Break all bidrectional relationships!
Use the unix mentality: do one thing and do it well.
Your constructors will evaporate into factory functions
Every one of your modules will transform into collections of abstract types and functions that operate on the types.
You shouldn’t be afraid if some of your methods end up being very very powerful.
Passing around functions, do it! Think of it as an object with only one method.

Mon, 01 Jan 0001 00:00:00 +0000

Chrome Dev Summit Keynote

MC’s monica &

5th year doing the summit
TIL Downasaur

Progressive Web Apps - trivago - 150% increase in engagement added to homescreen - Service worker in webkit / edge

Workbox

developers.google.com/ look for Workbox
has several caching strategies builtin

Https adoption ~63% of all traffic on Chrome

One-Tap Sign-Up - developers.google.com/identity - Signup on mobile device, automatically signs in back on desktop chrome. cross-platform.

The “Holy Trinity”

HTML: Web Components no real update, just polymer, yadda yadda
CSS: Grid
JS: ECMAScript 2015 fully implemented except for 1 feature (community has decided it was a badidea)
v8 - Chrome’s javascript engine. New version released. 22% faster on Speedometer. 5% faster on top 25 websites. 40% faster on ARES6.
v8 has support for WASM (WebAssembly). available on ALL browsers (last holdout was IE). So we now have another compilation* target
All documentation is moving to MDN. Centralizes across webkit, chrome, edge.

Dev tools

Lighthouse
Puppeteer (headless chrome instance) - anything that chrome can do puppeteer can do. CHECK THIS OUT!!! trypuppeteer.
Webpagetest - new version Chrome User Experience Report. try this out as well.

Keynote - underwhelmed. Very polished, not a lot of amazing announcments

A New Bar for Modern Web Standards

Thao Tran & Chris Wilson

Talking about Scuba Diving. Old way pretty terrible data access. Contrast with HighTide. Need to prioritize for user experience. Don’t just provide a service.

Ele.me featured as the Chinese Food & Ordering app. paint in 400ms, fully interactive in 2s. They have 260M users (the same as the entire population of the United States). They are the leader in food ordering. Built out as a PWA.

Jio (Indian Entertainment).

Still seems like the same messaging. PWA’s we’re big in 2G countries.

Mercado Libre (mercadolibre.com) big app in Brazil/Latin America.
Petlove.com.br
Flipkart
Starbucks PWA - able to pay via barcode. Don’t need a network connection.
m.uber.com - 3s interactive on 2G, 50kb core app
instagram - offline sync
ebay (exporing offline) great posterchild for problem with PWA. they aren’t investing in using the tech on the main site, instead doing it on ancillary pieces that doesn’t mean much.

Asking Permissions

Right now its an ignorable request that pops up at bottom of screen (mobile)
Moving to a modal dialog to force Block/Allow
dc - not sure this helps at all. We really need a better way to present context

West Elm

November 2016 Launch West Elm beta
March 2017 good feedback
May 2017 rollout to 10% traffic
June 2017 - rebuild with webcomponents
Rollout to 25% of traffic across ios and android
October 2017 Pottery Barn/WilliamsSonoma/West Elm retooling to new version

Building

Owen Campbell-Moore

good talk!

Wanted to make a cross-platform messaging app. Went from Mobile Chrome Apps -> Cordova -> Java extensions. Ended up with an app that was no longer cross-platform, and was super hacky.

Installing to home screen: launched to 100% on Android.

New image upload dialog. Really bad before.

Image Chooser (all we need to do is have input tag with type=“file” accept=“image/*” and you’ll get it for free.
Image Capture API
Background Sync API

One tap signup/signin. Cross browser and pretty awesome.

Faster Talk

Addy

Evolving RAIL
Switching to User-happiness (First Paint, First Meaningful, First interactive)
Parsing javascript is expensive
PRPL
5s time to full interactive is our time budget

Good tools to check out how your app is doing: - Calibre - Speedcurve - BundleSize - beta.httparchive

State of the web on mobile

Chrome User Experience Report - available as a Big Query result set - requires signing up for Google Cloud Platform :( - Lookup origin for yelp!

Take advantage of “fetch” and “preload” apis for handling slow network

Announcing two new members of the PWA family: - Pinterest - Tinder

dc side note - a lot of the PWA “wins” are comparing the cost of building the experience. E.g. 56MB for Mobile App on iOS vs. 250kb on PWA. Seems a bit disengenious to me.

Route-based code-splitting works! Something we might want to do at Yelp.

Service worker / Workbox

###

Using Workbox to make ServiceWorker configuration a lot easier.

workboxSW.strategies.staleWhileRevalidate

Chrome Dev Summit - Day 2

tag gives superscripts! blah blah blah howdy

Tooling

Paul Irish / Eric Bidelman

DevTools changes

Local caching for dev tool changes (overrides)
Performance Monitor - live realtime views of CPU/Memory/etc.
New Filter Sidebar on the the Console
Put “await” in front of any async call that returns a promise, have it resolve for you

Headless Chrome & Puppeteer

What is a headless browser? Well there’s no UI, invoked from cli.

–remote-debugging-port=9222
Chrome.launch({port: 92222, chromeFlags: [‘–headless’]
Selenium, PhantomJS, etc.
Testing Headless Chrome Easy: Puppeteer. It’s a node.js library
async/await, ES6 features, also supports Node 6
Zero config, bundles latest chromium
Chrome’s reference for using devtools
Super easy to take a screenshot (5 lines of code)
Easy page metrics embedding
Has debugging options
try-puppeteer.appspot.com

Future of performance on the web

Sam Saccone

Moving into single page granular assets. E.g. page1.js, page2.js, etc. etc.
TL;DR don’t do it. Chrome team tried to do this with moment.js. Unbundling caused an extra 100ms time.
Lots of stuff were proposals…not shipped, why even tell us about it?
Example “DOMChangeList” -

V8 Talk

22% better and 40% better on two major benchmarks since last year
Reducing Jank –> typically due to GC
V8 team is taking garbage collection off of the main thread
They leave some small slivers on the main thread, but now moved off and takes advantage of multiple cores

Real World WASM

A new capability for the web
Compile target for languages used in native apps
Binary format, safer than JS

Polymer Talk

Taylor Savage

The javascript-industrial-complex: feedback loop that encourages short term gains
“beware the local maximum”
path to a new web platform feature. very slooowwww…
spec proposal, implementation, ship, fix, wait for other browsers to catch up
web components & polymer

Javascript Frameworks Panel Discussion

Moderated by Addy

“Anything you are looking forward to” DOM Changelist! Moving more code to be executed on WebWorker and off of main thread.
Observables - Angular uses this a lot. Mutation observer etc., move to platform. Would be way better if we have primitives.
Moving from callbacks to promises. Being able to pass along an object.
Changelist + Worker DOM.
Async append -> build up a tree of DOM and asyncronously append into the dom.
Andrew Clark: (React) “Except for redesigning layout which we can’t, but maybe we will”. WTF?
What framework would you use?
Frameworks…you only have about 130kb, how much of that 130kb is your framework taking up?
Looks over at React.
Showing that there is a million+1 js frameworks, check out the stage presence at the panel discussion -