Caching

Note: This requires knowledge of both the taskcluster index api and artifacts.

Aside from the benefits of unified commands for hg/git heavy use of caches is the other goal of tc-vcs. Often times in a CI type environment various types of vcs operations are repeated (sometime many many times) tc-vcs can make these operations very cheap and fast depending on how you leverage the caches.

Remote caching

The best form of caching provided by tc-vcs is the full clone cache which is stored as a task artifact in taskcluster and then indexed. Refer to the clone namespace algorithm for how that is constructed.

The tc-vcs cli can easily create the caches for you (best to use as a task within docker worker)

# This assumes you are creating a cache via the docker-worker with the
# taskclusterProxy feature enabled...
tc-vcs create-clone-cache --task-id $TASK_ID --run-id $RUN_ID --proxy $URL

In addition the cli a docker image is available for use.

Here is an example task which caches gaia (note you should update deadline/created and use a tag other then latest for best results).

Remember that scopes are required to upload the artifact and index below is a very permissive example...

{
  "provisionerId": "aws-provisioner",
  "workerType": "gaia",
  "created": "2015-01-26T07:36:58.927Z",
  "deadline": "2015-01-26T08:36:58.927Z",
  "scopes": [
    "queue:*",
    "index:*"
  ],
  "payload": {
    "image": "taskcluster/taskcluster-vcs:latest",
    "command": [
      "https://github.com/mozilla-b2g/gaia"
    ],
    "maxRunTime": 3600,
    "features": {
      "taskclusterProxy": true
    }
  },
  "metadata": {
    "name": "cache",
    "description": "cache",
    "owner": "<owner>",
    "source": "<source>"
  }
}

The clone namespace algorithm

Given a url of https://github.com/mozilla-b2g/gaia strip the protocol from the string leaving 'github.com/mozilla/gaia' (the 'url alias') take the url alias and hash with md5. The namespace is:

tc-vcs.v1.clones.${md5('github.com/mozilla-b2g/gaia')}

For repo individual projects are cached using the name of the project + branch (in this case the master branch).

tc-vcs.v1.repo-project.${md5('github/mozilla-b2g/gaia/master')}

User directory caching

The $HOME/.tc-vcs/ folder contains copies of the remote caches (which are usually a tarball of some kind) this allows clone operations to transparently optimize away the (normally) largest and slowest operation you will run.

Aside from performance this allows you to use a "clean slate" (a fresh clone usually) for each clone operation with minimal overhead.