Git Clone without a Password Prompt

This is the cheatsheet for when you need to clone a private repository without an interactive password prompt.

These techniques are language agnostic and demonstrate how you can use git as a free private package manager - since all major package managers supports git.

Git Credentials Overview

These are all the ways and tools by which you can securely authenticate over git to clone a repository.

SSH Public Keys
- SSH_ASKPASS
API Access Tokens
- GIT_ASKPASS
- .gitconfig insteadOf
- .gitconfig [credential]
- .git-credentials
- .netrc
Private Packages (for Free)
- node / npm package.json
- python / pip / eggs requirements.txt
- ruby gems Gemfile
- golang go.mod
A Note on Security

Please send corrections, amendments, and suggestions to coolaj86@gmail.com.

The Silver Bullet™

Want Just Works™ for build, test, and deployment pipelines? This is the magic silver bullet:

Note: for local development you should probably use SSH Keys, as per usual.

Get your Access Token (described in the Git Token section below) and set it in an environment variable:

(preferably the Git user for this token would only have read-only access to the necessary repositories)

MY_GIT_TOKEN=xxxxxxxxxxxxxxxx

Set .gitconfig to handle all 3 styles of Git URL as authenticated https URLs. For Github URLs, copy and run these lines verbatim:

git config --global url."https://api@github.com/".insteadOf "https://github.com/"
git config --global url."https://ssh@github.com/".insteadOf "ssh://git@github.com/"
git config --global url."https://git@github.com/".insteadOf "git@github.com:"

Note: the "usernames" are arbitrary, but must be uinque, as the insteadOf syntax is reversed from what you might expect: the replacement string is key, and the pattern string is the value.

Create a GIT_ASKPASS script that echoes the token environment variable:

echo 'echo $MY_GIT_TOKEN' > $HOME/.git-askpass
chmod +x $HOME/.git-askpass

Set the environment to use it:

GIT_ASKPASS=$HOME/.git-askpass

Congrats, now any automated tool cloning git repositories won't be obstructed by username or password prompts, whether using https or either style of ssh url.

Not using Github?

For other platforms (Gitea, Github, Bitbucket), just change the URL. Don't change the usernames (although arbitrary, they're needed for distinct config entries).

Compatibility

This works locally in MacOS, Linux, Windows (in Bash), Docker, CircleCI, Heroku, Akkeris, etc.

More Info

For more information see the the ".gitconfig insteadOf" section below.

Security

For security concerns see "A Note on Security".

The Simplest Bullet™

Want the simplest solution for build, test, and deployment pipelines? This is it, hands down:

Note: for local development you should probably use SSH Keys, as per usual.

Create a Git Token and set it to an environment variable:

(preferably the chosen git user should only have read-only access to the necessary repositories)

TOKEN=xxxxxxxxxxxxxxxx

Set .gitconfig to handle all 3 styles of Git URL as authenticated https URLs, including the full authentication details in the URL:

git config --global url."https://api:$TOKEN@github.com/".insteadOf "https://github.com/"
git config --global url."https://ssh:$TOKEN@github.com/".insteadOf "ssh://git@github.com/"
git config --global url."https://git:$TOKEN@github.com/".insteadOf "git@github.com:"

Why isn't this The Silver Bullet™?

This approach is slightly less verbose, and slightly easier to understand, and is therefore a better choice in my opinion, and what I suspect many security experts would prefer.

However, saving a credential to a disk causes a knee-jerk reaction for a lot of people familiar with the "never save credentials to disk" dogma.

See "Security: Perception isn't Reality", as well as the notes in the security section far below.

SSH Public Key

To clone repositories like this:

git clone ssh://git@github.com/example/project.git

git clone git@code.example.com:project.git

How to get your ssh key:

[ ! -f "$HOME/.ssh/id_rsa.pub" ] && ssh-keygen
cat ~/.ssh/id_rsa.pub

How to add to the OS credential keychain:

ssh-add ~/.ssh/id_rsa

How to use it with Github:

Account > Settings > SSH and GPG keys > New SSH Key

https://github.com/settings/ssh/new

How to use it with Gitea:

Account > Settings > SSH / GPG Keys > Add Key

https://try.gitea.io/user/settings/keys

SSH_ASKPASS

How to create an SSH_ASKPASS script:

echo 'echo $MY_SSH_PASSPHRASE' > $HOME/.ssh-askpass

How to use it:

export MY_SSH_PASSPHRASE=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
export SSH_ASKPASS=$HOME/.ssh-askpass

Access Token

To clone repositories like this:

git clone https://github.com/example/project.git

Also for private packages in package.json, requirements.txt, go.mod, Gemfile, etc.

How get an API Token from Github:

Account > Settings > Developer Settings > Personal Access Tokens > Generate new token

https://github.com/settings/tokens

How get an API Token from Gitea:

Account > Settings > Applications > Manage Access Tokens > Generate Token

https://try.gitea.io/user/settings/applications

GIT_ASKPASS

How to create an GIT_ASKPASS script:

echo 'echo $MY_GIT_TOKEN' > $HOME/.git-askpass

How to use it:

export MY_GIT_TOKEN=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
export GIT_ASKPASS=$HOME/.git-askpass

git clone https://token@code.example.com/project.git

The script receives stdin in the form of:

Password for 'scheme://host.tld':

The script receives Git ENVs such as:

GIT_DIR=/Users/me/project/.git
GIT_EXEC_PATH=/usr/local/Cellar/git/2.19.0_1/libexec/git-core
GIT_PREFIX=

.gitconfig insteadOf

Replace a plain URL with a token-authenticated URL:

git config --global url."https://token:$MY_GIT_TOKEN@github.com/".insteadOf "https://github.com/"

Swap either style of SSH URL for an HTTPS token-authenticated URL:

git config --global url."https://ssh:$MY_GIT_TOKEN@github.com/".insteadOf "ssh://git@github.com/"
git config --global url."https://git:$MY_GIT_TOKEN@github.com/".insteadOf "git@github.com:"

User-based token swapping:

git config --global url."https://api:$MY_GIT_TOKEN@github.com/".insteadOf "https://api@github.com/"

Swap a dummy token with a live token:

git config --global url."https://live:$MY_GIT_TOKEN@github.com/".insteadOf "https://api:xxxxxxxx@github.com/"

Authenticate by organization:

git config --global url."https://token:$MY_GIT_TOKEN@github.com/myorg/".insteadOf "https://github.com/myorg/"

Authenticate by project:

git config --global url."https://token:$MY_GIT_TOKEN@github.com/myorg/project.git".insteadOf "https://github.com/myorg/project.git"

What the updated .gitconfig looks like:

~/.gitconfig:

[url "https://token:deadbeef@github.com/"]
    insteadOf = https://github.com/
[url "https://ssh:deadbeef@github.com/"]
    insteadOf = ssh://git@github.com/
[url "https://git:deadbeef@github.com/"]
    insteadOf = git@github.com:
[url "https://api:deadbeef@github.com/"]
    insteadOf = https://api@github.com/
[url "https://live:deadbeef@github.com/"]
    insteadOf = https://api:xxxxxxxx@github.com/
[url "https://token:deadbeef@github.com/myorg/"]
    insteadOf = https://api:xxxxxxxx@github.com/myorg/
[url "https://token:deadbeef@github.com/myorg/project.git"]
    insteadOf = https://api:xxxxxxxx@github.com/myorg/project.git

.gitconfig [credential]

Sets a default username for a given repository url.

(can't store passwords, not very useful)

.gitconfig:

[credential "https://code.example.com"]
    username = token

.git-credentials

Enable git's plaintext credential store:

git config --global credential.helper store

Single Repository: --local
Single User Account: --global
Entire System: --system

Save Token Credential for HTTPS Authentication:

echo "protocol=https
host=github.com
path=
username=token
password=$GIT_TOKEN" | git credential-store --file $HOME/.git-credentials store

Save SSH Passphrase for a shared private key (for pipeline deploments):

(unsafe for local developer accounts)

echo "protocol=ssh
host=github.com
path=
username=git
password=$MY_SSH_PASSPHRASE" | git credential-store --file $HOME/.git-credentials store

Save Token per Organization:

echo "protocol=https
host=github.com
path=myorganization/
username=token
password=$GIT_TOKEN" | git credential-store --file $HOME/.git-credentials store

Save Token per Project:

echo "protocol=https
host=code.example.com
path=project.git
username=token
password=$GIT_TOKEN" | git credential-store --file $HOME/.git-credentials store

Save Token per User:

(the username portion is ignored when using token authentication, so you can set it to any arbitrary string)

echo "protocol=https
host=code.example.com
path=
username=jackjack
password=$GIT_TOKEN" | git credential-store --file $HOME/.git-credentials store

What the updated .gitconfig looks like:

~/.git-config:

[credential]
    helper = store

What the updated .git-credentials looks like:

~/.git-credentials:

https://token:xxxxxxxxxxxxxxxx@github.com/
ssh://git:xxxxxxxxxxxxxxxx@github.com/
https://token:xxxxxxxxxxxxxxxx@github.com/myorganization/
https://token:xxxxxxxxxxxxxxxx@code.example.com/project.git
https://jackjack:xxxxxxxxxxxxxxxx@code.example.com/

Query to see which credentials will match a given repository URL:

echo "protocol=https
host=github.com
path=myorganization/" | git credential fill

Query which git credential helpers are in use:

(may neglect to show active system keystores)

git config --show-origin --get credential.helper

.netrc

Provides authentication for git, curl, ftp, heroku, akkeris, and many other system utilities.

(_netrc on Windows)

Set HTTPS Token for any site:

(does not match url path)

echo "machine github.com
  login token
  password xxxxxxxxxxxxxxxx
" >> $HOME/.netrc

echo "machine api.heroku.com
  login jon@example.com
  password 00000000-0000-4000-8000-000000000000
" >> $HOME/.netrc

Free Private Packages

You don't nee a paid package registry to host a private package.

Git is universally supported by all package managers and git credentials provide package authentication for free.

package.json: node, npm

Install private package with HTTPS Token authentication:

(master will be chosen if #commit-ish is omitted)

npm install 'git+https://github.com/example/project.git#v1.0.1'

Install private package with SSH Passphrase-Protected Key Authentication:

npm install 'git+ssh://git@github.com/example/project.git#v1.0.1'

The updated package.json will look like:

package.json:

{
    "dependencies": {
        "project": "git+https://github.com/example/project.git#v1.0.1"
    }
}

requirements.txt: python, pip, eggs

Install with HTTPS Token Authentication

pip install 'git+https://github.com/example/myproject.git@v1.0.1#egg=myproject'

Install with SSH Key Authentication

pip install 'git+ssh://git@github.com/example/myproject.git@v1.0.1#egg=myproject'

Update requirements.txt:

pip freeze --local | grep -v myproject > requirements.txt

The requirements.txt will update to include the editable git repository:

-e git+https://github.com/example/myproject.git@v1.0.1#egg=myproject

Gemfile: ruby, gems, bundler

(update Gemfile directly)

Install with HTTPS Token Authentication:

gem 'myproject', :git => 'https://github.com/example/myproject.git, :branch => 'master'

Install with SSH Passphrase-Protected Key Authentication:

gem 'myproject', :git => 'ssh://git@github.com/example/myproject.git, :tag => 'v1.0.1'

go.mod: Go

Install a private package

go get code.example.com/project

(requires .gitconfig "insteadOf" directive, only when using local SSH, or SSH Passphrase-Protected Shared Private Keys in a pipeline)

git config --global url."ssh://git@github.com/".insteadOf "https://github.com/"

A Note on Security

Passwords:

Passwords are for ignoramuses. Don't be an ignoramus.

(use keys, tokens, etc)

For Local Development:

Access Tokens and (Unprotected) Device Private Keys

Saving to local disk is expected (i.e. developer laptop)
- For SSH, Passphrase Protection is preferred
Do NOT commit to repository
Do NOT share between computers, or people

For CI/CD and Deployment Pipelines and Build Tooling:

Access Tokens and Passphrase-Protected Shared Private Keys

Don't save to local disk (i.e. developer laptop)
Don't commit Access Tokens to the repository
- (committing Passphrase-Protected Shared Private Keys may be awkward, but is not at all unsafe)
Saving to ephemeral disk is okay (i.e. during pipeline / Docker build)
- Do NOT make things complicated in order to avoid saving to ephemeral disk, it's moot

Security FUD:

Use standard practices, DO NOT make things more complicated than necessary. More complicated == Less secure

Security focus anywhere aside from the weakest link is an illusion. More complicated == Less secure

Security concerns are different between local and on-prem development from serverless and cloud deployments. In other words: If you lived in the desert, you wouldn't waste effort on preparing for a flood when it's obvious the more pressing concern is a draught - such a thing would be a sensless (and confusing) distraction. More complicated == Less secure

Local Systems:

High Access, High Risk
- Typically have greater access with fewer controls (i.e. passwords saved in browsers to sites which can access management consoles)
Need physical access constraints (i.e. protect against theft)
Full Disk Encryption where possible (requires password login to decrypt)
Auto lock-screen
Download restrictions (phishing, viruses, and keyloggers are a threat)

Cloud Systems

Low Access, Low Risk
Disks are SSD (not magnetic) and ephemeral ("secure delete" and "shredding" are meaningless)
Disk Encryption rarely helpful (decryption must be automated on reboot for zero downtime, therefore the same as no encryption)
Deploy keys are scoped
Remote access is disabled
Separate Test/Build ENVs (i.e. GIT_TOKEN) from Deploy ENVs (i.e. PORT) when possible
- Ideally running applications shouldn't have access to the ENVs that built them

Exceptions:

Traditional, low-cost VPS/VPCs are not ephemeral, treat them more like local systems
If containers expose remote access (SSH), treat them more like local systems

What is more likely?

A deploy key being leaked from a containerized cloud service
A local laptop being compromised

Thus, make the local developer experience simple and easy to reduce mistakes.

Think less about "limiting" and more about enabling. This:

Make it easier to do the right thing, than the wrong thing, or even nothing at all.

See also: "Security: Perception isn't Reality" as well as the notes in the security section far below.

The End