Automate Git Repository Setup with Python & Fabric

Jun
21
2017

I love Git. I find Git to be an intuitive way to track many versions between many systems. Now all my default config files and scripts utilize Git. However, logging into a server and creating the Git repo, then cloning it is a number of steps that detracts from getting work done.

This tutorial outlines a basic example of using Fabric(Python automation module) to automate the process of creating a ‘mycode.git’ repository and cloning it. This tutorial was written for Cent OS 6.5, but could be adapted for Ubuntu and Fedora. Basic knowledge of Linux and Python 3 is required.

If you need help setting up Python3, pip, and Fabric on CentOS 6 please refer to my earlier tutorial: https://www.savelono.com/linux/how-to-install-python-3-and-pip-3-for-centos-6.html

Goals of this post:

Write Python/Fabric function to automate the Git setup process
Write Python/Fabric function to automate cloning Git repository on local host

Introduction to Fabric

Fabric is a Python library and SSH abstraction layer. It’s purpose is to easily automate local and remote shell commands as tasks with Python. Here is a description from www.fabfile.org:

www.fabfile.org

Fabric is a Python (2.5-2.7) library and command-line tool for streamlining the use of SSH for application deployment or systems administration tasks.

It provides a basic suite of operations for executing local or remote shell commands (normally or via sudo) and uploading/downloading files, as well as auxiliary functionality such as prompting the running user for input, or aborting execution

This type of automation is good for me because SSH access is the only common thread for accessing my customers Linux servers. Fabric excels in situations with many servers located on different networks. If all my servers were on one network, I’d probably go with Puppet or Chef or some other “state” based system.

Lets start with the Python modules and default Fabric options. The env.hosts should be set to your Git servers IP address and be configured in a similar manner to my previous tutorial. Also, notice we created a blank class for handling errors called, “FabricException”. We will use this later to capture Fabric: aborts, errors, & warnings.

The “env.*” variables are used by fabric to conduct automation tasks. If left blank you will be prompted by Fabric at execution.

fabfile.py:

#!/bin/python3


from fabric.api import *
from fabric.contrib import files


# Local git managment automation tools
# copyright matt birkland, Savelono.com 2017
# License: GPLv2
#
# 


# Set the username
env.user   = "git"

# Set the password [NOT RECOMMENDED]
env.password = ""


env.hosts = "10.1.10.10"


class FabricException(Exception):
    pass

env.abort_exception = FabricException

#!/bin/python3

from fabric.api import *

from fabric.contrib import files

# Local git managment automation tools

# copyright matt birkland, Savelono.com 2017

# License: GPLv2

# Set the username

env.user = "git"

# Set the password [NOT RECOMMENDED]

env.password = ""

env.hosts = "10.1.10.10"

class FabricException(Exception):

pass

env.abort_exception = FabricException

Below is our main Fabric function. Your fabfile.py can have as many Fabric functions as you want but I tend to divide them up by purpose. We will alternate between the “local()” and “run()(for remote tasks)” functions of which make up the backbone of Fabric automation. The function starts by Prompting for a project name, followed by install directory. It does some basic error checking and then displays your entry.

# 
#  newgit pseudo code
# ------------
#
# - Prompt for name of newcode
# - Prompt for install directory
# - Test that directory starts with /
# - Test for tailing / & remove it
# - Display Git Repo build info
# - Test to see if project already exsist
# - Create git folder and intialize
# - Clone to local folder of choice
#
#------------------------------------------
def newgit():
    while True:
        git_name = input("Please enter name. Example, 'myproject' :")

        if not git_name:
            print("You must enter a name.  Please try again or quit(ctrl-c).")
            print()
            continue

        else:
            break

    print()
    print()
    print("Current ~/git/ tree:")
    print("----------------------------------------------------")
    local("ls -al ~/git")
    print("----------------------------------------------------")
    print()

    while True:
        git_dir = input("Please enter folder(default ~/git). Example, '/home/matt/software' :")

        if not git_dir:
            git_dir = "~/git"
            break

# check for slash as first character in valid unix directory
        elif git_dir[:1] != "/" and git_dir[:1] != "~":
            print("ERROR: {}/{}".format(git_dir,git_name))
            print("Please enter valid directory path.  Example, '/home/matt/git/somedir' or ~/git/somedir.")
            continue

# check for trailing slash and remove it
        elif git_dir[-1:] == "/":
            git_dir = git_dir[:-1]
            break

        else:
            break

    print("====================================================")
    print("     Git Server info")
    print("---------------------------")
    print("Git Server: {}".format(env.hosts))
    print("Git User: {}".format(env.user))
    print("Git Project: {}:/git/{}.git".format(env.hosts,git_name))
    print()
    print("       Git Local info")
    print("---------------------------")
    print("Git Poject name: {}".format(git_name))
    print("Git Directory: {}/{}".format(git_dir,git_name))
    print("=====================================================")
    print()
    print()

# newgit pseudo code

# ------------

# - Prompt for name of newcode

# - Prompt for install directory

# - Test that directory starts with /

# - Test for tailing / & remove it

# - Display Git Repo build info

# - Test to see if project already exsist

# - Create git folder and intialize

# - Clone to local folder of choice

#------------------------------------------

def newgit():

while True:

git_name = input("Please enter name. Example, 'myproject' :")

if not git_name:

print("You must enter a name. Please try again or quit(ctrl-c).")

print()

continue

else:

break

print()

print("Current ~/git/ tree:")

print("----------------------------------------------------")

local("ls -al ~/git")

print("----------------------------------------------------")

print()

while True:

git_dir = input("Please enter folder(default ~/git). Example, '/home/matt/software' :")

if not git_dir:

git_dir = "~/git"

break

# check for slash as first character in valid unix directory

elif git_dir[:1] != "/" and git_dir[:1] != "~":

print("ERROR: {}/{}".format(git_dir,git_name))

print("Please enter valid directory path. Example, '/home/matt/git/somedir' or ~/git/somedir.")

continue

# check for trailing slash and remove it

elif git_dir[-1:] == "/":

git_dir = git_dir[:-1]

break

else:

break

print("====================================================")

print(" Git Server info")

print("---------------------------")

print("Git Server: {}".format(env.hosts))

print("Git User: {}".format(env.user))

print("Git Project: {}:/git/{}.git".format(env.hosts,git_name))

print()

print(" Git Local info")

print("---------------------------")

print("Git Poject name: {}".format(git_name))

print("Git Directory: {}/{}".format(git_dir,git_name))

print("=====================================================")

print()

Now that we have settled on a name we will create the install Git repository in /git. Remember, in my last tutorial(on installing Git) we created a soft link between /home/git/code and /git(to shorten the string when we connect remotely).

 while True:
        this_ok = input("Is this correct? ('yes' or 'no') :")

        if this_ok == "yes":
            try:
                with settings(warn_only=False):
                    check = run("mkdir ~/code/{}.git".format(git_name))
                    print()
                    print()
                    break

            except FabricException:
                print("Git repo already in use!  Please run again and select a different name for the git project")
                print()
                exit(1)



        elif this_ok == "no":
            print("Exiting...")
            exit()

        else:
            print()
            print("=====================================================")
            print(" Please Enter 'yes' or 'no'.  Lowercase only please. ")
            print("=====================================================")
            print()
            continue


# Initialize Git directory

    try:
        with settings(warn_only=False):
            intialize_git = run("git init --bare ~/code/{}.git".format(git_name))
            print()
            print("Remote Git repo configured!")
            print()
            print()

    except FabricException:
        print("Unknown error")
        print()
        exit(1)

while True:

this_ok = input("Is this correct? ('yes' or 'no') :")

if this_ok == "yes":

try:

with settings(warn_only=False):

check = run("mkdir ~/code/{}.git".format(git_name))

print()

break

except FabricException:

print("Git repo already in use! Please run again and select a different name for the git project")

print()

exit(1)

elif this_ok == "no":

print("Exiting...")

exit()

else:

print()

print("=====================================================")

print(" Please Enter 'yes' or 'no'. Lowercase only please. ")

print("=====================================================")

print()

continue

# Initialize Git directory

try:

with settings(warn_only=False):

intialize_git = run("git init --bare ~/code/{}.git".format(git_name))

print()

print("Remote Git repo configured!")

print()

except FabricException:

print("Unknown error")

print()

exit(1)

Finally we will create the local directory(if needed) and clone the repo there!

# Intialize Git directory

    try:
        with settings(warn_only=False):
            intialize_git = run("git init --bare ~/code/{}.git".format(git_name))
            print()
            print("Remote Git repo configured!")
            print()
            print()

    except FabricException:
        print("Unknown error")
        print()
        exit(1)



# Create Project directory

    try:
        with settings(warn_only=False):
            clone_git = local("mkdir {}".format(git_dir))
            print()
            print()

    except FabricException:
        print("Directory exist already!")
        print()




# Clone Git repo

    try:
        with settings(warn_only=False) and lcd("{}".format(git_dir)):
            clone_git = local("git clone git@10.1.10.10:/git/{}.git".format(git_name))
            print()
            print("Git repo cloned to {}/{}!".format(git_dir,git_name))
            print()
            exit(1)

    except FabricException:
        print("Unknown error")
        print()
        exit(1)

# Intialize Git directory

try:

with settings(warn_only=False):

intialize_git = run("git init --bare ~/code/{}.git".format(git_name))

print()

print("Remote Git repo configured!")

print()

except FabricException:

print("Unknown error")

print()

exit(1)

# Create Project directory

try:

with settings(warn_only=False):

clone_git = local("mkdir {}".format(git_dir))

print()

except FabricException:

print("Directory exist already!")

print()

# Clone Git repo

try:

with settings(warn_only=False) and lcd("{}".format(git_dir)):

clone_git = local("git clone git@10.1.10.10:/git/{}.git".format(git_name))

print()

print("Git repo cloned to {}/{}!".format(git_dir,git_name))

print()

exit(1)

except FabricException:

print("Unknown error")

print()

exit(1)

Here is a link to the entire fabfile.py unabridged.

Let’s test it!

[matt@mattcom1 newgit]$ fab newgit
[10.1.10.10] Executing task ‘newgit’
Please enter name. Example, ‘myproject’ :mygitproject

Current ~/git/ tree:
—————————————————-
[localhost] local: ls -al ~/git
total 48
drwxrwxr-x 9 matt matt 4096 Jun 20 02:58 .
drwx–x—+ 59 matt matt 12288 Jun 20 14:43 ..
drwxrwxr-x 6 matt matt 4096 Jun 20 03:04 fabric
drwxrwxr-x 3 matt matt 4096 Apr 9 22:48 nflrank
drwxrwxr-x 4 matt matt 4096 Dec 30 01:08 puppet
drwxrwxr-x 5 matt matt 4096 Apr 10 14:10 python
drwxrwxr-x 3 matt matt 4096 Jun 20 03:03 savelono-projects
drwxrwxr-x 4 matt matt 4096 Nov 24 2016 swnpc
drwxrwxr-x 4 matt matt 4096 Jun 2 11:42 voiceip
—————————————————-

Please enter folder(default ~/git). Example, ‘/home/matt/software’ :

====================================================
Git Server info
—————————
Git Server: 10.1.10.10
Git User: git
Git Project: 10.1.10.10:/git/mygitproject.git

Git Local info
—————————
Git Poject name: mygitproject
Git Directory: ~/git/mygitproject
=====================================================

Is this correct? (‘yes’ or ‘no’) :

Type “yes” if the information is as intended.

[10.1.10.10] run: mkdir ~/code/mygitproject.git

[10.1.10.10] run: git init –bare ~/code/mygitproject.git
[10.1.10.10] out: Initialized empty Git repository in /home/git/code/mygitproject.git/
[10.1.10.10] out:

Remote Git repo configured!

[localhost] local: mkdir ~/git
mkdir: cannot create directory ‘/home/matt/git’: File exists

Fatal error: local() encountered an error (return code 1) while executing ‘mkdir ~/git’

Aborting.
Directory exist already!

[localhost] local: git clone git@10.1.10.10:/git/mygitproject.git
Cloning into ‘mygitproject’…
git@10.1.10.10’s password:
warning: You appear to have cloned an empty repository.
Checking connectivity… done.

Git repo cloned to ~/git/mygitproject!

Disconnecting from 10.1.10.10… done.
[matt@mattcom1 newgit]$

The directory ~/git exist already, so our exception captured it and the program continued.

our server:

[root@git mygitproject.git]# pwd
/home/git/code/mygitproject.git
[root@git mygitproject.git]# ls
branches config description HEAD hooks info objects refs
[root@git mygitproject.git]#

Our Git client(from which we executed the fabfile.py):

[matt@mattcom1 mygitproject]$ pwd
/home/matt/git/mygitproject

I’ll create a file ‘savelono.py’, add it as a file Git tracks and push it to the upstream repo.

[matt@mattcom1 mygitproject]$ touch savelono.py
[matt@mattcom1 mygitproject]$ chmod a+x savelono.py
[matt@mattcom1 mygitproject]$ git add .
[matt@mattcom1 mygitproject]$ git commit
[master (root-commit) ea8c083] Add savelono.py
1 file changed, 0 insertions(+), 0 deletions(-)
create mode 100755 savelono.py
[matt@mattcom1 mygitproject]$
[matt@mattcom1 mygitproject]$ git push
git@10.1.10.10’s password:
Counting objects: 3, done.
Writing objects: 100% (3/3), 218 bytes | 0 bytes/s, done.
Total 3 (delta 0), reused 0 (delta 0)
To git@10.1.10.10:/git/mygitproject.git
* [new branch] master -> master
[matt@mattcom1 mygitproject]$

That’s it! Hopefully this comes in use to others that want to automate Git with Python/Fabric. Thank you for reading.

Written by mattb in: Automation,Linux,Scripts | Tags: centos, fabric, fabric3, git, python, Python automation, python3, SSH automation

No Comments »

RSS feed for comments on this post. TrackBack URL

Save Lono

Linux Stuff, IT News, and Open Source Software

Automate Git Repository Setup with Python & Fabric

No Comments »

Leave a comment

Recent Posts