Revisiting Git, Github, and the wordpress.org plugin repository

Some months ago, I wrote about using Git and Github with the wordpress.org plugin repository. Since that time, I’ve been refining my plugin development workflow. I now do all of my development with Git, using git-svn tools to do all svn management.


Preamble: Git as primary vs secondary

Before talking more about my workflow, I should make a sort of conceptual distinction. When I develop most of my plugins, I am working alone. In these cases, Git is primary. It’s where the meaningful version control happens, while svn.wp-plugins.org is just a gateway for wordpress.org/extend, the distribution channel. I am not using the wordpress.org svn repository for version control or code sharing in any interesting way. Likewise with a few team projects that I’m involved in, notably Anthologize. All of our code-sharing and true version control happens via Git and Github (here’s our repo), with the wordpress.org svn repo used only as a distribution mechanism.

In contrast, I also use Git to develop BuddyPress, but the strategy is quite different. In the case of BuddyPress (as in the case of WordPress), the svn repo is the officially sanctioned version control system for the project. Git, in this case, is a secondary, local versioning system – essentially, my development sandbox. This setup places additional restrictions on how the git-svn link is managed, raising issues such as juggling svn branches, tagging version off of an svn branch, exporting patches in an svn-compatible format, and so on.

In this post, I’ll be focusing on the first kind of development setup, in which the wordpress.org svn repo is serving merely as a distribution channel. It’s a bit simpler to start there. Moreover, I’ve been chatting with Mark Jaquith about Git for WP development, and I know from those chats that he’s planning on writing up a description of the second kind of workflow (which characterizes the WordPress core work that he’s more concerned with). So I’ll leave the sophisticated stuff to him.


Part One: Getting your repos set up

You’ll need to have Git installed on your computer, with git-svn compiled. Here’s a guide to compiling it yourself on OS X; it’s also available through a number of repos on various OSes.

Create a directory for the plugin in the plugins directory of your dev install. I’ll use my recently released plugin Unconfirmed as an example; you can follow along using the example (until it comes time to push and commit!), or use your own plugin.


cd wp-content/plugins

mkdir unconfirmed

The next step will require you to clone your wordpress.org svn repo. This assumes that you have requested and been granted space in the repository already. If you’re just starting plugin development and don’t want to request space yet, that’s OK – just begin your development in a normal Git repository. As long as you are pushing your Git changes somewhere (like Github), you’ll be able to wipe out your local copy when it comes time to send it to wordpress.org and start from this point as if you were starting from scratch.

Get a revision number for the plugin. You don’t want to force git-svn to crawl through the 300,000+ revisions on the wordpress.org repository. (For those keeping score, this is the only time you will have to run an svn-native command!)


svn log http://plugins.svn.wordpress.org/unconfirmed

Look for the first commit number, where the plugin was added to the repository. In my case, it’s r387893.


r387893 | plugin-master | 2011-05-23 04:10:19 -0400 (Mon, 23 May 2011) | 1 line

adding unconfirmed by boonebgorges

Clone the svn repository into the local plugin directory. Here’s the syntax, followed by some explanation.


git svn clone -t tags -b branches -T trunk -r387893 http://plugins.svn.wordpress.org/unconfirmed unconfirmed

  • git svn clone is the command itself
  • -t tags -b branches -T trunk tells Git about the directory structure of the svn repository that you’re cloning. The flag --stdlayout or -s is shorthand for the same thing, though I’ve had mixed luck getting it to work – so I just enter the whole thing explicitly.
  • -r387893 is the svn revision number where I want Git to clone. In the next step, we’re going to tell Git to fetch the svn revision history starting with this commit; that’s why we chose the first, rather than the last, commit from the unconfirmed repo.
  • http://plugins.svn.wordpress.org/unconfirmed is the address of the plugin on the wordpress.org repo. It’s important to use this address rather than the alias http://svn.wp-plugins.org. As I discovered (after much frustration), Git is not able to trace the branch/tag history from the wp-plugins.org address – you have to use plugins.svn.wordpress.org.
  • unconfirmed is the relative path to the directory where I want the repo cloned.

Next, fetch the svn commit history like so:

cd unconfirmed
git svn fetch

When you hit enter, git-svn queries the wordpress.org repo, starting with the initial revision number you pegged in the previous step, and walks all the way forward to the (entire) repo’s most recent revision, mirroring your plugin’s svn revision history in your local Git clone. If you’ve just received the space in the svn repo, this will only take a few seconds. If you’re doing this with a plugin that was put into the repo some time ago, or one that has a lot of revisions (and branches and tags) in the svn repo, you’ll have to wait a long time. Drink a beer or ten while you wait.

You’ll know the process is done when you are returned to your command prompt. Test to make sure that you’ve pulled in all of the remote tree by entering the following:

git branch -r

This will show you a list of all the remote branches being tracked by this Git repository. If it’s a fresh repo, you’ll only see the trunk. If it’s a repo with existing branch and tag history, you’ll see one Git branch corresponding to the trunk, one corresponding to each svn tag, and one corresponding to each svn branch.

If you’re not planning to use Github or to share your code with anyone else, you’re done. If you are planning to use Github (as I do), you’ll need to add your Github endpoint as a remote repository. Assuming you’ve already created a repo on Github,

git remote add origin git@github.com:boonebgorges/unconfirmed.git

origin is the name I’ve chosed for the Github endpoint, but you can call it whatever you want.

Now it’s time to reconcile your commit histories. Hopefully, you’ve only got one history to deal with – either in the Git or in the svn repo – so you won’t have too much trouble. In such a situation, something like the following should work:

git pull --rebase origin master

So, here’s the thing about --rebase. It rolls back, albeit temporarily, all of the changes on your local copy of the repository back to the last common commit, applies all of the remote revisions, and then attempts to apply your revisions on top of it. If you are working with a fresh Github repository, there will be no common commit in the history, and there won’t be any revisions from Git to rebase back onto the tree. Thus, the svn history will be rolled back, zero commits will be put onto the local repo, and the svn history will be laid back down. In other words, nothing will happen, except that the Github repository will establish a common ancestor revision, allowing you to push. Mutatis mutandis, if you have an active Github repo but an empty svn repo, zero commits will be rolled back, and your Github history laid on top of the local repo, with the zero commits laid back on top. In other words, it’ll sync with the Github history seamlessly, and allow you to commit back to the svn repo by establishing a common revision history. If you have active, separate commit histories in both svn and Github, may God have mercy on your soul.

In any case, use --rebase with great caution. Read the git-rebase docs and try to wrap your head around it before doing anything that will mess up the revision history.

Once you’ve successfully pulled from Github, your three repositories – the local Git repo, the Github Git repo, and the svn repo on wordpress.org – will all be aware of each other and in sync. You are ready to develop.


Part Two: Day to day development

Let’s say you’ve found a bug (OH NOES) and you need to fix it. Before digging into the code, get yourself to the command line to make sure you’re in good working condition.

The first thing I do before I touch any code is to see what’s happening in my local repo:

git status

There’s a lot of good information that git status can give you, for which I refer you to the docs or to Google. The important concept for our purposes is that we start a new local branch every time we want to fix a bug or develop a new feature. The notion of hyper-specific, temporary branches is one of Git’s biggest selling points, as well as one of the places where it differs the most from the way that many WP developers work with svn. Since branches are strictly local unless explicity pushed, and since Git’s merge and rebase tools are so nice, you can create, merge, and destroy branches at will, and keep your work separate. To add (and switch to) a branch called stupidbugfix, use this syntax:

git checkout -b stupidbugfix

This does two things: it creates the new branch stupidbugfix, and it checks out that branch (similar to svn switch). git status will show you that you are now On branch stupidbugfix.

Do your bugfixing as you would normally do. When you’re ready to commit the changes, do another git status. You might see a result that looks like this:

# On branch stupidbugfix
# Changed but not updated:
# (use "git add <file>..." to update what will be committed)
# (use "git checkout -- <file>..." to discard changes in working directory)
#
# modified: readme.txt
#
no changes added to commit (use "git add" and/or "git commit -a")

In this case, I’ve made changes to one file: readme.txt. In order to commit these changes, you’ll first need to stage the file with git add:

git add readme.txt

(git add is automatically recursive, so you can add whole directories this way too.) Now you can commit the changes:

git commit -m "Fixing that stupid bug."

If you’re confident that you want to stage all changed files to a commit, you can skip the explicit git add and use the -a flag:

git commit -am "Fixing that stupid bug"

Now, let’s get your changes from the stupidbugfix branch into the master branch, which we’ll use to push to our central repository. Then we’ll delete the temporary stupidbugfix branch.

git checkout master
git merge stupidbugfix
git branch -d stupidbugfix

If you are sharing your code on Github, you can push your commits at this point:

git push origin master

Part Three: Releasing a new version to the wordpress.org repo

If you’re following the steps I’ve outlined above, all of your development is happening in Git and Github. The only time you’ll need to touch the wordpress.org repository is when you want to release a new version. Here is the procedure I use. [EDIT 2012-09-14: See the Addendum below for an improved workflow for this step.] First, rebase the svn trunk to your current branch.

git rebase trunk

This small bit is crucial, and it took me many months of frustration before I finally stumbled upon this (cache only!) blog post, which put me on the right track. The concept here – so far as I understand it, at least – is that git-svn actually rewrites the MD5 hash that Git uses to identify the changeset, and when you rebase the trunk into your current local branch, you forcing git-svn to match Git-changeset MD5s with svn-changeset MD5s. If you don’t do this, you’ll get infinite merge conflicts.

Now, let’s send those commits to wordpress.org svn.

git svn dcommit

This sends all changesets (since the last time you dcommitted) up to the WordPress svn repository. (In other words, you’re mirroring the revision history.) Once this is done, the revision history on your svn trunk will match that of your current git branch.

Now we’re ready to tag the release in svn. (I’m assuming that you’ve already changed current version and stable tag numbers in your plugin files before dcommitting. If not, make those changes, git commit them, git rebase trunk, and git svn dcommit.)

git svn tag 1.1

This will do the same thing as when you svn cp your trunk into tags/1.1.

I like to keep tag history in Git as well. Git tags are metadata (while svn tags are really just branches), so they have to be created separately, and pushed up to Github.

git tag -a 1.1 -m "Tagging 1.1"
git push --tags

Finally, before making any more changes that can be pushed back to Github, you’ll have to rebase from Github – again, to make sure that Git understands that your Git MD5s have been overwritten by git-svn. Assuming you’re on your master branch:

git pull --rebase origin master
git push origin master

Again, this rebase business is scary and dangerous, so try to understand the possible ramifications if you are working on projects that involve other people.


Conclusion

This might seem like a lot to learn. But developing in Git is hugely beneficial, well worth the learning curve. Git’s agile branching allows a kind of focused, compartmentalized development strategy that can’t easily be replicated with svn. And having Github as a bugtracker and changeset-viewer is also really great. Plus, working with Git in my own development means that I don’t have to code-switch (pardon the pun) when working with clients who use Git for collaborative work, which, I’m finding, is increasingly common. Learning about Git with your own WP plugins is a great way to learn some of the finer points of Git.


Addendum (14 Sep 2012): Be a better wordpress.org citizen and squash

The biggest headache in the git/wordpress.org workflow has always been lining up the revision histories. Git revision histories are non-linear, and SVN does not understand them. So the process of rebasing the development branch into the trunk-tracking branch is always precarious, as trunk is a flattened, svn-friendly branch. Moreover, even when the rebase does work, you end up reduplicating revision histories in both Git and SVN, when the real purpose of wordpress.org SVN here is really just distribution. (This makes Otto sad.)

So, for the last six months or so, I’ve used a modified version of the release workflow described in Part Three. First, I have a local branch, which I call svn, which tracks trunk:

git checkout -b svn trunk

At release time, I check out the svn branch, and do a squash merge. This means that all changes since the last commit to the svn branch are laid on top of svn as if they were a single set of changes. You can then commit these changes as a single changeset (thus keepin’ it linear, and keepin’ it short for Otto’s sake):

git checkout svn # if you're not already on it
git merge --squash master
git commit -m "Merging changes from Git for 1.1 release"

Note that you might get a message from Git that there were merge conflicts, in which case you’ll need to use mergetool to clean up. But this is a subject for another post.

Then, do your git svn dcommit and git svn tag from the svn branch. It’s important that you never rebase to or from the svn branch, or do any development there; all changes on that branch should be squash-merged from a git-only dev branch.

Revisiting Git, Github, and the wordpress.org plugin repository by Boone Gorges, unless otherwise expressly stated, is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License.

37 thoughts on “Revisiting Git, Github, and the wordpress.org plugin repository

  1. Pingback: Teleogistic / Using Github with wordpress.org plugin SVN

  2. Kunal

    Thanks for writing this post! Just one suggestion: instead of `-t tags -b branches -T trunk` git allows you to write `–stdlayout` as a shortcut.

    Reply
  3. Pingback: Revisiting Git, Github, and the wordpress.org plugin repository | hakre on wordpress

  4. huyz

    Excellent write-up.
    You saved me tons of time.

    The only thing I would add is maybe a note about username/password for the “git svn dcommit” command. I was prompted for a password with my local username. I didn’t know that I had to hit enter to be able to enter my wordpress.org username and password.

    Reply
  5. Pingback: WP-Footnotes anywhere in the content - huy'z

  6. Pingback: Using Git for WordPress Development SCM Instead of Subversion « CatN

  7. Ollie Armstrong

    Hi, having an issue with this and working with GitHub. Just like to say how great your guide is by the way, helped me so much!

    However slight issue when I pull commits down from GitHub that my colleagues have committed. When I pull them, I then do a git rebase trunk ready to dcommit to the svn repo, however I get conflict errors on all of the commits that have been pulled from GitHub.

    I’ve taken a look at them and they should conflict, but I think it is something to do with the git/svn repo not having a common commit or something.

    Is there another command that I can run that should merge the two repos or something?

    Appreciate any help,
    Ollie

    Reply
    1. Boone Gorges Post author

      Ollie – Glad the guide is helpful!

      If you’re working with other people, who are using Git exclusively, then you’ve got a whole host of other worries to deal with. From my understanding, when you rebase against the svn trunk, git-svn renames each commit to be in accordance with svn’s linear commit numbering. That means that when you commit a change with Git hash 12345, and then svn dcommit (or git rebase svn/trunk), the commit ID is changed to something else – say 54321. If you push that change back up to Github (at least, if you –force push it), you are effectively replacing your pure Git history. This is a problem for collaborators who are getting their information through Git, because commit 12345 (which they had already pulled) is in conflict with (because it is identical to) 54321.

      You can get around some of this by omitting the –rebase flag from your git pull. Git merge will detect your conflicts and will usually do a pretty good job of working around them automatically. However, you may find that there is still some funny business – you might end up, for instance, with two versions of every commit (the Git version and the SVN version) in your commit logs.

      For your setup, you might consider using SVN in a slightly dumber way, where at release time you copy over your files from Git into a fresh svn checkout of your trunk, svn add any new stuff, and then just commit the changes. That’ll mean that your svn commit history will not be in sync with your Git history, but at least it means that you won’t have any problems with being out of sync with your collaborators.

      Let me know what you figure out – I’m still learning as I go along with this stuff!

      Reply
  8. Ollie Armstrong

    Boone,

    Thanks, I actually did omit the rebase but because its actually being merged by GitHub its not really having the effect I want. I guess I will have to do as you suggested although its not ideal! This should be ten times easier…

    Cheers for the great guide and help!
    Ollie

    Reply
  9. Dave Doolin

    Github is amazing. I’m only using svn now to upload to WordPress itself. (Merging with a third party tool.)

    Very glad to see some of the power hitters, committers and other assorted vips in WP now using git & github as well.

    Reply
  10. Pingback: Deploying from git to svn | scribu

  11. Pingback: Git SVN Wordpress Plugin Conflict [closed] | SeekPHP.com

  12. Ronak

    Hi, Thanks for this wonderful guide. I have moved my plugin development from svn to git and am now using Github to host my plugin. I won’t be going back to svn at all.

    I have a very basic question about the workflow. Right now my git repo on the development machine is outside the wordpress installation. When I’m working on the plugin, I work directly on the plugin files inside WordPress installation. When I’m done with all the testing, I move the changes to my git repo and commit/push. Should I make move my git repo as the plugin directory itself inside the WordPress installation? What is the recommended way to develop in such a scenario?

    Reply
    1. Boone Gorges

      Glad it’s helpful, Ronak!

      My setup is similar to yours, in that my plugin files don’t live in my development WP installation. But instead of doing the complicated copy-paste, I use symlinks. The way I do it is that I have one “canonical” wp-content folder on my computer, and then in all of my WP dev installations, I remove the default wp-content folder, and create a symbolic link to the “canonical” version. But you could do this on a plugin-by-plugin basis, too. So if you have a plugin called ‘ronak-plugin’ that lives in /home/ronak/wp-plugins/ronak-plugin/, you can cd to your WP installation’s wp-content directory, and

      ln -s /home/ronak/wp-plugins/ronak-plugin/ ./ronak-plugin/

      WP will think that the plugin lives in wp-content, even though it really doesn’t.

      Reply
  13. Tareq

    Hi,
    What do I need to do extra when I’ve already started developing the plugin with git. After completing the writing, then I went to wordpress.org and created the plugin create request. How do I manage exisitng git repo to that newly approved svn repo? I am kinda stuck.

    Reply
    1. Boone Gorges

      Tareq – The only way I know to do this involves having a remote Git repo, like Github. Here’s the process I’ve used:

      – Make sure that your local changes have been committed and pushed to Github. (Triple-check!)
      – Delete the local plugin directory and create an empty one
      – Using the instructions above, git svn clone the empty wordpress.org svn repo into the new directory
      – Once that’s done, git remote add your Github repo
      – Fetch and pull from the Github master branch. This will merge your true repo (from Github) into the local clone of the wp.org repo

      At that point, you’re pretty much done. For development, I would suggest using Git exclusively, and releasing to the wordpress.org repo using a special branch that only receives merges:

      – If you haven’t done so yet, git checkout -b svn trunk. This creates a new local branch “svn” that tracks the remote wordpress.org trunk.
      – Develop on another branch. At release time, git checkout svn and git merge master (assuming you’re developing on ‘master’)
      – From the svn branch, git svn dcommit. It should commit a single merge commit, rather than your whole Git history

      Good luck!

      Reply
      1. Tareq

        Thanks Boone, that helps :)

        I always had pain when releasing it to wordpress.org svn repo. I wish they will support git someday.

  14. kavin

    Hi Boone,

    Thanks for this very detailed and helpful guide. I followed every step carefully on my WP plugin, up to the rebase step git rebase trunk. Whenever i do this, i get infinite merge conflicts for each and every file. Tried almost everything on the web and found no avail.

    I even tried mergetool git mergetool -y, but something went horribly wrong and files went missing/obsolete files reappeared in the rebase.

    Finally I gave up and ran git svn dcommit, Thankfully it ran without issues.

    What will i be loosing with git svn rebase? Is it safe to run svn dcommit alone? Or is there a way to redo the process?

    Reply
  15. Pingback: Revisiting Git, Github, and the wordpress.org plugin repository | Teleogistic - Dougal Campbell's geek ramblings

  16. Samuel "Otto" Wood

    Instead of doing a `git svn dcommit`, which mirrors your entire revision history, can you change your process to squash your many git changes down to a single commit to the SVN instead? If you’re working with a git repo for your changes anyway, then really you’re only using the WordPress SVN as a “release” repository, in which case it really doesn’t need your whole commit history.

    The basic problem I have with mirroring all those changes on our SVN is that they’re pretty worthless to the SVN if you’re doing the actual development somewhere else. The wp.org release system has to then crawl through each and every one of your changes when you do that, which slows down the whole thing for everybody.

    According to this Stack Exchange answer, you could do this using a `get rebase -i` or possibly `git-merge –squash` to do this (note: I am unfamiliar with git, do not use it, and can not tell you the “right” way for certain):

    http://stackoverflow.com/questions/158514/is-it-possible-to-make-git-svn-dcommit-result-in-a-single-svn-commit

    Reply
    1. Boone Gorges Post author

      Otto – Yes. Since writing this article, I got tired of svn’s inability to understand Git changelogs, so I switched to a more streamlined method. Basically, I keep a separate Git branch called ‘svn’, and at release time, I do a flat merge from my development branch into the ‘svn’ branch. Then all dcommitting (but no rebasing) happens from the svn branch. For most situations, this is equivalent to doing rebase -i + a big ol’ squash.

      As this post seems to have become one of the go-to resources on the web for Git + wordpress.org/extend, I’ll write an update in the near future.

      Reply
  17. Pingback: Yet another post about Git and wordpress.org SVN | Teleogistic

  18. Pingback: Using Git locally for a Subversion-based project (like BuddyPress) | Teleogistic

  19. Pingback: Git in my Subversion | danielbachhuber

  20. Pingback: Svn meets Git | Jan Renz

  21. Mani

    Thank you very much for this great article. My only concern is how to use github’s author name and email address while fetching from SVN. I know it is doable by using `authors.txt` file, however I am not sure if it will mess with the upcoming `svn dcommit`. My plugin has a 6 year old SVN history and I want to be sure that replacing author name and emails will not make any trouble.

    Reply
  22. chasewiseman

    Thanks for the great article! Does anyone have any insight on using this (updated) commit workflow along with the SVN assets folder? I have my git and SVN repositories synced up, but now I need to update the screenshots for the dot org repository to go along with a new release. I assume that if I do a pure SVN commit to update these then everything will get out of sync. I don’t see a way to update the assets folder via git-svn either. Any help would be appreciated!

    Reply
  23. Nico

    @cheasewiseman as far as i know you can checkout and commit the /assets folder with SVN seperatly. You could just use SVN in some temp folder to just handle the Screen shots. its not a everyday thing so …

    There are things i found in this tut are not working on this command:
    `git svn clone -t tags -b branches -T trunk -r387893 http://plugins.svn.wordpress.org/unconfirmed unconfirmed`

    if i try to clone revisions i end up with a empty folder. I found out that i have to use “-r HEAD” instead of revisions “-r387893″ for it to work. This pulls only from the latest revision if i understand it right. Also it changes the url to its base so i additionally have to use –no-minimize-url

    This is the full command i used to get it to work a (ubuntu 12.04 if that matters somehow). I use a dot in the end to pull inside the dir i am currently in

    `git svn clone -s –no-minimize-url -r HEAD http://plugins.svn.wordpress.org/regenerate-thumbnails-reminder .`

    What do not understand now what “trunk” is in this context. Is it a normal branch like master or is it “remotes/trunk”? Please someone explain it to me what this is.

    Reply
  24. Pingback: Git and GitHub for SVN developers - Mario Peshev on WordPress Development

  25. Pingback: Ben Lobaugh Online » Creating a two way sync between a Github repository and Subversion

  26. Pingback: WordPress.org vs. GitHub For Hosting WordPress Plugins and Themes

  27. Hervé Renault (@HerveRenault)

    Hi,

    I like the idea of using git as a primary tool and then git-svn only to publish.

    In you addendum, you write:
    git merge –squash master
    but the merge ends up with conflicts we have to edit manually, which is odd because we don’t touch the svn branch.
    So, don’t we need to git merge –squash -X theirs master
    ?

    Reply

Leave a Reply