Our Infrastructure

This post originally appeared on the Software Carpentry website.

As described in these posts from May, and October 2012 (which build on this one from April 2012), we use a complicated collection of tools to manage our work:

  • Our web pages and blog posts are Jinja2 templates stored in three GitHub repositories: one for the main web site, one for the Version 4 videos and slides, and one for the Version 3 notes. (We split material between repositories because we didn't want people to have to download the earlier versions in order to write a blog post—they're fairly large. In practice, though, the web site compiler doesn't work if you don't have everything at hand.)
  • We use a 950-line Python script to compile those templates into HTML pages. (We'd like to use Pelican, a blog compiler written in Python, but need a few patches to land before we can easily integrate the plugins we need.)
  • Our tutorial videos are on YouTube and embedded in our web pages.
  • We use Google Analytics to watch web site traffic.
  • And Disqus for managing comments.
  • And the Open Badges infrastructure for the digital badges we issue. (More about this later.)
  • We use EventBrite to handle registration about two thirds of the time; the rest of the time, hosts take care of it themselves (typically with an internal system they're required to use).
  • Our site is hosted by Dreamhost. Their service has been increasingly erratic over the past ten months, so we're probably going to move to another hosting service this summer.
  • We use Mailman to manage mailing lists, primarily because it's what Dreamhost provides. Unfortunately, what Dreamhost doesn't provide is access to Mailman's command-line API: they host mailing lists on a different set of servers than the ones used for web sites and shell access, so we can't write scripts to synchronize mailing lists with our database. This is another reason to move to another hosting service…
  • And speaking of databases, we have yet another version control repository that holds our contact database (as an SQL file we can load into SQLite), notes about bootcamp prospects, and other information that can't be public (because it includes personal email addresses, phone numbers, and so on). Using an SQLite database makes it really easy for us to ask questions like, "Which bootcamp attendees in 2012 became instructors in 2013?", but keeping it up to date is tedious.
  • Which reminds me that there are three other repos as well. assets stores all the "extra" public files that don't go into the web site, such as the vector source for our logo and the LaTeX and Markdown versions of our papers. guide holds the source for the instructors' guide that was supposed to be finished five months ago. Like the web site itself, it is written in HTML with Jinja2 templates and compiled using a little Python script. Finally, boot-camps holds actual teaching materials, plus the web pages that instructors create for their bootcamps. This repo is a mess right now, and cleaning it up is one of the major jobs for our summer break.
  • We use a Google Map to display the locations of past and future bootcamps, and a Google Calendar to display their dates. A little bit of custom Javascript populates the former using data extracted from bootcamp pages in the web site, while our web site compilation script creates the .ics file that fills the latter.
  • And oh yeah, we tweet. When we were using WordPress, tweets were sent automatically when new blog posts appeared; we do that manually now (and sometimes forget). We also sometimes forget to post notices about newly-announced bootcamps.

As you can tell, there are a lot of moving parts. Here are the steps involved in several common activities:

Write and publish a blog post.
  1. Create a new branch in your clone of the GitHub website repo.
  2. Write the post as an HTML file with a bit of Jinja2 boilerplate.
  3. Run the compilation script to make sure it's valid, properly linked, etc.
  4. Commit your changes, then send a pull request.
  5. Once it's merged, get an admin to log in to the server and recompile the public-facing version of the web site.
  6. Tweet about the post.
Set up a new bootcamp.
  1. Create a new event on EventBrite.
  2. (Optional) create a new mailing list for the instructors and helpers (so that they can coordinate with each other), and add the appropriate people to it.
  3. Create a new branch in your clone of the GitHub website repo, add an HTML page for the bootcamp, send us a pull request, and recompile the site on the server once it's merged (or in other words, do everything you'd do for a blog post). Our calendar and map are automatically populated using this data.
  4. Blog and tweet about the bootcamp so that people know they can register. (We forget to do this sometimes.)
  5. Create a new branch in the boot-camps repo to hold teaching materials.
  6. Add a page for the bootcamp to the gh-pages branch of that repo (because that's the way GitHub does things).
  7. Update the notes we keep in the private administrators' repo for each prospect to say that a bootcamp has been scheduled.
Wrap up a bootcamp.
  1. Add participants names and email addresses to the SQLite database file.
  2. Invite them to join the announcements mailing list (by uploading their addresses to a widget in Dreamhost's online control panel, which then sends the invites and handles responses).
  3. Write a blog post about how it went. (We sometimes forget this step too, or people don't send us a post.)
Add someone to the online study group.
  1. Add their name and email address to the SQLite database file.
  2. Add them to the appropriate mailing list (by hand, since Dreamhost doesn't let us use the Mailman API).
  3. Make them an author on the teaching blog.
Issue someone an instructor's badge.
  1. Add an entry to our SQLite database file.
  2. Add an includable snippet of HTML describing them to the web site, along with a photo of some kind, that we can use on the roster page.
  3. Add a line to the badge directory's index file (which handles everything else).

There are far too many manual steps in these lists, but I'm not sure what to replace it with. WordPress didn't meet many of our needs, and while customer relationship management packages like SugarCRM would be useful for managing bootcamp prospects and attendees, they don't handle blogging, event Registration, or badging.

It's tempting to build something ourselves, but that way madness lies we don't have the days needed to build, test, deploy, and maintain something. Even if we did, it always seems we'd be better off using them to create new lesson material or actually teach. We can't go on like this, but I honestly don't know what we should do instead.


Infrastructure Software Carpentry

Archive

Dialogue & Discussion

Comments must follow our Code of Conduct.

Edit this page on Github