Don’t forget the feed

Google killed Reader just over five years ago. This week, Digg decided to mark this Wood Anniversary by taking a club to its own Reader. Yet again, we oldsters find ourselves shaking our fists at Kids Today. See, for example, Bryan Alexander's thoughtful post on finding a way forward for RSS readers and RSS in general.

I celebrated this turn of events by struggling to set up an instance of Tiny Tiny RSS on my web server. Having finally figured it out, I'm pretty pleased with the functionality. One nice bit is the notice regarding "update errors":

Oh noes!

TT-RSS then has a tool for viewing a list of broken feeds. Looking through this list has been an interesting experience. For one thing, it's given me the opportunity to think about a few dozen people who have not otherwise entered my consciousness in a number of years.

More importantly, it's convinced me that the real threat to RSS is not the lack of reader software; you only need a couple of good ones to support an ecosystem. The bigger issue is that RSS content is disappearing.

The more mundane kind of problem is link rot and broken redirects. Many domains have gone away, some have moved in such a way that RSS links no longer work, some are now hidden behind password protection (Blogspot seems to be a particular offender). Web sites come and web sites go, so some of this is to be expected.

The more worrisome trend is content that's not available through RSS simply because there's no feed mechanism. A shamefully large number of my geekier aquantainces have moved their blogs to Jekyll and other static-site-generation tools, which don't appear to have feed support out of the box; and – this is the "shameful" part – since these folks, geeky as they may be, think so little of RSS, they don't bother setting up the secondary plugins or whatever necessary to serve feeds. I expect that kind of behavior from lock-up-my-content companies and technically-clueless organizations that rely heavily on proprietary and bespoke software, but not from people who ought to know better.

For all of its lumbering bloatedness, one of the truly wonderful things about CMSes like WordPress is that they give you things like RSS – along with a pile of other boring-but-critical-to-the-future-of-the-open-web tools – by default. You don't need to make the decision to support RSS readers (or responsive images, or markup that is accessible to assistive technologies, etc) – the system provides them for you, and you have to go out of your way to turn them off.

Those who build their own systems for old-school things like bloggish content distribution, or who rely on teh new hotness to do these tasks in ways that are slicker than the old-school tools, should beware the dangers of discarding the automated systems that are the result of many years and many minds and many mistakes. If you must reinvent the wheel, then do your due diligence. RSS feeds, like other assistive technologies, should not be an afterthought.

2017

Previously: 2016, 2015, 2014, 2013, 2012, 2011, 2010, 2009.

As 2017 began, I felt fairly despondent about what the year would bring. Looking back at the way 2017 actually turned out, it looks like the despondence forced me to shift focus in a few ways that, on balance, have been major positive changes.

A case in point. For the first half of the year, I was having problems sleeping. I struggled with feverish nightmares related to the news and the state of the world. The information consumption habits I thought so enlightened last year (no news after 6pm!) were not, it turned out, enough to release me from the power of the information flow. This summer, I started reading the physical local newspaper, and in the fall, I stopped reading news and magazines online altogether. My one engagement point with news during the day is at breakfast, over coffee. When I walk away from the table, I walk away from the manufactured anxiety of constant contact. It's been marvelously therapeutic. And I actually feel *better* informed on most subjects, since I'm able to judge them at some distance rather than succumbing to the endorphin rush of "breaking news" and the "conversation" that accompanies it. I am increasingly disconnected from memes and collective-outrage-du-jour and other aspects of the zeitgeist, but I consider this too to be an improvement in my quality of life.

In place of the online world that I'm increasingly leaving behind, I've been searching out new channels for engagement that feel more authentic. An oddly underattended college reunion and some related contact with college buddies has caused me to reflect on the communities that we move in and out of through the course of our lives, and to think about how these dormant connections might serve as the organic seed for meaningful future relationships. I've been devoting more time to music and to a new-found interest in the visual arts. I've reached out to some old friends, and after stumbling across a cache of old letters from high school and college, have started building the idea of private correspondence – letter writing for its own sake! – back into my life.

Regarding work, I had a somewhat busier year than I'd hoped, and I devoted less time to free software than I have in the past few years. (See my Hard G post for more.) My relationship with WordPress and BuddyPress is something that I have struggled with in 2017, though it's become somewhat clearer in the last few months. When I first started contributing to free software, my spirit was nourished by the personal connections I made to people who were using the software. As I became busier and more "important", I justified to myself that I had bigger things to worry about than those connections, focusing instead on some vague ideas about "leadership" and "product". Beginning this fall, I started approaching my free software work with a different attitude, starting with a dose of humility about who I am and the value I bring to the projects. I started spending much more time in the support forums. I began approaching bug reports and PRs with an amount of empathy that felt forced but eventually gave way to true feelings of kindness. And I've started thinking more about the ways that I can use my years of experience in these projects to connect the humans working in disparate areas in a spirit of actual cameraderie. I'm unsure how my time committments to these projects will or won't change in 2018 (though I have a feeling I'll be spending proportionately more time on BuddyPress, where I feel like I've got the ability to make a more direct impact), but I feel strongly that any time spent on them should be purposeful and humane.

With my career more generally, I have found some solace by being explicit with myself that I will not be doing this work forever. Thinking about an off-ramp in the next few years gives a bit more urgency and purpose to the work that I do today, and I think this is a good thing no matter what happens.

One small resolution for 2018 that I've already put in place is: more enforced time off. I've already slated a couple of weeks for the first half of the year when I will be totally AFK, no screens, no internet, maybe no newspaper. Just having this on the schedule has made the everyday load feel lighter.

Happy new year to all!

2016

Previously: 2015, 2014, 2013, 2012, 2011, 2010, 2009.

Good bye and good riddance to 2016!

Some highs:

  • My kids and my wife are amazing. My five-year-old started kindergarten and is learning to read. My almost-two-year-old went from a baby on January 1 to a sweet, talkative, out-of-control kid on December 31. My wife is most of the way through her PhD coursework. I’m very lucky.
  • I had a pretty successful year business-wise. I made more money and did some work that I found interesting and gratifying. In the next couple weeks, I’ll kick myself into gear and write about some of the client work I did in 2016, probably over on my Hard G blog.
  • I remained pretty active in the WordPress and BuddyPress projects. I managed to scale back a bit, especially as regards WordPress, when compared to 2015, but this was mostly by design, so counts as a good thing. I wrote a separate post laying out this work in 2016.

The lows? It was a rough year in so many ways. Over the last few months, I’ve reacted to the more awful parts of 2016 with a couple of changes to my routine. I guess these are kind of like “resolutions” that are already in progress.

The main theme is the idea of reclaim. I went through a period a few years ago where I attempted to take control of my digital life by moving from proprietary, third-party services to free-as-in-speech software that’s more under my control, a process I called Project Reclaim. Today, I’m focused less on technology, and more on energy and time. There are just so many hours in a day, and I’ve only got so much emotional and intellectual energy to spare. I’ve been taking steps to make sure that these resources don’t go to waste.

  • The news – I have a reflex to read the news when I have a few minutes to spare. Under normal circumstances, this habit would be a time-waster. In 2016, it’s become actively harmful to my well-being. Reading the news is something I need to brace myself for, so I’m now doing it just two or three times per day, at times I’ve set aside for information consumption. And never after 6pm! It keeps me up at night.
  • TwitterI stopped using Twitter actively a few years ago. But in 2016, I slipped into periods where I’d waste time scrolling through once or twice a day. This, despite the fact that nearly every time, I find it emotionally devestating. So, around the beginning of October, I stopped looking at the site altogether – aside from a weekly check to see if I have any mentions to respond to.
  • Magazines etc – 2016 was a Year of Thinkpieces, and I read too many of them for my own health. I’m going to mostly cut them out in 2017. Aside from other problems with the genre, I find myself disgusted with the smug slactivism that oozes from thinkpiece culture, the idea that engaging pithily with election analysis or with apocalypse porn or with red state travelogues amounts to doing something productive with your intellectual angst. I’m going to spend this energy engaging in my community instead.
  • The phone – I ditched my smartphone in 2014. After having a second kid and moving to Chicago, I caved and decided to get another smartphone late in 2015. Mainly, I wanted a convenient camera, so that I could have more pictures of my youngest. But I slowly found myself using the smartphone for other reasons, and feeling all the old awfulness creep back. Reading the news and reading Twitter and reading email are all potentially traumatic experiences, and having all those things in my pocket – even just potentially – is emotionally crippling for me. On November 10, I switched back to my dumbphone, and it felt soooo good.
  • Books – In 2015, I read 45 or 50 books. In 2016, I read maybe 10 or 15 – mostly for the reasons described above. I need to do better in 2017.
  • Separation – All year, I teetered on the edge of work-related burnout. This is partly because I treated free software contribution as “hobby”, at least in part: I spent a fair number of evenings catching up on ticket backlogs. In 2017, I’ll be more disciplined in this area, treating free software work as part of my workweek, not as something I’ll get around to if I have “free time”.

I can’t remember a January 1 where I felt so uneasy about the upcoming year. I’m reclaiming my energies and focusing them on things like my family and my community rather than letting them be exploited by the internet, a strategy that I hope will help me to make the best of 2017.

Handling LaTeX in WordPress and React.js

I’m in the midst of building a WordPress plugin that interfaces with WeBWorK, a web application for math and science homework. The plugin features a JavaScript app powered by React and Redux, which lives inside of a WordPress page template and communicates with WP via custom API endpoints. The project is for City Tech OpenLab and it’s pretty cool. I’ll write up some more general details when it’s closer to release.

The tool is designed for use in undergraduate math courses, so LaTeX-formatted content features heavily. LaTeX is beautiful when used on its own, but it’s hard to handle in the context of web applications. A few lessons and strategies are described below.

Delimiters

I’m dealing with “mixed” content: plain text that is interspersed with LaTeX. Some of this content is coming directly from WeBWorK, which formats LaTeX to be rendered with MathJax. WeBWorK is sending markup that, when simplified, looks like this:

This is plain text.
Here is an inline equation: <script type="math/tex">E = mc^2</script>
And here is the same equation as a standalone block:
<script type="math/tex; mode=display">E = mc^2</script>

It’s also possible for users to enter LaTeX from the front-end of the WP application. In this case, we’ve settled on the verbose delimiters \begin{math} and \end{math}, but we also support a shorthand delimiter borrowed from Jetpack’s LaTeX renderer:

This is plain text.
Here is an inline equation: \begin{math}E = mc^2\end{math}
And here is the same equation as a standalone block:
$latex E = mc^2$

Some of these delimiter formats are more fragile than others, for various reasons – script tags that cannot be escaped, false positives due to use of literal dollar signs, etc – so I standardized on some invented delimiters for data storage. Before saving any LaTeX content to the database, I’m converting all delimiters to the following formats (of my own invention, which explains their Great Beauty):

This is plain text.
Here is an inline equation: {{{LATEX_DELIM_INLINE_OPEN}}}E = mc^2{{{LATEX_DELIM_INLINE_CLOSE}}}
And here is the same equation as a standalone block:
{{{LATEX_DELIM_DISPLAY_OPEN}}}E = mc^2{{{LATEX_DELIM_DISPLAY_CLOSE}}}

This way, the delimiters are unique and easy to process on display.

Slashes

I’m using WordPress custom post types to store data. The lowest-level function available for saving post data is wp_insert_post(). Thanks to a glorious accident of history, this function expects data to be “slashed”, as in addslashes() and magic quotes. If you’re looking for a fun way to spend a few hours, read through this WordPress ticket and figure out a solution, please.

LaTeX uses \ as its escape character, which makes LaTeX-formatted content look “slashed” to WordPress. As such, attempting to save chunks of LaTeX using wp_insert_post() will result in slashes being removed and formatting being lost. So I had two choices: don’t use wp_insert_post(), or don’t use slashes.

I chose the latter. Before saving any content to the database, I identify text that appears between LaTeX delimiters, and I replace all slashes with a custom character. In all its glory:

public function swap_latex_escape_characters( $text ) {
    $regex = ';(\{\{\{LATEX_DELIM_((?:DISPLAY)|(?:INLINE))_OPEN\}\}\})(.*?)(\{\{\{LATEX_DELIM_\2_CLOSE\}\}\});s';
    $text = preg_replace_callback( $regex, function( $matches ) {
        $tex = str_replace( '\\', '{{{LATEX_ESCAPE_CHARACTER}}}', $matches[3] );
        return $matches[1] . $tex . $matches[4];
    }, $text );

return $text;
}

So a chunk of LaTeX like E = \frac{mc^2}{\sqrt{1-\frac{v^2}{c^2}}} goes into the database as

E = {{{LATEX_ESCAPE_CHARACTER}}}frac{mc^2}{{{{LATEX_ESCAPE_CHARACTER}}}sqrt{1-{{{LATEX_ESCAPE_CHARACTER}}}frac{v^2}{c^2}}}

They get converted back to slashes before being served via the API endpoints.

Rendering in a React component

I decided to stick with MathJax for rendering the LaTeX. MathJax has some preprocessors that allow you to configure custom delimiters, but I had a hard time making them work inside of my larger JavaScript framework. So I decided to skip the preprocessors and generate the <script type="math/tex"> tags myself.

React escapes output pretty aggressively. This is generally a good thing. But it makes it hard to print script tags and unescaped LaTeX characters. To make it work, I needed to live dangerously. But I didn’t want to print any more raw content than necessary. So I have two components for rendering text that may contain LaTeX. Click the links for the full source code; here’s a summary of what’s going on:

  1. FormattedProblem performs a regular expression to find pieces of text that are set off by my custom delimiters. Text within delimiters gets passed to a LaTeX component. Text between LaTeX chunks becomes a span.
  2. LaTeX puts content to be formatted as LaTeX into a script tag, as expected by MathJax. Once the component is rendered by React, I then tell MathJax to queue it for (re)processing. See how updateTex() is called both in componentDidMount() and componentDidUpdate(). Getting the MathJax queue logic correct here was hard: I spent a lot of time dealing with unrendered or double-rendered TeX, as well as slow performance when a page contains dozens of LaTeX chunks.

A reminder that React’s dangerouslySetInnerHTML is dangerous. I’m running LaTeX content through WP’s esc_html() before delivering it to the endpoint. And because I control both the endpoint and the client, I can trust that the proper escaping is happening somewhere in the chain.

I use a very slightly modified technique to provide a live preview of formatted LaTeX. Here it is in action:

output

Pretty cool TeXnology, huh? ROFLMAO

Improved ‘equalto’ validation for Parsley.js

I’ve been playing with Parsely.js for form validation on a client project. It’s pretty nice, but I was unhappy with the ‘equalto’ implementation. ‘equalto’ allows you to link two fields whose entries should always match, such as when you have password or email confirmation fields during account registration. parsley-equalto is not symmetrical. If you enter some text into A, and enter non-matching text into B, B will not validate. If you correct B so that it matches A, then B will validate. So far, so good. But if you correct A so that it matches B, it won’t change the validation.

So I wrote a custom implementation that triggers validation on the paired field, making the link between the fields symmetrical. It’s pretty ugly (to avoid recursion) and doesn’t have any error handling, but it should point you in the right direction. (I’ve called it iff, which you can look up.)

The markup:

<input
  name="password"
  id="password"
  data-parsley-trigger="blur"
  data-parsley-iff="#password-confirm"
  data-parsley-iff-message=""
/>

<input
  name="password-confirm"
  id="password-confirm"
  data-parsley-trigger="blur"
  data-parsley-iff="#password"
  data-parsley-iff-message="Passwords must match."
/>

The validator:

var iffRecursion = false;
window.Parsley.addValidator( 'iff', {
    validateString: function( value, requirement, instance ) {
        var $partner = $( requirement );
        var isValid = $partner.val() == value;
        if ( iffRecursion ) {
            iffRecursion = false;
        } else {
            iffRecursion = true;
            $partner.parsley().validate();
        }
	return isValid;
    }
} );

Indirect funding and the limits of free software patronage

A few thoughts about direct and indirect funding for free software development.

Proprietary software is often sold at retail, which makes for a diverse economic model. Let’s say that a million people buy a $10 yearly license for your iWidgetFactoryPro™. This year you’ll have $10,000,000 dollars, much of which can be used to fund future iWidgetFactoryPro™ development. If you lose fifty thousand users next year, you’ll have $500,000 less to work with. That’s a lot of cabbage. But you’ll still have $9,500,000, enough to carry on your product development, perhaps with a slightly reduced scope. Retail pricing spreads the risk around.

Most free software is not sold in a retail fashion. A single company might pay for the development of a tool, and then decide to release it under a free license. Or a tool’s builders may fund development themselves, with their own money or their own free time. Or a project that starts off as a labor of love may become important enough that a number of companies volunteer resources to improve it. In any of these cases, the funding model is highly centralized. Instead of a million users who share the financial burden roughly equally, as with retail software, a free tool may have a million users but just a small handful of funders, each of which is footing a disproportionately large part of the bill. It’s a precarious setup not merely because the sheer number of funders is so small, but also because the costs are distributed in a way that’s so uneven – and unfair! – that individual funders have arguably more inclination to walk away than the retail licensee who’s paid a lousy ten bucks for a copy of iWidgetFactoryPro™.

The asymmetrical funding model for free software is the cause of much hand-wringing among the individuals who maintain free software projects. How does a volunteer, a solopreneur, a small businessperson, an underfunded or unfunded Desperado, take on the huge and often unrewarding task of maintaining a popular project, while still managing to make money? My friend Daniel Bachhuber has written a number of posts recently in which he struggles with this problem, and is experimenting with a couple different models:

The key idea behind Sparks is to create a space where WordPress-based businesses can contribute to an open source roadmap, collaboratively prioritize, and then share the cost of development and maintenance […] When I explain the concept of Sparks to a prospective customer, they get it. It makes a ton of sense to share the burden of building and maintaining boring, business-critical infrastructure.

The strategy is to hedge against some of the instability of the “patronage” model – free software tools being supported by a very small group of generous funders – toward a more diversified financial base.

The structure of “Sparks” – crowdfunding, but where the members of the “crowd” are businesses with budgets – moves toward retail software in two ways that are worth considering. First, by getting interested individuals to take an equal share in funding the software, it tries to make the funding model more fair. Second, by tying the decision of which tools to build to the number of voters (or backers, or whatever) in support of that specific tool, it tries to make the funding model more direct.

Can this work? Direct funding models are kind of like health insurance exchanges: the economics only work if there’s a mandate that everyone participate. Proprietary software licensing is one such mandate. With free software, it’s an uphill slog, and a hard sell.

I have mostly given up on direct client funding for my free software work. There are a couple of interconnected problems:

  • Clients can be convinced to pay for something new and shiny and released with public credit to their name. But fewer want to pay for maintaining something old and boring and anonymous.
  • Once a project is released and in broad use, the client’s ongoing needs for improvement often (usually) diverge with the needs of the broader community.
  • When you are a maintainer of a large software project, quid pro quo contributions – “we’ll pay you to add this feature to WordPress” – are fraught with ethical and practical difficulties.
  • The things that clients want to build are not usually the things that I want to build, or the things that I think need to be built.

As direct funding has become less attractive and more difficult to manage, I’ve turned more toward indirect models. I’ve spoken and written at length about what I call “the reputation cycle”. This is the idea that time spent contributing to free software can improve your reputation, which allows you to increase your rates, which allows you to bill fewer hours, which allows you to contribute more time to free software. Over the last few years, I’ve ratched myself up to the point where I spend roughly 50% of my working time doing work that is not paid for by a client.

Or, at least, not paid for directly. Client work subsidizes free software work, but the subsidy is indirect. This indirectness avoids most of the problems sketched above. My decisions about how and where to contribute to the projects I’m involved with are made based on my own interests and my own assessment of project priorities and needs. Since the work is not being done under the aegis of any specific client relationship, I’m not bound by any specific client expectations.

If not framed correctly, the indirect model can feel vaguely dishonest. It involves charging a higher rate to paying customers, without providing any direct benefit for the increased cost. In one sense, this feeling is clearly misguided; the software I choose to work on in my “free” time is the very same software that powers my clients’ sites. They’re reaping benefits that are indirect – but not that indirect.

More importantly, the sense of dishonesty is misplaced because there’s no deception involved. The model is indirect, but it’s not implicit. My message for potential clients is pretty explicit: When you hire me, you are not only buying top-quality technical work, but you are also funding the more general improvement of the free software projects in which I’m involved. It’s part of the brand. It’s less Robin Hood – illicit redistribution of funds – than, say, buying organic milk: you pay more for a slightly better product, knowing that part of the extra cost goes toward the normalization of a system of production that’s superior to the conventional system.

Can this funding model – indirect, but explicit about it – be scaled? Probably not. Like patronage, it depends on the good will of a fairly small number of benevolent folks – developers and clients – to shoulder the burden of the other 99% of users. But there’s something noble about it too. A totally “fair” system, in which each user pays an equal amount to use a piece of software, ignores the fact that not all users have equal resources. One of the beauties of free software is that the generosity of those who can afford to contribute can benefit those who cannot.

 

WordCamp Chicago 2016 slides

I just finished giving a talk at WordCamp Chicago titled “Backward Compatibility as a Design Principle”, in which I discussed WordPress’s approach to backward compatibility, how it’s evolved over the years, and its costs and benefits when compared to the alternatives. I’m not sure that the slides are very helpful in isolation, but someone asked me to post them, and I am not one to disappoint my Adoring Fans. Embedded below.

wc-chicago-2016

download as pdf

BuddyPress Docs 1.9.0 and Folder support

BuddyPress Docs is one of my more popular WordPress plugins. For years, one of the most popular feature requests has been the ability to sort Docs into folders. Docs 1.9.0, released earlier this week, finally introduced folder functionality.

The feature is pretty cool. When editing a Doc within the context of a group, you can select an existing folder, or create a new one, in which the Doc should appear. Folders can be nested arbitrarily. Breadcrumbs at the top of each Doc and each directory help to orient the reader. And a powerful, AJAX-powered directory interface makes it easy to drill down through the folder hierarchy. (Folders are currently limited to groups, which simplifies the question of where a given folder “lives”. An experimental plugin allows individual users to use folders to organize their personal Docs.)

docs-folders

I’ve got a couple reasons for drawing attention to this release. First, the Folders feature was developed as part of contract work I did for University of Florida Health. They use WordPress and BuddyPress for some of their internal workspaces, and the improvements to BuddyPress Docs have helped them to build a platform customized to their users’ specific needs. My partnership with UF Health is a great instance of a client commissioning a feature that then gets rolled into a publicly available tool – the type of patronage that demonstrates the best parts of free software development as well as IT in the public sector.

A bonus side note: UF Health Web Services is currently hiring a full-time web developer. If you know PHP, and want a chance to work with cool people on cool projects – including WordPress and BuddyPress – check out the job listing.

The other fun thing about this release is that it’s the first major release of Docs where I’ve worked closely with David Cavins, master luthier and BuddyPress maven. He’s a longtime contributor to Docs, and has done huge amounts of excellent work to bring 1.9.0 to fruition. Many thanks to David for his work on the release!