Developer Update #4: Wait, it’s March‽
In our first update of 2022, Brian is deeply confused about time, offers hints of an unexpectedly manic development period due to a secret project, and thinks about the pragmatics of technical debt.
The problem with manic deadlines in lieu of leave is that they leave very little capability for “normal work.” Updating the blog, unfortunately, is that sort of “normal work” — important outreach, but always second priority.
While we’re expecting a public announcement soon, I’m going to have to be somewhat cagey in discussing why we have unexpectedly had a 6 weeks of extra development. But that… belongs in the status update!
Status Update
The status quo ante1, when we wrote dev diary 3, was that we were going to start an ARDC-funded development period roughly a week ago, and would continue for about ten weeks. That development period would bring us to public beta and we would be running focus groups2 during this time to start accumulating user feedback. The team would have taken a month of leave in December/January and we would be refreshed and ready to face 2022.
Instead, one of our long-term contacts came to us with what can only be described as an incredibly urgent opportunity. If we were able to deliver a functional notebook for a major project — they would fund the rapid acceleration of our dev program.
As such, rather than the thinnest possible bridge between a theoretical notebook creation and export, we now have a server with dedicated infrastructure hosting 21 instances of FAIMS3 (to support 20 different groups and testing) each compiled to Google Play with a robust field manual and a well-tested exporter to jsonl. The 6 weeks of panicked development got us server-level authentication (using DataCentral as an example authentication service), many many sync improvements, and many many many bugfixes.
We have shipped FAIMS3 to prod(ish)!
While there are still many features on our roadmap, such that we’re not comfortable offering our electronic field notebook as a generalised software as a service — we are able to offer concierged notebook experiences to those willing to live with few bells and whistles. Hence, FAIMS3 is “in production”-ish. We’re going to resume ARDC-sponsored development shortly, but this was an amazing opportunity to concentrate on fit and polish to deliver something specifically useful for a nation-wide data-sampling campaign.
We’re all also busy playing catchup on our other “normal work”, so more details later.
Thinking About Technical Debt
When Martin Fowler wrote about the challenges of dealing with Technical Debt during scaleup it immediately put me in mind of the developmental challenges we faced during January. As they kept articulating example after example, I felt that our current situation felt awfully similar.
This is not a problem.
Technical debt is the accumulated externalities of decisions made under resource constraints: time, money, attention, or material. Tradeoffs happen and usually the decisions are made intentionally or are forced as a function of the larger operating environment. I certainly see how decisions I’ve made around the planning and implementation of FAIMS3 have expressed elements from Company A (ship it fast!) and from Company B (Do everything right from the start!).
Instead, I want to muse briefly on the fortune/blessing/providence/circumstances that forced our hand to ship to production much earlier than expected. The demands when shipping code to customers with their boots on, ready to step into the field, are very different than those when we are building a product appropriate for an MVP (minimum viable product).
A persuadable demo is heavy on features or capability. By insisting on a notebook generator and on round-trip capability, I was sacrificing both notebook-specific features and a deep exploration into increasing reliability in the interests of being able to tell a compelling story. My plan was to have enough functionality to be able to demonstrate our product in carefully controlled focus groups that would then be able to have good discussions as to routes forward and unforeseen design consequences.
Instead, by changing our focus to a need to ship to prod now! now! now!, there has been a focus on data scale, server automation, and the pragmatics of authentication and sync. These features, while necessary, are not headline features during a demo — but they are far more fundamental for the day-to-day data collection needs. Thus, a large chunk of development was the unexpected facing of technical debt, rather than the planned and smooth reduction of debt possible when external deadlines were not screaming at us. At the end of the day, however — the urgency of shipping to prod focused our minds wonderfully and we were able to achieve a pilot product accepted by the clients.
While there were certainly things I would have quite grumpily communicated to my past self — how could I have been so stupid as to focus on [redacted] — we had relatively little debt to pay off and were able to do so. While we haven’t even entered the exponential elbow of scaleup to more than one client yet, I’m far less concerned that we will be unable to do so.
Thanks mystery client! By forcing our hand to ship to prod very early, we’ve paid off our technical debt far earlier than expected! This was very useful, even if it was a little frantic!
I’m not sure what my advice would be to other projects who are also facing a runway to shipping their MVP. The continual advice that we got was to ship early. Our project is so complex that I think the 30-odd weeks of dev was the earliest we could have credibly shipped the project. I think my advice is ‘Get someone to pay you to ship to prod when it is uncomfortable but not impossible for you to do so.’ Good luck implementing that advice!
Reading this $unit_of_time
Somehow the $unit_of_time which defaults to ‘biweekly’ is now… ‘quarter’ — I think I need to file a bug report.
Naomi Kritzer writes an excellent little short story on The Dragon Project
Simon Willison shares an interactive explainer of SHA256 by Domingo Martin.
Charity shares some useful “bad smells” to be really concerned about during an interview.
Wouter writes about different sorts of backups — an excellent explainer to pass onto your colleagues.
Chris Lee reflects on the challenges of applying a new curriculum.
Matt Blaze unpacks3 The Cryptography of Orphan Annie and Captain Midnight
danah boyd4 both uses text emotes like I do ::cheers:: and has transcribed one of their lectures about responsible data science.
Marianne Bellotti shows how organisational hierarchies make certain sorts of technical debt extremely predictable.
Filippo Valsorda provides a strongly worded wakeup call around professional maintenance in the open source community.
Cedric Chin discusses tacit knowledge versus deliberate practice.
Amy Shira Teitel researches alternate lunar rovers from the 1960s! Futures that could have been!
The “current situation, before”.
In positive news, we still plan to run focus groups! Numbers are limited for the first round in April/May but others will follow so contact Penny if you’re keen to try out FAIMS3!
Sorry about the use of ‘unpack’ — but it was just so boring writing “writes about”.
danah also had a major impact on my PhD a decade ago when they wrote about social steganography. Also, if you’ve followed this footnote, you might be interested in the Philosophy of Data! (Reach out if you are.)
For more news, subscribe!
