Tips and Points

This summer I've been working on a project with my wife called Tips and Points. The idea is to analyze irish dance competition results (and, eventually, other kinds of results) and glean some insight into how one can improve their dancing or where they're getting marked off the most at competition.

I've had a lot of fun working on it, and one of my goals going into it was to enjoy the development process and stack as much as the end result. In that vein, I decided to tackle the project in PureScript despite not having written as much tooling for the stack. In the way of compromise, I've been using a strangler approach and only rewriting things in PureScript as needs dictate. PureScript's foreign function interface is one of my favorite features of the language, and it makes the strangler pattern an easy grab, since calling back and forth between JavaScript and PureScript can often be done with almost no extra effort.

I've even started treating JavaScript as an "effects DSL" sometimes, preferring to write certain kinds of dynamic expressions in JavaScript directly without attempting to make more generalized bindings. For example, when working with node's child process API, there are a lot of dynamic options that are fairly idiomatic, but don't map well to PureScript. For those cases, I'll just use the FFI to bind to specific functions that fix specific options, rather than generating the options from PureScript. This doesn't cost much in terms of type safety, since effects heavy code already punches through type safety. For example, if I write a program to execute a command with some arguments, I'm already to going to need to test that manually to have any level of confidence that it works, because otherwise the surface area for failure is too high: the program might not be on the path, I might have passed in the wrong arguments strings, or maybe the process never finishes running and hangs the parent. Adding to this the possibility that I supplied the wrong options to the call really doesn't add any testing overhead, but the code is much easier to write in JavaScript.

Stuff That's Working

pg-boss for jobs

It's kind of bananas how much more software sophistication you can support once you throw a good job queue in the mix. Add to that the ability to use transactions, and you can buy some really nice guarantees about your production code. pg-boss is a little bit underdocumented, but it's been exactly what I need out of a job queue.

I wrote some lightweight bindings in PureScript for the createQueue, send, and work methods, and that's been plenty for my needs. Unfortunately, using the dashboard conflicted with my version of React. I got around this by installing the binary globally, and I wrote a little wrapper around that runs it and then opens a browser window. Rather than hosting the dashboard on my production server, I'm running it locally (for myself, I don't really see any downside to this approach, and it's less work).

Prefer my own ctl program over npm scripts

In the past, I've followed the usual pattern of adding all of my "tooling" scripts to my package.json in the scripts field. However, I got the feeling that the practice of trying to cram all of my logic into a bunch of one-liners was pushing me away from sophistication in these scripts. So, instead I went ahead and used optparse to write a command-line tool to handle all of my common workflows. I think this has worked out well, and I'm definitely adding more sophistication to these tools now, which makes the developer experience feel much nicer. I don't really like writing ./bin before all of my commands, nor do I really like things like the direnv approach of updating paths whenever you change directories (it doesn't work well when you launch new terminals, and also the overhead of loading shit whenever you cd is kind of annoying). So, instead, I settled on alias ctl="./bin/ctl" as a really low-tech approach, and I've been happy with it so far. If I decide to write more of these control programs for other repos, my dumb little alias will still work just fine. I thought about adding ctl to my global path or using an alias that'll work from outside of the directory. However, I kind of like that this alias pins the script to only work when you're in the right directory, because a lot of my scripts rely on relative paths and will rm -rf to do cleanup; I think this makes it harder to fuck up.

Writing SQL Instead of Using an ORM/FRM Layer

I've long felt that ORMs are overrated, because they're such a leaky abstraction. Databases are really very not object oriented, and trying to work with them that way has always felt clunky to me. However, it's definitely true that working without an ORM means writing more code, but I've found that, once the idioms are established, more code doesn't really mean more time.

In PureScript, this has meant writing a lot of encoders and decoders, and I've accepted this as the cost of doing business. I've embraced a style of defining my sql query, encoder, and decoder all in the were block of a function that defines each query that I need. Here's what it looks like in practice:

selectLastImportStatus
  :: ∀ c
   . Connection c
  => c
  -> UserId
  -> Aff (Maybe GetLastPayloadImportStatus)
selectLastImportStatus conn userId =
  do
    fRows <- Pool.query conn sql [ UserId.encode userId ]

    ε (Decoder.first decode fRows)
  where
  sql = Sql
    """
    SELECT *
    FROM tp_payloads
    WHERE user_id = $1
    ORDER BY created_at DESC LIMIT 1;
    """
  decode f = do
    id <- Decoder.property "id" PayloadId.decode f
    importStatus <- Decoder.property "import_status" PayloadImportStatus.decode f
    createdAt <- Decoder.property "created_at" Decoder.jsDate f

    pure { id, importStatus, createdAt }

Looking at the code above, I feel confident that I know exactly what I'm getting, and if I want to use some more advanced feature of SQL, it's right there. This also makes it so that I can mostly grab my queries directly and run them against my development database to see if everything is working as intended. I guess you could argue that you can do this in the console with an ORM, too, but I'd definitely prefer to work with the tables directly.

If you're familiar with PureScript, then you might also notice that in the code above the decoder is actually an Effect. I wrote an article about this earlier, but generally speaking, if my decoder is failing, then that means that there's some bug in my application somewhere, and I really want it to crash, so working with a layer that decodes into something pure (like an Either) is a bunch of performance overhead, extra monadic complexity, it obscures the stack trace, and is kind of the opposite of what I actually want. Just leaning into effects and crashes has worked great.

Incrementally Layering PureScript Over Express

Rather than writing a whole set of bindings, I've been layering on just what I need from Express. For the most part, this has been pretty easy, and has made it possible to keep moving and iterate on what I'm learning.

Omitting the Application Monad

A lot of idiomatic PureScript examples will write an AppM monad and then layer functionality onto it by using type classes to support new behaviors incrementally. I don't like the level of investment required by this approach -- it just feels a bit heavy-handed in general. Instead of using a reader monad, In my code, I've standardised the practice of passing around a context object for the application, so that signatures look like: TipsAndPoints -> ... -> Aff <something>. This is essentially the same concept as a reader monad, but without making it a monad and needing to do your work there. I also kind of like the fact that, with this pattern, the inputs to the function are represented in the function's params rather than its return value.

Manual Development Testing + Admin Tools

I've long thought that automated tests tend to be bloated and not actually all that useful (especially in the presence of a strong type system). For one thing, automated tests typically work by removing all but very specific effects from the equation (we can't call out to production or even development services when running automated tests, because we need a reproducible setup), but those effects are typically also where real bugs occur, so the situation is one where we're specifically testing all of the things that are unlikely to break. Automated tests are also not cheap, and part of that is that they're a leaky abstraction by definition: Automated tests ask us to bypass our program's built-in abstractions and force leaks so that we can eliminate effects and other behaviors in order to "isolate" tests. For obvious reasons, breaking your program's behaviors on purpose and changing them to something else tends to be error prone.

When I'm testing, I really want to get my hands dirty as much as possible, and I find that manual testing lets me get my hands on the most stuff without the added cost of writing tests. But, sometimes it's difficult to isolate the behaviors that you want test with manual tests. My philosophy on this is generally:

• Write tools for an admin console to aid in manual testing. For example, you can send a fake payload to a job or whatever.
• "Redirect" effects rather than using test fakes. IO redirection is a low-level primitive in computing and IO can probably always be "redirected". For example, if we're writing to a database, then we can write to a dev database instead of production (this is an approach that probably warrants a whole post)