Today I worked on some internal tooling for my software development, revisiting two small command-line utilities that I originally created some time ago:

Both tools were originally created fairly quickly to solve specific problems I had at the time. They worked, so naturally they remained largely untouched after that!

They are now undergoing fairly heavy revision, including refactoring, improving code quality, adding tests and generally bringing them closer to something I would consider suitable for use outside my own development environment.

Despite their slightly utilitarian origins, both have proved useful enough that I think they are worth continuing to develop.

Both are written in Go, released under the MIT licence, and can be found on my GitHub account.

db-concat#

db-concat is a small command-line utility for assembling multiple files and pieces of inline text into a single output file.

I originally created it primarily for building SQL database scripts, where a deployment might consist of many individual schema, table, stored procedure, data and migration scripts that ultimately need to be assembled into a single file.

Rather than simply concatenating every file in a directory, db-concat uses a small line-oriented Domain Specific Language (DSL) to describe exactly how the output should be constructed.

The DSL supports:

  • Concatenating files in a defined order.
  • Including other DSL instruction files.
  • Embedding blocks of text directly into the output.
  • Defining and modifying parameters.
  • Substituting parameters into filenames and generated text.
  • Loading parameters from external parameter files.
  • Supplying parameters from the command line.
  • Conditional if, else and endif blocks.
  • String equality and numerical comparisons.
  • Writing parameter values directly into the output.
  • Emitting text and escaped characters such as newlines and tabs.
  • Scoping DSL commands using optional command prefixes.

Parameters can come from several places, with command-line parameters having the highest precedence. This means a common DSL script can define sensible defaults while a build or deployment process can override particular values externally.

The tool itself does not execute SQL or attempt to understand the contents of the files being assembled. Its job is simply to construct an output file according to the instructions it has been given.

An example of the db-concat DSL is shown below:

# Example DSL Instruction File

# Define a default schema name
param DEFAULT_SCHEMA=app_schema

# Set the output file, using a parameter for the version
param DB_VERSION=1.0.0
output ./build/schema_v${DB_VERSION}.sql

# Include common setup instructions
include common/base_setup.dsl

# Conditionally include environment-specific data
if ENVIRONMENT=development
  concat data/dev_users.sql
  set CURRENT_SCHEMA=dev_schema
else
  concat data/prod_users.sql
  set CURRENT_SCHEMA=${DEFAULT_SCHEMA}
endif

# Print the current schema being used
text-begin
-- Current Schema: ${CURRENT_SCHEMA}
text-end

# Concatenate core tables
concat tables/products.sql
concat tables/orders.sql

# Inline text for a stored procedure
text-begin
CREATE OR REPLACE FUNCTION calculate_total(order_id INT) RETURNS DECIMAL AS $$
BEGIN
    RETURN (SELECT SUM(price * quantity) FROM order_items WHERE order_id = $1);
END;
$$
LANGUAGE plpgsql;
text-end

# Another conditional block based on a different parameter
param FEATURE_FLAG=true
if FEATURE_FLAG=true
  concat features/new_feature_tables.sql
  print FEATURE_FLAG
endif

# Final cleanup script
concat cleanup.sql

As can be seen from the example, the output can be constructed differently depending on parameter values.

Parameters can be defined within the DSL script itself, loaded from one or more external parameter files, or supplied on the command line when db-concat is executed.

This was particularly useful for database deployment scripts where I might want to build slightly different outputs for development, testing and production environments while still maintaining one common set of source scripts.

For example, development builds could include test data that would obviously be inappropriate for production, while environment-specific configuration or migration scripts could be selected using the same mechanism.

Although SQL was the original reason for creating the tool, there is actually very little about db-concat that is inherently SQL-specific.

At its core it reads instructions and assembles pieces of text into an output stream. There is therefore nothing stopping it from being used to construct configuration files, source files, deployment scripts or other text-based artefacts.

That more general use is something I may explore as I continue working on it.

repo-status#

repo-status solves a different problem.

Over time I tend to accumulate a fairly large number of Git repositories on my development machines. Remembering where everything is located — and checking whether I have forgotten to commit something somewhere — can become surprisingly annoying.

repo-status provides a simple way of defining those repositories in one central YAML configuration file.

A simplified configuration looks something like this:

repositories:
  - name: db-concat
    location: C:\Development\db-concat

  - name: repo-status
    location: C:\Development\repo-status

  - name: another-project
    location: C:\Development\another-project

Each repository is given a unique name and a local filesystem location.

Once configured, running repo-status against the manifest checks the Git status of the repositories listed within it.

There are options to use Git’s short status format or, more usefully, display only repositories containing uncommitted or untracked changes.

For example, that provides a quick way of asking the fairly important end-of-day question:

Have I forgotten to commit anything?

The utility also contains a list command for displaying the configured repositories and a path command for locating a repository by either its name or its position in the manifest.

The latter means it can also be incorporated into shell commands to quickly change into the directory containing a particular project.

Perhaps the more useful feature, however, is exec.

This runs a command inside multiple configured repositories.

For example:

repo-status exec --async repos.yaml git fetch

will run git fetch across the configured repositories in parallel.

A subset of repositories can also be selected by name or position:

repo-status exec --repos 1,api repos.yaml git fetch

Commands can be executed sequentially or asynchronously, and there is a --dry-run option which shows what would be executed without actually running it.

Most of the major commands can also produce JSON output, which makes the utility potentially useful as part of scripts or other development tooling rather than only for interactive use.

At the moment repo-status deliberately deals only with local repositories. It isn’t intended to replace GitHub, GitLab or any other repository hosting system. It is simply a small utility for managing the collection of working copies sitting on a development machine.

Like db-concat, it exists because I had a small repetitive problem and it was easier to write a tool than continue doing the job manually.

Why I Still Build Small Tools#

Neither of these projects is intended to become a huge application.

In fact, part of their value is that they are small.

They each have a fairly clearly defined responsibility, they can be called from scripts and build processes, and they solve problems that occur frequently enough in my own workflow to justify their existence.

They are also useful projects for experimenting with software engineering approaches without having to introduce those experiments into a larger production system.

The current work is therefore less about adding large numbers of features and more about improving the implementations: breaking responsibilities apart, improving error handling, increasing test coverage and generally making the code easier to understand and maintain.

There will undoubtedly be some additional functionality added along the way, but I would rather keep both utilities focused than allow them to gradually turn into general-purpose applications.

Both projects are MIT licensed, so if you find either of them useful, feel free to use or fork them.

Changes and improvements are also welcome. If a proposed change still fits the general purpose of the project, feel free to submit it and I will take a look.

If what you want to build is a substantial departure from the original purpose of the tool, however, a fork is probably the better option.

And if you keep your fork MIT licensed, there’s always the possibility that I might end up borrowing some of your ideas back!