I’ve been developing software for a long time and have put dozens of systems into production. I’ve also created many systems that are no longer in use. Sometimes a business was bought by another company; sometimes the company went out of business. Other times the software simply stopped being needed.

It’s clear software has a limited lifespan — and with AI making software development easier than ever, the cost and effort required are falling. I don’t think it’s yet easy for anyone to write production-quality software, but it’s getting closer. So what does this mean for the value of software?

Martin Alderson’s article “Has the cost of building software just dropped 90%?” argues that Jevons Paradox suggests lower costs will lead to more software being written. Companies that could never afford custom tools may suddenly be able to build them. Looking at the myriad spreadsheets most companies use to run their businesses, there’s clearly a lot of value in automating those processes.

In the short term, I expect companies — even ones wary of AI — to become less nervous about letting AI help build software. At first we’ll see local tooling: email workflows, automated customer outreach, and similar tasks. As those succeed, some of those small tools may be consolidated into bespoke systems.

For example, I use QuickBooks for accounting, but it’s often dramatic and expensive overkill for my needs. With a few hours’ work I could build bespoke software that does exactly what I need and nothing more. If the cost of building that software is low enough, why not replace a subscription? I’d love to stop paying $50/year to Intuit.

I don’t expect anyone to replace SAP or Oracle with in-house software — not yet. But there’s plenty of mid-sized software that could be replaced. It will be interesting to see how this plays out.

If you happened to read my previous post about where AI is going then you might have picked up on my concerns about the rising cost of memory and compute. For the last few years any time I’ve recommened hosting to a client I’ve recommended some variant of AWS, Azure, or GCP. I’ve done this because I remember how painful it used to be to get a server allocated from an overstressed IT department. When I was on internship years ago for a large petro-chemical company we planned had to plan for deployment 6 or even 12 months in advance. IT had to figure out capacity and plan, plan plan.

The cloud make this so much easier. We could spin up a server in minutes and with platform as a service we didn’t have to worry about underlying compute or updates to operating systems, or patching. It was a dream. I’d tell clients “You’re an insurance company leave being an IT company to Microsoft/Amazon/Google”.

Recently though I’ve been looking at the cost of hosting an application in Azure. Even for our small applications it is surprisngly expensive. We had a tendency to vastly overbuild the hosting when for applications that have only hundreds or low thousands of users. What really gets us is the cost of the database.

This isn’t a huge or highly performant database, but it is still costing us almost $400/month. For a small application with only a few hundred users this is a huge cost. Add on dev and test environments and the cost is even higher. Painful.

I wanted to understand if I was still recommending the right thing to people so I started looking at alternatives. My first stop was looking at a VPS provider. I wanted something in Canada becasue this is where I live and I wanted to support local businesses. I found a provider called OVH that had a VPS with 8 vCores, 24 Gig of memory and a 200 Gig SSD for $32 a month off contract.

Now obviously vCores aren’t a standard measurement unit so it’s hard to compares that with something from another offering like Azure but on the surface this feels similar to a D12 from Azure which is $380/month and that’s in USD so closer to $500/month CAD. The OVH offering is about 1/15th the cost of Azure.

Yikes!

So I figured I’d sign up. Maybe the added cost here would be becasue it was really arduous to get signed up. I’ll admit going through the process wasn’t as polished as a cloud provier. Little things like this

Is that a last name field or a field for people who only have a single name? Is Madonna buying hosting a frequently? The 2FA setup, though, was probably the best I’ve ever done. It had the really nice feature of letting you record the name of the app you were using for 2FA. I have multiple apps for 2FA and this is a really nice feature.

I did spot this warning, though

Okay so there might be some latency in getting this server provisioned. That sucks, but perhaps I’ve gotten too used to the instant gratificaiton of demanding compute and getting it 5 minutes later. The email I got said that the server would be provisioned within the next 7 days. That’s a long time to wait but whilte we do let’s talk about why this VPS might not be a great idea. What are we losing by not using cloud hosting?

Well first off it’s clear that scaling up and down is going to be so much harder. I can’t scale the machine up instantly when there is a spike in traffic. Equally, if that spike subsites I can’t scale down quickly. So I’m going to have to be comfortable knowing that I’ve got to pay for excess capacity. A lot of the line of business applications I work with have almost no traffic on the weekend and evenings but then predictable stable load during the day. I’m going to have to over provision to handle the maximum load. Maybe I can mine bitcoin or something on the weekends. Is that SETI@home project still a thing?

Next I’m losing the platform as a service benefits. I’m going to have to patch the operating system and database myself. I get to do backups, restores all that myself. So I’m going to trade my time for reduced hosting costs. Let’s imagine that I’m setting up a database server and a web server which is kind of typical for a line of business application. This is going to run to $700/month in Azure(I’m going to use Candian dollars from here out). So that gives me $650 a month to play with for manual patching and updating and all that jazz. If I pay myself $200/hour (woo, massive pay raise) then I can spend 3 hours a month on that work and still come out ahead. I don’t know if that scales well to dozens of servers but for a small application it might be worth it.

I’m likely going to have an increased cost for the initial set up of the server. I’ve got to figure out which web server to use, probably some default settings for PostgreSQL and tie it into a build pipeline. I suppose we’ll be deploying via SFTP or something like that.

Finally, I’m losing the reliability and security of the cloud. I don’t have the same level of redundancy and failover that I would get with a cloud provider. If my server goes down, it’s on me to get it back up. If there is a security breach, it’s on me to fix it. This is a big risk for any business, but especially for small businesses that may not have the resources to deal with these issues.

The other piece of the puzzle here is that the knowledge and expertise required to manage a VPS is higher than that required to use a cloud provider. I’ve been doing this computer stuff for year and I’m confident in my ability to teasily manage this but then I’m also still a bit sad about how Open Solaris failed to take off. I think that a lot of this gap can be closed using AI agents these days.

It’s a calculated risk.

So, dear readers, I pulled the trigger and signed up for the VPS. Once it is provisioned then we’ll return here and talk through setting up and deploying to it.

There is so much money tied up in AI these days that it seems impossible that all these AI comapnies are going to survive. Eventually the venture capital money is going to dry up and who is going to be left holding the bag?

At the same time the progress on models seems to be slowing down. When I use Cursor I’ve got a varierty of different models to pick from and I don’t see a super meaningful difference between them. Perhaps I’m just oblivious but they’re all pretty spectacular. And that’s part of the problem, changing models comes at almost zero cost. So today I can use Gemini and tomorrow GTP-5 so what moat do these companies have other than capacity? Even if I’ve got an applicaiton that is using an AI API then switching to a different provider is trivial.

Meanwhile running models locally is getting easier. Sure they’re not as good as the big models but for many applications they’re good enough. And running locally means no ongoing costs, no API limits and no data privacy issues. If you’re a small or medium business then buying a few GPUs to act as a farm for employeees to use is a one time cost that might make more sense than paying for API calls forever.

I honestly belive that the future is going to be a hybrid approach: some operations will run on local models and then fewer will be farmed out to a larger cloud model. But the number of application which make sense to run on a cloud model is going to be fewer and fewer as local models get better and better. Summarizing emails, writing cover letters, cleaning up data and reports are all things that can be done with small models. So that’s going to really undercut the market for OpenAI and others.

But here is where things get a little interesting: I was reading that OpenAI is purchasing something like 40% of all the memory being produced by 2030. And that’s just one company, if Microsoft, Google, Amazon and Meta all jump on this bandwagon then what memory will there be left for everyone else? What if that’s the intention? We’re already seeing consumer memory prices spike.

Here are the average prices for DDR5-6000 2x32GB kits over the last 18 months (from PC PartPicker):

That seems like a hockey stick graph to me. And this memory is used in everything from desktops to laptops to phones to TVs. I best stupid toasters and fridges use memory now too. I’m remminded that during COVID a lot of tech companies were hiring anybody they could, not because they had work for them, but because they wanted to hoard talent and keep it away from competitors. What if the same is happening with memory and compute? What if these big AI companies are buying up all the resources they can get their hands on to keep it away from competitors and startups? And perhaps they’re buying them to keep them out of the hand of consumers so they cannot run models locally?

That seems like a terrible thing for everybody. This strategy, if it is indeed a strategy and not a wild conspiracy theory, is going to cost billions. And just how long is venture capital going to pay for hording resources? At some point these companies are going to have to make money. And if they’re spending all their money on hoarding resources then there won’t be anything left for R&D, marketing or sales.

I’ve started playing around a bit with the source control tool Jujutsu which is commonly referred to as jj. Git has been my go to tool for what seems like decades now but in the before times I worked as a release engineer and made use of a huge stable of source control tools as our code base was spread over many versions and had been created from purchasing lots of other companies. For a while there I was working on a daily basis with

• ClearCase
• Perforce
• Subversion
• CVS
• Visual Source Safe
• Mercurial
• Git
• CCC/Harvest

I’m using Jujutsu at my day job now because it just layers transparently on top of git so I don’t need to go seeking permission. I’m only a few days into using it and I’m not thoroughly convinced yet that it is better than git but I’m willing to keep trying.

Here are some of the commands I’m using so far:

Get the latest version of the code from a central repository locally

jj git fetch

Start new work from the latest mainline

jj new main@origin -m "Whatever I'm going to work on"

Bookmark the work with a name I’m going to use as a branch in git

jj bookmark my-feature-branch

Push my work up to Github

jj git push --allow-new

Create a new commit before my current one that I can squash into

jj new -B @ -m "Some description of the work"

Push individual files into the parent change

jj squash path/to/file1 path/to/file2

I’ll keep expanding this document with new commands as I uncover them.

Postgres Data Masking and Anonymization

Predicting the performance of a web application is always a little bit difficult. If it’s a question of how the site will perform under load then you can use things like Artillery to throw requests at it. But sometimes problems arise from increasing amounts of data. This was the case for a site I helped develop a little while back.

It had been in production for a couple of years and was starting to have problems with some of the queries. I profiled some of them and found some query optimizations which cleared it up. But it was annoying that this hadn’t been caught earlier and that it was difficult to replicate in lower environments because they simply didn’t have enough data.

We could have generated data but for any sort of complex data model it’s difficult to create realistic data. Fortunately I knew of a place we could get really well structured data which would have the same performance profile as production: production. But this data contains sensitive information like phone numbers, addresses names and salaries. I didn’t want to just copy that over to the lower environments so I needed a way to clean up the data.

Initially I started with just throwing FakerJS at the data but performance of updating every row with values was not great. After some research I found the Postgres extension PostgreSQL Anonymizer which looked like it would fit the bill.

PostgreSQL Anonymizer

PostgreSQL Anonymizer is a Postgres extension which allows you to mask data in a variety of ways. It can be used to anonymize data or to mask it. It has a number of built-in functions for common tasks like replacing names with random names, addresses with random addresses and so on.

Getting it installed on Azure Postgres Flexible Server was a little tricky and I’ll probably post on that in a separate article. But once it was installed it was easy to use.

There are a couple of modes it can run in Dynamic and Static. In the dynamic mode it will leave the data in the table put but will apply rules based on the current user role to mask the data when queried. This is a pretty handy thing and you could use it for something like masking SSNs for any user other than the admin. Static mode will actually update the data in the table to a new value, per your rules. This is what I opted for as by default the masking wasn’t deterministic.

Using PostgreSQL Anonymizer

The script I put together to run after I had restored a backup into the test environment looked like this:

-- Start up the extension
CREATE EXTENSION anon;

-- Rules
SECURITY LABEL FOR anon ON COLUMN household.contact_email
  IS 'MASKED WITH FUNCTION anon.fake_email()';
SECURITY LABEL FOR anon ON COLUMN household.contact_phone
  IS 'MASKED WITH FUNCTION anon.random_int_between(10000000,90000000)';

SECURITY LABEL FOR anon ON COLUMN address.street1
  IS 'MASKED WITH FUNCTION anon.fake_address()';
SECURITY LABEL FOR anon ON COLUMN address.street2
  IS 'MASKED WITH FUNCTION anon.fake_address()';

...

-- Anonomize database statically
SELECT anon.anonymize_database();

You can see the sections there first enable the extension, then create a series of rules which will replace the data in the columns with fake data. Finally SELECT anon.anonymize_database(); will actually kick off the changes. A few seconds of crunching later and the data is all faked up and we can hand over the database to QA or developers without having to worry about sensitive data leaking.

Every once in a while I run into the need to generate a C# client for some API which has been nice enough to provide me with OpenAPI specifications. But it’s one of those things that I do so infrequently that I always forget how to do it. So I thought I would document it here.

The first thing to do is to install the OpenAPI generator. It’s written in Java which is obviously a decision I’m solidly against. As I run a Mac most of the time I prefer to use brew since it’s easier that trying to figure out how to build stuff with Maven.

brew install openapi-generator

Now comes the fun part: figuring out the options to use. There are bunch of different generators for different languages and on top of that each generator has options. The C# generator is called csharp and the options are described in some detail here https://openapi-generator.tech/docs/generators/csharp. In my case I was looking to generator a client for a US Government API that they seem to be really snippy about people getting their hands on documentation about it without going through a heap of hoops so we’ll just call it USGovAPI because this administration is not one I want to be on the wrong side of.

In addition I wanted to generate one for .NET 8 because the project is still running on the long term support version of .NET. So the command ended up being

openapi-generator generate -i swagger.json -g csharp -o out/usgoveapi --additional-properties=packageName=USGovAPI,targetFramework=net8.0

And with that we get a nice little C# client that we can use to call the USGovAPI. It even includes some unit tests to go along with it.

Just last week .NET 9 was realeased to much fanfare. There are a ton of cool and exciting things in it but for my current project I want to stick to a long term support version of .NET which is .NET 8. We might update later but for now 8 is great. Unfortunately dependabot isn’t able to read my mind so it was continually proposing updating to .NET 9 packages.

Fixing this is easy enough. I needed to add a couple of lines to my dependabot file to limit the sorts of updates it did to just minor and patch updates. Notice the update-types section.

updates:
  - package-ecosystem: "nuget"
    directory: "/."
    groups:
      all_packages:
        update-types:
          - "minor"
          - "patch"
        patterns:
          - "*"
    open-pull-requests-limit: 50
    schedule:
      interval: "weekly"

With this in place dependabot is only proposing minor and patch updates to my .NET packages. It does mean that if we see other major version updates to non-Microsoft packages we’ll have to manually update them.

I needed to get Particular Service Control up and running on our k8s cluster this week. Part of that is to get an instance of RavenDB running in the cluster and this actually caused me a bit of trouble. I kept running into problems where RavenDB would start up but then report that it couuld not access the data directory. What was up?

I tried overriding the entry point for the container and attaching to it to see what was going on but I couldn’t see anything wrong. I was able to write to the directory without issue. Eventually I stumbled on a note in the RavenDB documentation which mentioned a change in the 6.x version of RavenDB which meant that Raven no longer ran as root inside the container.

K8S has the ability to change the ownership of the volume to the user that the container is running as. This is done by setting the fsGroup property in the pod spec. In this case Raven runs as UID 999. So I updated my tanka spec to include the fsGroup property and the problem was solved.

...
deployment: deployment.new($._config.containers.ravendb.name) {
      metadata+: {
        namespace: 'wigglepiggle-' + $._config.environment,
        labels: {
          app: $._config.containers.ravendb.name,
        },
      },
      spec+: {
        replicas: 1,
        selector: {
          matchLabels: $._config.labels,
        },
        template: {
          metadata+: {
            labels: $._config.labels,
          },
          spec+: {
            securityContext: {
              fsGroup: 999,
              fsGroupChangePolicy: 'OnRootMismatch',
            },
            containers: [
              {
                name: $._config.containers.ravendb.name,
                image: $._config.containers.ravendb.image,
                ports: [{ containerPort: 8080 }, { containerPort: 38888 }],
                volumeMounts: [
                  {
                    name: 'data',
                    mountPath: '/var/lib/ravendb/data',
                  },
                ],
                env: [
                  {
                    name: 'RAVEN_Setup_Mode',
                    value: 'None',
                  },
                  {
                    name: 'RAVEN_License_Eula_Accepted',
                    value: 'true',
                  },
                  {
                    name: 'RAVEN_ARGS',
                    value: '--log-to-console',
                  },
                  {
                    name: 'RAVEN_Security_UnsecuredAccessAllowed',
                    value: 'PrivateNetwork',
                  },
                ],
              },
            ],
            volumes: [
              {
                name: 'data',
                persistentVolumeClaim: {
                  claimName: $._config.containers.ravendb.name,
                },
              },
            ],
          },
        },
      },
    },
...

This generated yml like

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: service-control-ravendb
  name: service-control-ravendb
  namespace: wigglepiggle-dev
spec:
  replicas: 1
  selector:
    matchLabels:
      app: service-control
      environment: dev
  template:
    metadata:
      labels:
        app: service-control
        environment: dev
    spec:
      containers:
      - env:
        - name: RAVEN_Setup_Mode
          value: None
        - name: RAVEN_License_Eula_Accepted
          value: "true"
        - name: RAVEN_ARGS
          value: --log-to-console
        - name: RAVEN_Security_UnsecuredAccessAllowed
          value: PrivateNetwork
        image: ravendb/ravendb:6.0-latest
        name: service-control-ravendb
        ports:
        - containerPort: 8080
        - containerPort: 38888
        volumeMounts:
        - mountPath: /var/lib/ravendb/data
          name: data
      securityContext:
        fsGroup: 999
        fsGroupChangePolicy: OnRootMismatch
      volumes:
      - name: data
        persistentVolumeClaim:
          claimName: service-control-ravendb

I had a report today from somebody unable to log into our Grafana instance. Super weird because this this has been running fine for months and we haven’t touched it. So I jumped onto the machine to see what was up. First up was just looking at the logs from Grafana.

docker logs -f --tail 10 5efd3ee0074a

There in the logs was the culprit No space left on device. Uh oh, what’s going on here? Sure enough the disk was full.

df -h

Filesystem                         Size  Used Avail Use% Mounted on
tmpfs                              1.6G  1.9M  1.6G   1% /run
/dev/mapper/ubuntu--vg-ubuntu--lv   96G   93G     0 100% /
tmpfs                              7.8G     0  7.8G   0% /dev/shm
tmpfs                              5.0M     0  5.0M   0% /run/lock
/dev/sda2                          2.0G  182M  1.7G  10% /boot
tmpfs                              1.6G   12K  1.6G   1% /run/user/1001

This stuff is always annoying because you get to a point where you can’t run any commands because there is no space left. I started with cleaning up some small parts of docker

docker system prune -a

Then cleaned up docker logs

sudo truncate -s 0 /var/lib/docker/containers/**/*-json.log

This then gave me enough space to run docker system df and see where the space was being used. Containers were the culprit. So next was to run

docker ps --size

Which showed me the web scraper container had gone off the rails and was using over a 100GiB of space.

7cf14084c56a   webscraper:latest       "wsce start -v -y"       7 weeks ago   Up 20 minutes               0.0.0.0:7002->9924/tcp, [::]:7002->9924/tcp                                webscraper       123GB (virtual 125GB)

This thing is supposed to be stateless so I just killed an removed it.

docker kill 7cf14084c56a
docker rm 7cf14084c56a
docker compose up -d webscraper

After a few minutes these completed and all was good again. So we’ll keep an eye on that service and perhaps reboot it every few months to keep it in check.

My life is never without adventure but unfortunately it isn’t the living on a beach sort of adventure. No it’s the installing yarn packages. I wanted to have a package installed in my project which was one I’d published from another repository. In this case the package was called @stimms/uicomponents. There were a few tricks to getting Github actions to be able to pull the package: first I needed to create a .yarnrc.yml file. This gives yarn instructions about where it should look for packages.

nodeLinker: node-modules

npmScopes:
  stimms:
    npmRegistryServer: "https://npm.pkg.github.com"

npmRegistries:
  "https://npm.pkg.github.com":
    npmAlwaysAuth: true

Now in the build I needed to add a step in to populate the GITHUB_TOKEN which can be used for authentication. I found quite a bit of documentation which suggested that the .yarnrc.yml file would be able to read the environment variable but I had no luck with that approach. Instead I added a step to the build to populate the GITHUB_TOKEN in the .npmrc file.

- name: Configure GitHub Packages Auth
    run: echo "//npm.pkg.github.com/:_authToken=${GITHUB_TOKEN}" > ~/.npmrc
    env:
    GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- name: Build
    run: |
    yarn install
    yarn lint
    yarn build

The final thing to remember is that by default the GITHUB_TOKEN here doesn’t have read permission over your packages. You’ll need to go into the package settings and add the repository to the list of repositories which can use the package. You just need read access. If you don’t do this step you’re going to see an error like error Error: https://npm.pkg.github.com/@stimms%2fuicomponents: authentication token not provided