How I Reduced My OPEX By 99.5% Using Go

Previously

I wrote about How I Processed 666K Pages Of Flattened PDFs into a Full Text Search Engine called the Apario Writer. Upon the conclusion of the last segment, I was able to optimize the compilation time of the original collection of data by rewriting the Sidekiq Ruby pipeline script into a dedicated Go Application. Regardless of what compiling the PDF assets would look like, I still needed to serve those assets - and that's where the Writer did little to nothing to actually address the OPEX of the project from 2020.

Given the size of the data set, the 666K pages ended up compiling into a directory of ~1.13TB in size. This was held in storage that was distributed across several high volume storage dedicated servers on OVH behind MinIO. This provided an S3 compatible API directly.

What I Know About OPEX

OPEX (Operational Expense) is how you describe spending money that is used explicitly for the operations of the business versus a capital expense. Hardware was considered a CAPEX (Capital Expense).

So when Bit Fry Game Studios needed their DevOps pipeline upgraded for the 9‑hour game builds into a 30‑minute private enterprise cloud build, it required a CAPEX investment of $69K plus trust in me in order to achieve a -$15K/month OPEX savings. Annualized over a hardware lifecycle, over $472K can be recovered from OPEX by making a small CAPEX expense up front.

One of the first projects that I ever worked on was in PHP and MySQL on Ubuntu 8.04. It was to balance the budget of a department that had ACME Bucks so to speak. It required me to write a finance module, fully tested, that managed Blue, Green and Black dollars.

• Blue dollars were for OPEX.
• Green dollars were for CAPEX.
• Black dollars were for external vendors where money left the company (versus moving between departments).

Black depreciated instantly - meaning 100% of it was paid immediately. Blue dollars were borrowed over a 12‑month pay‑back period. Green dollars were borrowed over a 36‑month pay‑back period. The department was given $N per month of Blue, Green, and Black dollars, and there were n‑Commitments existing on the books that were depreciating individually from past purchases that summed into the total monthly budget.

I wrote a utility that would express this information plainly to multiple Director of Engineer colleagues that I collaborated with closely through a dashboard interface to see monies coming in and going out for the department. That was attached to greater projects and used elsewhere - but the lesson that stayed with me since those early days at Cisco was the use of budgeting between OPEX and CAPEX. Cisco's size and volume of diverse departments made Blue and Green dollars possible, but in other businesses that are smaller, that's OPEX and CAPEX. I've built tooling around reporting reliable OPEX & CAPEX information for Engineering Directors.

The Costly OPEX Problem

That brings us into the Writer and how there was a SaaS model of the PhoenixVault that I built using Ruby on Rails on a Zoom call with a former Disney animator, watching me type rails new phoenixvault 🐦‍🔥.

When this launched, I had a rough budget of about:

• $1K for ElasticSearch subscription fees
• $2K for MongoDB Atlas subscription (M30)
• $3K for OVH hosting expenses for 12 bare metal hosts in US East & US West regions
• ~$1K per month for other expenses related to the operations of the business

All $7K/mo was an OPEX expense that I engineered a solution out from. Understand what I just said. I engineered a solution out of the problem that I wrote myself into by doing rails new phoenixvault without fully owning the dependency chain of what I was doing in the first place.

In order for me to reduce the OPEX of the project from $7K per month down to $33/month on OVH, required me to strip all dependencies out and seek that 99.5% OPEX cost savings by rewriting the Ruby on Rails app into Go. Not Rust. Go.

My Solution To The OPEX Problem

Okay, so lets actually look at the code that solved my OPEX problem. You ready? I'm using Rocky 9 Linux with SELinux enforcing. 😍

mkdir -p ~/workspace/ProjectApario
git clone g**@github.com:ProjectApario/reader.git ~/workspace/ProjectApario/reader

Running this application requires you to build a config.yml file and then execute the following:

sudo mkdir -p /opt/apario/{app,logs,config,database}
sudo chown -R $(whoami):$(whoami) /opt/apario
# copy your compiled writer output into /opt/apario/app
touch /opt/apario/config/config.yml

Then, you'll need to go back and compile the binary:

cd ~/workspace/ProjectApario/reader
go build -o reader .
sudo chmod +x reader
sudo mv reader /usr/local/bin/reader
which reader

Once you have successfully installed the reader, then you can run it against your config file:

cd /opt/apario/config
touch config.yml

Paste the following into the file for now:

---
database : "/opt/apario/app"
buffer : 1776
limiter : 1776
directories-limiter : 47369
pages-limiter : 47369
directory-buffer : 47369
site-title : PhoenixVault Reader
unsecure-port : 8080
secure-port : 8443

You can reference the config.go if you want to see every configuration option available. This example only scratches the surface of what the binary offers. This post will get into much of what is not here.

Let's begin with PING!

That's a great starting point.

config.yml
enable-ping : true

Enable the /ping endpoint of your application to return PONG. This is useful if you need to do a frequent health check on the binary in order to determine responsivity (new word? how sensitive to response it can be determined I guess) to PING.

CORS - Cross-Origin Resource Sharing Capability Built In

I needed to port over CORS from the Rails offering that came with using Ruby. I also needed to port over CSP.

First - CORS, it stands for cross-origin resource sharing and content - second - CSP, it stands for content security policy. Both of these are cybersecurity offerings built into the binary itself. Historically, when configuring a PHP based website, I would wire up either Nginx or Apache/HTTPD to define these values there. However, for the reader - we specify those values in the config.yml itself!

# existing configs
enable-cors : true
cors-allow-origin : "*"                          # value for Access-Control-Allow-Origin
cors-allow-methods : "GET, POST, PUT, DELETE, OPTIONS"   # value for Access-Control-Allow-Methods
cors-allow-headers : "Origin, Content-Type, Content-Length, Accept-Encoding, X-CSRF-Token, Authorization" # value for Access-Control-Allow-Headers
cors-allow-credentials : false                    # bool value for Access-Control-Allow-Credentials

CSP - Content Security Policy Capability Built In

The Front‑End Engineer's best friend - CSP! Alas, yes, even when you're running the reader on Rocky 9 Linux with SELinux enforcing and a properly configured firewalld, fail2ban and physical firewall policy - you can further harden the protection of the reader endpoint by defining the values of the CSP policy yourself, including the path that the report writes to.

# existing configs
enable-csp : true
csp-domains-csv : " "                        # List of CSP domains in CSV format
csp-thirdparty-csv : " "                     # List of third party domains in CSV format
csp-thirdparty-styles-csv : " "              # List of third party domains in CSV format
csp-ws-domains-csv : " "                    # List of Web Socket domains in CSV format
csp-script-unsafe-inline : true              # Enable/Disable Unsafe Inline script execution via CSP
csp-script-unsafe-eval : false               # Enable/Disable Unsafe Eval script execution via CSP
csp-child-unsafe-inline : true               # Enable/Disable Child SRC Unsafe Inline script execution via CSP
csp-style-unsafe-inline : true               # Enable/Disable Style SRC Unsafe Inline script execution via CSP
csp-upgrade-insecure : true                  # Enable/Disable automagically upgrading HTTP to HTTPS for requests via CSP
csp-block-mixed-content : true               # Enable/Disable automatically blocking mixed HTTP and HTTPS content for requests via CSP
csp-report-uri : "/security/csp-report"      # Path for content security policy violation reports to get logged

Now, asking somebody to set up Caddy or Nginx or Apache to run and deploy the binary itself with CSP protections built‑in provide you as the Site Reliability Engineer the capability to assert through code and policy that there are two barriers to cross through. The first belonging to nginx and the second belonging to the binary. For me, I chose not to imply that burden of additional dependencies just to run the Apario Reader.

SSL

The next thing that comes up are SSL certificates. That brings us to this cool functionality:

This is the basic SSL functionality that most people are familiar with. This allows for you to rely on an SSL file, lets say you use /var/lib/ssl/acme_company_name/2026/mydomain.com/public.crt, (same path)/private.key and (same path)/ca-bundle.crt, you can directly reference them using the config.yml file.

auto-tls : false                    # rely on file based SSL certificate resources
tls-public-key : " "                # Path to the SSL certificate's public key. It expects any CA chain certificates to be concatenated at the end of this PEM formatted file.
tls-private-key : " "               # Path to the PEM formatted SSL certificate's private key.
tls-private-key-password : " "      # If the PEM private key is encrypted with a password, provide it here.
force-https : false                 # force-https when true will redirect any request into --unsecure-port to --secure-port using middleware

However, that requires you to have money up front, and being a project dedicated to the public service of all, how does putting any gate that requires money anything but antithetical to open‑source? Well, that is where Go comes in that Ruby could only dream of doing as well.

The reader has this capability built‑in where I will generate for you an SSL certificate on the fly that is self‑signed but most importantly configurable. What do I mean by this? Well, for instance, if you've done replication in MongoDB, then you'll know that you'll need a SAN IP issued SSL certificate in order to use TLS properly between your nodes in a large enterprise setting - at least - that's what the lived reality was in 2020 when I had the SaaS model of PhoenixVault 🐦‍🔥 in the wild.

In the self contained binary version of that same idea, I can generate an SSL certificate for you and include extra domains, IP addresses, and the ability to auto‑rotate the certificate without restarting the binary.

auto-tls : true                     # Create a self-signed certificate on the fly and use it for serving the application over SSL.
tls-life-min : 72                   # Lifespan of the auto generated self signed TLS certificate in minutes.
tls-expires-in : 8760               # Auto generated TLS/SSL certificates will automatically expire in hours.
tls-company : "ACME Inc."           # Auto generated TLS/SSL certificates are configured with the company name.
tls-domain-name : " "               # Auto generated TLS/SSL certificates will have this common name and run on this domain name.
tls-san-ip : " "                    # Auto generated TLS/SSL certificates will have this SAN IP address attached to it in addition to its common name.
tls-additional-domains : " "        # Auto generated TLS/SSL certificates will be issued with these additional domains (CSV formatted).

If you relied on the file‑based SSL certificates, the auto‑reloading capability is preserved through the manner in which the HTTPD server is spawned in the Go reader.

Once we have the SSL configured on the host, we can move onto the next part of the configuration that is to the advanced search. Now, I didn't just give it this name because I wanted to be cute with you.

Search

The reader was designed to let you configure search in a manner that offers you choice. Since the binary is designed to run as an appliance on the host, handling many tasks all in one, there are many semaphores used in the reader. In this case, setting the limiter to 30 means that 31 will see a waiting room - a manual refresh is required to see if you've been allowed in. I was inspired by this design limitation to build room.

concurrent-searches defaults to 30 and that's for a 64GB RAM server that is not $33/mo from OVH. If you're using something like a SYS‑3, then I'd set this number to 10.

The config.yml file allows you to customize your reader appliance.

search-concurrency-buffer : 369      # buffer channel size for search results ; default = 369
search-concurrency-limiter : 9       # concurrent keyword processing per search query ; default = 9
search-timeout-seconds : 30          # maximum seconds to spend on a search
concurrent-searches : 30             # maximum number of allowed concurrent searches before a waiting room appears
search-algorithm : "jaro_winkler"    # values are wagner_fisher, ukkonen, jaro, jaro_winkler, soundex, hamming ; default is jaro_winkler

There are 6 different search-algorithm choices to pick from, and each of them have different implications. soundex and hamming are quite interesting when applied to internationalized data sets.

In addition to the algorithm being used, you can expand it further with:

# 1 - jaro
search-threshold-jaro : 0.71               # 1.0 means exact match 0.0 means no match; default is 0.71
# 2 - jaro-winkler
search-threshold-jaro-winkler : 0.71        # using the JaroWinkler method, define the threshold that is tolerated; default is 0.71
search-jaro-winkler-boost-threshold : 0.7   # weight applied to common prefixes in matched strings comparing dictionary terms, page word data, and search query params
search-jaro-winkler-prefix-size : 3         # length of a jarrow weighted prefix string
# 3 - ukkonen
search-ukkonen-icost : 1                   # insert cost ; when adding a char to find a match ; increase the score by this number ; default = 1
search-ukkonen-scost : 2                   # substitution cost ; when replacing a char increase the score by this number ; default = 2
search-ukkonen-dcost : 1                   # delete cost ; when removing a char to find a match ; increase the score by this number ; default = 1
search-ukkonen-max-substi