INFRASTRUCTURE

This diagram shows the current changelog.com setup:

%% https://fontawesome.com/search
graph TD
    classDef link stroke:#59b287,stroke-width:3px;

    %% Secrets
    secrets(( fa:fa-key 1Password )):::link
    click secrets "https://changelog.1password.com/"
    secrets -.-> |secrets| app
    secrets -.-> |secrets| repo

    %% Code & assets
    subgraph GitHub
        repo{{ fab:fa-github thechangelog/changelog.com }}:::link
        click repo "https://github.com/thechangelog/changelog.com"

        cicd[/ fa:fa-circle-check GitHub Action - Ship It \]:::link
        click cicd "https://github.com/thechangelog/changelog.com/actions/workflows/ship_it.yml"

        automation[\ fab:fa-golang Dagger Go SDK /]:::link
        click automation "https://github.com/thechangelog/changelog.com/blob/master/magefiles/magefiles.go"

        registry(( fab:fa-github ghcr.io )):::link
        click registry "https://github.com/orgs/thechangelog/packages"

        chat(( fab:fa-z Zulip )):::link
        click chat "https://changelog.zulipchat.com/#narrow/channel/455097-kaizen/topic/Code.20deploys"

        repo -.-> |.github/workflows/ship_it.yml| cicd
        cicd --> |magefiles/magefiles.go| automation

        cicd --> |success #kaizen code| chat
    end

    repo -.- |fly.io/changelog-2025-05-05| app

    registry ---> |ghcr.io/changelog/changelog-prod| app
    runner --> |flyctl deploy| app

    repo -.- |fly.io/pghero-2024-03-27| pghero

    %% PaaS - https://fly.io/dashboard/changelog
    subgraph Fly.io
        proxy{fa:fa-globe Proxy}
        proxy ==> |https| app

        pipedream[ fa:fa-bolt changelog.com cdn.changelog.com ]:::link
        click pipedream "https://changelog.com"

        app(( fab:fa-phoenix-framework IAD & EWR changelog-2025-05-05.fly.dev )):::link
        style app fill:#488969;
        click app "https://fly.io/apps/changelog-2025-05-05"

        pghero([ fa:fa-gem PgHero 2024-03-27 ]):::link
        click pghero "https://fly.io/apps/pghero-2024-03-27"
    end

    app <==> |Postgres| dbrw
    pghero --> dbrw

    subgraph Neon.tech
        dbrw([ fa:fa-database main branch primary ]):::link
        click dbrw "https://console.neon.tech/app/projects/orange-sound-86604986/branches/br-wandering-smoke-78468159"

        dbro1([ fa:fa-database main branch replica ])
        dbrw -.-> |replicate| dbro1
    end

    subgraph Namespace.so
        runner([ fa:fa-person-running GitHub Runner ]):::link
        click runner "https://cloud.namespace.so/9s8hfvousnlae/ghrunners"

        automation --> |runs-on: namespace-profile-changelog| runner
        runner --> |ghcr.io/changelog/changelog-runtime| registry
        runner --> |ghcr.io/changelog/changelog-prod| registry

    end

    %% Search
    search(( fa:fa-magnifying-glass Typesense ))
    app -..-> |search| search

    %% Exceptions
    exceptions(( fa:fa-car-crash Sentry )):::link
    click exceptions "https://sentry.io/organizations/changelog-media/issues/?project=5668962"
    app -..-> |exceptions| exceptions

    subgraph AWS.S3
        logs[ fab:fa-aws changelog-logs ]
    end
    pipedream -..-> |logs| logs

    %% Object storage
    proxy ==> |https| pipedream ==> app
    subgraph Cloudflare.R2
        assets[ fab:fa-cloudflare changelog-assets changelog.place ]
        feeds[ fab:fa-cloudflare changelog-feeds feeds.changelog.place ]
    end
    pipedream ===> |https| assets & feeds

    %% Monitoring
    subgraph BetterStack
        status[ fa:fa-layer-group status.changelog.com ]:::link
        click status "https://status.changelog.com"

        monitoring(( fa:fa-table-tennis Uptime )):::link
        click monitoring "https://uptime.betterstack.com/team/133302/monitors"
        monitoring -...-> |monitors| pipedream
        monitoring -.-> |monitors| proxy
        monitoring -.-> |monitors| status
    end

    %% Observability
    observability(( fa:fa-bug Honeycomb )):::link
    click observability "https://ui.honeycomb.io/changelog/datasets/changelog_opentelemetry/home"
    app -.-> |traces| observability
    pipedream -.-> |logs| observability

[!TIP] Continue live editing this Mermaid diagram

Let's dig into how all the above pieces fit together.

A three-tier monolith

TL;DR:

Front-end
- Fly.io Proxy
- Cloudflare R2
Application
- Elixir / Phoenix
- Typesense search
Database
- PostgreSQL (Neon.tech)

changelog.com is a monolithic Elixir application built with the Phoenix web framework. It uses PostgreSQL for persistence & Node.js to digest & compile static assets (CSS & JS).

Static assets, including all our mp3 episodes, are stored on Cloudflare R2. They are served via Pipedream, specifically https://cdn.changelog.com.

Pipedream (cdn.changelog.com)
↓
Cloudflare R2 (changelog.place)

The production instance of our application is running on Fly.io, as 2 instances spread across 2 regions: IAD & EWR. All https://changelog.com requests are served via Pipedream, which is Varnish HTTP Cache behind Fly.io Proxy.

Fly.io Proxy
↓
Pipedream (changelog.com)
↓
Application (changelog-2025-05-05.fly.dev)

The production database - PostgreSQL - is running on Neon.tech. It is replicated setup, with one leader (RW) & one replica (RO). We are currently not using the replica, and since Neon.tech scales down to 0, this doesn't cost anything.

Application (changelog-2025-05-05.fly.dev)
↓
PostgreSQL Leader (RW)
↓
PostgreSQL Replica (RO)

Production deploys

Each commit made against our primary branch gets deployed straight into production. The "Ship It!" GitHub Actions workflow is responsible for this. From the workflow jobs perspective, it is fairly standard:

1/2. CI/CD
- Uses Dagger Go SDK so that it works exactly the same locally as it does in GitHub Actions
- Uses a Namespace.so GitHub Runner so that caching is reliable & persistent across workflow runs
- A successful run publishes a container image to https://ghcr.io/thechangelog/changelog-runtime & https://ghcr.io/thechangelog/changelog-prod
- Deploys to Fly.io
2/2. Notify
- Notifies #kaizen / Code deploys channel in changelog.zulipchat.com if CI/CD succeeds

Secrets

All our secrets are stored in 1Password, in the changelog vault. We are declaring a single secret in Fly.io, OP_SERVICE_ACCOUNT_TOKEN, and then loading all other secrets into memory part of app boot via op & env.op.

In GitHub Actions secrets, we are still pasting them manually.

[!NOTE] We should use op here too.

Metrics & observability

Since our application & database are running on Fly.io, we benefit from free infrastructure metrics: https://fly-metrics.net

All logs from Pipedream are streamed into Honeycomb.io. This allows us to ask unknown questions about how various HTTP clients interact with our content. It also helps us explore how Pipedream interacts with the app.

We also send app traces via OpenTelemetry to Honeycomb.io.

App errors - e.g. Plug.Conn.InvalidQueryError - show up in Sentry.io.

BetterStack.com monitors our public HTTPS endpoints & alerts us when they become unhealthy.

Search

We use Typesense for search. It's near-instant & it just works.

What is missing?

The above is what we have so far. While we like to keep things simple, our setup is a constant work in progress. We keep making small improvements all the time, and we talk about them every few months in the context of our Ship It! Kaizen episodes.

For example, this diagram and document were created in the context of 🎧 Kaizen 8: 24 improvements & a lot more. If you would prefer to stay in reading mode, check out GitHub discussion #433.

If anything on this page is missing, or could be clearer, please open an issue. Thank you very much!

How to create a new app instance?

Start by creating a new app, e.g. flyctl apps create changelog-2025-05-05 --org changelog
Copy the existing app instance config, e.g. cp -r fly.io/changelog-{2024-01-12,2025-05-05}
Run all following commands in the app directory, e.g. cd fly.io/changelog-2025-05-05
Update the app name in e.g. fly.toml to match the newly created app
Set APP_INSTANCE=experimental env var in fly.toml

Otherwise Oban and other production components will treat this app as current production

Set the one secret required by the app to be able to access all other secrets

 flyctl secrets set --stage \
     OP_SERVICE_ACCOUNT_TOKEN="$(op read op://changelog/op/credential --account changelog.1password.com --cache)"

Deploy the latest app image from https://github.com/thechangelog/changelog.com/pkgs/container/changelog-prod
```
 flyctl deploy --ha=false --image=ghcr.io/thechangelog/changelog-prod:<LATEST_IMAGE_SHA>
```

How to promote a new app instance to production?

Update APP_PROD_INSTANCE in .envrc to match the newly created app name, e.g. changelog-2025-05-05
```
 direnv allow
 env | rg APP_PROD_INSTANCE
```
Ensure that the app is scaled across multiple regions & is resilient to a single region failure:
```
 just prod-region-resilient
```
Set APP_INSTANCE=production env var in fly.toml

Remove any env variables that should not be there (e.g. URL_HOST, STATIC_URL_HOST, etc.)

Update the app origin in the CDN with the new app instance URL, e.g. https://changelog-2025-05-05.fly.dev
Update the previous app instance reference everywhere in this repository - starting with the diagrams in this file.
Update APP_PROD_INSTANCE in GitHub Actions

How to branch the production db instance?

See Enable changelog.com devs to create prod db forks with a single command.