Reading the quality note: three handlers, three notes

Every code deploy on Zerolith now comes back with a note: a score out of 100 and a letter from A to E. It shows up in the “Quality note” column of your functions list and at the top of each function's page.

Before anything else, what it is not: it is not a security audit, it has never blocked a deploy, and it changes nothing about what runs. It is an indicator. It exists because the code deployed here is often written fast, sometimes written by an agent, and because the mistakes that cost the most are also the easiest ones to spot mechanically.

The best way to understand what it measures is to look at three versions of the same handler — a webhook receiver that checks whether an order exists — and the notes they actually get.

47/D — the webhook written fast

import hashlib
import os

import libsql_client

API_TOKEN = "sk_live_9f2c1a77b430"


def handler(request):
    payload = request.json()
    signature = hashlib.md5(request.body).hexdigest()

    client = libsql_client.create_client_sync(
        url=os.environ["DATABASE_URL"],
        auth_token=os.environ["DATABASE_AUTH_TOKEN"],
    )
    rows = []
    try:
        rows = client.execute(f"SELECT id FROM orders WHERE ref = '{payload['ref']}'").rows
    except Exception:
        pass

    return {"ok": True, "sig": signature, "found": bool(rows)}

This code works. It deploys, it answers, it returns the right JSON. The scan reports four problems:

Line	Severity	What is flagged
19	high	The SQL query is assembled inside an f-string
6	medium	An identifier looks like a hardcoded credential
11	low	MD5
18	low	An except that does nothing

100 − 30 − 15 − 4 − 4 = 47, which is a D.

Only the first one really matters. payload["ref"] comes from the request body, it is interpolated into the SQL string, and so anyone who can call this webhook chooses the end of the query. That is the kind of line you write without thinking about it on a Tuesday afternoon, and it is exactly what the note is there to surface.

MD5, by contrast, only builds a display fingerprint here. It is not a hole, and it costs four points: the scale is built so that a low-severity problem is visible without swamping the note.

81/B — nearly clean, but not quite

import hashlib

import requests


def handler(request):
    ref = request.query.get("ref", "")
    upstream = requests.get(
        "https://partner.example.com/orders/" + ref,
        timeout=5,
        verify=False,
    )
    etag = hashlib.md5(upstream.content).hexdigest()
    return {"order": upstream.json(), "etag": etag}

Two problems: verify=False (medium, 15 points) and MD5 (low, 4 points). 100 − 15 − 4 = 81, a B.

This is the most interesting of the three, because it is the note you get without doing anything outrageous. verify=False almost always arrives the same way: an awkward certificate in development, a line added to unblock the afternoon, and nobody took it out. The note is not saying this code is bad; it is saying there is still a line in it you no longer remember.

100/A — the same function, fixed

import hashlib
import hmac
import logging
import os

import libsql_client

logger = logging.getLogger("orders")


def handler(request):
    expected = hmac.new(
        os.environ["WEBHOOK_SECRET"].encode(), request.body, hashlib.sha256
    ).hexdigest()
    if not hmac.compare_digest(expected, request.headers.get("X-Signature", "")):
        return 401, {"error": "bad signature"}

    payload = request.json()
    client = libsql_client.create_client_sync(
        url=os.environ["DATABASE_URL"],
        auth_token=os.environ["DATABASE_AUTH_TOKEN"],
    )
    try:
        rows = client.execute("SELECT id FROM orders WHERE ref = ?", [payload["ref"]]).rows
    except libsql_client.LibsqlError:
        logger.exception("lookup failed")
        return 503, {"error": "storage unavailable"}
    finally:
        client.close()

    return {"ok": True, "found": bool(rows)}

Four changes, none of them hard. The value is passed as a bound parameter instead of into the string. The credential becomes an environment variable of kind secret — it travels to the cluster and is never readable back, not through the dashboard and not through the API. SHA-256 replaces MD5. And the except catches a specific error, logs it, and returns an honest status.

That last one is not cosmetic. An except Exception: pass around a database call turns a real failure into {"ok": true}: your caller sees a success where there wasn't one.

How the number is worked out

Start at 100 and subtract a weight per problem, according to its severity:

Severity	Weight	Examples
high	30	eval, os.system, shell=True, pickle.loads, SQL built by concatenation
medium	15	hardcoded credential, random for a token, debug=True
low	4	MD5/SHA-1, swallowed exception, tempfile.mktemp

The letter follows the number: A from 90, B from 75, C from 55, D from 30, E below that.

Two choices are worth explaining, because they are counter-intuitive.

The score does not divide by file size. An earlier draft did, and it looks fairer: a mistake in two thousand lines would matter less than a mistake in twenty. Except that an injection is an injection whatever the length of the file, and dividing by size hands everyone a way to improve their note by padding their code. Density is a good metric for triaging an existing codebase; it is the wrong metric for a handler you are being told is dangerous.

A high-severity problem can never be shown as an A or a B. With today's weights the arithmetic takes care of that on its own — a single high-severity finding drops you to 70, so to a C — but the rule is written separately, so that it survives a future re-tuning of the weights.

The same code, the other language

The first example, translated to Node with the libSQL client and crypto, comes back with exactly the same note: 47/D, the same four problems at the same severities.

That is not a coincidence: it is why Python and JavaScript are analysed by one engine and one ruleset, rather than by a specialised tool on each side. Two tools mean two severity scales — and a note that is not comparable across languages is not a note.

What an A does not say

An A means “no rule fired”. It does not mean “this code is safe”.

The analysis is static: it reads your source without running it, knows nothing about your data, and does not follow a value across ten functions. So it misses things — a forgotten authorisation check, wrong business logic, a credential read from a file — and it sometimes flags things wrongly, when a variable is called token without being one.

Three practical details to finish:

• The note always describes the currently deployed code. Redeploy and a fresh scan is queued; if you redeploy while a scan is running, the old code's scan is dropped rather than pinned onto the new code.
• Nothing backfills. A function deployed before the feature arrived has no note, and its cell stays empty — no dash, no E — until its next deploy.
• Your source is never kept. The scanner re-reads it at scan time and retains only the line a problem quotes.

The full details are in the documentation, and the rest of the platform's security model — per-tenant isolation, write-only secrets, authentication at the edge — is on the security page.