Evidencing Compliance: Release Registers and Security Testing Per Release

The email nobody is ready for

There is a moment I have watched happen in a dozen small companies. An email arrives from a big customer's procurement team, or from the insurer at renewal, or from an auditor who has been asked by a bank to look at the accounting system. It contains a questionnaire. Question fourteen says something like: describe your secure development lifecycle and provide evidence of security testing for the current production version.

And the owner, who genuinely does take security seriously, who pays for good hosting and has a developer who knows what SQL injection is, realises they have nothing to attach. Not because nothing was done. Because nothing was written down in a form anyone else could check.

This article is about closing that gap. Not the law. I have written about the Cyber Resilience Act, NIS2 and the AI Act elsewhere and those pieces tell you what is required. This is about the boring machinery that lets a small firm say, with a straight face and a PDF, here is what we released, here is who tested it, here is the report, and here is the version that is running right now.

I run this for every client in my portal, from an energy cooperative with accounting logic that an auditor checks, to a SaaS product with a handful of enterprise customers, to shops that just want their insurer to renew without a fight. It is not complicated. It is just relentless, and it only works if it is built into how software gets released rather than bolted on when the questionnaire lands.

What evidence actually means

Let me define the word, because most of the confusion lives here.

"We take security seriously" is a claim. "We use best practices" is a claim. "Our developer is very careful" is a claim. A policy document that says releases are tested before deployment is also a claim, and a surprisingly weak one, because anyone can write a policy.

Evidence is a record that a third party can check without trusting you. It has a date. It names a specific version of a specific system. It names who did the thing and, ideally, points at the artefact they produced. It exists in a place where it could not have been quietly changed afterwards.

The difference sounds pedantic until you are on the other side of the table. An auditor is not paid to believe you. An insurer's claims handler, when something has gone wrong and there is money on the line, is actively looking for the gap between what you said at renewal and what you can show. A procurement officer at a corporate has a checklist and a legal department, and "trust us" does not tick a box.

So the whole programme below is designed around one question: for any production system, at any moment, can I produce a record that says which version is running, what changed in it, who tested it for security and when, and where the report is? If yes, almost every questionnaire I have ever seen becomes a matter of copying and pasting. If no, every questionnaire is a scramble.

The one habit everything hangs off

Every production release gets a version number and release notes. That is it. That is the habit.

I mean every release. Not major ones. Not the ones with a feature in them. The Tuesday afternoon deploy that fixed a typo gets a version, because next month when someone asks what changed between the version the pen tester looked at and the version that is running now, the answer has to be a list, not a shrug.

The versioning scheme matters less than the discipline, but pick one and write it down. For products that ship on a schedule, semantic versioning, the familiar 2.4.1 pattern, works fine. For applications that deploy continuously, several times a week, I have moved to calendar versions: the year, the month, and a counter, so 2026.11.3 is the third release in November 2026. It tells you at a glance how old the running version is, which is exactly the question an insurer asks.

The release notes are not marketing. They are three lists: what was added, what was changed, what was removed. Plus a timestamp and who approved it going out. In the portal I keep for clients this is a release register: one entry per production release per project, visible to the client, exportable as a PDF in English and German so it can be handed to an auditor who has never seen the system.

Everything else in this article attaches to that register. Security tests reference a version. Critical path approvals reference a version. Audit log queries are bounded by versions. Take the register away and the rest is a pile of documents with no spine.

Security testing is a property of a version, not a policy

Here is where the thinking shifts, and it is the shift that makes the questionnaire easy.

Most small firms, if they test security at all, think of it as an event. We had a pen test in 2024. There is a PDF somewhere. The problem is that the system tested in 2024 does not exist any more. It has been deployed over forty times since. The report is evidence that a system which no longer runs was once reasonably secure.

The programme treats security testing as something that is recorded against a version. Every release version must be security tested, before it goes live or promptly afterwards. The record says: version 2026.11.3 of the customer portal, assessed on this date, by this person or tool, this is the report, pass or fail.

What the assessment contains depends on the change and the stack. For the Rails applications I build, the baseline is:

Static analysis. Brakeman runs on every build and fails it on new high confidence warnings. It catches the boring, common, expensive things: SQL built from strings, unescaped output, mass assignment, redirects to user supplied URLs. It is free and it takes a minute. There is no excuse for not having it.

Dependency audit. Bundler audit against the advisory database, plus the equivalent for the JavaScript side. A shocking share of real breaches come through a library nobody has looked at in two years. This also runs on every build.

Injection and cross site scripting sweeps. Automated scanning of the running application, not just the code, against the common injection classes. This is where the automated tools earn their keep, because they try the thousand variations a human would not bother with.

A penetration test. A person, with a brief, trying to get in. For a major release or a new system this is the full exercise. For a routine release it might be a focused look at what changed. The point is that a human with intent looked at it, and wrote down what they found.

Load and resilience drills where relevant. If the system is something whose unavailability costs money, and NIS2 explicitly cares about availability, then someone has pushed it until it fell over and written down where that was.

The reports live in the product's documentation repository, in git, with the version they relate to. That location is deliberate. A git repository is a ledger. Every change is timestamped and attributed, and altering history is both hard and visible. When an auditor asks "how do I know this report was not written last week", the commit history is the answer.

And the portal links to them. Each client company has a cyber security section listing every production application, the version currently running, the most recent assessment, and a link to the report. When I release, I bump the current version in the portal. When the assessment is done, I record it. Two clicks, and the evidence chain is intact.

The traffic light

This is the part clients like most, and it is the simplest.

In the portal sidebar, every company has a traffic light. Red means no security assessment has ever been recorded for that company. Orange means the version currently running in production has not been successfully assessed, which usually means we released something and the assessment is still pending. Green means every production application's current version has a passing assessment on record.

That is the whole rule. It cannot be gamed by writing a nice policy. It cannot be green because someone feels good about security. It is green when the running versions have been tested and the records exist, and it goes orange the moment a release ships, until the assessment catches up.

The reason it works is that it turns compliance from an abstract obligation into a state that an owner can see in one glance and feel mildly annoyed about when it is wrong. Orange for three weeks is a conversation. Red is an emergency. Green is what you screenshot and attach to the questionnaire.

I want to be honest about the incentive it creates: it makes me release less casually. If every release turns the light orange until I have done the assessment, I batch small changes and I test properly. That is exactly the behaviour the Cyber Resilience Act is trying to produce with its vulnerability handling obligations, and it turns out a coloured dot does it more reliably than a directive.

Who is asking, and what they actually want

The reason I am confident this shape is right is that I have sat through enough of the questionnaires to notice they are all the same six questions in different clothes.

The auditor. Usually a financial auditor, or an IT auditor working for one, who needs assurance that a system producing financial figures does what it claims and has not been tampered with. They ask: what version is running, what controls exist over changes to it, who can change it, and is there a record of every change. They also ask about the logic itself, which is where verified critical paths come in, below. What they want to see is a change log, access controls, and evidence that tests ran before releases. A release register with test run records attached is close to the whole answer.

The insurer. Cyber insurance questionnaires have converged over the last few years, and they ask: do you use multi factor authentication, do you patch within a defined window, do you have backups tested for restore, do you log access and changes, have you had a penetration test in the last twelve months, and do you have an incident response plan. At renewal they want ticks. At claim time they want proof that the ticks were true on the day. The assessment record with dates and the audit log export are what make the ticks defensible.

The NIS2 supplier request. If one of your customers falls under NIS2, they are obliged to manage risk in their supply chain, which means they send you a letter. It asks whether you have a vulnerability handling process, how quickly you patch, whether you test releases, whether you can notify them of incidents within a set time, and whether you can provide evidence. The release register and the per version assessment are the evidence, and the notification obligation is a paragraph in the incident plan.

Enterprise procurement. The longest form, usually a spreadsheet with two hundred rows, of which a hundred and fifty do not apply to a company of your size and the rest ask about secure development, access control, encryption, logging, business continuity and subprocessors. The trick with procurement forms is that a "no, but" with evidence beats a "yes" without it. "We do not have a dedicated security team, but every release is assessed, here is the register and here are the last three reports" gets through. "Yes we take security seriously" gets a follow up.

The Datenschutzbehörde. Rarer, but if a GDPR complaint or a breach lands you in front of a supervisory authority, they ask what technical and organisational measures you had in place under Article 32 and whether you can demonstrate them. Demonstrate is the key word. Article 32 does not require perfection. It requires appropriate measures and the ability to show them. Same register, same reports.

The bank. Increasingly, lenders and payment providers ask a shortened version of the insurer's questions before extending terms.

Six audiences. One register, one set of assessment records, one audit log, one vault of signed documents. That is not a coincidence. It is because they are all, in the end, asking the same question: can you show me, or do I have to take your word for it.

Verified critical paths, for when the logic itself is the risk

Some systems have a piece of logic that, if it is wrong, is not a bug but a liability. The arithmetic that allocates an energy cooperative's costs across its members. The mapping that decides which legal documents a customer must accept for which product. A tariff calculation. A commission split.

For these, testing that the code does what the developer intended is not enough, because the question is whether what the developer intended is what the client, and the client's auditor, believe the rule to be. So the programme has a second mechanism: a verified critical path.

A verified critical path is a plain language description of how a piece of critical logic works, written for the client rather than for developers, alongside the executable tests that prove the system does that. The client reads it and approves it. At the moment of approval it is frozen: the content is hashed, the hash and the approver and the timestamp are recorded, and from then on any change to the underlying logic or the description is flagged. A flagged change is not a release blocker in itself, but it is visible, and it may require the client to approve again, and in some cases a third party to re approve.

The portal produces a summary page and a PDF, in English and German, that can be handed to an auditor. What the auditor gets is not "the developer says it works" but "here is the rule in words, here is the client's approval of that rule on this date, here are the tests that check the system follows it, and here is proof that nothing has changed since".

I have found this does something unexpected. Writing the plain language description forces a conversation between the client and me about what the rule actually is, and about one time in three, that conversation finds a case nobody had thought about. The compliance artefact turns out to be a design tool.

Test gated releases, and why the tests are for the client

I will not rehearse the whole argument for test driven and behaviour driven development here, I have done it in my Cucumber piece and the Rails testing guide. The compliance angle is narrower and it is this.

No production release goes out until the full test suite has run green. Staging is exempt, because staging is where you find out. Production is not. And the fact that the suite ran, when, on which commit, with what coverage, is recorded in the portal as a test run, scoped to the company and the application, with a link to the report.

The reason to record it rather than just do it is the same reason as everything else here. "We have tests" is a claim. "Version 2026.11.3 was released after a run of 1,412 scenarios at 84 percent coverage on this date, report attached" is evidence, and it happens to be exactly what the auditor's question about change controls is looking for.

The behaviour driven scenarios have a second role. They are written in plain language, so the client's tester, and I insist every client names one, can read them and confirm they describe what the business wants. That confirmation, on staging, before the release, is recorded too. Then the signatory, the person with authority to accept the release, accepts it. Request, tests, staging confirmation, acceptance, release, assessment. Every step has a name and a date attached.

That sounds heavy. In practice it is a few clicks per release and it is the difference between a change history that means something and one that does not.

Audit logging that an insurer can actually use

Most applications have some logging. Almost none of it is useful when it matters.

When an insurer is assessing a claim, or an auditor is chasing a discrepancy, the questions are specific: who changed this record, when, from where, and what was it before. Application logs full of request timings and stack traces do not answer that. What answers it is an audit trail: an append only record of who did what to which business object, kept separately from the data itself.

In the portal, every action that touches an accounting relevant record, a user account, a permission, a document or a release is written to an audit event with the actor, the target, the change and the timestamp. Logins and failed logins are recorded separately. The retention rule is written down: events are append only, nothing deletes them, and they are kept for a defined period beyond any regulatory minimum.

And they are exportable. This is the part that most systems miss. An insurer does not want to log into your admin. They want a file, filtered by date range and by system, that their own analyst can read. The export exists for that reason, and I would rather build it once than build it at two in the morning during an incident.

There is a GDPR wrinkle here worth naming. Audit logs contain personal data, the actor's identity at minimum, so they need a lawful basis and a retention policy like anything else. Legitimate interest in security and accountability covers it comfortably, but write it down in the records of processing. An audit log you cannot explain under GDPR is a liability of its own.

The vault: signed documents in one place

The last piece is the least technical and the one most often missing.

Every engagement produces documents that someone will eventually ask for: the signed contract, the data processing agreement, the insurance certificates, the offboarding clauses, the approval PDFs from the verified paths. In most small firms these are in email attachments, a Dropbox folder with an inconsistent naming scheme, and a solicitor's filing system.

The portal has a document vault per company. Signed contracts between the parties, insurance documents, the professional indemnity and public liability schedules, and the legal and release documents generated by the programme itself, all in one place, with the client able to see their own. Documents that are signed are marked as immutable and carry the signature metadata, so nobody can quietly upload a new version of a contract over the old one.

When procurement asks for your insurance certificate and your DPA, the answer is a link, not an afternoon. I wrote about the vendor side of that question in how data processors can get you fined, and the vault is the mirror image: being the vendor who can answer.

What this looks like on a normal Tuesday

Let me make it concrete, because a list of mechanisms sounds heavier than the thing is.

A client's tester writes a request in the portal: the invoice export should include the customer's VAT number. I write the behaviour scenario and the tests. The tests run red, I make them green, the suite runs on the branch, the change goes to staging. The tester checks it on staging and confirms. The signatory accepts. I release, the version bumps to 2026.11.4, the release notes get the one line, the light goes orange. Brakeman and the dependency audit ran on the build and passed. I run the injection sweep against production that evening, record the assessment, link the report, and the light is green again. If the change had touched the VAT calculation itself, the verified critical path would have flagged, and the signatory would have re approved it before release.

Elapsed overhead beyond the actual work: perhaps twenty minutes. And in that twenty minutes, every question from the six audiences above has already been answered for that release.

The Monday morning version

You do not need my portal to do this. You need the habit and a place to write things down. Here is the version any small firm can start next week.

Tag every production release in git. A tag with the version number and a message with the three lists. That is your release register, and git is a perfectly good ledger. If you deploy from a branch, tag the commit you deployed.

Keep a spreadsheet with one row per release. Version, date, what changed in a sentence, who approved it, test suite result, security assessment done yes or no, date of assessment, link to report. Five minutes per release.

Add Brakeman and a dependency audit to your build. Or the equivalents for your stack. Make the build fail on new findings. This is a morning's work once and it runs forever.

Get one penetration test done on the current version and put the report in a folder with the version number in the file name. Then, for every release after that, write in the spreadsheet whether the change was in scope of that test or whether it needs another look.

Start an audit log if your application does not have one. For Rails there are mature libraries that do it in an afternoon. Make it append only. Write down how long you keep it.

Put the signed documents in one folder with a fixed naming scheme and no deletions. Contract, DPA, insurance schedules, and a text file listing what is there and when it was signed.

Write a one page policy that describes the above. Not aspirations. What you actually do. The policy exists so that a questionnaire can reference it, and it is only worth anything if the spreadsheet proves you follow it.

That is the whole minimum. It will get a firm of five people through an insurer's renewal, a NIS2 supplier letter, and most procurement forms. It will not get you through a financial audit of critical logic, and it will start to hurt at about the point where you have three systems and two people releasing, because the spreadsheet stops being trusted the moment two people are editing it.

How it grows

The portal version is what the spreadsheet becomes when the spreadsheet is no longer enough. The transitions are predictable.

The spreadsheet becomes a register in a system when someone other than the person filling it in needs to trust it. Usually that is the first client who asks to see it.

The folder of reports becomes linked assessments when you have more than one production application and the question "is the running version tested" needs an answer per application rather than per company.

The traffic light appears when the owner wants to know the state without asking.

Verified critical paths appear the first time an auditor asks not whether the system was tested but whether the rule it implements is the right rule.

The audit log export appears the first time an insurer or an auditor asks for one, and you would rather it appeared before that.

The vault appears when the third procurement form in a year asks for the same four documents.

None of this needs to be built before you need it. All of it needs to be built before the email arrives, which is a different thing, and the honest answer for most firms is that the minimum version above buys you the time to build the rest.

What I would not bother with

A few things that show up in compliance conversations and that I would leave alone at this size.

Certification. ISO 27001 is a real thing and a genuine signal, and for a company under fifty people it is also a year of a manager's time and a five figure sum, for a badge that most of your customers will not require. Do the programme above first. If a customer makes certification a condition of a contract that pays for it, do it then, and you will find the programme has done most of the groundwork.

Tooling before habits. There is an entire industry selling compliance automation platforms to small firms. They are fine. They are also useless if nobody in the company tags releases and records tests, and if everybody does, a spreadsheet works for longer than the vendors would like you to believe.

Policies nobody reads. A forty page information security policy that was written for the questionnaire and describes a company that does not exist is worse than a one page policy that is true. Auditors can tell, and when something goes wrong, a policy you did not follow is evidence against you.

The point of all of it

I want to end on why I actually do this, because it is not primarily about questionnaires.

The programme makes releases better. Knowing that every release will be versioned, tested, assessed and recorded makes me, and the developers I work with, release deliberately. The verified paths find real bugs in the conversation, before they are code. The audit log has resolved more "who changed the price" disputes in a minute than any amount of email archaeology. The vault has saved a client a week of searching during a due diligence.

The compliance is a side effect of running software properly and writing it down. The questionnaires just happen to be the moment when writing it down pays for itself.

If you would like to see the portal with a login, or you want a second opinion on your own minimum version, get in touch. I run this for every client in my growth programme and I can tell you in an hour where the gaps are.