Your auth tests are testing your test double

Last time I wrote about five Keycloak misconfigurations that never produce an error — a missing audience mapper, an issuer string that doesn’t quite match, roles nested somewhere .NET doesn’t look. All silent. All expensive.

The obvious follow-up is the question I didn’t answer: how would you ever catch one? I have a Keycloak lab with a .NET API and a test suite sitting over it, so I went and checked whether my own tests would have noticed any of them. For a while the honest answer was no — and the reason generalises well past my code.

Why? Because the standard way to test authorization in ASP.NET Core is to replace the authentication handler with a fake one. You register a test scheme, stamp a ClaimsPrincipal onto every request, and assert that your endpoints admit the right people. It’s fast, it needs no key material, and it genuinely does test your policies. It also deletes the entire layer where Keycloak integrations actually go wrong.

Every one of those five traps lives in the layer the fake removes. Which means a suite built that way stays completely green against a configuration that would reject every request in production.

The shortcut everybody takes

If you have tested a protected endpoint in .NET, you have probably written this, or pasted it:

public class TestAuthHandler(
    IOptionsMonitor<AuthenticationSchemeOptions> options,
    ILoggerFactory logger,
    UrlEncoder encoder)
    : AuthenticationHandler<AuthenticationSchemeOptions>(options, logger, encoder)
{
    protected override Task<AuthenticateResult> HandleAuthenticateAsync()
    {
        var claims = new[]
        {
            new Claim(ClaimTypes.Name, "alice"),
            new Claim(ClaimTypes.Role, "doc.editor"),
        };
        var principal = new ClaimsPrincipal(new ClaimsIdentity(claims, "Test"));
        return Task.FromResult(AuthenticateResult.Success(
            new AuthenticationTicket(principal, "Test")));
    }
}

Registered over the top of the real thing in your factory:

services.AddAuthentication("Test")
    .AddScheme<AuthenticationSchemeOptions, TestAuthHandler>("Test", _ => { });

I want to be fair to this pattern, because it is not stupid. It is quick, it has no key material to manage, it runs offline, and it will correctly tell you that a doc.reader cannot reach an endpoint marked RequireRole("doc.admin"). If what you are testing is your policy wiring, it does the job.

The trouble is what it implies about everything else.

What it quietly deletes

HandleAuthenticateAsync returns Success without ever looking at the Authorization header. There is no token. There is nothing to validate. So none of this runs:

options.TokenValidationParameters = new TokenValidationParameters
{
    ValidateIssuer = true,
    ValidIssuer = authority,
    ValidateAudience = true,
    ValidAudience = audience,
    ValidateLifetime = true,
    ValidateIssuerSigningKey = true,
    ClockSkew = TimeSpan.FromSeconds(30),
    ValidAlgorithms = [SecurityAlgorithms.RsaSha256, SecurityAlgorithms.RsaSsaPssSha256],
    NameClaimType = "preferred_username",
    RoleClaimType = ClaimTypes.Role,
};

Nor does MapInboundClaims = false, nor the IClaimsTransformation that flattens Keycloak’s nested realm_access and resource_access into the role claims .NET actually reads.

Now go back through that list with the five silent traps in mind. The audience mapper is ValidateAudience. The issuer mismatch is ValidIssuer. The nested roles are the claims transformation. Three of the five are checks the fake handler skipped, and the other two are things it never reaches. The tests pass because the fake stamped doc.editor on the request itself. Of course they pass. You wrote the answer on the exam paper.

flowchart LR
    R["<b>Request</b><br/>Authorization: Bearer …"]

    subgraph F["With a fake handler"]
        direction TB
        F1["<i>token ignored</i>"] --> F2["stamp a<br/>ClaimsPrincipal"] --> F3["policy check"] --> F4["endpoint"]
    end

    subgraph V["With the real JwtBearer handler"]
        direction TB
        V1["verify signature"] --> V2["ValidIssuer"] --> V3["ValidAudience"] --> V4["ValidateLifetime"] --> V5["ValidAlgorithms"] --> V6["claims<br/>transformation"] --> V7["policy check"] --> V8["endpoint"]
    end

    R --> F
    R --> V

    style F1 fill:#fde2e2,stroke:#c33
    style F2 fill:#fde2e2,stroke:#c33

The two lanes agree on the last two boxes and nowhere else. Everything the fake lane skips is configuration that fails silently.

Swap the key, not the handler

The fix is smaller than the problem. You don’t need a fake handler; you need a key you control. Keep the real JwtBearer handler, the real transformation, the real policies, and swap only where the signing key comes from:

builder.ConfigureTestServices(services =>
{
    services.Configure<JwtBearerOptions>(JwtBearerDefaults.AuthenticationScheme, options =>
    {
        // No network: hand the handler our public key directly instead of letting
        // it discover one. Keeps the unit suite fast and offline.
        options.Authority = null!;
        options.MetadataAddress = null!;
        options.RequireHttpsMetadata = false;
        options.Configuration = new OpenIdConnectConfiguration();
        options.TokenValidationParameters.IssuerSigningKey = Tokens.SigningKey;
        options.TokenValidationParameters.IssuerSigningKeys = [Tokens.SigningKey];
    });
});

Two details in there are load-bearing.

The first is that it mutates TokenValidationParameters rather than assigning a new one. Every production setting — the audience, the lifetime check, the 30-second ClockSkew, the algorithm allowlist — survives into the test. Replace the object instead and you have quietly rebuilt the fake handler with extra steps.

The second is options.Configuration = new OpenIdConnectConfiguration(). Setting Authority to null is not enough on its own; the handler will still try to build a ConfigurationManager and fetch metadata. Handing it a pre-built, empty configuration short-circuits that, and the suite never opens a socket. No JWKS endpoint, no discovery document, no container.

Now you get to be the attacker

Here is the part I did not expect. Once the test process owns the signing key, a hostile identity provider is easier to stand up than a friendly one.

private readonly RSA _rsa = RSA.Create(2048);

public TestTokenIssuer()
{
    SigningKey = new RsaSecurityKey(_rsa) { KeyId = "test-key-1" };
}

That is the whole issuer. Twenty-odd lines further on it can mint a token with any issuer, any audience, any expiry, signed with any key you hand it. A real Keycloak will never issue you a token for the wrong realm — that’s the point of it — so if you want to prove your API rejects one, you have to forge it yourself.

The tokens have to be shaped right, though, and this is where a naive fake falls down. Keycloak emits roles as nested JSON objects, not repeated flat claims:

// Keycloak emits these as nested JSON objects, not as repeated flat claims.
// Reproducing that shape exactly is the whole point — it is what
// KeycloakClaimsTransformation has to cope with.
claims.Add(new Claim(
    "resource_access",
    JsonSerializer.Serialize(new Dictionary<string, object>
    {
        [DefaultAudience] = new { roles = apiRoles },
    }),
    JsonClaimValueTypes.Json));

If your test tokens carry flat ClaimTypes.Role claims, your transformation is never exercised — and the transformation is the piece most likely to be wrong.

Five tokens Keycloak would never issue

With that in place, the security suite stops asserting configuration and starts asserting behaviour. The canonical one:

[Fact]
public async Task Rejects_an_unsigned_alg_none_token()
{
    // The canonical JWT attack: strip the signature and set alg to "none". A naive
    // decoder that trusts the header would grant this forged doc.admin token.
    var response = await factory
        .CreateClientWithToken(TestTokenIssuer.CreateUnsignedToken())
        .GetAsync("/documents");

    Assert.Equal(HttpStatusCode.Unauthorized, response.StatusCode);
}

No library will emit that token for you, so it is built by hand — header, payload, empty signature, three parts and two dots:

var header = B64("""{"alg":"none","typ":"JWT"}""");
var payload = B64(/* … sub, iss, aud, exp, and resource_access … */);

return $"{header}.{payload}.";

Note what the forged payload asks for: doc.admin, the highest role in the realm. A forgery that requests nothing proves nothing — if it were accepted you might never see a difference. Ask for the keys to everything, and a passing test means the rejection was real rather than incidental.

The other four vary exactly one axis each:

• An unknown signing key. An attacker who mints their own keypair must not be able to impersonate the IdP.
• A different audience. A token legitimately issued for another API in the same realm must not be replayable against this one. This is precisely what the audience mapper exists to enable.
• A different realm. Multi-realm deployments make this a live risk: same server, same signing infrastructure, different issuer.
• An expired token, well outside the 30-second ClockSkew.

Five tests, each one a real attack or a real misconfiguration, and every one of them passes through the production validation path.

Two details that cost me an evening

Expired tokens are harder to build than they look. A JWT carries nbf as well as exp, and nbf must precede exp or the token is malformed rather than merely expired — which means your “expired token” test starts failing for the wrong reason:

// notBefore must precede expiry, including when a test deliberately asks for an
// ALREADY-EXPIRED token. Anchoring it to `now` would make such a token
// unconstructable rather than merely invalid; anchoring it to expiry would push it
// into the future for ordinary tokens, making them not-yet-valid. Take the earlier.
var notBefore = expiry < DateTime.UtcNow
    ? expiry.AddMinutes(-1)
    : DateTime.UtcNow.AddMinutes(-1);

ValidIssuer has to be stated, not inherited. I set the test issuer through configuration, as you would, and every single test returned 401. Under minimal hosting the application’s own appsettings.json is applied after the test host’s in-memory source, so the override loses and the production issuer stays in place. Set it directly on TokenValidationParameters and the problem disappears. That one cost me an hour and now carries a comment so it never costs anyone else one.

What this still doesn’t prove

Worth being straight about the limits, because they are real.

These tests prove the API rejects what it should. They do not prove Keycloak emits what the API expects. A realm with no audience mapper at all would still pass every test above — the test issuer happily writes aud: docvault-api, because I told it to. The suite validates one side of a contract using tokens I wrote myself.

So the other half is a different job. In my lab CI boots a real Keycloak in Docker, applies the Terraform realm to it, then asserts the two things that fail silently — that acr_values_supported really contains silver, and that a client_credentials token really carries aud: docvault-api. Neither can be proven by a forged token, and neither shows up as an error anywhere else.

I’ll also admit the alg:none test is redundant. Modern Microsoft.IdentityModel rejects unsigned tokens regardless of what you configure. I keep it because it documents the expectation and because it will fail loudly the day somebody relaxes ValidAlgorithms to “fix” an integration problem. It is a regression guard, not a discovery.

Try it, then break it

git clone https://github.com/MagnusJohansson/keycloak-poc
cd keycloak-poc
make test

No Docker, no network, no cloud account — 74 tests, and the assertions themselves run in under half a second.

Then do the thing that actually builds the intuition. Open AuthenticationExtensions.cs, set ValidateAudience = false, and run the suite again:

Failed DocVault.Api.Tests.Security.TokenValidationTests
       .Rejects_a_token_minted_for_a_different_audience
Failed: 1, Passed: 28

One test, and its name is the production misconfiguration you just introduced. Now run that same experiment against a suite built on a fake authentication handler, and watch it stay green.

That difference is the whole argument.

Earlier in this series: why your realm belongs in Terraform, the misconfigurations that never produce an error, OAuth beyond the SPA, and Keycloak on Azure Container Apps.