Lesson 3: Middleware Pipeline and Request Processing in ASP.NET Core 🔄

Introduction

Welcome to Lesson 3! After mastering async/await and dependency injection in the previous lessons, you're ready to explore one of the most powerful architectural features of ASP.NET Core: the middleware pipeline. Understanding how requests flow through your application is crucial for debugging, performance optimization, and building robust web APIs. 💻

In a typical .NET developer interview with 4+ years of experience, you'll likely face questions about:

• How the middleware pipeline processes requests
• Creating custom middleware components
• The order of middleware execution and why it matters
• Performance implications of middleware design
• Short-circuiting the pipeline

🤔 Did you know? The middleware pipeline in ASP.NET Core was heavily inspired by Node.js's Express framework and OWIN (Open Web Interface for .NET), but Microsoft refined it to be more performant and type-safe!

Core Concepts

What is Middleware? 🧩

Middleware is software that's assembled into an application pipeline to handle requests and responses. Each middleware component:

• Can perform operations before and after the next component in the pipeline
• Can choose whether to pass the request to the next component
• Has access to both incoming requests and outgoing responses

🌍 Real-world analogy: Think of middleware like airport security checkpoints. Your luggage (HTTP request) passes through multiple stations: check-in → security scan → customs → gate. Each station can inspect, modify, or even reject your luggage before passing it to the next station. The return journey (response) goes back through some of these stations in reverse!

The Request Pipeline Architecture 🏗️

┌─────────────────────────────────────────────────────────────┐
│                     HTTP Request ⬇️                          │
├─────────────────────────────────────────────────────────────┤
│  ┌──────────────────────────────────────────────────────┐   │
│  │ Middleware 1 (e.g., Exception Handler)              │   │
│  │  ├─ Before next()                                   │   │
│  │  │                                                   │   │
│  │  │  ┌────────────────────────────────────────────┐  │   │
│  │  │  │ Middleware 2 (e.g., Authentication)        │  │   │
│  │  │  │  ├─ Before next()                          │  │   │
│  │  │  │  │                                          │  │   │
│  │  │  │  │  ┌──────────────────────────────────┐   │  │   │
│  │  │  │  │  │ Middleware 3 (e.g., Authorization)  │  │   │
│  │  │  │  │  │  ├─ Before next()                │   │  │   │
│  │  │  │  │  │  │                                │   │  │   │
│  │  │  │  │  │  │  ┌─────────────────────────┐   │   │  │   │
│  │  │  │  │  │  │  │   Endpoint/Controller   │   │   │  │   │
│  │  │  │  │  │  │  │   (generates response)  │   │   │  │   │
│  │  │  │  │  │  │  └─────────────────────────┘   │   │  │   │
│  │  │  │  │  │  │                                │   │  │   │
│  │  │  │  │  │  └─ After next()                 │   │  │   │
│  │  │  │  │  └──────────────────────────────────┘   │  │   │
│  │  │  │  │                                          │  │   │
│  │  │  │  └─ After next()                           │  │   │
│  │  │  └────────────────────────────────────────────┘  │   │
│  │  │                                                   │   │
│  │  └─ After next()                                    │   │
│  └──────────────────────────────────────────────────────┘   │
├─────────────────────────────────────────────────────────────┤
│                     HTTP Response ⬆️                         │
└─────────────────────────────────────────────────────────────┘

Middleware Registration in Program.cs 📋

In ASP.NET Core 6+, middleware is configured in Program.cs using the WebApplicationBuilder pattern:

var builder = WebApplication.CreateBuilder(args);
var app = builder.Build();

// Middleware registered here - ORDER MATTERS!
app.UseExceptionHandler("/error");
app.UseHttpsRedirection();
app.UseStaticFiles();
app.UseRouting();
app.UseAuthentication();
app.UseAuthorization();
app.MapControllers();

app.Run();

⚠️ Critical concept: The order of middleware registration determines the order of execution. This is one of the most common sources of bugs in ASP.NET Core applications!

The Three Ways to Create Middleware 🔧

1. Inline Middleware (Use/Run)

Use() - Adds middleware that calls the next component:

app.Use(async (context, next) =>
{
    // Before logic
    await next(); // Call next middleware
    // After logic
});

Run() - Adds terminal middleware (doesn't call next):

app.Run(async context =>
{
    await context.Response.WriteAsync("Request ends here");
});

2. Convention-based Middleware

A class with a specific structure:

public class RequestTimingMiddleware
{
    private readonly RequestDelegate _next;
    
    public RequestTimingMiddleware(RequestDelegate next)
    {
        _next = next;
    }
    
    public async Task InvokeAsync(HttpContext context)
    {
        var stopwatch = Stopwatch.StartNew();
        await _next(context);
        stopwatch.Stop();
        
        context.Response.Headers.Add("X-Response-Time", 
            $"{stopwatch.ElapsedMilliseconds}ms");
    }
}

3. Factory-based Middleware (IMiddleware)

Implements the IMiddleware interface:

public class ApiKeyMiddleware : IMiddleware
{
    private readonly IConfiguration _configuration;
    
    public ApiKeyMiddleware(IConfiguration configuration)
    {
        _configuration = configuration;
    }
    
    public async Task InvokeAsync(HttpContext context, RequestDelegate next)
    {
        // Implementation here
    }
}

💡 Tip: Use factory-based middleware when you need to inject scoped services (like DbContext). Convention-based middleware is instantiated once per application lifetime, so it can only inject singleton services in the constructor!

HttpContext Deep Dive 🔍

The HttpContext object is your gateway to everything about the current request:

+------------------+
|   HttpContext    |
+------------------+
        |
        ├─ Request (HttpRequest)
        |    ├─ Method (GET, POST, etc.)
        |    ├─ Path
        |    ├─ QueryString
        |    ├─ Headers
        |    ├─ Body
        |    └─ Cookies
        |
        ├─ Response (HttpResponse)
        |    ├─ StatusCode
        |    ├─ Headers
        |    ├─ Body
        |    └─ ContentType
        |
        ├─ User (ClaimsPrincipal)
        ├─ Session
        ├─ Items (per-request cache)
        ├─ RequestServices (DI container)
        ├─ Connection
        └─ RequestAborted (CancellationToken)

Detailed Examples

Example 1: Request Logging Middleware ✏️

Let's create comprehensive logging middleware that captures request details:

public class RequestLoggingMiddleware
{
    private readonly RequestDelegate _next;
    private readonly ILogger<RequestLoggingMiddleware> _logger;

    public RequestLoggingMiddleware(
        RequestDelegate next,
        ILogger<RequestLoggingMiddleware> logger)
    {
        _next = next;
        _logger = logger;
    }

    public async Task InvokeAsync(HttpContext context)
    {
        // Generate unique ID for this request
        var requestId = Guid.NewGuid().ToString();
        context.Items["RequestId"] = requestId;

        // Log request details
        _logger.LogInformation(
            "Request {RequestId}: {Method} {Path} started at {Time}",
            requestId,
            context.Request.Method,
            context.Request.Path,
            DateTime.UtcNow);

        // Important: enable buffering to read body multiple times
        context.Request.EnableBuffering();

        var stopwatch = Stopwatch.StartNew();
        
        try
        {
            await _next(context);
            stopwatch.Stop();

            _logger.LogInformation(
                "Request {RequestId}: completed with {StatusCode} in {ElapsedMs}ms",
                requestId,
                context.Response.StatusCode,
                stopwatch.ElapsedMilliseconds);
        }
        catch (Exception ex)
        {
            stopwatch.Stop();
            _logger.LogError(ex,
                "Request {RequestId}: failed after {ElapsedMs}ms",
                requestId,
                stopwatch.ElapsedMilliseconds);
            throw; // Re-throw to let exception handler deal with it
        }
    }
}

// Extension method for easy registration
public static class RequestLoggingMiddlewareExtensions
{
    public static IApplicationBuilder UseRequestLogging(
        this IApplicationBuilder builder)
    {
        return builder.UseMiddleware<RequestLoggingMiddleware>();
    }
}

// Usage in Program.cs
app.UseRequestLogging();

Key learning points:

• Middleware can inject singleton services (ILogger) via constructor
• context.Items is a per-request dictionary for passing data between middleware
• Always use try-catch in middleware to ensure logging even on failures
• Extension methods make middleware registration cleaner

Example 2: API Key Authentication Middleware 🔐

A common interview question: "How would you implement custom authentication?"

public class ApiKeyAuthenticationMiddleware
{
    private readonly RequestDelegate _next;
    private const string API_KEY_HEADER = "X-API-Key";

    public ApiKeyAuthenticationMiddleware(RequestDelegate next)
    {
        _next = next;
    }

    public async Task InvokeAsync(
        HttpContext context,
        IApiKeyValidator validator) // Scoped service injected here!
    {
        // Skip authentication for health check endpoints
        if (context.Request.Path.StartsWithSegments("/health"))
        {
            await _next(context);
            return;
        }

        // Check if API key exists in header
        if (!context.Request.Headers.TryGetValue(
            API_KEY_HEADER, out var apiKeyValue))
        {
            context.Response.StatusCode = 401;
            await context.Response.WriteAsJsonAsync(new
            {
                error = "API Key is missing",
                header = API_KEY_HEADER
            });
            return; // Short-circuit the pipeline!
        }

        // Validate the API key
        var apiKey = apiKeyValue.ToString();
        var validationResult = await validator.ValidateAsync(apiKey);

        if (!validationResult.IsValid)
        {
            context.Response.StatusCode = 403;
            await context.Response.WriteAsJsonAsync(new
            {
                error = "Invalid API Key"
            });
            return; // Short-circuit again
        }

        // Add user claims for downstream middleware
        var claims = new List<Claim>
        {
            new Claim(ClaimTypes.Name, validationResult.ClientName),
            new Claim("ApiKeyId", validationResult.KeyId.ToString())
        };
        
        var identity = new ClaimsIdentity(claims, "ApiKey");
        context.User = new ClaimsPrincipal(identity);

        // Continue to next middleware
        await _next(context);
    }
}

public interface IApiKeyValidator
{
    Task<ApiKeyValidationResult> ValidateAsync(string apiKey);
}

public class ApiKeyValidationResult
{
    public bool IsValid { get; set; }
    public string ClientName { get; set; }
    public Guid KeyId { get; set; }
}

Key learning points:

• Short-circuiting: Not calling _next() stops the pipeline
• Scoped services can be injected in InvokeAsync method, not constructor
• Setting context.User makes authentication info available downstream
• Always set appropriate status codes (401 vs 403)

🧠 Mnemonic: "S.A.M. - Singleton in Constructor, All scopes in Method" - Remember where to inject different service lifetimes!

Example 3: Response Caching Middleware 🗄️

Here's a more advanced example showing response manipulation:

public class CustomResponseCachingMiddleware
{
    private readonly RequestDelegate _next;
    private readonly IMemoryCache _cache;

    public CustomResponseCachingMiddleware(
        RequestDelegate next,
        IMemoryCache cache)
    {
        _next = next;
        _cache = cache;
    }

    public async Task InvokeAsync(HttpContext context)
    {
        // Only cache GET requests
        if (context.Request.Method != HttpMethods.Get)
        {
            await _next(context);
            return;
        }

        var cacheKey = GenerateCacheKey(context.Request);

        // Try to get from cache
        if (_cache.TryGetValue(cacheKey, out CachedResponse cachedResponse))
        {
            context.Response.StatusCode = cachedResponse.StatusCode;
            context.Response.ContentType = cachedResponse.ContentType;
            
            foreach (var header in cachedResponse.Headers)
            {
                context.Response.Headers[header.Key] = header.Value;
            }
            
            await context.Response.WriteAsync(cachedResponse.Body);
            return;
        }

        // Replace response stream to capture output
        var originalBodyStream = context.Response.Body;
        using var responseBody = new MemoryStream();
        context.Response.Body = responseBody;

        try
        {
            await _next(context);

            // Only cache successful responses
            if (context.Response.StatusCode == 200)
            {
                responseBody.Seek(0, SeekOrigin.Begin);
                var responseText = await new StreamReader(responseBody)
                    .ReadToEndAsync();

                var cached = new CachedResponse
                {
                    Body = responseText,
                    StatusCode = context.Response.StatusCode,
                    ContentType = context.Response.ContentType,
                    Headers = context.Response.Headers
                        .ToDictionary(h => h.Key, h => h.Value.ToString())
                };

                _cache.Set(cacheKey, cached, TimeSpan.FromMinutes(5));

                // Write response back to original stream
                responseBody.Seek(0, SeekOrigin.Begin);
                await responseBody.CopyToAsync(originalBodyStream);
            }
        }
        finally
        {
            context.Response.Body = originalBodyStream;
        }
    }

    private string GenerateCacheKey(HttpRequest request)
    {
        return $"{request.Path}{request.QueryString}";
    }

    private class CachedResponse
    {
        public string Body { get; set; }
        public int StatusCode { get; set; }
        public string ContentType { get; set; }
        public Dictionary<string, string> Headers { get; set; }
    }
}

Key learning points:

• Response stream replacement technique for capturing output
• Using MemoryStream as a buffer
• Always restore original stream in finally block
• IMemoryCache is thread-safe and can be injected as singleton

⚠️ Warning: Replacing the response stream is powerful but can impact performance. Consider ASP.NET Core's built-in ResponseCaching middleware for production use!

Example 4: Conditional Middleware Execution 🎯

Sometimes you need middleware that only runs under certain conditions:

public static class ConditionalMiddlewareExtensions
{
    public static IApplicationBuilder UseWhen(
        this IApplicationBuilder app,
        Func<HttpContext, bool> predicate,
        Action<IApplicationBuilder> configuration)
    {
        return app.Use(async (context, next) =>
        {
            if (predicate(context))
            {
                var branchBuilder = app.New();
                configuration(branchBuilder);
                var branch = branchBuilder.Build();
                await branch(context);
            }
            else
            {
                await next();
            }
        });
    }
}

// Usage example
app.UseWhen(
    context => context.Request.Path.StartsWithSegments("/api"),
    appBuilder =>
    {
        appBuilder.UseMiddleware<ApiKeyAuthenticationMiddleware>();
        appBuilder.UseMiddleware<ApiRateLimitingMiddleware>();
    });

app.UseWhen(
    context => context.Request.Headers.ContainsKey("X-Debug"),
    appBuilder =>
    {
        appBuilder.UseMiddleware<DetailedLoggingMiddleware>();
    });

Key learning points:

• app.New() creates a new branch in the pipeline
• Predicates allow runtime decisions about middleware execution
• This is more efficient than checking conditions inside every middleware

Common Mistakes ⚠️

1. Wrong Middleware Order 🔴

// ❌ WRONG - Authorization before Authentication!
app.UseAuthorization();
app.UseAuthentication();
app.MapControllers();

// ✅ CORRECT - Authentication first, then Authorization
app.UseAuthentication();
app.UseAuthorization();
app.MapControllers();

Why it matters: Authorization middleware checks if the authenticated user has permission. If authentication hasn't run yet, context.User will be empty!

2. Forgetting to Call next() 🔴

// ❌ WRONG - Pipeline stops here!
app.Use(async (context, next) =>
{
    await context.Response.WriteAsync("Hello");
    // Missing: await next();
});

// ✅ CORRECT
app.Use(async (context, next) =>
{
    await next();
    await context.Response.WriteAsync("Hello");
});

3. Modifying Response After It Started 🔴

// ❌ WRONG - Can't set headers after body is written!
app.Use(async (context, next) =>
{
    await next();
    context.Response.Headers.Add("X-Custom", "Value"); // Exception!
});

// ✅ CORRECT - Set headers before calling next
app.Use(async (context, next) =>
{
    context.Response.Headers.Add("X-Custom", "Value");
    await next();
});

💡 Tip: Use context.Response.OnStarting() callback if you need to modify headers after calling next():

app.Use(async (context, next) =>
{
    context.Response.OnStarting(() =>
    {
        context.Response.Headers.Add("X-Custom", "Value");
        return Task.CompletedTask;
    });
    await next();
});

4. Injecting Scoped Services in Constructor 🔴

// ❌ WRONG - DbContext is scoped, middleware is singleton!
public class MyMiddleware
{
    private readonly ApplicationDbContext _db; // Danger!
    
    public MyMiddleware(RequestDelegate next, ApplicationDbContext db)
    {
        _next = next;
        _db = db; // This will be disposed after first request!
    }
}

// ✅ CORRECT - Inject in InvokeAsync
public class MyMiddleware
{
    private readonly RequestDelegate _next;
    
    public MyMiddleware(RequestDelegate next)
    {
        _next = next;
    }
    
    public async Task InvokeAsync(
        HttpContext context,
        ApplicationDbContext db) // Scoped service here!
    {
        // Use db safely
    }
}

5. Not Handling Exceptions 🔴

// ❌ WRONG - Unhandled exceptions crash the app
app.Use(async (context, next) =>
{
    var data = await riskyOperation(); // Might throw
    await next();
});

// ✅ CORRECT - Proper error handling
app.Use(async (context, next) =>
{
    try
    {
        var data = await riskyOperation();
        await next();
    }
    catch (Exception ex)
    {
        _logger.LogError(ex, "Middleware error");
        throw; // Let exception middleware handle it
    }
});

Performance Considerations ⚡

Middleware Order Impact

+---------------------------------------+
|  Middleware Order for Performance    |
+---------------------------------------+
| 1. Exception Handler (catches all)   |
| 2. HTTPS Redirection (fast check)    |
| 3. Static Files (short-circuit)      |
| 4. Response Caching (avoid work)     |
| 5. Authentication                     |
| 6. Authorization                      |
| 7. Custom middleware                  |
| 8. Endpoint routing/Controllers      |
+---------------------------------------+

💡 Performance tip: Place middleware that can short-circuit requests (like static files, caching) earlier in the pipeline to avoid unnecessary processing!

Memory Management

• Avoid storing large objects in context.Items
• Be careful with response stream buffering (it uses memory)
• Use IMemoryCache with appropriate expiration policies
• Consider response compression middleware for large payloads

Key Takeaways 🎯

1. Middleware order is critical - Authentication before authorization, exception handling first
2. Three types of middleware: Inline (Use/Run), Convention-based, Factory-based (IMiddleware)
3. Singleton services in constructor, scoped services in InvokeAsync method
4. Short-circuiting by not calling next() stops pipeline execution
5. HttpContext is your gateway to request/response data and per-request services
6. Response modification must happen before headers are sent
7. Always handle exceptions in custom middleware
8. Use extension methods for clean middleware registration
9. Performance matters - order middleware to enable early short-circuits
10. Testing middleware requires mocking HttpContext and RequestDelegate

🧠 Memory technique - "EPIC-SAM":

• Exception handling first
• Performance-critical middleware early
• Injection rules matter (where you inject)
• Call next() unless short-circuiting
• Singleton in constructor
• All scopes in method
• Modify response before sending

📚 Further Study

1. Microsoft Official Docs - ASP.NET Core Middleware: https://learn.microsoft.com/en-us/aspnet/core/fundamentals/middleware/
2. Andrew Lock's Blog - Understanding Middleware: https://andrewlock.net/exploring-the-code-behind-iaplicationbuilder/
3. ASP.NET Core Source Code - Middleware: https://github.com/dotnet/aspnetcore/tree/main/src/Http/Http.Abstractions/src

📋 Quick Reference Card

╔══════════════════════════════════════════════════════════════╗
║         ASP.NET CORE MIDDLEWARE QUICK REFERENCE             ║
╠══════════════════════════════════════════════════════════════╣
║ MIDDLEWARE TYPES                                             ║
║ • Use()/Run() - Inline middleware                            ║
║ • Convention - Class with InvokeAsync method                 ║
║ • IMiddleware - Factory-based interface                      ║
║                                                              ║
║ REGISTRATION ORDER (typical)                                 ║
║ 1. UseExceptionHandler                                       ║
║ 2. UseHttpsRedirection                                       ║
║ 3. UseStaticFiles                                            ║
║ 4. UseRouting                                                ║
║ 5. UseAuthentication                                         ║
║ 6. UseAuthorization                                          ║
║ 7. MapControllers/MapEndpoints                               ║
║                                                              ║
║ DEPENDENCY INJECTION RULES                                   ║
║ • Constructor: Singleton services only                       ║
║ • InvokeAsync: Any service lifetime                          ║
║                                                              ║
║ SHORT-CIRCUITING                                             ║
║ • Don't call next() to stop pipeline                         ║
║ • Useful for: auth failures, caching hits, static files      ║
║                                                              ║
║ HTTPCONTEXT KEY PROPERTIES                                   ║
║ • Request - incoming data                                    ║
║ • Response - outgoing data                                   ║
║ • User - authentication/claims                               ║
║ • Items - per-request dictionary                             ║
║ • RequestServices - DI container                             ║
║                                                              ║
║ COMMON PITFALLS                                              ║
║ ✗ Wrong middleware order                                     ║
║ ✗ Forgetting await next()                                    ║
║ ✗ Modifying response after it started                        ║
║ ✗ Scoped services in constructor                             ║
║ ✗ Not handling exceptions                                    ║
╚══════════════════════════════════════════════════════════════╝

🔧 Try this: Create a middleware that logs the time taken by each subsequent middleware component. Store timestamps in context.Items and calculate differences after calling next()!

---

You now have a solid understanding of ASP.NET Core middleware! This knowledge is essential for building production-grade applications and will definitely come up in senior developer interviews. In the next lesson, we'll explore advanced Entity Framework Core patterns including change tracking, query optimization, and the Unit of Work pattern. 🚀