Serve response of an HTTP request after receiving another request

My use case is to serve response of an HTTP request after receiving another request from separate server.

1. I want to do this best possible way keeping scaling in mind.
2. We are using Golang 1.19 with Gin Framework.
3. Server will have multiple pods thus channels will not work.
4. There will be timeouts for all request making initial request timed out after 60 seconds.

My current solution is to use a shared cache where each pod will keep checking the cache. I believe, I can optimize this with channels where rather than checking in cache one by one, system periodically checks for any completed response.

I would also like to know how it could have been achieved in other programming languages.

PS: This is design based query, I have some reputation here to share bounty thus asking here. Please feel free to edit if question is not clear.

Answer

tl;dr

problem description

So assuming your server application called server_app for instance, has 3 pods:

     +---------------------+
     |  server_app_service |
     +---------------------+
     |  server_app_pod_a   |
     |  server_app_pod_b   |
     |  server_app_pod_c   |
     +---------------------+

your service receives a request called "request A", and decides to pass it to server_app_pod_a. Now your server_app_pod_a forwards the request to some gateway, and waits for some sort of notification, to continue the processing of client’s response. And as you already know, there’s no assurance that when gateway does the request B, the service passes it to server_app_pod_a again. And even if it did so, your application’s state management would become a difficult task to do.

Messaging

As you might’ve noticed, I bolded the word “notification” in the past paragraph, that’s because if you really think about it, the request "B" looks more like a notification with some message rather than a request for some resource. So my number 1 choice would be a message queue like kafka (there are plenty of those, again, as you know). And the idea is, if you could define an algorithm to calculate unique keys for your requests, you can expect the resulting notifications in your exact same pod. This way, state management would be much simpler, and also, the chance of getting the notification in the same pod would be much higher (this depends on many factors of course, like the state of the message queue). Taking a look at your questions:

1. I want to do this best possible way keeping scaling in mind.

Sure thing, you can use these message queues like kafka, to achieve scaling and fewer data loss, both for the message queue and your application.

4. There will be timeouts for all request making initial request timed out after 60 seconds.

This one depends on how you manage timeouts in your codebase, using contexts would be a good idea.

I would also like to know how it could have been achieved in other programming languages.

Using message queues is a general idea, which would be applicable to almost any programming language, but depending on the programming paradigms of a language, and language-specific libraries and tools, there might be some other approaches to this problem. For instance in Scala, if you use some specific tool called akka (which provides actor model programming paradigm), you can use something so called akka-cluster-sharding, to handle this problem. And the idea is pretty simple, we know that there must be some sort of superviser, which knows the exact location and state of its own subscribers. So when it receives some message, it just knows where and which actor (we’re talking about actor model programming) to forward the request to. In other words, it can be used to share state between actors spawned on a cluster, either on the same machine or not. But as a personal preference, I wouldn’t go for language-specific communications, and would stick to general ideas, because of the problems it might cause in the future.

Wrap-up

Long enough explanations :). Just to make some sense of what I’m talking about, let’s follow up the exact same scenario, with a difference in communication model:

1. Client sends request “A” to server_app service.
2. The service, choses one of the pods (server_app_pod_b for instance) to handle the request.
3. The pod then tries to define some key for the request, and passes it to the gateway, along with the request, and waits for a message with the key, to be published in the queue.
4. The gateway does what it’s supposed to, and sends a message with the key, to the message queue.
5. The exact same pod serer_app_pod_b receives the message with the key, fetches the data of the message, and continues to process the client’s request.

There are probably other approaches available to address this issue, but this is what I would go for. Hope that it helps!