Overview

The server has a specified Requests Per Minute(RPM) for api calls. Design a client to saturate server's allocated RPM.

Functional Requirements:

• Keep client's requests rate just below RPM to keep server saturated
• Provide latency and throughput report for various token sizes returned from server.

Non Functional Requirements:

• Optimum resource usage at client
• Keep load usage to the minimum Read/load task context into the memory only when needed.

Design:

Note: asyncio.Semaphore() is for concurrently active sessions. It cannot be used for RPM.

1. Simple solution:

Schedule a job at 1/(RPM) of a minute. Say API allows 120 RPM, ie 2 calls per second. So schedule one api call every 0.5s. Pros:

• Keeps code logic simple. Easy to implement, understand/explain, debug.
• Good for in-house server to distribute load over time. Cons:
• Under-utilizing allocated RPM from server.
• Not ideal for heavy work loads where we want to squeeze every out of the server.

2. Batched posting:

Post jobs in batches of RPM at the beginning of 1 min.

• Needs token bucket algo into scheduling
• Relatively simpler to implement
• utilizes allocated RPM from server.

3. Streaming posting:

Keep posting jobs as tokens become available.

• Need token bucket algo
• Best utilization of allocated RPM from the server.
• More suitable for streaming input when rate of incoming tasks/jobs is spread over the 1 minute duration.

The value of streaming becomes apparent when mulitple tangential rate-limit constraints are applicable

• Reqs Per Min
• Reqs Per Day
• Tokens Per Min
• Tokens Per Day
• Time of Day
• Day of Week, etc

Assumptions:

• No tasks takes more than a minute to complete, which is the token refresh frequency.
• Jitter or spreading out of requests is not considered. If server scale/load is a concern, this can be taken into account.
• Standard Deviation of latency is not a concern since these are batched jobs. Saturating the allocated RPM and token throughput are the only concerns.

Future:

1. Tasks scheduling: For multiple tangential rate-limit contraints, a scheduling algorithm would help maximize the returns. This is to ensure it always stays just below the various thresholds.
2. So far client resources/limits have not been an issue since there is no real job/compute intensive operation. If we do start hitting the bottlenecks at client, horizontal scaling can be considered.
3. Generate chart to help visualize how data-token-size and other params impact latency.
4. Logging should be conrolled by flag.
5. Moduler code layout, separating token-bucket and reporting into their own modules.

Installation:

cd api_benchmark
chmod +x ./run.sh
./run.sh
. .venv/bin/activate
make setup; make server

In a new shell,

cd api_benchmark
. .venv/bin/activate
make setup; make client

Just wait for the test to finish and generate report. Now you can play around with the input params to see impact of number of tasks vs data-token-size.