Your API Is Slow. Now What? A Practical Guide to Finding the Real Bottleneck

That message looks simple.

It isn’t.

When you’re responsible for a production system, one of the most frustrating incidents is an API that is sometimes slow. There is no obvious error. The service is still running. Kubernetes reports that all Pods are healthy. CPU usage looks normal. Memory looks fine. Nothing appears to be broken.

So you restart the Pods.

For a few minutes, everything looks better.

Then the problem comes back.

This is where many developers start changing things without really knowing what they’re fixing. They increase CPU limits, add more replicas, restart containers, increase database resources, or change a timeout.

Sometimes those changes appear to help.

But there is a much more important question to answer first:

Where did the eight seconds actually go?

That question changes the entire way you approach performance problems.

A slow API is not a root cause. It is a symptom.

The real engineering problem is finding which part of the system is responsible for the latency and why.

A Request Is a Journey Through Your System#

When we think about an API request, it’s easy to imagine something simple:

Client → API → Database → Response

Real production systems are rarely that simple.

Consider an e-commerce platform built with Next.js, an API Gateway, several backend services, Redis, PostgreSQL, and external payment services.

A request for an order might travel through something like this:

Customer   │   ▼Next.js   │   ▼Cloudflare   │   ▼Load Balancer   │   ▼Ingress   │   ▼API Gateway   │   ▼Order Service   │   ├───────────────┐   │               │   ▼               ▼ Redis         PostgreSQL   │   ▼Product Service   │   ▼External Service

Now imagine a customer calls:

GET /api/orders/12345

and receives a response after eight seconds.

Which component caused the delay?

It could be the API Gateway.

It could be the Order Service.

It could be PostgreSQL.

It could be Redis.

It could be a network connection.

It could be another microservice.

It could even be an external API that your application has no control over.

The request may have spent its time approximately like this:

Cloudflare             20msIngress                 5msAPI Gateway            30msOrder Service          40msRedis                  10msProduct Service       200msPostgreSQL           7,700ms                     --------Total                 8,005ms

From the customer’s perspective, the API took eight seconds.

But saying “the API is slow” doesn’t tell us much.

The useful information is that PostgreSQL consumed almost the entire eight seconds.

This is the first important habit to develop when debugging distributed systems:

Don’t ask only how slow the API is. Ask where the request spent its time.

Before Fixing Performance, Define the Problem#

There is another problem with saying:

“The API is slow.”

Slow compared to what?

100 milliseconds?

500 milliseconds?

Five seconds?

And is every request slow, or only a small percentage?

These questions matter.

Suppose our monitoring system reports:

p50 = 120msp95 = 300msp99 = 7.8s

If you haven’t worked much with percentiles, these numbers can look confusing at first.

The p50, or 50th percentile, means roughly half of the requests completed faster than 120 milliseconds.

The p95 means roughly 95% of requests completed faster than 300 milliseconds.

The p99 means roughly 99% completed faster than 7.8 seconds.

The interesting part is that last number.

Most requests are relatively fast, but a small percentage are extremely slow.

That explains a common production situation where developers say:

“It works fine for me.”

while customers are reporting:

“This page takes forever to load.”

Both can be correct.

The application may perform well for most requests while having terrible tail latency for a smaller group.

This is why production performance analysis often focuses on percentiles such as p95 and p99, rather than looking only at average response time.

Why the Average Can Hide a Serious Problem#

Imagine an API receives 100 requests.

Ninety-nine requests complete in 100 milliseconds.

One request takes 10 seconds.

The average is approximately:

(99 × 100ms + 10,000ms) / 100≈ 199ms

Someone looking only at the average might conclude:

“The API responds in about 200 milliseconds.”

Technically, that’s true.

But one customer just waited ten seconds.

This is one reason performance engineering isn’t simply about finding a single number and trying to make it smaller.

You need to understand the distribution of latency.

A useful starting point is:

p50 → typical experiencep95 → slower usersp99 → worst-case / tail experience

The right target depends heavily on the endpoint and business requirement. A search endpoint, payment operation, internal admin API, and health check may all have very different acceptable latencies.

Don’t Change Anything Yet#

When an API is slow, there is a strong temptation to immediately change infrastructure.

You might see:

API latency → 8 seconds

and respond with:

Increase CPUIncrease memoryAdd more PodsRestart containersIncrease database resourcesIncrease timeout

The problem is that none of these actions answer the most important question:

Why is it slow?

Suppose the real problem is a database query.

Adding more API Pods might temporarily reduce some queueing.

But it could also increase the number of database connections and make PostgreSQL even more overloaded.

Suppose the real problem is an external payment provider.

Adding CPU to your application won’t make the payment provider respond faster.

Suppose the real problem is an N+1 query pattern.

Adding more memory doesn’t remove the unnecessary network requests.

The better approach is to treat performance debugging as an investigation.

A useful mental model is:

Symptom   ↓Measure   ↓Locate   ↓Form a hypothesis   ↓Collect evidence   ↓Run an experiment   ↓Identify root cause   ↓Apply the fix   ↓Measure again

Notice that the fix comes near the end, not the beginning.

Case Study: The Order API That Sometimes Takes Eight Seconds#

Let’s look at a realistic example.

Imagine our Order Service needs to return an order together with information about the products inside that order.

The implementation looks something like this:

def get_order(order_id):    order = Order.objects.get(id=order_id)
    items = OrderItem.objects.filter(order=order)    products = []    for item in items:        response = requests.get(            f"https://product-service/products/{item.product_id}"        )        products.append(response.json())    return {        "order": order,        "items": products,    }

At first glance, this seems reasonable.

We retrieve the order, retrieve its items, and then retrieve the corresponding products.

But let’s think about what the program is actually doing.

Suppose an order contains five products.

The application makes approximately:

Order Service    │    ├── Product Service    ├── Product Service    ├── Product Service    ├── Product Service    └── Product Service

Five network requests.

Now suppose a customer has an order containing 50 items.

The application makes:

1 request → Order1 query   → Order Items50 requests → Product Service

That’s a very different workload.

If each Product Service request takes 150 milliseconds and they happen sequentially:

50 × 150ms= 7,500ms= 7.5 seconds

We’ve just explained the mysterious eight-second response.

And notice something important.

Nothing necessarily looks broken.

The containers can be healthy.

The Kubernetes Pods can be healthy.

CPU can be low.

Memory can be normal.

The network can be working correctly.

Every individual request is technically successful.

The architecture itself is causing the latency.

Why Is the API Only Slow Sometimes?#

This is where the original symptom becomes useful.

The API wasn’t always slow.

It was sometimes slow.

That tells us something.

Maybe small orders look like this:

5 products×150ms≈750ms

while large orders look like:

50 products×150ms≈7.5 seconds

Now the behavior isn’t random anymore.

Latency is correlated with the number of products in the order.

That gives us a hypothesis:

The number of downstream requests increases with the size of the order.

That’s much more useful than:

“The API is randomly slow.”

Fixing the N+1 Problem#

One possible solution is to change the Product Service API so it can retrieve multiple products at once.

Instead of:

GET /products/1GET /products/2GET /products/3GET /products/4...

we could provide something like:

POST /products/batch

with:

{  "ids": [1, 2, 3, 4, 5]}

The architecture becomes:

Order Service      │      │ one request      ▼Product Service      │      ▼Multiple products

We have changed the problem from:

N network requests

to:

1 network request

The exact implementation depends on the system. Sometimes batching is appropriate. Sometimes caching is better. Sometimes asynchronous processing makes more sense.

The important point is not the specific optimization.

The important point is how we discovered it:

Slow request    ↓Trace request    ↓Find many downstream calls    ↓Notice relationship with order size    ↓Identify N+1 pattern    ↓Change architecture    ↓Measure again

That’s performance debugging.

What If the Database Is the Problem?#

Let’s say we’ve investigated the application and its service calls.

The next obvious place to look is PostgreSQL.

Suppose our Order Service executes:

SELECT *FROM ordersWHERE customer_id = 12345;

Our database contains millions of orders.

We shouldn’t immediately create an index just because indexes are usually good.

First, ask PostgreSQL how it executes the query:

EXPLAIN ANALYZESELECT *FROM ordersWHERE customer_id = 12345;

Imagine we discover:

Seq Scan on ordersRows Removed by Filter: 9,999,000Execution Time: 4200 ms

Now we have evidence.

PostgreSQL is scanning a huge amount of data to find the rows we need.

If customer_id is frequently used for filtering, an appropriate index may dramatically reduce the amount of work:

CREATE INDEX idx_orders_customer_idON orders(customer_id);

Afterward, we run the query again.

Perhaps the result becomes:

Before:4,200ms
After:15ms

Now imagine the original incident report:

“The Kubernetes API is slow.”

But Kubernetes wasn’t the problem.

The root cause was a database query.

This is why performance debugging needs to cross application boundaries.

Low CPU Doesn’t Mean Your Application Is Healthy#

One of the most misleading signals during an incident is CPU.

Imagine you look at your Kubernetes dashboard and see:

CPU:     20%Memory:  40%Pods:    Running

Everything looks healthy.

But customers are waiting eight seconds for a response.

How?

Because an application doesn’t spend all of its time calculating.

A request might spend most of its lifetime waiting for something else:

API │ ├── waiting for PostgreSQL │ ├── waiting for Redis │ ├── waiting for another service │ └── waiting for an external API

While the process is waiting, CPU usage may remain low.

This is why:

CPU utilization is a useful signal, but it is not a complete definition of application health.

The same principle applies to memory.

A service can have plenty of free memory and still be slow because it is blocked on a database lock, waiting for a connection, waiting on network I/O, or throttled by another resource.

Connection Pool Exhaustion#

Here’s another problem that can produce surprisingly high latency without obvious CPU pressure.

Suppose our application has a PostgreSQL connection pool of 20 connections.

Then 100 requests arrive concurrently.

The first 20 requests may acquire connections:

Request 1   → DB connectionRequest 2   → DB connectionRequest 3   → DB connection...Request 20  → DB connection

The remaining requests have to wait:

Request 21  → waitingRequest 22  → waitingRequest 23  → waiting...

From the outside, the application may simply look slow.

You might see:

CPU       → normalMemory    → normalPod       → RunningPostgreSQL → healthy

But the application is spending time waiting for a database connection.

This is an important distinction:

Slow query

and:

Waiting for a connection before the query can even start

are different problems.

The solution is therefore different too.

You need to investigate things such as:

• connection pool configuration
• request concurrency
• query duration
• long-running transactions
• database connection limits
• application instance count

Again, the goal isn’t to guess the configuration.

It’s to understand where the waiting is happening.

External Dependencies Are Part of Your Latency Budget#

Microservices introduce another interesting problem.

Your service may be perfectly healthy while one of its dependencies is having a bad day.

Consider a checkout flow:

Customer   │   ▼Order Service   │   ▼Payment Service   │   ▼External Payment Provider

Normally, the external provider might respond in:

200ms

But during an incident, it might take:

6 seconds

If your application waits synchronously for that response, your API inherits that latency.

Your Kubernetes metrics might still look perfectly normal:

CPU       → 25%Memory    → 45%Pods      → Healthy

The application isn’t doing anything wrong.

It’s waiting.

This is one reason production systems often need:

• timeouts
• retries with care
• circuit breakers
• asynchronous processing
• graceful degradation
• dependency monitoring

A five-second external API call should not necessarily be allowed to block your application indefinitely.

Now Kubernetes Enters the Investigation#

So far, we’ve looked mostly at application and dependency problems.

But sometimes Kubernetes really is involved.

Consider a container with:

resources:  requests:    cpu: "100m"  limits:    cpu: "200m"

The CPU request tells Kubernetes roughly how much CPU capacity the workload needs for scheduling purposes.

The CPU limit places an upper bound on how much CPU the container can consume.

If the application regularly needs substantially more CPU than its configured limit allows, it can experience CPU throttling and increased latency.

But here’s the important part:

Don’t see a CPU limit and immediately increase it.

First investigate.

Ask:

Is CPU actually saturated?Is the container being throttled?Is the workload CPU-bound?Is the node under pressure?Is the application doing unnecessary work?Would increasing CPU simply move the bottleneck somewhere else?

You want evidence before making the change.

Why Adding More Pods Can Make Things Worse#

Scaling horizontally is powerful.

Suppose one API Pod can handle approximately 100 requests per second.

With three Pods:

3 × 100=300 requests/second

If traffic grows to 500 requests per second, increasing the number of Pods may be the correct solution.

But what if PostgreSQL is already the bottleneck?

You scale:

3 Pods   ↓20 Pods

Now you potentially have many more application instances competing for database resources.

The result could look like:

More Pods    ↓More concurrent requests    ↓More DB connections    ↓Database saturation    ↓Longer query times    ↓Higher API latency

You scaled the API and made the system slower.

This is why distributed systems need bottleneck-aware scaling.

Scaling one component doesn’t necessarily increase the capacity of the whole system.

How Do We Actually Find the Bottleneck?#

This is where observability becomes extremely valuable.

Observability is often described using three major signals:

Observability                     │          ┌──────────┼──────────┐          │          │          │          ▼          ▼          ▼        Logs      Metrics     Traces

They answer different questions.

Logs#

Logs help answer:

What happened?

For example:

GET /api/orders/12345status=200duration=8120ms

That’s useful.

But it still doesn’t tell us exactly where the eight seconds went.

Metrics#

Metrics help answer questions such as:

How much?

How often?

Is it getting worse?

We might see:

request_count       = 10,000p99_latency         = 7.8scpu_usage           = 35%memory_usage        = 60%db_connections      = 95%error_rate          = 2%

Now we can start looking for relationships.

For example:

Traffic increases       ↓DB connections increase       ↓Latency increases

That correlation gives us something to investigate.

Traces#

Distributed traces answer one of the most important questions:

Where did this particular request spend its time?

For our order request, a trace might look like:

GET /api/orders/12345│├── API Gateway       20ms│├── Order Service     30ms│├── Redis             10ms│├── Product Service  200ms│└── PostgreSQL      6,900ms

Now the problem becomes much easier to reason about.

Instead of saying:

“The API takes eight seconds.”

we can say:

“This request takes eight seconds, and approximately 6.9 seconds are spent waiting on PostgreSQL.”

That’s a dramatically better debugging position.

A Complete Production Investigation#

Let’s put the entire process together.

A customer reports:

“The order API is sometimes extremely slow.”

We begin by looking at latency.

p50 = 120msp95 = 300msp99 = 7.8s

The problem appears concentrated in the slowest requests.

We inspect traces from those requests.

We discover:

Gateway             20msOrder Service       30msRedis               10msProduct Service    200msPostgreSQL       6,900ms

Now we know where to investigate.

We inspect the database query with:

EXPLAIN ANALYZE

The database shows a sequential scan over a large table.

We investigate the query and discover that a frequently used filter doesn’t have an appropriate index.

We add the index.

Then we measure again.

Before:

p99 = 7.8s

After:

p99 = 350ms

The investigation is complete.

Notice how different this is from:

API slow   ↓Increase CPU   ↓Restart Pods

The second approach changes the system without understanding it.

The first approach builds a chain of evidence.

A Practical Debugging Framework#

When you receive a vague report such as:

“The API is slow.”

start by making the statement more precise.

Ask:

Is every request slow?#

If not, look for characteristics shared by the slow requests.

Maybe they’re:

• large orders
• specific customers
• particular endpoints
• requests with many database records
• requests involving an external dependency

What does the latency distribution look like?#

Check:

p50p95p99

A high p99 with a normal p50 tells you a very different story from a high p50.

Where is the time being spent?#

Use application metrics and distributed tracing to break the request into components.

Is the application computing or waiting?#

Look at:

CPUDatabaseNetworkExternal APIsConnection poolsLocksI/O

Is Kubernetes actually involved?#

Only after you have evidence should you investigate things such as:

CPU throttlingMemory pressureOOMKilledNode pressureSchedulingNetworkPolicyService routingIngress

This ordering matters.

Kubernetes is part of the system, but it isn’t automatically the cause of every performance problem.

The Difference Between a Workaround and a Root-Cause Fix#

Suppose your API is slow because a PostgreSQL query is inefficient.

You add more API replicas.

Latency improves from eight seconds to five seconds.

Did you fix the problem?

Not necessarily.

You may have changed the workload enough to temporarily improve the symptoms.

The database query is still inefficient.

A proper fix might involve:

Query optimization+Appropriate indexing+Connection management+Caching where appropriate+Better application behavior

And then you verify the result using measurements.

This distinction matters in production because a workaround can become permanent infrastructure.

Six months later, nobody remembers why the system has 20 replicas.

The original problem may still be sitting underneath it.

Performance Debugging Is About Evidence#

The deeper lesson here isn’t about PostgreSQL.

It isn’t about Kubernetes.

It isn’t even about observability.

It’s about how you think when a production system behaves unexpectedly.

A useful investigation looks like this:

Symptom                       │                       ▼                  Measurement                       │                       ▼                 Localization                       │                       ▼                  Hypothesis                       │                       ▼                    Evidence                       │                       ▼                   Experiment                       │                       ▼                  Root Cause                       │                       ▼                     Fix                       │                       ▼                 Verification

Each step reduces uncertainty.

You don’t need to know the answer immediately.

You need to know what question to ask next.

That is one of the most valuable skills you can develop as a backend or platform engineer.

The Next Time Someone Says “The API Is Slow”#

Don’t start by restarting the Pod.

Don’t immediately increase CPU.

Don’t blindly add replicas.

Start with:

“Which requests are slow?”

Then:

“How slow are they?”

Then:

“Where is the time being spent?”

Then:

“What evidence tells us why?”

And finally:

“How do we know our fix actually worked?”

The goal of performance engineering isn’t to make a system faster by changing random variables until the graphs look better.

It’s to understand the system well enough that you can explain why it was slow, what changed, and why the change fixed it.

That mindset becomes increasingly important as your architecture grows.

A monolith can hide many of these problems inside one process.

A microservice architecture spreads them across services.

Containers add another layer.

Kubernetes adds another.

Cloud infrastructure adds another.

And suddenly a request that looks like:

GET /api/orders/12345

is actually a journey through dozens of components.

When something goes wrong, the engineer who understands that journey has a significant advantage.

So the next time you see:

🚨 API latency: 8 seconds

don’t ask:

“What should I restart?”

Ask:

“Where did those eight seconds go?”

That’s where the real investigation begins.