What Are Some Key Concepts That Every Backend Engineer Should Know?

Techletter #19 | April 20, 2023

As my interest in backend engineering has elevated, I am giving my spare time to enhance my understanding of this field. I do write a lot of code in both React and nodejs. So, I need to learn a lot of things every day in order to solve different kinds of problems that I come across.

Recently I have started watching a course by Hussain Nasser.

I have documented my short notes below. Hope you will pick up some wonderful concepts and satisfy your curiosity along he way.

Request Response

The request and response are the basic building blocks of communication between a client and a server. In the context of web development, the HTTP protocol is commonly used for exchanging requests and responses between clients and servers.

• Its simple & almost everywhere.

Client Sends a request -> Server parses the request -> Server Processes the request -> Server sends the response -> Client parses the response & consume.

• Parsing the request is not cheap

• Parsing XML costs more than parsing JSON

• Its used in

  • Web, HTTP, DNS, & SSH
  • RPC (Remote Procedure Call)
  • SQL & Database Protocols
  • APIs (REST/SOAP/GraphQL)
  • Implemented in variations

• Anatomy

  • A request structure is defined by both cient & server.
  • Request has a boundry
  • Defined by a protocol and message format.

• Building an image upload service with req & res

  • Send a large request with image
  • Send image in chunks

• Req & Res doesn’t work everywhere

  • Notification Service | Polling don’t scale well
  • Chatting Application | Empty requests can make the network slow & conjusted
  • If there is very long requests
  • What if Client Disconnects?

This is just a short note. Will be explaining them in detail in the upcoming articles

Synchronous & Asynchronous I/O

• Synchronous I/O

  • Caller sends a request & blocks
  • Caller cannot execute any code meanwhile
  • Reciever responds, Caller unblocks
  • Caller & Reciever are in sync

• Asynchronous I/O

  • Caller sends a request
  • Caller can work until it gets a response
  • Caller either:
    • Checks if the response is ready(epoll)
    • Receiver calls back when it’s done (io_uring)
    • Spins up a new thread that blocks
    • Caller & Receiver are not in sync

Push

• Req/Res isn’t always ideal. Client wants real time notification from backend. So push model is great for certain cases.

What is push?

• Clients connect to server. Server sends data to client. Client doesn’t have to request anything. Protocol must be bidirectional.

• Used by RabbitMQ

Pros and cons of push

• Pros
  • Real Time
• Cons
  • Clients must be online
  • Clients might not be able to handle
  • Requires a bidirectional protocol
  • Polling is a preferred for light clients

You can use web socket for this.

Polling

• Short Polling
  • Polling is a good communication style

What is Short Polling?

• Client Sends a request
• Server responds immediately with a handle
• Server continues to process the request
• Client uses that handle to check for status
• Multiple “Short” request response

Pros - Simple to build - Good for long running requests - Client can disconnect Cons - Too chatty - Network bandwidth - Wasted backend resources

Lets implement a short polling

const app = require('express')();

const jobs = {};

app.post('/submit', (req, res) => {
    const jobId = `job:${Date.now()}`
    jobs[jobId] = 0;
    updateJob(jobId, 0);
    res.end("\n\n" + jobId + "\n\n");
});

app.get('/checkstatus', (req, res) => {
    console.log(jobs[req.query.jobId])
    res.end("\n\nJobStatus:" + jobs[req.query.jobId] + "%\n\n")
});

app.listen(8080, () => console.log("Server is listening"));

const updateJob = (jobId, progress) => {
    jobs[jobId] = progress;
    console.log(`updated ${jobId} to ${Progress}`);
    of (progress === 100) return;
    this.setTimeout(() => updateJob(jobId, progress + 10), 3000);
} 

Long Polling

• Client sends a request
• Server responds immediately with a handle
• Server continues to process the request
• Client uses that handle to check for status
• Server won’t reply until it has response

Pros - Less chatty & backend friendly - Client can still disconnect cons - Not real time

Wil be implementing the long polling in the upcoming articles.

Server Sent Events

• A response has start and end
• Client sends a request
• Server sends logical events as part of response
• Server never writes the end of the response
• It is still a request but an unending response
• Client parses the streams data
• Works with request/response (HTTP)

Pros - Real time - Compatible with request/response Cons - Client must be online - Client might not be able to handle - Polling is preferred for light clients - HTTP/1.1 problem (6 connections)

Publish & Subscribe

── Video Upload/
    ├── Upload Service
    ├── Compress Service
    ├── Format Service
    ├── Notification Service
    └── Copyright Service

When you want to communicate between multiple services then req/res is not good.

Upload -> Upload service

Raw MP4 -> Compress Service -> Compressed video -> Format Service -> 480P, 720P, 1080P, 4K video -> Notification Service

Pros - Scales with multiple receivers - Great for microservices - Loose coupling - Works while clients not running

Cons - Message delivery issues - Complexity - Network saturation

Consider this as a simple list of concepts. There are many things which cannot be simply explained in one article. Hope, you have grasped few concepts, in turn you can question yourself and find answers which will staisfy your curosity.