Techletter #51 | December 16, 2023
How to build data engineering pipelines at scale?
Data is the new oil. But, to get that oil you need tools that can extract and get what can be useful. Similarly, data pipelines are required to efficiently handle the data that is coming at a large scale. Managing data is a challenging process. Once the company grows, the unstructured data grows. So how do you handle the large-scale data?
Is GO a system programming language?
Systems programming, or system programming, is the activity of programming computer system software.
The primary distinguishing characteristic of systems programming when compared to application programming is that application programming aims to produce software that provides services to the user directly (e.g. word processor),
whereas systems programming aims to produce software and software platforms that provide services to other software, are performance constrained, or both (e.g. operating systems, computational science applications, game engines, industrial automation, and software as a service application).
https://medium.com/@erik-engheim/is-go-a-systems-programming-language-c243c80eb6f9
Why handling date time is tricky?
Calculating time based on your locality is a complex process. Why? Because, how you store the time matters a lot. At least, I have made mistakes and understood (still I am learning) about time.
Scheduling global meetings between different time zones is a very tedious process. It is very simple for the end user, but an engineer should think a lot under the hood while handling it.
For example, a meeting is scheduled between two members A & B for December 16, 2023, 9:00 AM IST. Member A is in India & Member B is in USA. So at what time does Member B need to join to complete the meeting successfully? Member B needs to join at December 15, 2023, 10:30 PM EST(I have used golbaltimeconverter(dot)com & freeconvert(dot)com). The database date time would be December 16, 2023, 3:30 UTC (24hr format).
One of the best ways to handle time is through UTC(Coordinated Universal Time/ Universal Time Coordinated). While storing the time (ex createdAt time) the database stores it in UTC (You can use timestamp as well). Using the UTC you can always convert the time by adding the number of hours for the current time zone. In the above example, you can see, UTC -5 for EST and UTC +5:30 for IST.
Dayjs & moment are two libraries that will save a lot of your time while handling date time in JavaScript. But, believe me, handling date time is not as easy as we think. It requires a great amount of effort and attention while you deal with it.
https://www.freecodecamp.org/news/synchronize-your-software-with-international-customers/
https://medium.com/geekculture/database-should-we-store-datetime-or-timestamp-f054d8d4d3fe
How come Amazon S3 handles 100 million requests per second?
(Building and operating a pretty big storage system called S3)
Amazon’s S3 may be a simple service for the end user. But, how is it built and scaled. How can it handle large amounts of data & requests? How the data is physically stored? You need to think about hard disk mechanics, firmware, and the physical properties of storage media at one end, to customer-facing performance experience and API expressiveness at the other. It’s definitely not a simple software system.
https://www.allthingsdistributed.com/2023/07/building-and-operating-a-pretty-big-storage-system.html
What’s next after Java/React?
I have been a JavaScript developer for ~2 years now. But, after a while you feel stuck just with react, nodejs etc. What should we learn after that in order to keep going? Especially to me I have gained more interest in backend engineering & software infrastructure. This articles talks about not only technical skills that we need to develop as well as soft skills that we need to have in order to keep growing.
https://levelupsoftwareengineering.substack.com/p/whats-next-after-java-and-react