Sketech #8 👁️ I crafted this to show you how a zip works, why throughput ≠ latency and how CI/CD workflows come to life
Sketech crafted to make concepts unforgettable
Welcome to this week’s edition of Sketech — Visual Thinking for Engineers
In this edition, you’ll find:
Ever wonder How a ZIP file really works? Let’s break it down (Recap)
Insights on Throughput ≠ Latency: Why Both Matter in System Performance (Recap)
From Code to Production, How CI/CD keeps your Workflow Smooth (Recap)
File Compression Explained
Zip reduces data size by encoding it into a more compact form, enabling efficient storage management and optimized data transmission.
Exploring Compression Types:
↳ Lossless: Formats like ZIP, PNG, FLAC ensure no data is lost during compression. This type is used for text, executables or any data to guarantee the integrity.
↳Lossy: Formats like JPEG, MP3 and many video codecs discard less critical data to achieve higher compression ratios. For multimedia where slight degradation is acceptable.
Where to Apply Compression?
1/
File Handling and Storage: If your app handles large data (text files, images, databases, logs) compression algorithms can optimize storage and reduce file transfer times.
↳ File Compression: Use libraries like zlib, gzip or 7-zip to compress files before storing or sending over a network.
↳ Databases: PostgreSQL and MongoDB support data compression, but you can also implement column or record compression using algorithms to save storage.
↳ Compressed Logs: Automate log compression with tools like gzip to save disk space.
2/
Media Optimization: Compression is key to improving load times and user experience, especially in streaming or real-time content.
↳ Images: Use JPEG or PNG to reduce file sizes without losing too much quality. Libraries like Pillow (Python) or Sharp (Node.js) can compress images efficiently.
↳ Streaming: Implement codecs like H.264 or H.265 using libraries like FFmpeg to compress video while conserving bandwidth. Ideal for video calls, live streaming, or platforms.
↳ Audio: Apply lossy compression with formats like MP3 or AAC to reduce file sizes while maintaining audio quality.
3/
Compression in APIs and Data Transfers: When working with APIs that handle large datasets, compression can reduce network traffic and improve performance.
↳ REST APIs: Use GZIP or Brotli to compress API responses, reducing data sent to clients.
↳ WebSockets: Use compression to reduce the amount of data sent over WebSocket connections, optimizing real-time message transmission.
↳ File Transfers: Implement compression for large file transfers between client and server to reduce upload/download times.
4/
Cloud Storage and Processing: Apply compression to save resources and reduce costs in cloud environments.
↳ Cloud Data Compression: Services like Amazon S3 or Google Cloud Storage allow storing compressed files. Automatically compress files before uploading to save space.
↳ Data Transfer in Microservices: Compress HTTP or gRPC payloads to reduce latency and improve scalability in microservice-based systems.
Check out the LinkedIn post – you won’t believe the engagement
Throughput ≠ Latency: Why Both Matter in System Performance
In system design, two words often pop up: throughput and latency. While they might seem similar, they’re actually quite different and impact performance in unique ways.
What Each Metric Means
↳ Throughput: This is about volume. It’s the amount of data a system can process in a given period.
↳ Latency: This is about speed. It’s the time it takes for a single data packet to travel from the source to its destination. Lower latency means data reaches its destination faster, but it doesn’t indicate the total volume that can flow through.
Key Optimization Techniques → To achieve optimal performance, different techniques can improve either throughput or latency:
Throughput:
• Bandwidth aggregation
• Load balancing across servers
• Data compression for efficient transfer
Latency:
• Content Delivery Networks (CDNs) to bring content closer to users
• Protocol optimization, like TCP/IP tuning
• Edge computing to process data near the source
Measurement Tools → Both metrics require specific tools for accurate measurement:
• Throughput: Tools like iPerf and Netperf are ideal for network performance testing, showing the volume of data a system can handle over time.
• Latency: Ping, traceroute, and Speedtest are useful for measuring the time it takes for data to travel between two points, providing insights on speed and response time.
Throughput and latency each have their place in building efficient systems, but there’s another question: how does fault tolerance factor into this balance?
Ensuring a system can withstand failures without sacrificing throughput or latency is the next step in resilient design.
Take a look at the LinkedIn post
Lessons Learned in Building CI/CD Pipelines
After working with CI/CD Processes for years, here are the most valuable lessons I’ve learned:
↳ 1/ Automation is Non-Negotiable: Automating tests and deployments minimizes human error and speeds up development cycles.
↳ 2/ Frequent Integrations Catch Issues Early: Merging code regularly is key to detecting conflicts and preventing last-minute surprises.
↳ 3/ Staging Environments Aren’t Optional: A robust staging environment that mirrors production reduces deployment risks significantly.
↳ 4/ Monitoring is More than Just Logging: Tracking performance in production provides insights that logs alone can’t, helping with future improvements.
↳ 5/ Small, Frequent Releases Are Better: Incremental changes are easier to manage, test and roll back if something goes wrong.
These insights have shaped my approach to CI/CD, making each step smoother and more resilient.
We’ve reached the end of this week’s edition! I trust these visuals helped simplify the concepts and made them easier to implement. Until next time – keep pushing forward
Nina
Sketech Newsletter Crafted to make concepts unforgettable 🩵