In this article, we are going to see how to save the SNS messages to S3 using Kinesis Data Firehose The first step of building a data engineering pipeline in your organization is storing the unorganized data with long retention and partitioning. Therefore storing the SNS data with Long-term storage[...]
Read moreLatest Posts
Kubernetes NGINX Sidecar ReverseProxy - Connection refused Localhost
While using NGINX reverse Proxy as a Sidecar container for your microservice, We often use localhost to connect to the application container on the same POD but it can potentially harm your application availability in certain systems and lead to a lot of 502 or connection refused issues Here is[...]
Read moreRun SQL queries directly on S3 backup of AWS RDS - using Apache Drill
Apache Drill is a revolutionary product that identifies itself as a Schema-free SQL Query Engine for Hadoop, NoSQL and Cloud Storage - While it was often represented or perceived as a tool for Data Engineers or Analysts. It is not just for Data Engineers In this article, we are going[...]
Read moreCloudflare Ansible Example - To create, update, delete DNS records
In this article, we will discuss how to use Ansible to automate CloudFlare DNS management. Ansible has modules for almost everything and it includes Cloudflare the popular CDN and DNS provider Since Cloudflare is part of the community.general collection of ansible - it is likely you would have it installed[...]
Read moreAWS IMDS - What is Link-Local IP address 169.254.169.254
In the world of networking, IP addresses are crucial for devices to communicate with each other over the Internet. However, not all IP addresses are created equal, and some serve unique purposes. One such IP address is 169.254.169.254, which is commonly referred to as a link-local address. In this article,[...]
Read moreComparing a Kid and AI - Learning AI with real life examples
If you are reading this, am sure you are trying to get into the mystery world of AI and puzzled by the various terminologies of the AI world. AI Model GPT - Generative Pre-trained Transformer LLM - Large Language Model FM - Foundation Model Text Generation Text Summarization Transformers Embeddings[...]
Read moreHow to Find and Convert Latin1 encoded rows to UTF8 - MySQL
In this brief article, I want to share a few tricks and lessons I have learnt on Converting Latin1 Encoded rows to UTF8 I had a table with a field named username with millions of records and we recently converted that table from latin1 charset to UTF8 First, let's start[...]
Read moreUsing SOPS with AWS KMS - Encrypt and Decrypt files | Devops Junction
Secret Management is an essential part of infrastructure and DevOps. how do you store your Secrets and Environment variables define how safe your infrastructure is As a good engineering practice, most companies guide their developers to not hard code the secrets and environment variable directly but out of convenience or[...]
Read moreAWS S3 LS Examples - List and Browse S3 buckets | Devops Junction
Amazon Simple Storage Service (AWS S3) is a highly scalable and cost-effective object storage service provided by Amazon Web Services (AWS). It allows users to store and retrieve vast amounts of data, making it a popular choice for developers, businesses, and organizations worldwide. To interact with S3 buckets and objects[...]
Read moreEC2 Instance Connect Endpoint - SSH the easy way | Devops Junction
How do you SSH to an EC2 instance which is on a Private Subnet? You would say either of the following options Create a Bastion Host in Public Subnet then SSH to the Bastion Host and then SSH to the EC2 instance in Private Subnet Setting up EC2 instance Connect[...]
Read more