PDFstract - Easy way to extract and chunk documents for RAG
As you all know I have been working on this AI and Data Space for a while now and I have seen various bottlenecks and complexities in the data engineering and augmentation layer - at both startup and large enterprise clients When you think of where to use AI or get...
JSON performance MYSQL vs PostgreSQL - Testing with FastAPI
It all started when I wanted to check the performance of JSON of MYSQL and JSONB of PostgreSQL I am a big fan of PostgreSQL for its power and I have read N number of articles showing why PostgreSQL is better than MySQL Various Benchmarks and research done to prove...
How to Subscribe Private Endpoint Internal HTTP URL to SNS - AWS
In this article, we will see an interesting use case or system design where we want to connect AWS SNS to an internal HTTP or HTTPS service. SNS is a public service designed to connect to external public HTTPS endpoints. SNS is purpose-built for external...
How to find Unused Load Balancers - Based on Traffic | AWS FinOps
In this article, we are going to see how to find unused load balancers - based on Cloud Watch usage metrics like request count, active flow count We are going to get the usage metric from CloudWatch API using Python Boto and write it as a CSV report for all types of...
How to configure Sendgrid as an SMTP Relay on Metabase
In this article, we are going to see how to configure SendGrid as an SMTP relay for the Metabase If you are here I assume that you already are aware of Metabase and a little basics of SendGrid Mail service provider. In Metabase, the SMTP server or Email Channel plays...
How to Send SNS messages to S3 - using AWS Kinesis Data Firehose
In this article, we are going to see how to save the SNS messages to S3 using Kinesis Data Firehose The first step of building a data engineering pipeline in your organization is storing the unorganized data with long retention and partitioning. Therefore storing the...
Kubernetes NGINX Sidecar ReverseProxy - Connection refused Localhost
While using NGINX reverse Proxy as a Sidecar container for your microservice, We often use localhost to connect to the application container on the same POD but it can potentially harm your application availability in certain systems and lead to a lot of 502 or...
Run SQL queries directly on S3 backup of AWS RDS - using Apache Drill
Apache Drill is a revolutionary product that identifies itself as a Schema-free SQL Query Engine for Hadoop, NoSQL and Cloud Storage - While it was often represented or perceived as a tool for Data Engineers or Analysts. It is not just for Data Engineers In this...
Cloudflare Ansible Example - To create, update, delete DNS records
In this article, we will discuss how to use Ansible to automate CloudFlare DNS management. Ansible has modules for almost everything and it includes Cloudflare the popular CDN and DNS provider Since Cloudflare is part of the community.general collection of ansible -...
AWS IMDS - What is Link-Local IP address 169.254.169.254
In the world of networking, IP addresses are crucial for devices to communicate with each other over the Internet. However, not all IP addresses are created equal, and some serve unique purposes. One such IP address is 169.254.169.254, which is commonly referred to as...
Comparing a Kid and AI - Learning AI with real life examples
If you are reading this, am sure you are trying to get into the mystery world of AI and puzzled by the various terminologies of the AI world. AI Model GPT - Generative Pre-trained Transformer LLM - Large Language Model FM - Foundation Model Text Generation Text...
How to Find and Convert Latin1 encoded rows to UTF8 - MySQL
In this brief article, I want to share a few tricks and lessons I have learnt on Converting Latin1 Encoded rows to UTF8 I had a table with a field named username with millions of records and we recently converted that table from latin1 charset to UTF8 First, let's...
Using SOPS with AWS KMS - Encrypt and Decrypt files | Devops Junction
Secret Management is an essential part of infrastructure and DevOps. how do you store your Secrets and Environment variables define how safe your infrastructure is As a good engineering practice, most companies guide their developers to not hard code the secrets and...
AWS S3 LS Examples - List and Browse S3 buckets | Devops Junction
Amazon Simple Storage Service (AWS S3) is a highly scalable and cost-effective object storage service provided by Amazon Web Services (AWS). It allows users to store and retrieve vast amounts of data, making it a popular choice for developers, businesses, and...
EC2 Instance Connect Endpoint - SSH the easy way | Devops Junction
How do you SSH to an EC2 instance which is on a Private Subnet? You would say either of the following options Create a Bastion Host in Public Subnet then SSH to the Bastion Host and then SSH to the EC2 instance in Private Subnet Setting up EC2 instance Connect and...
Articles for DevOps Beginner
Clone EC2 instance using Terraform - How to | Devops Junction
This is a familiar requirement for everyone I believe. How to clone an EC2 instance and create a new instance with the same configuration. While we...
Pulumi AWS Example - Creating VPC, Subnets, EC2, Bastion host
In this article, we are going to see how to set up compute infrastructure consisting of virtual private cloud, subnets, internet gateway, NAT...
Terraform For Each Examples - How to use for_each | Devops Junction
In this post, we’ll try to cover Terraform looping constructs. We’ll cover the looping constructs that specifically work at the resource level. They...
Terraform Create Multiple EC2 with different Configs - for_each and count together
Those who have been using Terraform might already know that you can create multiple numbers of the same resources with the help of count or for_each...
Terraform AWS EC2 user_data example - aws_instance| Devops Junction
In this article we are going to see how to create AWS EC2 instance with user_data. While launching AWS Ec2 instances we might want to add some...
How to Deploy Tomcat on Kubernetes Step by Step
Overview In this post, we are going to see how to deploy tomcat on Kubernetes. How to Deploy Tomcat based web application into Kubernetes. We are...
Terraform AWS Example - Create EC2 instance with Terraform
Terraform, An outstanding and innovative product from hashicorp and it is a leader in Infrastructure as Code tools Segment. Before I go any...
Ansible EC2 Example - Create EC2 instance with Ansible
Infrastructure as Code is getting all attention it deserves and everyone is trying to find their way to the `Completely automated Infrastructure...
Docker network between containers - Docker Networking Example
In this post we are going to see How to establish docker network between containers, How to Link Docker Containers, How to Access one Container from...
Ansible Playbook Examples - Sample Ansible Playbooks | Devops Junction
In this post, we are going to see examples of Ansible playbook and various different modules and playbook examples with various modules and multiple...
Packer AWS Example with Terraform
In this article, we are going to see a quick packer aws example setup and provide the steps to create an AWS Image (AMI) using Packer and we are...
Ansible AD HOC Command Examples - Ansible Cheat Sheet | Devops Junction
Ansible Vagrant Example - Testing Ansible with Vagrant | Devops Junction
In this article, we are going to see how to use Vagrant to create a development of Virtual machines and start practising Ansible. This is A Beginner...