← Back to Home

Jenkins on Kubernetes

Jenkins on Kubernetes
View on GitHub

Tech Stack

Kubernetes (Minikube with VirtualBox driver), Jenkins, SonarQube, Nexus Repository Manager, Maven, Docker, Docker Hub, Ingress Controller (NGINX), Services (ClusterIP, NodePort), PersistentVolumes, PersistentVolumeClaims, GitHub

jenkins CICD

Project Goal

Implement a complete Continuous Integration and Deployment pipeline inside a Kubernetes cluster using Minikube. The goal is to automate code build, quality analysis, artifact versioning, and storage, leveraging Jenkins, SonarQube, and Nexus, all containerized and orchestrated within Kubernetes. This project serves as a practical, local DevOps lab environment to solidify real-world CI/CD workflows using open-source tools and infrastructure-as-code.

jenkins-k8s-architecture

Project Description

This project builds and automates a full DevOps lifecycle with Jenkins as the orchestrator, SonarQube for static code analysis, and Nexus as an artifact repository. All components are deployed in Kubernetes using YAML manifests. The infrastructure was set up in Minikube using the VirtualBox driver, simulating a production-like environment locally.

Jenkins Deployment and Configuration

Persistent Storage Setup
  • Created a PersistentVolume (PV) and PersistentVolumeClaim (PVC) for Jenkins to store job history and plugin data.
  • The volume uses hostPath storage due to the local Minikube setup.
Jenkins Deployment
  • Defined a jenkins-deployment.yaml with resource limits, volume mounts, and container port exposure.
  • Used a Docker image with OpenJDK 17 and Maven 3.9 pre-installed, along with Docker CLI.
  • Mounted JDKs into /opt/jdks to allow Jenkins to switch between JDK 11, 17, and 21 within the same pod.
Service Exposure
  • Created a ClusterIP service for internal access.
  • Exposed externally via Ingress using path-based routing at /jenkins.
Jenkins Configuration
  • Installed essential plugins: Git, Maven Integration, SonarQube Scanner, Nexus Artifact Uploader.
  • Configured credentials and global tools (Maven, JDKs) via the Jenkins UI.
  • Created a Freestyle job and later migrated to a Declarative Pipeline using a Jenkinsfile.

SonarQube Deployment and Integration

Database Setup
  • Deployed PostgreSQL in a separate pod as a dependency for SonarQube.
  • Used sonarqube-db-secret.yaml for secure credentials.
  • Created a PVC for persistent database storage (sonar-postgres-pvc.yaml).
SonarQube Deployment
  • Used the official sonarqube:10.2-community image.
  • Applied system tuning parameters (vm.max_map_count) using an init container.
  • Mounted a PVC to persist SonarQube data.
Service Exposure
  • Exposed via ClusterIP and routed externally through Ingress at /sonarqube.
Jenkins Integration
  • Configured a webhook in SonarQube to notify Jenkins about Quality Gate results.
  • Set up SonarQube Scanner in Jenkins with project key and authentication token.
  • Added a sonar-scanner stage in the Jenkins pipeline to analyze code and prevent artifact deployment if the Quality Gate fails

Nexus Repository Deployment and Usage

Nexus Deployment
  • Deployed Nexus using a single nexus.yaml manifest with volume mounts and required environment variables.
  • Created a PVC to persist repository data.
Service Exposure
  • Exposed via ClusterIP and routed externally using Ingress at /nexus.
Initial Setup
  • Retrieved the admin password from /nexus-data/admin.password.
  • Configured a Maven repository (vprofile-repo) for internal artifact storage.
Pipeline Integration
  • Installed the Nexus Artifact Uploader plugin in Jenkins.
  • Added a pipeline stage to upload .war files to Nexus using Jenkins credentials and the Nexus service domain.

Pipeline Flow and Implementation

Jenkinsfile Pipeline

The pipeline includes the following stages:

  1. Fetch code from GitHub
  2. Build using Maven (skip tests)
  3. Run Unit Tests
  4. SonarQube Static Code Analysis
  5. Quality Gate Check
  6. Docker Image Build
  7. Docker Image Push to Docker Hub
  8. Upload .war to Nexus
  9. Deploy Minikube
Code Analysis Integration
  • SonarQube performs code linting, bug detection, code smells, and vulnerability scanning.
  • If the Quality Gate fails, the pipeline stops before deploying artifacts.
Artifact Handling
  • Artifacts are versioned using build metadata (BUILD_ID, BUILD_TIMESTAMP).
  • .war files are uploaded to Nexus under /vprofile-repo/QA/vproapp/.

Ingress and Networking Setup

  • Installed the NGINX Ingress Controller in the Minikube cluster.
  • Defined a shared devtools-ingress.yaml to route paths /jenkins, /sonarqube, and /nexus.
  • Edited /etc/hosts on the host machine to map jenkins.local, sonarqube.local, and nexus.local to the Minikube IP.
  • Ensured session persistence and proper HTTP path handling for Jenkins and other tools.

Automation and Debugging Tools

  • Used kubectl apply -f with the correct order to deploy resources: Secrets → PVCs → Deployments → Services → Ingress.
  • Verified resource status using kubectl get, describe, and logs.
  • Temporarily exposed NodePorts for browser access during development.
  • Enabled Ingress logging and used NGINX logs for troubleshooting.

Future Improvements

  • Add a dedicated Jenkins agent to reduce load on the main controller pod.
  • Migrate Nexus and SonarQube data to a more advanced PersistentVolumeProvisioner.
  • Use a Kubernetes Job to deploy the final Docker image to a live application pod.
  • Implement post-build notifications via Slack or email.

Outcomes

  • Built and deployed a full CI/CD workflow in a local Kubernetes cluster.
  • Practiced using Ingress and Services to expose multiple tools under a single IP.
  • Integrated Jenkins with SonarQube and Nexus using plugins and credentials.
  • Strengthened understanding of build, test, and deploy stages in CI/CD pipelines.
  • Simulated a real-world DevOps setup using only local resources and open-source tools.