REANA at CMU T3¶

For Admins¶

For this test, we will deploy REANA on a single machine using the local installation. The reason is because, if properly deployed REANA in the machine, it will take all the resources, and for the first test, we want to be able to share the machine with other users.

Steps to install dependencies¶

```bash sudo apt-get update

* Install Docker: 
Kubernetes requires a container runtime to run containers. Docker is a popular choice.

1. Install Docker:

```bash
sudo apt-get install -y docker.io

2. Enable Docker to start on boot:

sudo systemctl enable docker
sudo systemctl start docker

Install kubectl: The command-line tool for interacting with the Kubernetes API server to manage and control the cluster.
Add the Kubernetes signing key:

sudo curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.32/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg

2. Add the Kubernetes repository:

echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.32/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list

3. Update the package list again:

sudo apt-get update

4. Install kubectl:

sudo apt-get install -y kubectl

Install kind: kind is a tool for running local Kubernetes clusters using Docker container “nodes”. kind was primarily designed for testing Kubernetes itself, but may be used for local development or CI.

[ $(uname -m) = x86_64 ] && curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.27.0/kind-linux-amd64
chmod +x ./kind
sudo mv ./kind /usr/local/bin/kind

Deploy REANA¶

Follow the instructions from the website.

wget https://raw.githubusercontent.com/reanahub/reana/maint-0.9/etc/kind-localhost-30443.yaml
kind create cluster --config kind-localhost-30443.yaml

Only for the CMU T3: We want to stored the workspaces in a different disc in the host. Then, modify the kind-localhost-30443.yaml:

apiVersion: kind.x-k8s.io/v1alpha4
kind: Cluster
nodes:
  - role: control-plane
    extraPortMappings:
      - containerPort: 30443
        hostPort: 30443
        protocol: TCP
    kubeadmConfigPatches:
      - |
        kind: InitConfiguration
        nodeRegistration:
          kubeletExtraArgs:
            node-labels: "ingress-ready=true"
    extraMounts:
      - hostPath: /mnt/scratch/reana/workspaces
        containerPath: /var/reana
        propagation: HostToContainer
      - hostPath: /cvmfs
        containerPath: /cvmfs
        propagation: HostToContainer

Continue with the installation

wget https://raw.githubusercontent.com/reanahub/reana/maint-0.9/scripts/prefetch-images.sh
sh prefetch-images.sh
helm repo add reanahub https://reanahub.github.io/reana
helm repo update
helm install reana reanahub/reana --namespace reana --create-namespace --wait

wget https://raw.githubusercontent.com/reanahub/reana/maint-0.9/scripts/create-admin-user.sh
sh create-admin-user.sh reana reana john.doe@example.org mysecretpassword

How to monitor the cluster¶

In the terminal, use k9s. To install it in Ubuntu:

wget https://github.com/derailed/k9s/releases/download/v0.32.7/k9s_linux_amd64.deb && apt install ./k9s_linux_amd64.deb && rm k9s_linux_amd64.deb

and then just type: k9s.

Install Metrics Server for Resource Monitoring¶

To see CPU and memory usage in k9s, you need to install the metrics server:

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

For Kind clusters, you may need to add the --kubelet-insecure-tls flag. Create a patch file:

cat <<EOF > metrics-server-patch.yaml
spec:
  template:
    spec:
      containers:
      - name: metrics-server
        args:
        - --cert-dir=/tmp
        - --secure-port=4443
        - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
        - --kubelet-use-node-status-port
        - --metric-resolution=15s
        - --kubelet-insecure-tls
EOF

Apply the patch:

kubectl patch deployment metrics-server -n kube-system --patch-file metrics-server-patch.yaml

Wait for the metrics server to be ready:

kubectl wait --for=condition=available --timeout=300s deployment/metrics-server -n kube-system

Verify it's working:

kubectl top nodes
kubectl top pods -A

Now in k9s, you can:

Use :top pods to see real-time CPU/memory usage
Use :top nodes to see node-level metrics
Press Ctrl+D in pod view to toggle resource columns

To change setting in the REANA cluster¶

The easiest way is using helm. First install this plugin repo:

helm plugin install https://github.com/databus23/helm-diff

Then create a values.yaml file and modify what you need. The helm chart (list of parameters) is here

Then check the diff and upgrade the changes:

``bash helm diff upgrade reana reanahub/reana -n reana -f values.yaml --version 0.9.4 helm upgrade reana reanahub/reana -n reana -f values.yaml --version 0.9.4

This is the values.yaml that worked on the CMU cluster:

```yaml
compute_backends:
  - kubernetes
kubernetes_jobs_memory_limit: 2Gi
kubernetes_jobs_max_user_memory_limit: 10Gi

workspaces:
  retention_rules:
    maximum_period: 365
    cronjob_schedule: "0 2 * * *"

components:
  reana_workflow_controller:
    environment:
      IMAGE_PULL_SECRETS: "my-cern-gitlab-secret,my-dockerhub-secret"
      REANA_JOB_HOSTPATH_MOUNTS: '[{"name": "cvmfs", "mountPath": "/cvmfs", "hostPath": "/cvmfs"}]'

How to check the helm chart template¶

helm template reana reanahub/reana -n reana -f values.yaml --version 0.9.4

Configure private registries¶

Follow the instructions here.

In your local machine (or wherever you have your docker and gitlab registry accounts) loging into your docker and follow the instructions:

docker login  ### this will login into dockerhub
docker login gitlab-registry.cern.ch   ### this into gitlab

These two commands will create a ~/.docker/config.json file. You need to split them, one for each registry, i.e config_gitlab.json and config_dockerhub.json. If you are outside the cluster, copy that file into your area using scp, and then in the cluster you need to add it as a secret, like:

kubectl create secret generic my-cern-gitlab-secret --from-file=".dockerconfigjson=config_gitlab.json" --type="kubernetes.io/dockerconfigjson" -n reana
kubectl create secret generic my-dockerhub-secret --from-file=".dockerconfigjson=config_dockerhub.json" --type="kubernetes.io/dockerconfigjson" -n reana

Finally, add into your values.yaml file (or create one if you haven't) the following:

components:
  reana_workflow_controller:
    environment:
      IMAGE_PULL_SECRETS: "my-cern-gitlab-secret,my-dockerhub-secret"

and run:

helm upgrade reana reanahub/reana -n reana -f values.yaml --version 0.9.4

Add reana-admin-access-token¶

First check if this value exist:

kubectl get secret reana-admin-access-token -n reana -o jsonpath='{.data.ADMIN_ACCESS_TOKEN}' | base64 --decode

If not, create a token and encoded it in base64:

ACCESS_TOKEN=$(openssl rand -hex 32)
echo "Generated Token: $ACCESS_TOKEN"  # VERY IMPORTANT: Verify the token
BASE64_TOKEN=$(echo -n "$ACCESS_TOKEN" | base64 | tr -d '\n')
echo "Base64 Encoded Token: $BASE64_TOKEN"

And include it in the ADMIN_ACCESS_TOKEN variable when modifying:

kubectl edit secret reana-admin-access-token -n reana

Check if the ADMIN_ACCESS_TOKEN contains 64bytes.

kubectl describe secret reana-admin-access-token -n reana

Cleanup Instructions¶

To remove the current REANA deployment and start from scratch, follow these steps:

Delete the REANA Helm Release:

helm uninstall reana -n reana

Delete the Kubernetes Namespace:

kubectl delete namespace reana

Delete the Kind Cluster:

kind delete cluster --name kind

Remove Temporary Files:

rm -f kind-localhost-30443.yaml prefetch-images.sh create-admin-user.sh

Clear Helm Cache:

helm repo remove reanahub

Remove Docker Images:

docker images | grep reana | awk '{print $3}' | xargs docker rmi -f

Clear Kubernetes Configurations:

rm -f ~/.kube/config

Remove Kind Configuration:

rm -rf ~/.kind

Verify Cleanup:

kubectl get all --all-namespaces
docker ps -a
docker images

After completing these steps, you can redeploy REANA following the instructions above.