drumm.sh: Recent Blogs RSS Feed

Grievt – Continued Collaboration

Wed, 13 Nov 2024 00:00:00 GMT

A short backstory

Grievy is an app that serves as a digital companion after the loss of a loved one. It was developed through a collaboration between Nele Stadtbäumer, who came up with the idea, Professor Drumm, who facilitated the project, and a team of six students from Fachhochschule Aachen, who turned Nele’s ideas into a software product. The idea had such momentum, that Nele and two of the students founded a startup and refined their idea and product. Some features, including support with bureaucratic tasks, had to be dropped in this refinement process. Now, two years after the launch of grievy, the startup has decided to work together with FH Aachen again, to realize these organizational features in a sister app for grievy. Grievt gives users handy advice surrounding funerals and other topics, an editable checklist of things that need to be done after the death of a loved one, and a feature for automatically generating letters of termination for different contracts.

Forming – Storming – Norming…

Within Scrum, you need a product owner, a scrum master and a development team. Daniel Bachmann, one of the students who worked on grievy and went on as cofounder of the startup, now took the role of product owner, and Professor Drumm took the role of Scrum Master. For the development team, six students from FH Aachen were chosen, but after the first two weeks of the semester, only four of us remained. This worried us in the beginning, but with time we came to appreciate working in a small team. While many things were new for us, we benefitted from the fact that Daniel and Professor Drumm had not only worked on this kind of project before but also worked together before.

In any kind of group work, there needs to be time spent storming and norming between group members, before the stage of performing is entered. In the end, the size of our team and the previous experience of Daniel and Professor Drumm helped us become productive quite fast.

What we learned

Working on grievt provided invaluable learning opportunities. We gained experience using Typescript and the Ionic framework, Github, SAST tools and developing and testing for the Apple appstore and Google playstore, and most importantly working with Scrum.

In the second half of the semester, Herr Drumm started tracking our accuracy and velocity, teaching us about the usual trajectory of these measures and emphasizing the importance of correct evaluation and the advantage of well-written user stories. There were times when we believed we would finish a particular user story within a sprint, but some details kept us pulling our hair. Within the last two weeks of the project, a lot of bug fixing occurred and last-minute amendments were made. Suddenly, issues that took us hours only a month ago were solved in a fraction of that time. We had gotten to know the technology and our app so much better that we could come up with solutions quicker and more creatively.

The App

At the end of January, we showed the final result of our app to our peers in a live presentation. It was exciting to take a step back and see the full picture of what we worked on closely all semester. While app store deployment still awaited review at this time, grievt became available to the general public two weeks later. This marked the success and endpoint of our project. Today, when I navigate to grievt in the app store, it notifies me of the fact that I am an internal tester, a sweet reminder of the work I put into the app.

Using SAP AI Services in Python 🐍

Sun, 16 Oct 2022 00:00:00 GMT

Between April and June 2022 my colleague Stephan and I taught a Python MOOC on the openSAP platform. The course was named Python for Beginners. It was very popular with over 40.000 participants and very well received. Many participants asked the following questions in the forum:

Why is SAP teaching Python
How can Python be used in the SAP ecosystem
What is a possible next step for course participants

In this blog post I try to answer the second and third question in the list. I will show one possibility of using Python in the SAP ecosystems. In particular, I choose a topic which does not require any prior expertise in the SAP ecosystem.

SAP and Python

Today, a search for Tutorial related to Python in the tutorial section of the SAP Developer Center returns 18 tutorials from a variety of areas. There is even a tutorial showing how to set up a connection between SAP HANA and a Juypter Notebook. From the different available option I choose the SAP AI Business Services as the basis for this blog post. The reason being the these services are easy to consume and do not require any expertise with other SAP products like SAP ERP or SAP S/4HANA.

SAP AI Business Service

The SAP AI Business Services are a collection of six services:

Business Entity Recognition
Data Attribute Recommendation
Document Classification
Document Information Extraction
Personalized Recommendations
Service Ticket Intelligence.

The remainder of this blog post will use the Business Entity Recognition service as an example. Using the content form this blog post it should easily be possible to use the other services as well.

So what exactly is Business Entity Recognition? The documentation states that Business Entity Recognition is used to:

Detect and highlight entities in unstructured text using machine learning.

Given the following plain text as input:

Christian Drumm works at the FH Aachen.
The FH Aachen is located at the Eupener Str. 70 in Aachen, Germany"

The Business Entity Recognition service could, for example, identify the following concepts:

Address: Eupener Str. 70, Aachen, Germany
Person: Christian Drumm
Organization: FH Aachen

Exactly which concepts can be identified depend on the variant of the Business Entity Recognition service used. A discussion of the differences and possibilities is beyond the scope of this post and can be found in the documentation.

For testing the Business Entity Recognition service two possibilities exist. Either a public sandbox can be used or a free trail of the service. I would recommend using the free trial. While it requires some initial set up (which is nicely described in this tutorial) it has a much better performance then the sandbox system. Depending on when you try to use the service the sandbox sometimes requires waiting several minutes for results.

SAP API Business Hub

The first step to testing the Business Entity Recognition service is to understand the API of the service. All APIs offered by different SAP products are listed on the SAP API Business Hub. The SAP API Business Hub provides two main features:

discovering available APIs
documentation and test functionality for the APIs.

Important notice: In order to follow use the test functionality it is necessary to register and login to the SAP API Business Hub. Otherwise some of the functionality is not available.

API of Business Entity Recognition Service

The image below shows the overview page of the API of the Business Entity Recognition Service on the API Business Hub.

A detailed specification of the API is available on the API Reference page. The Try Out page enables the testing of the service directly in the SAP API Business Hub.

The API Reference page show which operations are offered by the Business Entity Recognition service. The service offers operations related to:

Datasets
Training
Models
Deployments
Inference.

The operations related to datasets, training, models and deployments are relevant when training own models. For testing the service these operations are not necessary. Only the operation related to inference are used in the following.

Testing the API

In order to understand the Business Entity Recognition service I will not describe how to test the service in the SAP API Business Hub. Instead I will explain how to test the API using HTTPie (other options to test the API are tools like Postman or the Rest Client for Visual Studio Code).

Although HTTPie provides a Web-based tool and a native app I will be using the CLI client in the following. The reason for this is that #TheFutureIsTerminal 😉.

Testing the Inference of the Business Entity Recognition Service

In order to test the inference of the Business Entity Recognition service the following steps are necessary:

Getting an OAuth access token for the service
Posting the inference data
Getting the result of the inference.

Each of this steps is described in detail in the following sections.

Get an OAuth Access Token

Describing the details of OAuth is beyond the scope of this blog post. This video provides a nice introduction to the topic.

The documentation describes the steps necessary to get an OAuth bearer token. In order to get a token an HTTP GET request needs to be send to the path /outh/token. In order to get the bearer token the parameter grant_type needs to be set to client_credentials.

The important question is now where to send the GET request. In the SAP BTP this information is contained in the service key of the respective service. To find the service key navigate to Instances and Subscriptions. Select your instance there and open the details of the service key.

The service key is in JSON format ans should look similar to this

{
    "url": "https://business-entity-recognition.cfapps.eu10.hana.ondemand.com",
    "swagger": "/api/v1",
    "uaa": {
        "clientid": <your-client-id>,
        "clientsecret": <your-client-secret>,
        "url": "https://<your-instance-id>.authentication.eu10.hana.ondemand.com",
        ...
    }
}

Using the URL given in the uaa part of the service key a bearer token can be requested. Using HTTPie the GET request looks like this:

http -a <your-client-id>:<your-client-secret> \
"https://<your-instance-id>.authentication.eu10.hana.ondemand.com/oauth/token?grant_type=client_credentials"

In order to save some typing I usually store the client id and client secret in en environment variable. But this is not required. Furthermore, it might be necessary to surround the values with single quotes '. If everything works the response should look something like this:

HTTP/1.1 200 OK
....
{
    "access_token": "eyJhbGciOiJS...",
    "expires_in": 43199,
    "jti": "27294b49bf47456380850bcf68ccaa04",
    "scope": "uaa.resource ner-production!b30772.default",
    "token_type": "bearer"
}

The important part of the response is the access token. This very long token is used to invoke the different operations of the Entity Recognition service.

Posting the Inference Data

Recognising entities using the Business Entity Recognition service is a two step process. First the data is uploaded to the service. The service performs the recognition of the entities asynchronously. After the recognition is complete, the result can be retrieved. Therefore, the next step is now to upload the inference data. For this blog post I will use the following text as inference data:

Christian Drumm works at Eupener Str. 70 in 52072 Aachen, Germany.

To upload this data to the Business Entity Recognition service a POST request to the path /inference/jobs needs to be executed. The URL for the service can again be found in the service key. The URL consists of the two parts url and swagger. Authentication for the service is performed using the access token retrieved in the previous step.

The actual data is uploaded as an JSON object. HTTPie automatically build a JSON object form the provided parameters.

In addition it is required to define which inference model should be used. In this example the sap_address_entity is used.

So in summary the data can be uploaded using the following command.

http -A bearer -a <your-bearer-token> POST \
https://business-entity-recognition.cfapps.eu10.hana.ondemand.com/api/v1/inference/jobs \
text="Christian Drumm works at Eupener Str. 70 in 52072 Aachen, Germany." \
modelName="sap_address_entity"

If everything works the service returns a response like the one below:

{
  "data": {
    "freePlanUsage": {
      "inferenceCharactersUsage": "Maximum limit of 120000 , utilized 471 Inference characters and remaining 119529",
      "inferenceRequestsUsage": "Maximum limit of 30 , utilized 3 Inference count and remaining 27"
    },
    "id": "e74fbde9-11e7-4e04-94db-dbf56768ec4e",
    "message": "Inference job has been submitted",
    "modelName": "sap_address_entity",
    "modelVersion": null,
    "status": "PENDING"
  }
}

The response show that the inference job has been submitted. The ID of the job is returned in the id.

Getting the results

In order to get the result of the inference service a GET request is executed to the path /inference/jobs/<job-id>. The job id in this path is the job id from the previous service. So in order to get the result the following HTTPie command can be used.

http -A bearer -a <your-bearer-token> \
https://business-entity-recognition.cfapps.eu10.hana.ondemand.com/api/v1/inference/jobs/<job-id>

Below is the result of this request.

{
  "data": {
    "createdAt": "2022-10-14T10:26:45Z",
    "freePlanUsage": {
      "inferenceCharactersUsage": "Maximum limit of 120000 , utilized 603 Inference characters and remaining 119397",
      "inferenceRequestsUsage": "Maximum limit of 30 , utilized 5 Inference count and remaining 25"
    },
    "id": "684ad559-b42f-462a-b865-65958e1c2865",
    "modifiedAt": "2022-10-14T10:26:58Z",
    "result": [
      {
        "cityTownVillage": [
          {
            "confidence": 1.0,
            "value": "Aachen"
          }
        ],
        "country": [
          {
            "confidence": 0.9,
            "value": "Germany"
          }
        ],
        "customerName": [
          {
            "confidence": 0.92,
            "value": "Christian Drumm works"
          }
        ],
        "district": [],
        "houseNumber": [
          {
            "confidence": 0.99,
            "value": "70"
          }
        ],
        "stateProvince": [],
        "street": [
          {
            "confidence": 1.0,
            "value": "at Eupener Str"
          }
        ],
        "zip": [
          {
            "confidence": 1.0,
            "value": "52072"
          }
        ]
      }
    ],
    "status": "SUCCESS"
  }
}

The status in line 52 show that the inference job completed successfully. In addition each identified entity is returned together with confidence value. For example, line 31 ff shows that the concept house number has been identified with a confidence of 0.99 and the value of the house number is 70. Line 44ff shows that the concept zip has been identified with a confidence of 1.0 and the value is 52072.

Using the Service in Python

After using HTTPie to invoke the service the next step is to build a simple Python program to do the same. The following listing contains a simple Python program executing the previous steps.

import requests
import time
import json

# Constants
CLIENTID = "<your-client-id>"
CLIENTSECRET = "<your-client-secret"

AUTH_SERVICE = "https://<your-instance>.authentication.eu10.hana.ondemand.com/oauth/token?grant_type=client_credentials"
BER_SERVICE = "https://business-entity-recognition.cfapps.eu10.hana.ondemand.com/api/v1/inference/jobs"

#Get input from the use
user_input = input("Please enter a text: ")

# Step 1: get the bearer token
authentication = requests.get(AUTH_SERVICE, auth = (CLIENTID, CLIENTSECRET))

# Check if we got an authentication token
if authentication.status_code == 200:

    #Step 2: Submit the inference job
    payload = {"text": user_input, "modelName": "sap_address_entity"}
    headers = {"Authorization": "Bearer " + authentication.json()["access_token"]}

    submitted_job = requests.post(BER_SERVICE, json=payload, headers=headers)

    #Step 3: Wait until the job is finished
    job_finished = False
    while not job_finished:
        job_result = requests.get(BER_SERVICE + "/" + submitted_job.json()["data"]["id"], headers=headers)
        if job_result.json()["data"]["status"] == "SUCCESS":
            job_finished = True
            print("Job finished")
        else:
            print(".", end="", flush=True)
            time.sleep(2)

    # Print the job results
    format_jason = json.dumps(job_result.json(), indent=2)
    print(format_jason)

To call the service in Python the requests library is used. This library is imported in line 1. Lines 6 - 10 defines a few constants. The values <your-client-id>, <your-client-secret> and <your-instance> need to be replaced with the values for your service. Line 13 get the text from the user to be analysed.

The first step (line 16) is to get the bearer token using the service URL. The authentication for this service is performed using the client id and client secret. If the request was successful (line 19) the POST request is created in lines 22 - 25.

The while loop in lines 28 - 36 checks the job result (line 30) until the status is "SUCCESS". If the job ended successfully the result is formatted using the json library (line 39) and printed.

Summary

I hope this blog post showed how easy it is to consume SAP service in Python. While I used the Business Entity Recognition service, using other services works using the same approach. So now you are able to apply the Python knowledge from the Python for Beginners course in the SAP ecosystem.

If you have any question please feel free to ask them in the comments.

Christian

Blog Posts from the CBIS Lecture

Mon, 07 Mar 2022 00:00:00 GMT

This post contains pointers to the different blog posts written by students during the lecture Cloud-based Information Systems in the winter term 2022. Some of the blog posts are hosted on this Web site, some on external sites.

The table below contains the links to the blog posts as well as the names of the authors.

Topic	Students
How to use REST-APIs in PowerApps	Ben Pauly, Fabian Lösch
Power FX Tutorial	Marvin Lützeler
Share-Point Online	Vijitha Susilakumar, Artur Sichwardt
oee.ai and PowerApps	Anas Kudaish, Heram Kritharan, Wesam Habib
PDF Creator in Power Apps	Anton Zajcev, Fabian Kirchhoff
How to compare two Excel files using Power automate Desktop	Ali Ayadi, Hassen Trabelsi
Power Automate und Office-Skripte	Muteba Kayembe, Ahmed Wafi
Cloudbasierte Informationssysteme – DKV Mobility	Evangelos Sarantis, Martin Nolte
Webhooks	Dorian Loeben

Production order and confirmation integration – a university project

Tue, 01 Mar 2022 00:00:00 GMT

Production order and confirmation integration – a university project

This blog post is part of a series of posts, describing an integration project that connects the oee.ai Manufacturing Intelligence Platform with the production planning capabilities of an on-premise S/4HANA System. The project was realized by students from the FH Aachen University of applied Sciences, with great support from project partners at oee.ai and SAP.

The blog post is available on the SAP Community.

From semester project to start-up

Sat, 13 Nov 2021 00:00:00 GMT

The beginning of the project

As part of the interdisciplinary project, I had the opportunity to get to know many different interesting projects. From the beginning, grievy particularly caught my eye. On the one hand, I had a direct connection to the idea of grievy, as a digital accompaniment after a death, since I had to mourn a death in my family a few years ago. On the other hand, grievy was a start-up with an innovative idea and a young, interesting team. With a little luck of the draw (I was not the only one who found grievy very interesting) I made it into the team of my wishes.

The realization of the project

Together with 5 fellow students I was part of the developer team, while Nele acted as product owner and Christian Drumm was the scrum master. Since none of us knew anything about programming an app, it took a while until we found our way into the project. But with each sprint we became more productive, organized and efficient. We also got to know each other better and improved our collaboration more and more. At the end of the project, we had a very good product, which we were able to present to the other teams in a final presentation.

What I learned

In addition to my programming skills, I was also able to improve collaboration with other project participants. Furthermore, I was able to enhance my self-organization and time management. Last but not least, the project taught us the agile Scrum method, as we used it every week. For the changing requirements within the project, this method was optimal.

What happened next

I was very pleased with the way the project went. The result was excellent and I also had a lot of fun during it. A face-to-face final event, which unfortunately couldn't take place due to the pandemic, would have been the cherry on the cake.

But the project should not be over for me yet. Nele, the product owner in the project and founder of grievy, was so happy with the result of our collaboration that she asked me and a colleague if we would like to continue working on grievy in the long term. I didn't have to think long, for me grievy is a great opportunity and I have fun working on it. So our partnership continues.

Where we stand today

Since the end of the interdisciplinary project, we have continued to work on grievy. Aenis, my colleague and friend, and I are responsible for the development of the app and all the technical aspects around it. Together with Nele we have further developed the app itself and grievy in general.

In October we finished the first test version of our app. A closed group of testers was able to put the app under the microscope. Among the testers were funeral directors, grief counselors, software developers and of course the original team from the interdisciplinary project. The response was very positive and we received a lot of praise. But of course there are still things we need to correct and optimize. We are very grateful for the positive feedback and constructive criticism. At the moment we are working on implementing the feedback and completing the app. We are very excited to finally release the app next year.

As you can see, a lot has changed at grievy since the prototype from the interdisciplinary project. The most work for me was programming and continuously improving the app. But there was also a lot to do organizationally. We applied for scholarships, successfully participated in competitions for founders, planned the long-term development of grievy, and much more.

In addition to our work as software developers for grievy, Aenis and I, together with Nele, will also be involved in the start-up as co-founders. To put it in a nutshell, grievy and the interdisciplinary project are a great opportunity for me and I look forward to the journey ahead.

Contact us

More Bitwarden CLI Shell Scripting

Tue, 31 Aug 2021 00:00:00 GMT

In my previous blog post I started to explore shell scripting by adapting the Bitwarden CLI to my needs. Luckily, DJ and markhepburn also wrote blog posts on the topic (cf. here and here). Based on their blog posts I tried a few more ideas. Finally, I was able to improve my initial shell script a bit (at least in my opinion 😉).

Additional Requirements

After reading DJ's and Mark's blog post I added the following requirements to my initial list.

If a TOTP token

is available for an entry, my custom CLI should also copy this token.

Simply my shell script by using some to the features DJ used in his script.
Distinguish between different username / passwords for the same Web site.

These requirements didn't look like much work. However, especially the first one required me to change the approach I took in the shell script.

My New Solution

Before going into more details here is the resulting script. The complete script is also available on GitHub.

#!/bin/zsh

set -e

copy_data () {
  local id=$1
  local login=$2
  local sessionkey=$3
  local totp

  echo "Name: $(jq -r ".name" <<< $login)"

  # Copy the username to the clipboard
  echo "> Copying Username"
  jq -r ".login.username" <<< $login | pbcopy

  # Wait for user input before coping the password
  echo "> Press any key to copy password..."
  read

  # Copy the password to the clipboard
  echo "> Copying Password"
  jq -r ".login.password" <<< $login | pbcopy

  # Copy a TOTP Token if available
  totp=$(jq -r ".login.totp" <<< $login)

  if [[ $totp != "null" ]]; then
    # Wait for user input before coping the totp token
    echo "> Press any key to copy TOTP Token..."
    read
    echo "> Copying TOTP Token"
    bw get totp $id --session $sessionkey | pbcopy
  fi
}

main() {
  local searchterm=$1
  local sessionkey logins login id

  #Unlock the vault an store the session key
  sessionkey=$(bw unlock --raw)

  # Search for passwords using the search term
  logins=$(bw list items --search $searchterm --session $sessionkey)

  id=$(jq -r '.[] | "\(.name)\t\(.login.username)\t\(.id)"' <<< $logins \
    | fzf --reverse --with-nth=1,2 --delimiter="\t" --select-1 --exit-0 \
    | awk -F"\t" '{print $3}'
  )

  if [[ -n $id ]]; then
    login="$(jq ".[] | select(.id == \"$id\")" <<< $logins)"
    copy_data $id $login $sessionkey
  fi
}

main "$@"

Main Function

The first change I implemented is to introduce a main function (cf. lines 37 – 56). This idea I copied directly from DJs blog post. The execution of the script starts in line 58 by invoking this main function. The special parameter $@ is used to pass all parameters of the script to the main function.

Besides that, I also qualified all the variables with local (cf. lines 6 – 9 and 38 – 39). Also coping the positional parameters (e.g. $1 or $2) to a local variable at the beginning of the function seems to be a good idea (cf. lines 6 – 8).

Here String <<<

In the previous blog post I described how I ran into some problems due to the handling of special characters like \n by echo. My first solution was to use printf instead. In DJ's blog post I learned about here strings. I used these in the new version of the script as well (e.g. lines 15, 23, 26 or 47).

However, currently I'm unsure if I like the syntax of here strings or of printf better. The advantage of here strings is conciseness. The disadvantage in my point of view is, that suddenly line can't be read from left to right anymore. For example, line 15 could be nicely read from left to right using printf.

# Copy the username to the clipboard
printf "%s" "$login" | jq -r ".login.username" | pbcopy

With the here string I need to read until the <<< in order to understand where the input of jqis coming from. Nevertheless, I stuck with the here strings for the new version of the script just to get used to them.

# Copy the username to the clipboard
jq -r ".login.username" <<< $login | pbcopy

fzf-Features

In the initial script I used a conditional to distinguish between the situations that one or many results are returned by the search. This complexity is unnecessary and was removed in the current version of the script. Instead, the parameter --select-1 is used to tell fzf to immediately return when only one element was passed.

In order to distinguish between logins with the same name and different passwords, used the --delimiter parameter DJ used in his blog post.

In the script the name, the login name and the id of the entries are passed to fzf separated by a tab. This input is generated using jq string interpolation). For each entry returned by bw list items in line 45 a string consisting of the name, a tab, the username, a tab and the id is generated.

jq -r '.[] | "\(.name)\t\(.login.username)\t\(.id)"' <<< $logins

The --with-nth=1,2 parameter tells fzf to only display the first two elements of the input. The resulting invocation of fzf is shown in line 48. The following screenshot shows the resulting user interface when the search returns multiple entries for the same system with different user names.

More Bitwarden CLI

The major changes of the new version of my script are related to the Bitwarden CLI

Whenever the password vault is locked and a command like get or list is executed, the Bitwarden CLI asks for the password to unlock the vault. In my initial script this was not a problem. It only accessed the fault once using bw list <searchterm>. However, the JSON data returned by this command doesn't contain the current TOTP token for an entry. To get a TOTP token bw get totop <ID> needs to be executed with the ID of a vault entry. As I don't want to enter my password multiple times for one entry in the password vault the script needs to use the log in functionality of the Bitwarden CLI.

This log in functionality consists of two steps. First, the password vault is unlocked in line 42 using bw unlock. This command requires entering the password to unlock the vault. If the correct password is entered, a session key is returned. In my script this session key is then stored in the local variable sessionkey. For all subsequent invocations of the Bitwarden CLI (cf. line 45 and 33), the session key is passed to the command using the --session parameter.

Final Result

The following GIF shows the new script in action.

Summary

Using what I learned from DJ I was able to improve my initial script quite a bit. Right now I'm pretty happy with the result. Let's see if I'll continue with more shell scripting in the future.

Christian

Adapting the Bitwarden CLI with Shell Scripting

Wed, 25 Aug 2021 00:00:00 GMT

Bitwarden has been my password manager of choice since it was recommended to me by Frank about two years ago. I like most of its functionality. However, one thing kept annoying me. To be honest this was already the case with my previous password manager.

The integration of Bitwarden into the browser is great. When I'm not in the browser the interaction with Bitwarden often felt cumbersome. For example, I quite often work or in a virtual machine running Windows¹ or in the terminal in full screen mode. In order to get a password from Bitwarden when I'm not in the browser I need to do the following steps:

Switch to the browser
Click on the Bitwarden icon
Enter the master password (in most of the cases)
Enter a search string
Click on the icon to copy the password
Switch back to my work environment.

Especially, when working in the terminal, the need to suddenly switch to using the mouse annoyed me.

Bitwarden CLI

One of the nice features of Bitwarden is, that there is also a command-line interface available. I had a look at this CLI a few weeks back, but I did not find it useful. With the ID of an entry in my vault the Bitwarden CLI would allow me to look up the password using

bw get password <ID>

In order to get the necessary ID I could use

bw list items <search-string>

However, the bw list items command returns a JSON string containing all the information of the entries matching the <search-string>. In particular, I didn't see an easy way to get the ID from the results and use it to get the password. So after playing with the CLI for about half an hour, I gave up.

Shell Scripting to the Rescue

A few days ago I read the blog post Bringing the Unix Philosophy to the 21st Century by Kelly Brazil (found via @qmacro of cause). After reading this post i thought: "hold up, wait a minute". The Bitwarden CLI returns JSON. I have already used command-line tools jq and fzf a bit. It should be possible to adapt the Bitwarden CLI to my needs with those tools.

Of course, being a good software developer, I started with a Web search first. And, surprise, I found a blog post by Ben Aaron Goldberg that describes almost exactly what I wanted to have. Using this blog post as a basis I was sure I could create the CLI for Bitwarden that I needed.

My Requirements

What features did I want to have in my bespoke Bitwarden CLI?

I want to be able to search for usernames and passwords using a search string
If only one entry is found, first the username and after that the password

for the entry should be copied to the clip board.

If more than one entry is found I want to be able to search through the

results and select the entry I was looking for. After that the username and password should again be copied to the clip board.

The following screenshot shows what I had in mind for the CLI.

My Solution

My requirements didn't sound overly complicated. So I stared building a shell script implementing them. This is what I came up with after quite some trial and error. I created a shell script called bwc - a mnemonic for Bitwaren Copy.

set -e

copy_uname_and_passwd () {
  # Print the name of the selected login
  echo "Name: $(printf "%s" "$1" | jq -r ".name")"
  echo "> Copying Username"
  # Copy the username to the clipboard
  printf "%s" "$1" | jq -r ".login.username" | pbcopy
  echo "> Press any key to copy password..."
  # Wait for user input before coping the password
  read
  echo "> Copying Password"
  # Copy the password to the clipboard
  printf "%s" "$1" | jq -r ".login.password" | pbcopy
}


# Search for passwords using the search term
logins="$(bw list items --search $1)"

if [ $(printf "%s" "$logins" | jq ". | length") -eq 1  ]
then
  login="$(printf "%s" "$logins" | jq ".[0]")"
else
  name="$(printf "%s" "$logins" | jq -r ".[].name" | fzf --reverse)"
  login="$(printf "%s" "$logins" | jq ".[] | select(.name == \"$name\")")"
fi

copy_uname_and_passwd $login

Lines 3 - 15 are mostly copied from the blog post mentioned above. The only thing I did was wrapping it in a function that I could reuse.

The execution of the script starts in line 19. The Bitwarden CLI is used to search for items using the search term passed to the script as a parameter. The result is stored in the variable logins.

Next, the length of the resulting JSON array is calculated using jq ". | length". If only one entry is found, this entry is stored in the variable login (line 23). If more than one entry is found the names of these entries are passed to fzf. The name of the selected entry is stored in the variable name. This variable is used line 26 to get the selected login from the result array logins and store it in the variable login.

Finally, the function copy_uname_and_passws is called with the login as a parameter.

The following two GIFs show the script in action. First, with only one result returned from the search:

Second, with multiple results returned by the search:

Be careful with echo

One of the problems I was facing during development of the script was this error message from jq:

parse error: Invalid string: control characters from U+0000 through U+001F must be escaped at line 6, column 13

After lots of trial and error I noticed the problem was the echo I was using in the script. echo does some special handling of characters like \n.

For example, initial line 21 contained the following code:

$(echo $logins | jq ". | length")

Due to the special handling of \n the JSON piped from echo to jq was not valid. Once I changed echo to printf everything worked as expected.

Summary

I now have exactly the Bitwarden CLI I was looking for. Furthermore, creating the shell script showed me, again, the power of the Unix philosophy of having a program do exactly one thing and do this well.

As this is my first shell script I'm sure there my things to improve. I would be happy to hear any suggestions by some of the command-line and shell scripting magicians 🧙‍♀️ out there.

1 This is due to the fact that VMware doesn't have a working version of its proprietary VPN solution for any recent MacOS.↩

Microcontrollers at 3°C

Mon, 23 Aug 2021 00:00:00 GMT

Networking on multiple levels

Ever spent a week at 3°C with microcontrollers and sensors? We have. We are Corinna Aufderheide and Anne Mayer and in this blog post we would like to tell you about our experiences in the lecture "Industrial Production and Industry 4.0" and give you some insights into our last semester.

"Industrial Production and Industry 4.0" is a lecture in the second semester of the 4-semester master's program Industrial Engineering. Beforehand, we were given the task to order some hardware components like a ESP8266 Microcontroller, breadboards and a sensorkit with seemingly endless possibilities. Due to the lack of prior experience with microcontroller and sensor, we were overwhelmed by the variety of the sensors to choose from. During an introductory session Prof. Drumm and Prof. Meinecke gave us various examples of what can be done with the sensors. For example, we learned how to trigger an LED when pressing a button or how to measure temperature and humidity in a room.

The following image shows a few of the components we used in our project.

At this point We were very glad that we took the Python introductory lecture in the previous semester. This enabled us to quickly program the microcontroller using PyCharm and to analyse data using Jupyter Notebooks. In addition, we were also introduced to the AWS IoT Services. These are cloud services provided by Amazon to collect and process large amounts of data.

Industry 4.0

But the question that interested everyone in the course: What do I do with all this in real life?

Industry 4.0 - as the fourth industrial revolution is known - has become an integral part of the industrial context. It is about digitalization and the intelligent networking of machines and processes using internet technologies. The possibilities and potentials that Industry 4.0 holds for industrial production were the central topic of the lecture taught by Prof. Drumm, Prof. Meinecke and Prof. Luft. In addition to the introductory session, we were provided with 12 learning videos on different topics in the context of Industry 4.0. The topics included cyber-physical systems, cloud computing, Big Data, business analytics and the autonomization of production. The underlying question always was: How can industrial production be supported and improved using those technologies?

A real-world use case

Building on this question, we were asked to find a use case for an IoT project. The goal was to investigate a real-world problem by recording data using sensors and microcontrollers, storing the data using AWS IoT service, and then analyzing the data to generate some insights. If possible, the use case should be in an industrial context. This was a big challenge for most of the students due to the pandemic situation. We were lucky and were able to conduct our project together with a company. Through research and several discussions, we came into contact with a logistics service provider that specializes in the storage and transport of frozen goods. For this company we implemented a project in which we examined temperature changes in a cold storage hall for fresh products. For the project microcontrollers were attached to all doors of the cold storage hall as well as to the ceiling. The microcontrollers were connected to 5 motion sensors at the gates and 9 temperature sensors distributed throughout the storage hall at heights of 2 and 8 meters. The images below show the prototypical setup with a temperature sensor and a motion sensor respectively.

Within a week, 2.5 million data points were collected using those 14 microcontrollers and sensors. All this data had to be analysed afterwards.

Some difficult problems

We worked several hours per week on the project, and it was a strain on our nerves. In contrast to the "laboratory conditions" at home, WLAN interruptions, daily resets and inexplicable phenomena in the industrial context were more difficult to analyse and solve on site. But we also learned first-hand what to look out for when carrying out a project in an industrial setting. For example, we learned a lot about data protection, the right choice of sensors and that working in a cold storage hall at 3 °C can be really tough.

We also had to deal with the problems that arose from the people working in the storage hall. For example, we discovered that an infrared sensor also reacted to the body temperatures of the employees behind the gates and provided unusable data for the project. We therefore decided to use ultrasonic sensors instead.

We also had to deal with WLAN dropouts at night. These can be seen in the image below. For some unexplained reason WLAN connection of the microcontroller stopped every night. A software solution was not successful here and the application of a hardware reset was not feasible in the short project phase. Therefore, the only option for us was a manual reset every microcontroller every morning with the help of an employee of the logistics company in order to record suitable data.

During programming and analysis, further problems arose due to the high number of data. All the data stored in AWS had to be cleaned and changed to the same format. This resulted in high run times for the data analysis program. As a result we had to learn about parallel processing in Python on our own to be able to present a descriptive and fast solution.

Our results

After the analysis, we presented our results to the logistics company and made the analysis tool available to them. Based on our analysis the logistics company will start to work on the detected problems and thereby improve the cold chain for fresh products.

Some thoughts on the lecture

The project was a real challenge through which we were able to learn what it means to carry out a project in practice. Through our perseverance, we were ultimately able to create value for our project partner. In summary the lecture gave us the opportunity to gain valuable experience in practice and to experience for ourselves how Industry 4.0 technologies can be used and what their potential is. With the expansion of our network, knowledge and experience, the high effort of the module was definitely worth it.

Building my Python Dev Environment with VIM

Sat, 29 May 2021 00:00:00 GMT

A few months ago I finally took the time to learn VIM. This whole endeavour was inspired by DJ Adams through his Hands-on SAP Dev videos. After seeing all the incredible things DJ did in the terminal using VIM, I wanted to be able to do this as well.

In summary, learning some of the command line tools I saw in DJs videos really changed the way I'm working. Today, I spend a lot more in the terminal time using ranger, tmux and VIM.

However, one thing I did not do so far was customizing VIM to my requirements. While I installed quite a few plugins, I never took the time to really adjust VIM and the plugins to my needs. Last week I decided to create a default project setup for my Python 🐍 projects. As a result, I also tried various VIM plugins. Here is what my current development environment in VIM looks like and the settings I decided to use. The whole configuration is available on Github.

Basic OS Settings

One of the first things I did when learning VIM was to remap the <caps lock> key. I highly recommend doing this as:

<caps lock> is really not useful unless you are developing

ABAP

you will need to use the <esc> key a lot in VIM
VIM offers out of the box functionality for

switching cases of characters using <g><U> and <g><u>

Basic VIM Settings

The most important setting for me is to activate the syntax highlighting and automatic indentation. This is done by adding this code to the .vimrc:

syntax on
set autoindent

Note: The simplest approach to editing the .vimrc file is by executing :e $MYVIMRC

Next, of course, is a proper color there. After trying different ones, I finally settled for the Dracular Theme. As I'm using the vim-plug as the plugin manager, I installed the theme using:

Plug 'dracula/vim', { 'as': 'dracula' }

The color scheme is activated by adding colorscheme dracula to the .vimrc. The following screenshot shows the dracula theme in action while editing this blog post.

Next, I enabled relative line numbers by adding set number relativenumber to the .vimrc. This way it is easy to see how many lines I need to move up or down to put the cursor on a certain line. Additionally, highlighting the current line with set cursorline helps me to quickly spot the current line.

Finally, I added keyboard shortcuts to simplify moving between panes by pressing <ctrl> and <h>, <j>, <k> and <l>.

map <C-j> <C-W>j
map <C-k> <C-W>k
map <C-h> <C-W>h
map <C-l> <C-W>l

VIM Plug Ins

With the basic configuration in place, the next step is adding plugins to extend the functionality of VIM. As mentioned above, I use vim-plug as my plugin manager. In my current configuration I use the following plugins:

Improved Status Bar

vim-airline offers a highly customizable status bar. I simply installed this plugin with its default configuration. The following screenshot shows my status bar while writing this blog post.

Project / File System Explorer

A useful feature all IDEs offer is a file system explorer for projects or directories. NERDTree provides exactly this feature for VIM. Again, I simply installed the plugin in its default configuration. In contrast to IDEs there is no concept of a project in NERDTree. Instead NERDTree simply show the directory from which VIM was launched. As all of my projects are organized in different directories anyway, this is all I need.

One of the nice feature of NERDTree is the action menu, which can be accessed by hitting <m> inside the NERDTree window. This action menu provides quick access to functions like creating, renaming, moving and deleting files. The following screenshot shows the NERDTree and the actions menu in the directory for this blog.

To enable me to quickly access NERDTree inside VIM I added the following mapping to my .vimrc. This mapping allows me to toggle the NERDTree by hitting <ctrl>-<n>.

map <C-n> :NERDTreeToggle<CR>

FZF - the Command-line Fuzzy Finder

Having a project tree is nice. However, searching files in large project or somewhere on the file system is quiet tedious this way. This is when fzf becomes very useful. Using fzf inside VIM requires the installation of two plugins.

Again, I simply used the default configuration for this plugin. With the plugins installed there are a few new commands available in VIM. :FZF starts fzf in the directory of the current file. :Files in contrast allows to specify a directory from which fzf starts to search. The following screenshot shows the fzf pop up window after executing :Files in the root directory of my website project and entering a few search characters.

Autocompletion

Of course, every development environment needs proper auto-completion. For VIM, there a lots of different plugins offering this kind of functionality. I decided to use Conquer of Completion (CoC). CoC support the Language Server Protocol which is the protocol VS Code uses as well. Because of this, CoC offers a similar feature set to the auto-completion in VS Code.

I added the following configuration to my .vimrc. I copied this configuration from the CoC documentation. Lines 2 - 12 enable <tab> to complete suggestions. Line 15 allows to trigger the suggestion popup using <ctrl>-<space>. Lines 23 - 31 allow displaying documentation for the word under the cursor using <K>. Finally, lines 35 - 42 add keyboard shortcuts to enable the navigation to definitions, implementations and references.

" Use tab for trigger completion with characters ahead and navigate. 
inoremap <silent><expr> <TAB>
      \ pumvisible() ? "\<C-n>" :
      \ <SID>check_back_space() ? "\<TAB>" :
      \ coc#refresh()
inoremap <expr><S-TAB> pumvisible() ? "\<C-p>" : "\<C-h>"

function! s:check_back_space() abort
  let col = col('.') - 1
  return !col || getline('.')[col - 1]  =~# '\s'
endfunction

" Use <c-space> to trigger completion.
inoremap <silent><expr> <c-space> coc#refresh()

" Use <cr> to confirm completion, `<C-g>u` means break
" undo chain at current position.
" Coc only does snippet and additional edit on confirm.
inoremap <expr> <cr> pumvisible() ? "\<C-y>" : "\<C-g>u\<CR>"

" Use K to show documentation in preview window
nnoremap <silent> K :call <SID>show_documentation()<CR>

function! s:show_documentation()
  if (index(['vim','help'], &filetype) >= 0)
    execute 'h '.expand('<cword>')
  else
    call CocAction('doHover')
  endif
endfunction

" GoTo code navigation.
nmap <silent> gd <Plug>(coc-definition)
nmap <silent> gy <Plug>(coc-type-definition)
nmap <silent> gi <Plug>(coc-implementation)
nmap <silent> gr <Plug>(coc-references)

" Remap keys for applying codeAction to the current buffer.
nmap <leader>ac  <Plug>(coc-codeaction)
" Apply AutoFix to problem on the current line.
nmap <leader>qf  <Plug>(coc-fix-current)

The following GIF shows the plugin in action in a small python project.

Note: CoC uses its own plugin manager. This means that CoC plugins are not installed using the VIM plugin manager. Instead the command :CocInstall is used to install the plugins. In my environment I currently have the following CoC plugins installed:

coc-markdownlint

coc-json

coc-html

coc-tsserver

coc-pyright

coc-css

Linting and Fixing

With code completion in place I wanted to integrate linting and automatic fixing of linting errors as well. For this purpose I installed the Asynchronous Linting Engine (ALE). ALE basically works out of the box. However, ALE also supports the Language Server Protocol a providing code completion. In my setup I wanted to use CoC for code completion and ALE for linting. Therefor, I followed the suggestion from the ALE FAQs and added the following line to the CoCConfig:

{
  "diagnostic.displayByAle": true
}

In addition, I disabled the usage of the Language Server Protocol for ALE by putting the following line at the very beginning of my .vimrc:

let g:ale_disable_lsp = 1

Finally, I configured which fixers ALE should use based on different file types. For all files I want to remove trailing lines and trailing white space. Javascript should be fixed using eslint. Python 🐍 files are fixed using isort and black. Maybe I should also add flake8 to this list 🤔.

Setting g:ale_fix_on_save to 1 enables the automatic execution of the fixers when a file is saved.

let g:ale_fixers = {
      \   '*': ['remove_trailing_lines', 'trim_whitespace'],
      \   'javascript': ['eslint'],
      \   'python': ['isort', 'black']
      \}

" Set this variable to 1 to fix files when you save them.
let g:ale_fix_on_save = 1

Terminal

Although I mostly run VIM inside tmux I also installed Neoterm. Neoterm provides a nice integration of a terminal as well as the REPLs of different programming languages.

I added the following configuration to my .vimrc. The first line specifies that the terminal should be opened in a new window below the current one (cf. documentation). The key mappings allow to toggle the terminal in all modes using <ctrl>-<y>.

let g:neoterm_default_mod = 'botright'
nnoremap <c-y> :Ttoggle<CR>
inoremap <c-y> <Esc>:Ttoggle<CR>
tnoremap <c-y> <c-\><c-n>:Ttoggle<CR>

Unit Testing

The plugin that completes my VIM setup is Test.VIM. This plugin allows to run unit test in different languages and using different unit test frameworks. For Test.VIM I added the following configuration to my .vimrc.

The first line specifies that I want to use pytest as the test runner for python projects. Line 4 configures Test.VIM to run all commands using Neoterm. Lines 7 - 11 define keyboard shortcuts to execute different commands like running a test file (<t>, <ctrl>-<n>) or running a whole test suite (<t>, <ctrl>-<s>). Finally, lines 13 - 19 defines an auto-command to execute all unit tests in a file whenever the buffer is saved.

let test#python#runner = 'pytest' 

" make test commands execute using neoterm
let test#strategy = "neoterm"

"keyboard short cuts
nmap <silent> t<C-n> :TestNearest<CR>
nmap <silent> t<C-f> :TestFile<CR>
nmap <silent> t<C-s> :TestSuite<CR>
nmap <silent> t<C-l> :TestLast<CR>
nmap <silent> t<C-g> :TestVisit<CR>

" auto commoand to run tests automatically
augroup test
  autocmd!
  autocmd BufWrite * if test#exists() |
        \   TestFile |
        \ endif
augroup END

Summary

With the configuration and plugins describes in this post I have now a really nice development environment based on VIM. I hope this post helps others to configure VIM to their needs without the need for all the trial and error I went through.

Useful Resources

vimawesome: collection of awesome VIM plugins.
YT Playlist on using VIM as an IDE
VIM for Python
My .vimrc file

grievy - My Experiences as a Product Owner in the Digital Lab

Tue, 25 May 2021 00:00:00 GMT

The project

Through a death in my family, I have experienced what it means to lose a loved one. Many relatives feel overwhelmed with organizational matters and left alone with their grief. The scientific situation shows: the risk of mental illness is demonstrably increased after an experienced bereavement. The handling of the many organizational tasks, as well as the accompaniment through the grief have hardly arrived in the digital space.

My idea is to develop an app that accompanies, supports and demonstrably relieves people at exactly these points after the death of a loved one. My name is Nele Stadtbäumer, I am a psychologist and founder of grievy - the first digital companion after a death.

The beginning of the collaboration

For the implementation of the grievy app, I needed technically-savvy people who could bring the content created by our psychotherapists and designers into the format of an app. My goal was to give young people the chance to actively shape and collaborate on a practical project. Through the recommendation of a friend, I turned to Prof. Christian Drumm. Fortunately, it was just before the winter semester and the start of the project seminar. Luckily, I had the chance to become a project partner with grievy.

The collaboration

The collaboration started with a presentation of all projects and project partners for the students in seminar. All projects were very interesting and in total very diverse, so that there should have been something for every area of interest. My concern was that grievy is only an idea and a start-up in its infancy, and therefore not very interesting for students. Especially in contrast to the big companies involved. However, my concerns were completely unfounded. The project was very popular with the students and the participants had to be chosen through a lottery. Finally, a great team of 6 students as developers, Christian Drumm as Scrum Master and myself as Product Owner was founded.

We worked remote throughout the entire project (due to the pandemic). We organized all user stories via a common Trello board and communicated via mail and Zoom. As the collaboration progressed, we could see the students growing in their tasks and becoming more independent and organized. They really outgrew themselves over the course of the project seminar and, with no prior knowledge of app programming, created a first product version that was something to behold!

I enjoyed the collaboration very much and learned a lot myself. I felt informed at all times and was surprised every week how much the students achieved within a week. They brought in their own ideas and showed me different ways to implement each idea of mine. Our Scrum Master, Christian Drumm, steered us safely through the project and kept an eye on the time frame and the tasks.

Outlook

The final presentation in the plenum rounded off the project seminar. During the seminar we focused on the organizational part of the grievy app. The final state after the project seminar was an app with all organizational information and services, as well as placeholders for what is still planned in the future. We would not have expected this great result in such a short time!

Even today, two of the students are still working on grievy. I always look back on that time with pleasure and I am very happy that this collaboration came about. I am already thinking about how we can be involved again in the next project seminar. I am already looking forward to the next collaboration!

Docker Hates Me

Tue, 23 Mar 2021 00:00:00 GMT

In this blog I described what I learned when trying to replicate the examples from the third chapter of the Kubernetes in Action by Marko Luksa.

Unfortunately, I rarely stick to tutorials or descriptions in books. Most of the time I try to extend the examples just a little. This usually leads me down the next rabbit hole. And after lengthy debugging session and lots of searching on the internet is usually haven't exactly achieved what I wanted but also learned lots of stuff along the way. This is exactly what happen in this case 😉!

Python Echo Server

In order to have a little application to deploy to my Kubernetes cluster I decided to implement an echo server. But in contrast to the echo server described in the Kubernetes in Action book I wanted to use Python 🐍 and FastAPI to implement the echo server.

Implementing an echo server is straight forward with FastAPI. The following snippet shows my implementation. The code is also available on Github.

import uvicorn
import socket

from fastapi import FastAPI, Request

app = FastAPI()

@app.get("/")
async def root(request: Request):
  return {
    "client": request.client.host,
    "host": socket.gethostname(),
    "message": "Hello World from Python!"
  }

if __name__ == "__main__":
  uvicorn.run(app, host="0.0.0.0", port=8000)

The echo server can be run locally by executing python main.py. Once the server is running sending an HTTP Get request to it will return a simple JSON object containing client IP, the host name and a message (cf. lines 11 - 13 in the code snippet). Below is the output of running curl -s localhost:8000 | jq on my laptop. Note that I use the -s option to suppress progress output. Furthermore, the result is formatted for better readability using jq.

{
  "client": "127.0.0.1",
  "host": "Christians-MBP-2.local",
  "message": "Hello World"
}

Comparison between Virtual Machines and Containers

Before I continue with the explanation of how to deploy the echo server to Kubernetes it is important to understand what containers are and how they differ from virtual machines. The following figure ¹ shows a comparison of running applications in virtual machines and containers.

Virtual machines as well as containers are technologies for virtualisation. The goal of both technologies is to provide an isolated environment to run applications. In the case of virtual machines hardware virtualisation is used to achieve this. The host machine executes multiple virtual machines. Each virtual machine contains its own operating system. Furthermore, each virtual machine contains the necessary libraries as well as the applications. This is shown on the left side of the figure.

In contrast to that, containers rely on special features of the host operating system for the creation of isolates environments (e.g. cgroups in Linux). As shown on the right side of the figure, containers do not contain a guest operating system. Containers only contain the necessary libraries and the application. As a result containers have a smaller runtime overhead and are much smaller than comparable virtual machines.

Docker is a well known software for the creation of containers. Kubernetes was especially develop as a system

for automating deployment, scaling, and management of containerized applications.

Creating a Docker Image

The next step in order to deploy the echo server to my cluster is to create a Docker image. Docker images are created using a Dockerfile. This file specifies:

The base image on which the new image will be based
What libraries need to be installed
Which command is executed in order to run the application inside the container.

The following listing shows the contents of the Dockerfile I created for the Python echo server.

# Pull base image
FROM python:3.8

# Copy files
COPY *.py Pipfile* /

# Install dependencies
RUN pip install pipenv
RUN pipenv install --system --dev

# Run server
EXPOSE 8000
CMD ["python", "main.py"]

Line 2 specifies the python as the base image. Furthermore, the tag 3.8 details the version of the python image to use. Next, line 5 specifies that all *.py files and all files starting with Pipfile* should be copied to the root directory of the container. Lines 8 and 9 are responsible foe installing the necessary Python libraries using the pipenv tool. Finally, line 12 exposes port 8000 of the container and line 13 starts the echo server by executing python main.py.

Building the Docker Container

Based on the Dockerfile the container is created using the docker CLI.

docker build -t ceedee666/py-echo-server .

This command:

Builds a docker image using the Dockerfile in the current directory
Names the image with ceedee666/py-echo-server.

In the image name ceedee666 is my Docker Hub username. Adding the username to the name of the image is required in order to be able to push the image to Docker Hub. Once the image is build, it can be used to crate a container and run it.

docker run --name py-echo-container -p 1234:8000 -d ceedee666/py-echo-server

The command above consist of three main parts:

It creates and runs a container named py-echo-server
The container is based on the image ceedee666/py-echo-server
Maps the port 1234 of the host to the port 8000 of the container.

After the container is running it is now possible to test the echo server running inside the container using e.g. curl -s localhost:1234 | jq.

Pushing the Image to Docker Hub

The final step is to push the Docker image to the Docker Hub. This enables Kubernetes in the next step to use this image and deploy it to the Raspberry Pi cluster. Pushing an image can be done using the following command:

docker push ceedee666/py-echo-server

First Rabbit Hole - exec format error

After the Docker image is pushed to the Docker Hub the next step is to deploy it to the Raspberry Pi cluster. The easiest way to do this is using a deployment. Deplyoments can be created using the Kubernetes CLI. To following command creates a deployment named py-echo-server based on the Docker image ceedee666/py-echo-server.

k create deployment py-echo-server --image=ceedee666/py-echo-server

Once I executed this command I could see in the Kubernetes dashboard that a deployment, a replica set and a pod were created. However, the pod kept restarting and never got to the status running.

In the pod logs I found the following error message:

standard_init_linux.go:219: exec user process caused: exec format error

After I bit of searching it became clear that the Docker image was build for the wrong system architecture. I create the image on my Intel-base MacBook Pro and tried to run it on the ARM-based Raspberry Pi cluster.

Fortunately, Docker supports cross platform build using Buildx. For example, the following command can be used to build a Docker image for the ARM-based system.

docker buildx build --platform linux/arm/v7 -t ceedee666/py-echo-server .

Second Rabbit Hole - Github Actions

After build the image for the correct platform using Buildx I tried to push the image to Docker Hub again. For whatever reason, the push was suddenly extremely slow. It took more then 6 hour to push the small image. And I never found out why. While waiting for the push to complete I started reading about Github Actions for building Docker images.

Based on this blog and lots of experimenting I was able to create a Github Action in my repository that:

Build the Docker image for different platforms
Pushes the resulting images to Docker Hub.

The following listing show the YAML file for the action.

name: '[builder] CI for releases'

on:
  push:
    branches:
      - main

  workflow_dispatch:

jobs:
  release:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout
        uses: actions/checkout@v1

      - name: Dockerhub login
        env:
          DOCKER_PASSWORD: ${{ secrets.DOCKER_PASSWORD }}
          DOCKER_USERNAME: ${{ secrets.DOCKER_USERNAME }}
        run: |
          echo "${DOCKER_PASSWORD}" | docker login --username ${DOCKER_USERNAME} --password-stdin
      - name: Set up Docker Buildx
        id: buildx
        uses: crazy-max/ghaction-docker-buildx@v1
        with:
          version: latest

      - name: Build dockerfile (with push)
        run: |
          docker buildx build \
          --platform linux/amd64,linux/arm/v7,linux/arm64 \
          --push --tag ceedee666/py-echo-server:latest .

Line 1 sets the name of the action. Line 3 - 6 define that the action should be run on every push to the main branch. Line 8 enables the manual triggering of the action. Starting with line 10 the build process for the image is defined.

Line 18 - 23 read the Docker Hub password and username from Github secrets and use these to log on to Docker Hub. After setting up Docker Buildx, lines 30 - 34 perform the actual build of the images and push them to Docker Hub.

The following screenshot show the running action after it has been manually triggered. Using this action the image can now be build for multiple platforms and pushed to Docker Hub in about 5 minutes.

Deploying the Echo Server

With the Docker image for the processor architecture on Docker Hub I could finally create the deployment again:

k create deployment py-echo-server --image=ceedee666/py-echo-server

This time the pod started successfully as shown in the following screenshot.

In order to access the echo server outside the cluster it is necessary to expose the echo server as a service. There a different types of service available in Kubernetes. As on my local cluster no external load balancer is available the type NodePort can be used. The following command exposes the echo server on port 8000.

 k expose deployment/py-echo-server --type=NodePort --port 8000

After this command k get services returns the following list of services.

NAME             TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE
kubernetes       ClusterIP   10.152.183.1     <none>        443/TCP          53d
py-echo-server   NodePort    10.152.183.126   <none>        8000:31862/TCP   15d

Using port forwarding it is now possible to access the echo server. The following command forwards port 9090 to the py-echo-server.

k port-forward service/py-echo-server 9090:8000

Now it is possible to access the echo server running in the Kubernetes cluster. For example, executing curl -s http://localhost:9090 | jq returns:

{
  "client": "127.0.0.1",
  "host": "py-echo-server-5465f599f5-plvrd",
  "message": "Hello World from Python!"
}

Scaling the Echo Server

Now I'm able to scale the deployment up and down by specifying the desired replica count. The following command, for example, scales the replica set to 5. Consequently, 5 pods running the echo server will be started.

k scale deployment/py-echo-server --replicas=5

The following GIF shows the process of scaling the deployment. First, only one pod is running. Curl always hits the same pod. Next the deployment is scaled up to 5 pods. Curl starts hitting all of them. Finally, the deployment is scaled down to 2 pods. Consequently, curl only hits the two remaining pods.

Note, that I use curl -s http://pi-picard:31580 to access the echo server. The reason is, that performing a port forwarding of the NodePort service selects a target pod. Therefore, no load balancing is preformed. To see the load balancing, the service must be invoked via the control pane, which is running on pi-picard in my cluster.

Next Steps

Basically, my Raspberry Pi cluster is now up and running, and the first application has been deployed. Now I need to thinking about what to try out next. Currently, my list of ideas contains:

Deploying an SAP CAP application to the cluster
Try out OpenFAAS.

Let's see what the next blog will bring... 😉.

Christian

References

[1] The figure is based on M. Lukša, Kubernetes in action. Shelter Island, NY: Manning Publications Co, 2018. ↩

Exploring the Raspberry Pi Cluster

Mon, 08 Mar 2021 00:00:00 GMT

In the previous blog I described the initial steps to build and set up a Raspberry Pi cluster. With this initial setup in place, the next step is to connect the four Raspberry Pis together to form a Kubernetes cluster.

Creating a Kubernetes Cluster

As described in the Kubernetes documentation a Kubernetes cluster consists of the control plane and several workers nodes. The worker nodes execute the containerizes applications while the control plane is responsible for managing the worker nodes. The final configuration step in my Raspberry Pi cluster is to:

Define which node acts as the control plane (Note: in a productive environment the control plane would also be running on several computers)
Add the worker nodes to the control plane.

Note, that the following description is specific for MikroK8s. However, it should be easily adoptable to different Kubernetes variants.

Defining the control plane and adding worker nodes to the cluster is that last time requiring to log into the individual Raspberry Pis directly. Once the cluster is set up all interactions with the cluster are performed using the kubectl, the Kubernetes CLI.

Adding Worker Node

In my cluster the Raspberry Pi with the host name pi-picard is obviously going to run the control plane 🖖. To add worker nodes to a node it is necessary to log in using SSH (e.g. ssh ubuntu@pi-picard). Once you are logged into the Raspberry Pi that is going to be the control plane execute the following command (caveat: this is specific for MikroK8s):

microk8s add-node

The result of this command should be some joining instructions similar to the ones below:

  From the node you wish to join to this cluster, run the following:
    microk8s join 192.168.178.251:25000/b5379006e31afba0056940b2ceea37f3

  If the node you are adding is not reachable through the default interface you can use one of the following:
    microk8s join 192.168.178.251:25000/b5379006e31afba0056940b2ceea37f3
    microk8s join 10.1.2.64:25000/b5379006e31afba0056940b2ceea37f3

Therefore, the next step is to log into the worker nodes (pi-ricard, pi-data and pi-worf) in my case and the and execute the join command. Once the worker nodes are joined to the cluster, kubnectl can be used to show the nodes in the cluster.

In my cluster executing

microk8s kubectl get no

results in the following output:

NAME        STATUS   ROLES    AGE   VERSION
pi-data     Ready    <none>   30d   v1.20.2-34+c6851e88267786
pi-riker    Ready    <none>   30d   v1.20.1-34+97978f80232b01
pi-picard   Ready    <none>   30d   v1.20.2-34+c6851e88267786
pi-worf     Ready    <none>   30d   v1.20.2-34+c6851e88267786

Furthermore, using microk8s status shows some additional information regarding the cluster:

microk8s is running
high-availability: yes
  datastore master nodes: 192.168.178.251:19001 192.168.178.252:19001 192.168.178.253:19001
  datastore standby nodes: 192.168.178.254:19001
addons:
  enabled:
    dashboard            # The Kubernetes dashboard
    dns                  # CoreDNS
    ha-cluster           # Configure high availability on the current node
    metrics-server       # K8s Metrics Server for API access to service metrics
  disabled:
    ...

This output shows that in my cluster the high availability is configured and the services dns, dashboard and metrics-server are running. Disabling and enabling service can also be performed using the microk8s CLI. See the MicroK8s documentation for further details.

Configuring kubectl

From this point on it is possible to interact with the Raspberry Pi cluster using the Kubernetes CLI kubectl. In order to do so, kubectl needs to know on which server the control plane is running and what user credential to use to access the cluster. This information is stored in the kubeconfig file. According to the Kubernetes documentation the following process is used by kubectl to find the kubeconfig:

By default, kubectl looks for a file named config in the $HOME/.kube directory. You can specify other kubeconfig files by setting the KUBECONFIG environment variable or by setting the --kubeconfig flag.

I chose to store the kubeconfig file for my cluster in a different location and set the KUBECONFIG environment variable whenever I want to interact with the cluster.

export KUBECONFIG=~/Documents/development/rpi-cluster/kubeconfig

Creating the kubeconfig File

The interesting question is which information kubeconfig file needs to contain. Again, there are different approaches how to create a kubeconfig file. Especially, in more advanced scenarios than the one I described here, there are lots of different features available.

For my cluster I simply executed:

microk8s config

in the control plane node. The output of this command should look something like this:

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0tLS1CRUdJT...
    server: https://192.168.178.251:16443
  name: microk8s-cluster
contexts:
- context:
    cluster: microk8s-cluster
    user: admin
  name: microk8s
current-context: microk8s
kind: Config
preferences: {}
users:
- name: admin
  user:
    token: THNNNU...

Simply store the output of the command on the computer that is used to access the cluster and configure kubectl to use it.

Using kubectl

With the kubeconfig in place it is now possible to interact with the cluster using the kubectl command.

Adding a kubectl Shortcut

To interact with the cluster the kubectl command will be used a lot! As it in not exactly the shortest command to type it is useful to configure a shortcut for it. I simply added the following alias to my (ohmyz.sh) shell config:

alias k='kubectl'

With this alias in place I can now use k to invoke the kubectl command. For example executing k -h results in the following output:

kubectl controls the Kubernetes cluster manager.

 Find more information at: https://kubernetes.io/docs/reference/kubectl/overview/

Basic Commands (Beginner):
  create        Create a resource from a file or from stdin.
  expose        Take a replication controller, service, deployment or pod and expose it as a new Kubernetes Service
  run           Run a particular image on the cluster
  set           Set specific features on objects

Accessing the Kubernetes Dashboard

In order to explore the Kubernetes cluster a little and get familiar with kubectl, the next step is to access the Kubernetes Dashboard. The Kubernetes Dashboard is a web application that can be used to manage the cluster. Accessing the Kubernetes Dashboard requires a few steps:

Identifying the service name of the dashboard
Configuring port forwarding from the client to the cluster
Retrieving client secrets to access the dashboard.

Identifying the Service Name

Kubectl can be used to get the services running in the Kubernetes cluster. Executing

k get services

result in the following output in my cluster:

NAME          TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
kubernetes    ClusterIP      10.152.183.1    <none>        443/TCP          30d

Obviously, the output does not contain the dashboard service. The reason is, that the dashboard service is deployed in a different namespace. Using k get namespace a list of all the namespaces in the cluster can be retrieved. In my cluster the following namespaces are available:

NAME              STATUS   AGE
kube-system       Active   30d
kube-public       Active   30d
kube-node-lease   Active   30d
default           Active   30d

In MikroK8s the dashboard is deployed in the kube-system namespace. Executing

k get services -n kube-system

returns the services running in this namespace. In my cluster the command returns the following list of services.

NAME                        TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                  AGE
metrics-server              ClusterIP   10.152.183.97    <none>        443/TCP                  30d
kubernetes-dashboard        ClusterIP   10.152.183.88    <none>        443/TCP                  30d
dashboard-metrics-scraper   ClusterIP   10.152.183.143   <none>        8000/TCP                 30d
kube-dns                    ClusterIP   10.152.183.10    <none>        53/UDP,53/TCP,9153/TCP   29d

Configuring Port Forwarding

Once the exact name and namespace of a service is know it is possible to configure port forwarding to this service. The kubnectl [documentation]/(https://kubernetes.io/docs/tasks/access-application-cluster/port-forward-access-application-cluster/) contains a detailed example how to use port forwarding to access applications running in the cluster. To access the Kubernetes Dashboard a port of the local machine needs to be forwarded to the appropriate service.

The following command forwards the local port 8080 to the port 443 of the service kubernetes-dashboard in the namespace kube-system.

k port-forward -n kube-system service/kubernetes-dashboard 8080:443

Once the port forwarding is running it is possible to access the Kubernetes Dashboard at the URL https://localhost:8080. Accessing this URL in a browser results in the login window shown below.

Retrieving Secrets to Access the Dashboard

In order to further explore the possibilities of kubectl I will use the authentication token to access the dashboard. In order to read the correct authentication token it is first necessary to find out which tokens exist. The following command returns a list of all secrets in the namespace kube-system.

kubectl -n kube-system get secret

Running this command returns a long list of available secrets. In my cluster the this is the first part of the returned list:

NAME                                             TYPE                                  DATA   AGE
calico-node-token-spnt5                          kubernetes.io/service-account-token   3      31d
resourcequota-controller-token-6tc67             kubernetes.io/service-account-token   3      31d
calico-kube-controllers-token-p8dlv              kubernetes.io/service-account-token   3      31d
job-controller-token-lhkst                       kubernetes.io/service-account-token   3      31d
ttl-controller-token-m96b7                       kubernetes.io/service-account-token   3      31d
attachdetach-controller-token-kw5fg              kubernetes.io/service-account-token   3      31d
pod-garbage-collector-token-xrjns                kubernetes.io/service-account-token   3      31d
disruption-controller-token-bv46c                kubernetes.io/service-account-token   3      31d

In order to access the Kubernetes Dashboard a secret of type service-account-token is required. In my cluster the respective secret is called kubernetes-dashboard-token-ccpfl. Using the following command this secret can be read:

k -n kube-system describe secret kubernetes-dashboard-token-ccpfl

Executing the above command returns the following information:

Name:         kubernetes-dashboard-token-ccpfl
Namespace:    kube-system
Labels:       <none>
Annotations:  kubernetes.io/service-account.name: kubernetes-dashboard
              kubernetes.io/service-account.uid: 11cca18d-288b-4166-b0fb-feb1d3e49053

              Type:  kubernetes.io/service-account-token

              Data
              ====
              ca.crt:     1103 bytes
              namespace:  11 bytes
              token:      eyJhbGciOiJSUzI1NiIsImtpZCI6InBsUVlzU...

Pasting the returned token into the login screen should provide access to the dashboard. The following screenshot shows the node overview of my cluster.

Outlook

Accessing the Kubernetes Dashboard concludes this blog. In the next blog I'll describe how to build a small Node.js application and deploy it into the cluster.

Christian

Building a Raspberry Pi Cluster

Fri, 05 Mar 2021 00:00:00 GMT

About a year ago I decided to build a Raspberry PI. I can't remember anymore how I got the initial idea. Basically, I though it is a quite nice project. Furthermore, I wanted to learn about technologies like Kubernetes and OpenFaas.

Of cause there are lots of different options to learn Kubernetes besides building a physical cluster:

Docker Desktop offers functionality to setup a local Kubernetes cluster
minikube is a tools to "helping application developers and new Kubernetes users" in setting up a local cluster
AWS,

GCP and Azure also provide the possibility to deploy Kubernetes clusters.

Nevertheless, I decided to get my hands dirty 👷 and went ahead to build a Raspberry Pi cluster. In this blog, which is hopefully the first in a series of blogs on the topic, I describe the necessary steps to build the cluster and install the required software.

Bill of Materials

There are a large number of blogs on building Raspberry Pi based clusters (e.g. here, here or here). I didn't do much research when building my cluster and just went for a straight forward setup with four Raspberry PIs. My cluster consist of the following components:

Component	Amount	Link
Raspberry Pi 4 Model B, 4GB	4	List of Resellers
SanDisk Ultra 16GB microSD	4
Anker PowerPort 6 (6-Port USB charger)	1	Anker Website
ICY BOX 4 Way Raspberry Pi 4 Case with Fan	1
D-Link DGS-108 8-Port Switch	1	D-Link Website

Of cause, I also bought a set of network and USB cables. Furthermore, a SD card reader is required in order to install the operation system on the SD cards.

Building the cluster was very simple. The most complex part was assembling the ICY BOX case. However, after about 30 minutes this was finished as well and the cluster was nearly ready.

Installing the OS

Basically, it is possible to install every OS on a Raspberry Pi that supports the ARM architecture. For novices the official Linux distribution Raspberry Pi OS (previously called Raspbian, available here is the best choice. However, no novice ever tried to build a cluster. So there are lots of different options to choose from.

For my cluster I chose to install the latest development release of Ubuntu. At the time of writing this blog this was Ubuntu Server 20.10. Of cause, it would also be possible to use the desktop version of Ubuntu. However, the cluster will mainly be accessed using different command line tools. Therefore, all the UI components of the desktop version would therefore never be used.

Installing the OS is straight forward. There is also a large number of tutorial available on this topic. I simply followed the nice step by step tutorial on the Ubuntu website. The only challenge when installing the OS was to execute the same steps four times. It would also have been possible to clone the SD cards. However, I was lazy and just installed the OS four times.

Kubernetes requires cgroups being enabled. To enable cgroups add the following line to the boot/firmware/cmdline.txt.

cgroup_enable=memory swapaccount=1 cgroup_memory=1 cgroup_enable=cpuset

Once the SD cards are prepared with the OS it is time to insert them into the Raspberry Pis and power up the cluster for the first time!

Configuring the OS

Once all Raspberry Pis are up and running there are a few configuration steps necessary. These steps are:

Change the default password of the ubuntu user
Set a hostname for each Raspberry Pi
(Optional) Copy SSH key to each Raspberry Pi

The first step ist to change the default password of the ubuntu user. In order to change the password it is necessary to log on to each Raspberry Pi using SSH. To log on to the Raspberry Pis their IP addresses are required. The installation tutorial describes one possibility how to identify the IP addresses of the Raspberry Pis. The default password for the user ubuntu is ubuntu. On the first login Ubuntu asks to change this password. In my cluster I used the same password for all four Raspberry Pis.

The next step is to set the hostnames of the Raspberry Pis. I used the following hostnames for my cluster:

pi-picard
pi-ricard
pi-worf
pi-data

As Michael pointed out on Twitter I missed the obvious pi-card 🤦.

Setting the hostname in Ubuntu is done using the following command. After setting the hostname I rebooted the Raspberry Pis. However, this is not necessary.

sudo hostnamectl set-hostname <new-computer-name-here>
sudo reboot

Finally, I copied my public SSH key to the Raspberry Pis to enable login without a password. I simply used my existing SSH key. Generation of an SSH key can be performed using the ssh-keygen command on the local machine (not on the Raspberry Pis). Copying the public SSH key to the Raspberry Pis is done using the following command:

sh-copy-id -i ~/.ssh/id_rsa ubuntu@<raspberry-pi-IP>

After this step the configuration of the OS is finished. Next I configured my local DNS to simplify connecting to the RaspberryPis.

Configuring local DNS

Accessing the Raspberry Pis using their IPs is quite cumbersome. This is especially true if due to the DHCP configuration the Raspberry Pis get different IP addresses on the local network all the time. Therefor I performed two configuration steps:

Configure a fixed IP address for each Raspberry Pi.
Added the hostnames to the local DNS server.

In the DHCP config of my local network I set gave each Raspberry Pi a fixed IP in the range 192.168.178.251 - 192.168.178.254.

In my local network I already have a Pi-hole instance running. Therefore, I could simply add the hostnames of the Raspberry Pis to the local DNS configuration of the Pi-hole. After that, the Raspberry Pis were accessible using their individual hostnames.

Note, that the last configuration step is very important. If the cluster nodes in an Kubernetes cluster are not addressable using their hostnames, certain commands in the Kubernetes CLI don't work. It took me quite a while to find out that the underlying reason for some of these error messages where due to the missing DNS configuration.

Installing Kubernetes

The final installation step is the installation of Kubernetes on the Raspberry Pis. Again, there are different options available:

For no particular reason I chose MicroK8s for my cluster. MicroK8s can be installed in Ubuntu using the following command:

sudo snap install microk8s --classic

Note, this command needs to be executed on all Raspberry Pis in the cluster. After this, the software installation on the cluster is finished. To complete the setup, it is necessary to install the Kubernetes tools (i.e. kubectl) on the computer that will be used to control the cluster. Installation instructions for different operating systems are available here.

Outlook

After build the Raspberry Pi cluster and installing the necessary software the cluster is ready for usage. The next step is to try out the different features of Kubernetes on the cluster. This will be described in the next blog of this series.

Christian

Adding comments to my blog

Wed, 03 Mar 2021 00:00:00 GMT

Allowing visitors to add comments to my blogs is a functionality, that was quite high on my product backlog from the start. Today I finally managed to add it. In this blog I describe the problem I had implementing the feature using Utterances. My problem where mainly due to me being a Gatsby / React / JavaScript novice. Maybe this blog will help novices in the future.

Comments Functionality

Allowing reads to comment is a quite common feature for blogging sites. In fact, it is so common, that the Gatsby documentation has an own section on adding comment. As described on that side there are many options for adding comments to a Gatsby site. I didn't spent much time comparing the different options, but after briefly reviewing different options, Utterances look like a very promising option.

The nice feature of Utterances is, that Github issues are used to store comments. Therefore, adding a comment requires a Github user. While this might discourage some people from commenting, this is a very nice prerequisite in my case. As mention in a previous blog I want to use my web site and blog also for teaching. Therefore, I can use the comment functionality in the future to introduce some Github features to students.

Adding Utterances

The Utterances website contains actually a code generator to help adding the Utterances to a website. After the initial configuration steps and providing some information an HTML fragment is generated. In my case the following fragment was generated:

<script src="https://utteranc.es/client.js"
        repo="ceedeee666/drumm.sh-comments"
        issue-term="pathname"
        theme="github-light"
        crossorigin="anonymous"
        async>
</script>

The first thing I tried was adding this fragment to my Layout component. However, nothing happened. After some searching (not Googling as I use DuckDuckGo as my main search engine) I found out that one can't simply add script tags to JSX. Further searching revealed a list of blogs that describe how to add Utterances to a Gatsby / React site in detail. Here are just a few examples:

Function components cannot have string refs

I basically followed the first blog. The reasons are:

I only used functional components so far in my site
it contained all the functionality in one component.

After implementing my Comments component I got the following error message:

Error: Function components cannot have string refs. We recommend using useRef() instead. Learn more about using refs safely here: https://fb.me/react-strict-mode-string-ref.

My Solution

Reading the React documentation on Refs and the DOM it was obvious I needed to use the useRef() function. After some experiments this is the solution I came up with (the complete source code of my site is available here):

import React, { useEffect, useRef } from 'react'
import { useStaticQuery, graphql } from 'gatsby'

const Comments = () => {
  const data = useStaticQuery(graphql`
    query RepoQuery {
      site {
        siteMetadata {
          commentsRepo
        }
      }
    }
  `)

  const commentsRef = useRef()

  useEffect(() => {
    const script = document.createElement('script')
    script.src = 'https://utteranc.es/client.js'
    script.async = true
    script.setAttribute('repo', data.site.siteMetadata.commentsRepo)
    script.setAttribute('issue-term', 'pathname')
    script.setAttribute('label', 'comment :speech_balloon:')
    script.setAttribute(
      'theme',
      'github-light'
    )
    script.setAttribute('crossorigin', 'anonymous')

    commentsRef.current.appendChild(script)

  }, [data])

  return (
    <div ref={commentsRef}/>
  )
}

export default Comments

The code cosists of the following parts:

Line 5: Here I create a query to read the Github repository from the gatsby-config.js.
Line 15: Creates a Ref using the useRef() function.
Line 17: The useEffect() functions builds the script element and adds it to the element identified by the commentsRef. It is important to note

that the DOM element in the Ref is available using the .current attribute.

Line 34: Finally, the <div> for the comments script is create and returned.

Whit this Comments component in place I can now add comment functionality to the pages of my website by just adding a <Comments />.

Result

The following screenshot shows the final result. Already including some comment by Tobias.

I hope this blog will help other when trying to add Utterances to their Gatsby website.

Christian

Why I Created My Own Website

Thu, 25 Feb 2021 00:00:00 GMT

This is the first (of hopefully a lot more to come) post on my brand new Website. In this post I describe the journey that brought this website into existence. Along the way I hope to clarify why building my first personal website was an inevitable choice.

How it all started

Every journey starts with the first step. So what was this first step in my case? About a year ago after reading some of Alex Ellis tweets and blogs I decided to build a four node Raspberry Pi cluster. Here is a photo of the result:

However, this project went down the road of many personal projects. It ended up sitting half finished on my desk. After the winter term of 2020/2021 was finished a few weeks ago I decided to reactive the cluster. After a few days I had it running on MicroK8s and was ready to start learning more about Kubernetes and cloud native development. This was about the time I published the following tweet:

In the thread of this tweet Frank requested a blog on the topic. Furthermore, DJ Adams encouraged me to blog even if I don't understand everything i write about in full detail. So this was decided. I was going to write a blog about Kubernetes on my Raspberry Pi cluster. At least such a blog would help me to remember what I did, which problems I encountered and how I was able to solve them.

Why no SAP Community Blog?

In the past I would have simply written this blog in the SAP Community. However, now I started thinking. There were several reasons for this:

The SAP Community is focused on SAP related topics. A blog on installing Kubernetes on a Raspberry Pi cluster doesn't really fit there.
I have several other topics in mind on which i want to write a blog. Most of them are related to my lectures at the FH Aachen and

would also not fit on the SAP Community. 3. In the last semester I asked two of my students to write a blog instead of taking an exam. They published these blogs on the SAP Community as well (here and here). I'm really happy with the result of this experiment and want to further expand it in the future. Again, the SAP Community doesn't seem to be the right place to post those blogs.

After one or two days I decided I needed a solution to write blogs. In my head I had collected the following requirements:

I want to be able to write blogs myself.
I want to easily allow my students to write blogs as well.
If possible I want to use Markdown to write the blogs.
Ideally the new solution would offer some kind of integration with Github for version control.

The last two requirements originated form some past experience. Firstly, I started writing my lecture materials using Markdown and publishing them on Github (cf End to End SAP ERP Process). I really like using Markdown and Github as it allows:

me to quickly correct errors in the lecture material and publish the updated material
enables students to submit improvements using pull requests.

Furthermore, I loath the Typo 3 installation we have a the FH Aachen. Creating and updating a personal website using it is just 🤬. Therefore, I was sure i needed something different.

Learning about the JAMStack (again)

At this point I sent a massage to Oliver. He is basically the only Web Dev person I know. Although he has done some real stuff, i.e. ABAP, in the past 😎. A few evening later we talk over Zoom and he recommended Gatsby and Netlify. After all, those are the tools he used to build openSIT.

After this chat I was sold on the idea of developing my own personal website using the JAMStack.

Gettings started with Gatsby

Getting started with Gatsby is quite easy. There are several good tutorials available. I worked trough the The Great Gatsby Bootcamp. After that I felt ready to go. And then I went through the usual loop many times:

I have an idea
It's more complex then I thought 🤔
I hate programming ☹️
I hate programming 😖
I hate programming 🤬
It works!
I love programming 😍
Goto 1.

What I learned

First and foremost, developing using Gatsby is really fun. From creating the frist hello world app until the final style adjustments I liked the whole experience. The main problems I ran into were due to me trying to learn 3 things in parallel. This site is built using:

Gatsby and JavaScript
Gatsby plugins
Gatsby uses Graphql
Bootstrap
SCSS

Besides JavaScript, I didn't know any of these technologies before. And my JavaScript skills are very bad as well. For everything I was able to find solutions online. Once I read the solution to a problem it most of the time seemed obvious. So in the end I was able to build this website. The source code is available here. If you notice any bad mistakes I'm looking forward to a pull request.

What's Next?

There is still a lot of work to do. The sections for teaching and projects are still empty. However, the next term starts in about four weeks. So at least the lecture section will need to contain some content soon. Furthermore, I hope to get back to regularly write blogs. I will start with a blog series about my Raspberry Pi cluster soon. And then there are lots of functions missing which I want to add. For example there should be the possibility to comment on blogs. I'm most likely going to implement it using utterances. Let's see where this journey will lead.

Christian

Hello World

Tue, 23 Feb 2021 00:00:00 GMT

Lorem Ipsum

Lorem ipsum dolor sit amet. Nam tempora officia sit ullam officia ut quia asperiores et aliquid consectetur est consequatur eligendi et quis amet. Amet mollitia ut quam eveniet qui fugiat dolorem ut unde rerum rem deleniti velit est minima galisum. Non consequatur omnis et animi eveniet est quia nemo eos quae molestiae. Ut esse beatae sit officiis doloribus et blanditiis natus et laudantium autem.

Ab dolorum nihil et animi ullam ad nemo dolores est iure internos At maxime voluptatem. Sed quae porro quo velit eligendi et praesentium rerum eum doloremque magnam vel impedit culpa. Non vero culpa ex corrupti asperiores rem velit porro.

Ut molestiae omnis et molestiae ratione aut officiis soluta. Qui omnis consectetur quo error provident et quae quis vel repellendus magnam At quam nihil. Qui quasi necessitatibus tempora consequuntur et nullpa rerum est molestiae dolorem non cupiditate officiis ut libero doloremque est eaque sint.