Stichword-Archiv: software development

Building Kubernetes Operators

November 17, 2020 4:33 pm Veröffentlicht von

In the past months ad Red Hat I dove a bit into operator development. During learning how to build Kubernetes operators myself I published a quick start article on opensource.com . Using Operator SDK is a good way to kick-start your own Kubernetes operator.

While the post itself is based on an older version of the SDK and usage has changed, you should be able to get such an example operator up and running quickly using this post and a later version of the Operator SDK.

While working on different operators, I wrote down patterns and best practices I learned or discovered in the different projects, which I wrote down on a blog post on openshift.com. The CFP CFP: How to build a Kubernetes operator that doesn’t break production is mainly based on this blog post. If you want to build your own Operator, those two blog posts may help you build an operator that is easy to build, maintain and run.

CFP: Things I love about SRE that I loathe about DevOps

Januar 28, 2020 8:32 pm Veröffentlicht von

DevOps & SRE – what is it?

Let’s define what those terms mean.

DevOps means, the same team building the software is responsible for running it. This can be easiest imagined for software that is operated by the vendor themselves, i.e. cloud services. The idea is, no one knows the software better than the developers, so no one could better operate and fix in-flight issues than them. On the other hand, developers have high interest in building software that is easiest to operate, if they operate it themselves. Issues found during operations can be addressed immediatly.

SRE is a concept where one team is responsible for running one or more services. Again, imagine a company building cloud services. They may all have similar requirements in operations, so why not create a dedicated team to run them all. This team contains software developers to automate common tasks that emerge when running the services to minimize manual efforts. SRE teams monitor the software to spot issues as soon as they appear and fix them in the best case before a customer notices them.

Which DevOps problems may SRE solve?

In DevOps, having a team that runs software and a separate team that builds it, is a well-known anti-pattern.

“If you have a DevOps team, you’re doing it wrong”

This quote can be found in many tweets, blog posts, and conference talks. But when it comes to SRE, this is a common pattern. One team is building software, another team is running it, and building software to improve running it.

In DevOps, you usually find a specific support role being rotated through the team. That means everybody is in that role for example for a week out of 10, given a team of 10 members.

Working in this support role usually has very different requirements than the usual development work.

The context can get completely lost during the transition into the support role (which often comes as surprise on Monday mornings). And once context is built up and the developer feels comfortable in that role, his shift ends and the support context is again lost.

This results in people hating to get into the support role, which also makes it less attractive to build things that improve the supportability of the software.

In SRE teams, people are on call as well every few weeks, often even 50% of their time. Why don’t they dislike it? The difference is the focus of the team. When not on call, developers in the SRE team work on improving the support. They are less involved in actual product code, but rather build software to ease the operations. For example, to automate updates and other maintenance tasks.

DevOps: supportability vs. new features
SRE: supportability = new features
DevOps: Dev team learning ops all together
SRE: Ops learning from devs and devs learning from Ops

Agile development in SRE teams

As SRE teams contain a fair portion of software development work, and get filled up by software developers, it is a natural move to also adapt agile software development practices. It depends heavily on the percentage of development work vs operations, which may be influenced by the team size, to find the right model to track and manage the project work. For example, in a small team where a high percentage of people is on call during the day, it might not make too much sense to plan sprints of 2 weeks if only a few backlog items are expected to get done in that timeframe.

Key Takeaways of this talk

By the end of this talk about the differences between SRE and DevOps working styles, attendees should have awareness (1) what the most significant differences between DevOps and SRE are, (2) A successful team running DevOps or SRE needs experienced Ops as well as Dev people, (3) If team members greatly dislike getting into the operating role, the team should work hard on improving the support experience.

Ruby best practice: Implementing operator == and ensuring it doesn’t break

März 8, 2019 12:42 pm Veröffentlicht von

In ruby, comparing hashes, strings and objects is a complicated topic. Should you use equal?, eql? or ==? There is plenty of help on this topic, but in this post, we will focus on the interesting behavior of the == operator and how you can make it behave as you need it for your use case.

When comparing Hashes in Ruby, the == operator compares the content of a hash recursively.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
 
my_hash = {
    :sub_hash => {
        :value => 42
    }
}

my_second_hash = {
    :sub_hash => {
        :value => 42
    }
}

my_third_hash = {
    :sub_hash => {
        :value => 21
    }
}

puts "my_hash == my_second_hash? #{my_hash == my_second_hash}"
puts "my_hash == my_third_hash? #{my_hash == my_third_hash}"


1
2
 
my_hash == my_second_hash? true
my_hash == my_third_hash? false

Unfortunately, when comparing objects of arbitrary classes, the default operator only compares the object identity.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
 
class MyClass
  def initialize(value)
    @value = value
  end
end

my_object = MyClass.new(42)
my_second_object = MyClass.new(42)

puts "my_object == my_second_object? #{my_object == my_second_object}"


1
 
my_object == my_second_object? false

If you want to do a deep comparison of objects of your class, you need to implement your own operator == by overriding the existing operator.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
 
class MyClass
  attr_reader :value

  def initialize(value)
    @value = value
  end

  def ==(other)
    other.respond_to?("value") && value == other.value
  end
end

my_object = MyClass.new(42)
my_second_object = MyClass.new(42)

puts "my_object == my_second_object? #{my_object == my_second_object}"


1
 
my_object == my_second_object? true

That was easy. But imagine this was a bigger class and someone else needed to add a property, not being aware of the existence of this operator and some other code depending on it to ensure no public member of the object changed. How can you ensure such a change doesn’t sneak in unnoticed?

I stumbled across the following solution when implementing an operator == for a class in the BOSH code together with my colleague Max.

As BOSH code is written in TDD – and your code should be as well – writing a test that breaks with a change as the one described above should ensure the operator to keep working. But how can such a test look like?

Consider the following change to our code above:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
 
class MyClass
  attr_reader :value
  attr_reader :value_new

  def initialize(value)
    @value = value
    @value_new = value
  end

  def ==(other)
    other.respond_to?("value") && value == other.value
  end
end

my_object = MyClass.new(42)
my_second_object = MyClass.new(42)

puts "my_object == my_second_object? #{my_object == my_second_object}"

To detect the variable @value_new has been added using rspec can be done with a test like the following:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
 
require './object_compare_op'

describe :MyClass do
  describe 'operator ==' do
    context 'when instance variables are modified' do
      let :obj do
        MyClass.new(42)
      end
      let :other_obj do
        MyClass.new(42)
      end

      all_members = MyClass.new(0).instance_variables.map { |var| var.to_s.tr('@', '') }
      all_members.each do |member|
        it "returns false when #{member} is modified" do
          eval <<-END_EVAL
            class MyClass
              def modify_#{member}
               @#{member} = 'foo'
              end
            end
          END_EVAL
          obj.send("modify_#{member}")
          expect(obj == other_obj).to(
            equal(false),
            "Modification of #{member} not detected by == operator.",
          )
        end
      end
    end
  end
end

The variable @value_new only has an attribute reader, so we cannot simply assign a new value. But this doesn’t stop you from changing the value. Not in Ruby. Using the eval in the test, we add a method for all existing instance variables of MyClass (one in each iteration) that modifies the member.

Afterwards, the newly added method is called to change the value of the member and the expect checks if the operator detects the modification. And – for our code above – will fail. Hence, whenever someone adds a new member to MyClass, he will be reminded to also it to the operator == by this test. Even if the code of test itself might not be as speaking, the output of the failing test is:

 Modification of value_new not detected by == operator.

In some situations you may want to exclude a member from this check as it is just internal or not important to the equality of two objects. To enable this, we added an exclude list for private members to the test. This adds a bit of complexity to adding new members to the class as the test will bother you and you also have to add the member to the exclude list, but it improves the safety of your operator ==.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
 
require './object_compare_op'

describe :MyClass do
  describe 'operator ==' do
    context 'when instance variables are modified' do
      let :obj do
        MyClass.new(42)
      end
      let :other_obj do
        MyClass.new(42)
      end

      all_members = MyClass.new(0).instance_variables.map { |var| var.to_s.tr('@', '') }
      private_members = %w[value_new]
      public_members = all_members - private_members
      public_members.each do |member|
        it "returns false when #{member} is modified" do
          eval <<-END_EVAL
            class MyClass
              def modify_#{member}
               @#{member} = 'foo'
              end
            end
          END_EVAL
          obj.send("modify_#{member}")
          expect(obj == other_obj).to(
            equal(false),
            "Modification of #{member} not detected by == operator.",
          )
        end
      end
    end
  end
end

With this kind of test, you can easily implement comparison operators for your classes that check for object equality rather than identity and ensure you do not forget to add new members of the class also to the comparison.
You can take a look at productive code in the BOSH code base hier kontaktieren. As you may see it’s not much different to what I presented here – it’s a universal approach to solve the problem.

A Christmas Poem

Dezember 19, 2018 7:31 am Veröffentlicht von 

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
 
#include <stdio.h>

/* A CHRISTMAS POEM
 *
 * Christmas is near,
 * brings relaxed atmosphere.
 * The devs are staying at home,
 * feeling bored like a stone.
 *
 * Left the winter outside,
 * but still not satisfied.
 * They are missing their code,
 * yet so tired of node.
 *
 * And if this feels like you - 
 * here is something to do.
 * Celebrate! It's written in C:
 * A compilable Christmas tree. */

              #
            define
           o printf
         #define O "%c"
        int main(){o(O,
       77);o(O,101);for(
      int l=0;l<2;++l)o(O
     ,114);o(O,121);o(" ")
    ;o(O,88);o(O,0x6d);o(O,
           0x61);o
           (O,115)
         ;o(O,0x0A);}

Blogpost: CloudFoundry Summit Europe 2018

Dezember 18, 2018 7:31 am Veröffentlicht von

After attending CloudFoundry Summit 2018 in Basel in October, I published an event summary together with my colleagues. This writing summarizes the talks that where interesting from the perspective of us as BOSH developers. You can find the post on the community page of SAP.

Recap and collected replays: Cloud Foundry Summit Europe 2018 — thanks @ManuelDewald and team! #cfsummit @cloudfoundryhttps://t.co/t7b6NjplyN

— SAP Cloud Platform (@sapcp) 6. November 2018

Watch all talks online via the youtube playlist.

Define interfaces in a duck typed language like ruby

Oktober 5, 2018 6:55 am Veröffentlicht von

In Java, it is very intuitive how interfaces are defined and used. You just create an interface in a similar way you would create a class and derive the classes, implementing the interface.

interface Drivable {
    public void drive(int meters);
    public void stop();
}

class Car implements Drivable {
    public void drive(int meters) {
        //start the engine
        //go for it
    }

    public void stop() {
        //stop the engine
    }
}

class Bagger implements Drivable {
    public void drive(int meters) {
        //start engine
        //start left track
        //start right track
    }

    public void stop() {
        //stop left track
        //stop right track
    }
}

class AutomatedDriver {
    public void forward(Drivable vehicle, int meters) {
        vehicle.drive();
        vehicle.stop();
    }
}

This results in a exlplicit class structure as depicted in the following class diagram.

However, in languages like ruby, interfaces are defined implicitly, which means that two classes implement the same interface as soon as they respond to the same interface. Take a look at our example as implemented in ruby:

class Car
    public void drive(meters)
        # start the engine
        # go for it
    end

    def stop
        # stop the engine
    end
end

class Bagger
    def drive(meters)
        # start engine
        # start left track
        # start right track
    end

    def stop
        # stop left track
        # stop right track
    end
end

class AutomatedDriver
    def forward(vehicle, meters)
        vehicle.drive();
        vehicle.stop();
    end
end

The classes Car and Bagger still both implement the Drivable interface. But as in ruby you use the so-called ducktyping, they both implement it implicitly by just responding to the same API, consisting of drive and stop. However, even in duck-typed languages, you might want to define and document your interfaces in a central point to make sure once you change it, all implementing classes do as well. You can do this by implementing unit tests to ensure the interface is fulfilled.

Following is an example of a rspec test to ensure our Drivable interface is implemented correctly.

shared_examples "a Drivable" do
    it {  expect(subject).to respond_to(:drive).with(1).argument }
    it {  expect(subject).to respond_to(:stop).with.no_args }
end

describe Car do
    it_behaves_like "a Drivable"
end

describe Bagger do
    it_behaves_like "a Drivable"
end

If the developer now changes something in the interface Drivable, he does so in the rspec test ensuring the interface. This test will fail for all classes that are expected to implement it but not yet do.

Even if it is not as intuitiv as it is in java, where your code just doesn’t compile if you fail to implement the interface, it is possible to define an interface and ensure it is implemented correctly in duck typed languages.

You might argue that you lose a bit of the flexibility of duck typing if you implement this for all your interfaces, and you are right! But in many cases, for example if the one defining the interface and the ones implementing it are different people, this is a very useful tool.

For example, imagine you are the author of a ruby library. A shared_example is a good and straight forward way to tell the users of your ruby gem what you expect their classes to behave like. Also, this will make them confident that if they upgrade to a newever version of your library, they will notice changes in the API by executing their test suite.

Use Ansible to clone & update private git repositories via ssh

Juli 7, 2018 7:21 am Veröffentlicht von

One of the first things I wanted to do when I started using Ansible was to clone a git repository on a remote machine as I keep configuration, scripts, and source code in github or gitlab repositories. Things that are not meant for the public, I store in private repositories that I want to clone via ssh. Cloning and updating them I now want to automate with Ansible.

There are different ways to go for this task:

  • Checkout the repo locally and copy it to the server via a Ansible synchronize task
  • Generate an ssh key on the server and allow cloning the repo with that key manually
  • Copy a local ssh key to the server and allow cloning the repo with that key
  • use ssh-agent to load the local key and forward the agent to the server
While it might be tempting to just copy an ssh key via Ansible to the remote server, I find this quite risky,  as it means you copy a secret to a persistent storage on a remote server. Also, if you version your Ansible playbooks in a git repository as well to be able to execute the playbook from somewhere else, the private key has to be versioned along with it.

Using ssh-agent, you can easily load your ssh key prior to provisioning the git repo on the remote server without copying it over, and without allowing access to your repo for a different key than the one you have granted access for development.
Let’s go through this via a simple example. Let’s say you want to run the following playbook, which includes ensuring the git repository github.com/ntlx/my-private-repo is up-to-date.

1
2
3
4
5
6
7
 
---
- hosts: webserver
  tasks:
      - name: Ensure repo is up-to-date
        git:
            repo: git@github.com/ntlx/my-private-repo.git
            dest: repos/my-private-repo
I assume you added your public ssh key to your github.com repository so you are able to clone and work on the repository locally. To clone the repository on the remote machine, you need to load your ssh-key to ssh-agent with the following command.

ssh-add ~/.ssh/id_rsa

Now we need to enable the forwarding of the ssh agent to the remote machine so we can access the loaded key remotely. There are different ways to do so, but I find it most useful to do it in your ansible.cfg like this:

1
2
 
[ssh_connection]
ssh_args=-o ForwardAgent=yes

That way, you allow the forwarding for all your Ansible-managed hosts at once.

Now you can go on executing your playbook and should be able to clone the repository on the remote host.

To make it even easier, we can add a task to load the ssh-key before executing the other tasks in the playbook. For this, add the local host to your Ansible inventory:

1
2
 
[local]
local_machine ansible_connection=local ansible_host=localhost

Now we can add a small shell task to load the ssh-key:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
 
---
- hosts: local
- name: load ssh key
  shell: |
      ssh-add ~/.ssh/id_rsa

- hosts: webserver
  tasks:
      - name: Ensure repo is up-to-date
        git:
            repo: git@github.com/ntlx/my-private-repo.git
            dest: repos/my-private-repo

When you now execute the playbook, you shouldn’t need to load the ssh-key before.

Guest Post on opensource.com: “Try this vi setup to keep and organize your notes”

Juni 19, 2018 9:14 pm Veröffentlicht von

Today my first blog post on opensource.com has been published, please have a look at it if your are, similar to myself, interested in an easy, intuitive, synchronized way to keep your notes.

You can find it on the opensource.com blog or just by following this link.

Thanks to the opensource.com team for collaboration and editing, I hope I will get the chance to write another one soon!

Thoughts on Members vs. Parameters

Mai 17, 2018 6:38 am Veröffentlicht von

Yesterday in a longer refactoring session, we stumbled across some open questions when it comes to member variables vs function parameters.

In a function with a huge number of parameters, we decided to create new class(es) to split this mess up a bit. As the new class first contained only one public function – because we moved one function out of a bigger class – we had to decide which of the parameters to choose for the input of the constructor and the actual function call, respectively.
Without being able to judge whether this is a good recipe currently, we split them up by the following classifications:

Does the variable change between calls of the public method?

If this is the case, in our it should be a function parameter. This was hard to decide for many of the input parameters, as in this state of the refactoring, every instance of the new class would be used only once to call the function.

Is the variable a member, a local variable, or a parameter in the caller?

We found it clean to treat all the parameters and local variables of the calling function that need to be passed to the new one as input parameters of this function. This should also make it easier to change the locally created object into a member of the calling class in future. Most of the member variables of the calling class have been turned into a member of the new class (although not all of them, as some could be decided based on earlier mentioned reasons).

Is the variable a pure input parameter, a complex object or subject of change?

We found it useful in our case to classify the variables we had to pass to the new class in three different types:

Pure input parameters

We created some structs to group the huge number of input parameters thematically, which turned out to be a good idea later in the process as we could find subfunctions and classes taking one of those groups and acting upon it. Also, we thought it would be a good idea to put those input only parameters into the interface of the function instead of the class constructor.

Complex Object

Some of the variables to handle are real objects, receiving messages from the new class. We decided to take them as members of the new class to enhance the object-oriented feeling of objects talking to each of their members.

Subjects of Change

There was at least one variable who changed it’s internal state which would be an input parameter to our new class. (We cannot be 100% sure, because we are coding in ruby, where we don’t have the possibility to const our variables and ask our compile, who might try to change it). As this would increase also the state, being held by the newly created class, we decided to put it as input parameter to the new method.

Quick and Dirty Java Service Template

Dezember 25, 2017 1:04 pm Veröffentlicht von

To get a java service running, you don’t always need to start a tomcat and deploy a war file, as described in an earlier blog post. Even if tomcat gives you a number of benefits, such as user management and database connection handling, sometimes you do not need this and just want to start up a small http server to provide a bit of functionality.

The template contents

To get such projects started as fast as possible, I created the quick and dirty java service template, where I collected stuff I find most useful to implement a new java service:

It uses the grizzly http server to provide the web server, which makes starting an http server running java code as easy as executing a jar file.
Often, communication between a web page and a java backend works easiest by sending a JSON document. This is why we include the gson JSON parser in the template.
In addition to some java code, we will also package static content, such as html files or images to the jar package. All this can be accessed via the same URL after starting the jar file.

Checkout the service template at https://github.com/NautiluX/JavaServiceTemplate.

Start up

To build and run the server, we use mvn. So compilation and start can be done using the following two maven commands:

mvn clean install

mvn exec:java

now we can access our small server at localhost:5678.

Customize

Now for some more details to show you where to put server code and static content.
The project is set up using maven. That means all our java code can be found in src/main/java. When we start the server, by executing mvn exec:java, the class org.ntlx.server.Main will be executed, which creates the http server.

package org.ntlx.server;

import com.sun.jersey.api.container.grizzly2.GrizzlyServerFactory;
import com.sun.jersey.api.core.*;

import java.io.IOException;

import org.glassfish.grizzly.http.server.*;

public class Main {

  public static void main(String args[])
      throws IllegalArgumentException, NullPointerException, IOException {
    Main main = new Main();
    main.startServer();
  }

  public void startServer() throws IllegalArgumentException, NullPointerException, IOException {
    ResourceConfig rc = new PackagesResourceConfig(“org.ntlx.service”);
    HttpServer myServer = GrizzlyServerFactory.createHttpServer(“http://0.0.0.0:9876/calc/”, rc);
    HttpHandler httpHandler = new CLStaticHttpHandler(HttpServer.class.getClassLoader(), “/ui/”);
    myServer.getServerConfiguration().addHttpHandler(httpHandler, “/”);
    try {
      myServer.start();
      Thread.currentThread().join();
    } catch (Exception e) {
      System.err.println(e);
    } finally {
      myServer.stop();
    }
  }

}

The http server is listening on port 9876 and is searching for service endpoints in the package org.ntlx.service.

GET endpoint

To implement the code for our service, we take a look at the example implementation of the service template. We want our GET endpoint as heartbeat to check if the backend is up and running. Therefore we want it to just return
{
    “getCall”: “succeeded”
}
The code for this simple get endpoint can be found in the class org.ntlx.service.SumService in the function dummyGet:

package org.ntlx.service;

import com.google.gson.*;
import javax.ws.rs.*;

@Path(“sum”)
public class SumService {

  @GET
  @Produces(MediaType.APPLICATION_JSON)
  public String dummyGet() {
    JsonObject result = new JsonObject();
    result.addProperty(“getCall”, “succeeded”);
    return result.toString();
  }

  …
}
We tell the http service, that we want requests that come in to the path /sum to be handled by this class. With the annotation @GET we tell it, that this function shall be called by GET http requests. With the @Produces annotation we get the http response header application/json. To create this simple json string, we use the Gson library.

POST/PUT/…

To create endpoints for other type of requests, we can do the same as for get. Lets have a look at the backend for our summing service, which should sum a list of numbers coming in as JSON.

  @POST
  @Produces(MediaType.APPLICATION_JSON)
  public String dummyPost(String body) {
    CalculatorInput input = new Gson().fromJson(body, CalculatorInput.class);
    JsonObject result = new JsonObject();
    result.addProperty(“result”, input.calculateSum());
    return result.toString();
  }

We tell the server we are listening for POST calls (the same could be done for POST, PUT, DELETE, etc.). We add a String parameter to the function, which will be filled up with the request body of the incoming POST call. To get the input, we again use GSON to instantiate an object of the CalculatorInput class, which executes our – very complex – workload in the function call calculateSum().

Static Content

Static content – like html files – must be located in the class path, so the http server can find it. That means we have to rebuild the jar file any time we want to change our static content. We told the http server where to search for it (/ui/) and where to place it (/). That means we can put our static content into the folder src/main/resources/ui, and they will be included in our jar file automatically and found by the http server under the path /. That’s why we are routed to the file created as src/main/resources/ui/index.html when we visit localhost:9876 after starting the server.