Nedtes

My ideas, thoughts and questions, but mostly just software

Running a Robots Cluster With ROS and Kubernetes

Kubernetes brings a lot of value to distributed systems. Including self healing, secrets management and tons of other things I am too lazy to mention. One area where kubernetes magic would bring so much value that is still in development is Robotics.

I have used ROS for more than 2 years and also used it in my thesis. And handling communication between robots, especially when we have a dozen of them can be a pain and since I have been using Kuberentes in my day to day job lately. I thought of experimenting with ROS and Kubernetes. But before I dive in, here’s a quick introduction to the two of them.

Kubernetes:

  • There are Nodes, those are worker machines (previously minions). This could be a VM or a physical machine
  • All nodes in a cluster are managed by a master node.
  • Pods are the smallest building blocks of Kubernetes. A Pod is a group of one or many containers.
  • A node can have one or multiple Pods running inside it.

ROS:

ROS uses a computational graph model with a pub/sub model for communicating data.

  • Nodes in ROS are processes that perform specific computational job. For example we can have a node responsible for moving the robot wheels and another for localization.
  • Master, this is the main component for a running ROS system. It ensures name resolution for nodes and maintaining communications between different Nodes.
  • Messages, Nodes communicate by sending messages. ROS provides a set of default messages types and also the ability to write our own custom message types.
  • Topics are the transport system between Nodes. A topic can have only one message type through it. And it can have subscribing nodes and publishing nodes.
  • To put it short, a Node will publish a Message through a Topic.

From reading previously about the big architecture points of both systems. We can come up with the idea how to run ROS on kubernetes.

  • We will need to have the master node tainted and run master ROS on it.
  • One Node will mean one robot. And we need to run one Pod per Node.

Here’s how we run ROS Master on kubernetes:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: ros-kinetic-master-deployment
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: ros-master
    spec:
      nodeSelector:
        dedicated: master
      containers:
      - name: ros-kinetic
        image: ros:kinetic-perception-xenial
        args: ["roscore"]
        ports:
        - containerPort: 11311
---
apiVersion: v1
kind: Service
metadata:
  name: master-service
spec:
  clusterIP: None
  ports:
  - port: 11311
    targetPort: 11311
    protocol: TCP
  selector:
    app: ros-master
  type: ClusterIP

You can see that we expose port 11311, which is the default port. Now any ROS Node can connect to the master. So far so good. We have a running ROS master that can manage ROS Nodes for us.

Now we have a problem: Service in kubernetes need to expose a specific port and don’t have support for random ports support.

That’s why we need to add a headless service. In a headless service, each pod gets its own DNS entry. Which will be helpful for us addressing specific pods.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: listener
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: listener
    spec:
      containers:
      -  name: listener
         image: ros:kinetic-perception-xenial
         args: ["rostopic","echo","such_topic"]
         env:
          - name: ROS_HOSTNAME
            value: listener
          - name: ROS_MASTER_URI
            value: http://master-service.default.svc.cluster.local:11311
---
apiVersion: v1
kind: Service
metadata:
  name: listener
spec:
  clusterIP: None
  ports:
  - port: 11311
    targetPort: 11311
    protocol: TCP
  selector:
    app: listener
  type: ClusterIP

And for the talker we run

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: talker
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: talker
    spec:
      containers:
      -  name: talker
         image: ros:kinetic-perception-xenial
         env:
          - name: ROS_HOSTNAME
            value: talker
          - name: ROS_MASTER_URI
            value: http://master-service.default.svc.cluster.local:11311
         args:
            - rostopic
            - pub
            - "-r"
            - "1"
            - such_topic
            - std_msgs/String
            - "Kubernetes is the best"
---
apiVersion: v1
kind: Service
metadata:
  name: talker
spec:
  clusterIP: None
  ports:
  - port: 11311
    targetPort: 11311
    protocol: TCP
  selector:
    app: talker
  type: ClusterIP

That’s it for now, you can check the listener logs to see the echo statements inside it.