• Virtual Build Setup Part2: TeamCity

    This is the 2nd part of the series about the virtual build setup behind VPN. The first part can be found her.

    Now that the VPN is working we have 3 Nodes communicating via VPN:

    • vpn1 VPN Entry Server
    • teamcity TeamCity Host
    • teamcity-node1 TeamCity Agent

    It’s time to fill them with live. We used Docker to easily do installation. Luckily there are already some ready-to-use containers for TeamCity but in order to get a compatible build we had to patch them a little bit.

    What should be build

    • Scala Play! Application with some NodeJS dependencies for client-side-code
    • C++-Applications
    • The build should create ready-to-use Docker containers, so we need the privileged Mode
    • The build shall be done inside Docker containers to get them easily reproducible.

    Pitfall 1: Swap-File

    The cheapest Vultr-VMs comes with only 768MB of RAM. This is OK for the VMs, but Java allocates more memory upon startup. It’s necessary to create a Swap file. This Manual is handy.

    Pitfall 2: Docker inside VPN

    We do not want the Docker servers to listen on the public eth0 net interface, which it does by default. You can however tell docker to listen on a specific IP Address directly, but this is error prone as you have to tell it on each Docker invocation.

    Much more easily you can set Docker to automatically listen on one interface, using following configuration:

     # This comes into /etc/default/docker.io
     # You have to replace 10.8.0.10 with your VPN ip adress.
     DOCKER_OPTS="--ip 10.8.0.10"
    

    It is maybe also possible to restrict all trafic using iptables, but it turns out that docker writes in your iptables too (in order to create FORWARD-Calls to it’s VMs). This is a bad as you cannot just create a meaningful set and then backup it but you also have to check after each Docker call, that your configuration is still Ok.

    Main TeamCity Server

    We are using the Dockerfile nob13/teamcity-docker. A database server is necessary, we choose Postgres.

    The following decking.json-File will start up the TeamCity server together with a Postgres SQL server

    {
        "containers":{
            "teamcity":{
                "image":"nob13/teamcity",
                "port":["8111:8111"],
                "mount":["/data/teamcity:/data/teamcity"],
                "dependencies":["postgres:postgres"]
            },
            "postgres":{
                "image":"postgres",
                "mount":["/data/postgres:/var/lib/postgresql/data"]
            }
        },
        "clusters" : {
            "main":["teamcity", "postgres"]
        }
    }
    

    You can start it with decking.io:

    # As root
    apt-get install npm nodejs-legacy # Node.js and NPM
    npm install -g decking.io
    
    # The following calls in the same directory as decking.json
    decking create
    decking start
    

    You can also choose Fig, the configuration would be similar. After starting TeamCity for the first time, you have to cop the Postgres JDBC lib found here into /data/teamcity/lib/jdbc. This cannot be done from Docker but must be done manually. Then just continue the installation (set your master password).

    The database user is postgres, there is no user set. The only directory which are needed to backup is /data. It contains the database together will all TeamCity related files.

    Team City Build Agent Node

    Here we use the following Dockerfile [nob13/teamcity-agent-docker]. It is forked from Sjoerd Mulder because we need the MongoDB-executable inside the container to use MongoFaker for application testing.

    The startup doesn’t need any additional Containers, so there is no need to use decking.io, just execute:

    docker run -d --name teamcity_agent --privileged -e TEAMCITY_SERVER=http://teamcity.lan:8111 -p 90 90:9090 nob13/teamcity_agent:latest
    

    teamcity.lan is the VPN DNS-Name of the TeamCity Server.

    As soon as the Node has been started, you can authorize it inside TeamCity Main server, and voila: The build setup is ready to run.

    The node is able to create docker images because it is privileged.

  • Virtual Build Setup Part1: VPN

    Usually our customers bring their own build infrastructure. For older C++ projects I was deploying a set of 3 virtual servers (Ubuntu 32bit+64bit + Windows). But all this stationary computers are standing in the way if you actually do coding from where and when ever you want. Either you have only small upload, high energy consumption or you family doesn’t like this old boxes in your flat :/

    So let’s build in the cloud :) Travis is a number to expensive if you aren’t doing OpenSource only and just want to try things out. I wanted to do something else than Jenkins, so this time it’s about TeamCity, or VPN at first.

    The only problem is: nobody likes managing servers. So the main target was that the build server is not really webfacing. It’s running in it’s own VPN. The VPN can be entered by the Laptop and so it’s only necessary to keep SSH and OpenVPN services always up to date, a task much easier than patching proprietary software like TeamCity or Jira.

    Here is the setup, currently 3 VMs which are building and testing one Scala Play! Application. They are all hosted on Vultr, with 3x5$ per Month is cheap enough and keeps room to scale a little bit. You can even scale it down to only one node when you run all three services on one machine.

    The VMs are all 64bit Ubuntu 14.04 LTS.

    Server Setup

    • One VPN entry server
    • One TeamCity Server
    • One TeamCity build agent (can be on demand)

    OpenVPN Configuration

    OpenVPN is easier to configure than you usually think of it. You need one server and multiple clients, and a simple Public Key Infrastructure. I used EasyRSA 3. There were however some pitfalls:

    • Googling for OpenVPN related stuff got tricky since everyone is using it for private surfing
    • You want passwords without a passphrase for your server. Just add nopass to the certificat generation
    • To use a Mac as a client: Tunnelblick is a good choice. For Yosemite, use the current Beta. To get DNS working, try out the different mechanisms offered by Tunnelblick.
    • Use topology subnet on the server, as this is the better default now.
    • Enable client-to-client on the Server so that clients can talk to each other.
    • Disable ns-cert-type server on the Clients as this is obsolete, it’s enabled on Ubuntu 14.04 by default but not compatible with the generated keys in EasyRSA3.
    • Enable remote-cert-tls server on the Clients to avoid MTM-Attacks.

    DNSMasq Configuration

    Remembering IP-Addresses in the VPN is not very convenient. One way to solve it, is to let your own DNS server running, I choosed DNSMasq. However this is still a weak issue on the setup, as it’s very manually.

    OpenVPN usually gives out the same IP adress for each client. Client IPs are stored in a file called /etc/openvpn/ipp.txt. From there you can copy them into the /etc/hosts/ file and make dnsmasq server as a DNS for them. You can configure it to only listen to the tun0 interface (OpenVPN) only and push it to your clients via

    push "dhcp-option DNS 10.8.0.1"
    

    in the OpenVPN server configuration.

    Configuration Files

    Both are for Ubuntu 14.04 LTS.

    Note: This OpenVPN configuration is not suitable for private surfing.

  • Moving to jekyll

    This blog moved to jekyll. There may be some posts broken now, I am feeling very sorry about that. Broken images should be fixed now.

  • JPA: ElementCollection on a table with composite primary key

    Just for the record, because Googling did not brought us far on this.

    If you ever have a JPA2/Hibernate table with multiple primary keys and want to add an ElementCollection on that

    @ElementCollection
    List<Integer> testCases = new ArrayList<>();

    Chances that you get the following
    error message:

    org.hibernate.MappingException: Foreign key (FK_b10frvpxbd5uu483xde8d5qso:testcase_ids [XXX])) must have same number of columns as the referenced primary key (XXX)

    The problem is that Hibernate does not automatically add all your primary keys to the created table for the embedded collection.

    But it is possible to tell JPA which columns to use:

    @CollectionTable(joinColumns={@JoinColumn(name="id", referencedColumnName="id"), @JoinColumn(name="revision", referencedColumnName="revision")})
    List<Integer> testCases = new ArrayList<>();

    These tells Hibernate to add "id" and "revision" (the both primary keys in my case) to the generated table.

  • MongoFaker released

    We are proud to release MongoFaker.

    It is a small Scala Library which starts a MongoDB server in ~250ms.

    All you have to do is supply some mongod executable inside your $PATH variable (on Debian/Ubuntu it is available automatically after installing mongodb), and you can start as easily as

    import net.reactivecore._
    val instance = MongoInstance.start(MongoConfiguration())
    // MongoDB server up and running in a
    // temporary test directory.
    val port = instance.port
    val address = instance.address
    // Connect to port & address
    // Test your code
    instance.stop()

    Hopefully it helps other developers saving some seconds on each integration test run.

  • CMake Presentation

    Together with Manuel Herrmann I gave a short presentation of the CMake buildsystem at the Berlin C++ UserGroup on 21.01.2014.

    The slides are online, you'll find them here: CMake.pdf

    The first sample project was this OpenGL project from Manuel.

    The second sample was a small math library called smallcalc I wrote for µoscalc on iPad, demonstrating cross platform compilation to iOS and Android.

    Thanks to Manuel.

  • Android: Bubble Chain Reaction

    Bubble Chain Reaction is online. Its a small turn based game written in Scala for Android, demonstrating Scalas Matureness for Android Development. You can play it with 2-4 Players or against the computer. Two players have to place bubbles inside a field of 3x3 (or bigger). Once there are enough bubbles in a field, it explodes and the player will own all the neighbor bubbles. This may lead to chain reactions. The player owning the whole map will win the game.

    There are no needed permissions and the app contains no Ads.

    Have fun with it.

    device-2013-09-12-180221device-2013-09-12-180348

    Note: there is a similar game called Chain Reaction in the Play Store (this happens if you look for a nam just before you release ;-) ). The rules are similar, but my variant needs one more bubble to explode a field. The game idea howewer is somewhat older, I saw the Game running on a KC 85/4 in the 90s.

  • Scala 2.10/Android SBT up and running

    Some notes after spending the whole day how to get fxxxxing Android+SBT+Scala+IntelliJ up and running.

    • Setup the Android SDK, set all necessary paths etc. Don't forget to install the all relevant API Versions (See Android Doc)
    • Setup IntelliJ for Android Development (See Stackoverflow)
    • The documentation of fxthomas seems to be the most accurate: http://fxthomas.github.io/android-plugin.
    • Versin 0.7 of the SBT-Plugin just released; grab it https://github.com/jberkel/android-plugin It's necessary to compile it and to deploy to local repository
    • If you need Intellij-Support, you'll need fxthomas' idea plugin fork as the current version is broken for Android. Get it here https://github.com/fxthomas/sbt-idea. You have to compile it and it needs the dependency to android-plugin above. If it doesn't find 0.7-SNAPSHOT, rename the dependency in sbt-idea to 0.7 (the name just changed)
    • Now you can start with your Android Project, I recommend this template https://github.com/fxthomas/android-scratch.
    • Add the idea plugin you compiled above to to project/plugins.sbt
      addSbtPlugin("com.github.mpeltonen" % "sbt-idea" % "1.5.0-SNAPSHOT")
    • Try out android build and running
      emulator-start [emulatorname]
      start

subscribe via RSS