Upgrading Ubuntu MATE from 18.04 to 20.04

The automated distro update feature of Ubuntu used to be very hit or miss.  It wasn't unusual to finish with a bricked system and have to start all over.  A while back, Canonical seemed to decide that this was too much trouble and removed it from Ubuntu.  For the last decade or so I got used to just doing a fresh install, maybe clearing the root file system if it was on a separate partition.

Now this feature is back so I thought I'd give it a try on my Ubuntu MATE 18.04 laptop.  I thought this would be a decent challenge.  It's a dual boot UEFI laptop with Windows 10.  It has a weird feature where as far as I can tell, it is hard-coded in the BIOS to boot /EFI/Microsoft/Boot/bootmgfw.efi.  (I've been copying /EFI/ubuntu/grubx64.efi to /EFI/Microsoft/Boot/bootmgfw.efi for 7 years.  Not making this up.  See Ubuntu – How to get the HP laptop to boot into grub from the new efi file).

This worked pretty good for me.  There was no trouble with the dual boot configuration.  

I needed to reinstall various things that are not part of the Ubuntu distro but that come from a PPA, things like Brave, Docker, Dropbox, Google Chrome, Percona, VirtualBox, but this was expected.

I needed to reinstall Postman using Snap.  I'm not exactly sure how I installed Postman in the first place.  I find Snap kind of annoying, why do we need two different package managers, apt and snap?  Before you know it, Linux will be like the Mac with half a dozen.  But anyway, this worked fine.  Chromium has also become a snap.

Hibernate (save to and resume from disk) can be tricky to set up, but the distro upgrade left my hibernate configuration alone and it continued to work, afterwards.

In the past I tried a technique to automate the re-installation of  packages from the old distro on the new one.  Example: What does the “apt-get dselect-upgrade” command do?  I didn't get good results from doing it this way because the inventory of packages changes a lot from one distro to the next and it still required some manual steps to get a collection of packages that would work in the new distro.  The distro upgrade takes care of all this for you now, keeping all the packages that are still supported, removing the ones that are not.

One thing the distro upgrade didn't catch was the Beats Audio configuration.  I don't think I bothered with this when I upgraded from 16 to 18 a couple of years ago, so that tells you something about how much I use audio on this computer.  Here is a page that outlines the basic approach: Beats Audio On Linux.  Here are the HDA Jack Retask settings that are working for my HP Envy 17t-j000:

Generally, I was pretty happy with this experience.  Expecting the distro install to figure out the Beats Audio configuration would be unreasonable since these pinouts are not documented anywhere.  Other than that, I probably spent no more than 2 hours on this, mostly updating the non-Ubuntu PPAs (not including downloading time, which was considerable, maybe another 2-3 hours).

I'm also feeling pretty upbeat about the computer itself, which is still soldiering on after over 7 years.  This is the longest I've ever had a laptop.  I'm not really feeling the obsolescence pressure to upgrade as before, which in the past has mainly come from inadequate memory.  The current laptop has 16 GB which is still plenty for the foreseeable future, assuming I don't start needing to work heavily with Kubernetes or something.   

My own attitude towards this has changed a lot, too.  It used to be like Christmas morning when I installed a new distro.  There's still a tiny bit of that, but mostly I just want things to work now, with a minimal investment of time and effort.  Part of the reason for that is my career got a lot more challenging after 2011 and I was spending more personal time either learning the new stack at work or cramming for the next interview, so the Linux hobby project had to go on the back burner.  Also, the changes these days are evolutionary rather than revolutionary.  Linux is still my favorite desktop OS, but it's mature now and a lot of the excitement is gone.

nvidia-smi now works out of the box for me, where before I had to manually load CUDA and experiment with a bunch of different nvidia drivers to get this to work:

Maybe I'll take another look at tensorflow...

Here's a few notable changes in Ubuntu 20.04:

• No more 32 bit support.  My 'travel' computer is a 32 bit Dell Latitude 630, so I might have to consider other Linux distros if it's still working in 2023...
• Nvidia drivers are included with the distro

Links:

• How to upgrade from Ubuntu MATE 18.04 or 19.10 to 20.04 LTS
  
• Ubuntu MATE 20.04 Release Notes
  
• What You Need to Know About Snaps on Ubuntu 20.04
  

Uberconf 3

On Day 3 I went to Brian Sletten's Machine Learning sessions:

Machine Learning: Overview

"The term machine learning refers to the automated detection of
meaningful patterns in data."

Machine Learning: Natural Language Processing

Machine Learning: Deep Learning

Research into neural networks began in the 50s with the concept of emulating a neuron by summing a set of weighted inputs, with a response being triggered if the combined values exceeded a threshold.  This was known as a Perceptron.  A perceptron is a single-layer neural network.

Deep Learning refers to the use of neural networks that are more than one layer deep.

Machine Learning: TensorFlow

This brought things a bit more down to earth with the focus on a particular software implementation of machine learning algorithms.

This topic probably comes closest to something I'd be able to apply at work.  As I mentioned earlier, there was not a very strong lineup of Big Data topics.

TensorFlow uses GPGPU (General-purpose computing on graphics processing units) to accelerate processing.  I was aware that bitcoin mining software did this but this was the first time I saw this acronym or really became aware that GPGPU was being used for a lot of other things.  This is your excuse for getting a really expensive, top-of-the-line graphics card.  If we start applying machine learning algorithms at work, my guess is we will build on top of Spark MLib since we already have such a big investment in the Hadoop, Scala, Spark stack.  It's weird that Brian didn't mention this at all.

There's a JavaScript API called TensorFlowJS that allows you to run ML in your browser!  Your phone!  IOT devices!  And here's another thing you can debug in Chrome DevTools.

Links

• A visual introduction to machine learning
• Keras: The Python Deep Learning library
• Dl4j Deep Learning for Java
• TensorFlow.js
• scikit-learn: Machine Learning in Python
• Brian Sletten's Data Science and Machine Learning resources
• Vulkan
• Welcome to the Jungle Or, A Heterogeneous Supercomputer in Every Pocket

Books

• Doing Data Science
• The Master Algorithm: How the Quest for the Ultimate Learning Machine Will Remake Our World
• The Theory That Would Not Die: How Bayes' Rule Cracked the Enigma Code, Hunted Down Russian Submarines, and Emerged Triumphant from Two Centuries of Controversy
• Machine Learning with R ("Make sure you get the second edition")
• Deep learning with Python
• Perceptrons by Marvin Minsky (downloadable)

A few miscellaneous comments about the show in general...

Women are definitely better represented than in the bad old days.  I would say the percentage of women has increased from around 5% to 20 or 25%.

The Mac is still a popular choice and I didn't notice anyone else running Linux.

Ditch Gradle, maybe reconsider the whole corporate sponsorship thing.

Uberconf 2

Today I did 5 back-to-back sessions with Venkat Subramaniam.  If you've never had the pleasure, he is a high-energy and fun presenter who seems to have mastered quite a wide range of software development topics.

Exploring Modern Javascript

This was a fast and fun presentation.  I'm up to speed enough on JavaScript that I was able to keep up and complete the labs in time.  Modern JavaScript has map, reduce and filter, cool.  I got a little more practice with Visual Studio Code.  Venkat was going pretty fast and asked us to take a short break and even so, he just barely finished, so this might be a longer session that was trimmed to 3 hours.

Creating React Applications

I didn't fare quite so well on this one.  Why do they schedule presentations during nap time?  Also, I really could have benefited from spending a couple of hours doing the React 'hello world' exercises beforehand, the way I did with Vue.  Venkat was still zooming through the material like the Energizer Bunny.  When we got to the first exercise I found myself just staring at an empty JavaScript file without a clue.  I finished up the lab at dinner (with the benefit of having the solutions in front of me).  This wasn't a total loss but I definitely could have gotten more out of it.

Taking polyglot programming to the next level with GraalVM

This is a very cool and promising new technology.  I didn't have any trouble bringing up "hello world" using the GraalVM javac and java and building the insanely fast native image.  Being able to debug other languages in Chrome DevTools is another cool feature.

A fire alarm livened up one of the afternoon sessions.  Never a dull moment at Uberconf!

Panel discussion

One of the better questions was 'What strongly held opinion have you done a 180 on?'  Several speakers had responses along the lines of 'I used to think it was all about technical talent and skill, but now I realize relationships, collaboration and soft skills are more important.'  One panelist said he had pulled a 180 on Microsoft ('I used to be an irritatingly passionate Apple fanboy').

Here are some of the books mentioned by the panelists:

• Delta-V by Daniel Suarez
• You Are Not A Gadget by Jaron Lanier
• Heart of the Machine: Our Future in a World of Artificial Emotional Intelligence by Richard Yonck
• Thinking Machines: The Quest for Artificial Intelligence--And Where It's Taking Us Next by Luke Dormehl
• The Broken Earth Trilogy by N K Jemisin
• The Coddling of the American Mind: How Good Intentions and Bad Ideas Are Setting Up a Generation for Failure

There was an amusing moment where several of the panelists were describing their awkward manual systems (e.g. spreadsheets) for keeping track of books they want to read, books they have read, etc.  There was a barely audible comment from Brian Sletten 'goodreads.com'.  Goodreads is an awesome tool to manage your reading and share ratings and reviews with your friends about books you have read that I have been using since 2012 or so.  Be my goodreads friend.

Uberconf 1

This was the longest day, 7:30-20:00.

Highlights:

A Vue Perspective (3h) and Testing Vue with Raju Gandhi.
Based on my extremely limited exposure, Vue seems like the most elegant of the "big three" client side frameworks.  It's the framework that lacks a big corporate sponsor (Angular comes from Google and React comes from Facebook).

I used Visual Studio Code for the coding parts.  It feels like I am getting over the hump with this tool.  This free Microsoft IDE has really taken off in the last 2 years or so.  The thing about it that first caught my attention is its support for Spring Tool Suite 4.

Craig Walls' Securing Spring REST & OAuth was excellent.  I have used Spring Security at work a bit but was still feeling like it hadn't quite clicked for me, and this helped.  Oauth is a riddle wrapped in a mystery inside an enigma, but Craig makes it seem easy.

My final 'Uberconf after dark' presentation was Neal Ford 'Stories Every Developer Should Know'.  This was an entertaining collection of cautionary tales about projects that went disastrously wrong and lessons we can learn from them.  I never knew that Enterprise JavaBeans came out of something called the San Francisco Project at IBM.

Here's a couple of presentations from Spring One Platform 2017 about STS 4 and Visual Studio Code:
Spring Tools 4 - Eclipse and Beyond 
Erich Gamma at SpringOne Platform 2017 Visual Studio Code 

The "Keynote" with Hans Doktor of Gradle was just a naked infomercial.

Uberconf 0

Really impressive performance by the keynote speaker, Jessica Kerr.  All the audiovisual stuff died and she spoke from memory for 45 minutes or an hour.
The Origins of Opera and the Future of Programming Or: Collective problem solving in music, art, science, and software. appears to be identical or very close to the keynote speech.

Night before Uberconf 2019

Uberconf

It's been almost 5 years since my last NFJS event.

Back then I was complaining that there wasn't much new under the sun in the Java world.  The picture is a bit different today.  Java 8 and to some degree 9-12 have mostly addressed the concerns about the stagnation of the Java language itself.

The insane proliferation of JavaScript frameworks has shaken out somewhat with Angular, React and Vue emerging as the "big 3".  I work in AngularJS now and am feeling a bit less angstful about the whole client side than I did back in 2014.

One thing that is conspicuous to me in this year's lineup is quite a few more women presenters than in the past.  It will be interesting to see if there are more women attending this event which has traditionally been 95% male.

As always there are a bunch of agonizing choices to make.  Anybody got a Time-Turner?

A big focus at my employer right now is big data.  We are using Hadoop, Spark and Scala.  I think I first heard about Hadoop at NFJS 2010.  I see Scala on the schedule (cool language in its own right aside from big data) and a bunch of Machine Learning, but otherwise not that much.  I guess big data is ancient history these days and not that much of a focus in and of itself.

There's several talks about the recent Java releases and modular Java which kind of makes sense.  I gather a lot of users are used to sitting on the sidelines for the first year or so of a new Java release and aren't really up to speed on the new 6-month cadence.

Looks like Kotlin is a fairly hot topic.  I would kind of like to jump on this bandwagon but I'm still striving for Scala mastery... maybe next year.

There's a ton of Angular, React, Vue and JavaScript and I will probably be spending a bunch of time in these.  I'm doing better with client side than I have been for a long time but still see this as a weak area, especially with things moving so fast.

Craig Walls is going to be there with half a dozen or so Spring presentations.  Spring has been a special focus of mine for a long time.  In a way I'm working in Spring almost every day but all the new features like Reactive aren't that useful to my employer's business model, so I'm not really learning and getting hands on with something new every day.

I'm going to try to catch at least one of the GraalVM presentations, sounds cool and could be the wave of the future.  There's a lot of emphasis these days on performance and getting the fastest possible startup time for microservices.

Microservices, architecture, dev/ops, cloudy stuff, agile, security, soft skills...  It looks like NoSQL is pretty much played out as a topic.  There's still a couple of AMQP talks though Kafka is the new messaging hotness.

One last thing I'll mention, there's a talk called Web Components: The Future of Web Development is Here.  I went to a couple of talks about web components and polymer back in 2014, and, blown away by the clean, component-based model, wrote "The way most web development is done is going to completely change in the next year or so."  Well, that might be true, but not because of the widespread adoption of Web Components I envisioned.  But now, 5 years later, they really mean it!

Links

• React vs Angular vs Vue.js — What to choose in 2019?

Spring Boot, Spring Data JPA, jackson-dataformat-xml

At work, we had a RESTful web service application implemented in Groovy/Grails.  This probably started out as a SOAP web service and was then ported to Grails 1.1.1 (2009).  It worked, but had become almost unmaintainable due to the archaic tech stack.  Also, there was a desire to move away from Grails, which not many of the team knew, in favor of Java 8 and Spring.

This was about a one month project.  It built on the experience I got doing RESTful Web Service in Spring.

Since I was using Spring Boot, this web application could have run as a stand-alone Java app using the embedded servlet container support, and this is how I was debugging it at first, but the requirement was to have a drop-in replacement for the old Grails WAR file running in Wildfly, so that was an additional wrinkle.

The original service, which my program had to emulate closely, did not use HTTP methods in a RESTful manner, but instead mostly used GET and POST for everything.  This alone ruled out the use of Spring Data REST, so I needed to write controllers.

I started by creating a suite of integration tests which could be pointed at either the old or the new service.  This was invaluable for verifying that the new service performed identically to the old one.  Many problems in my implementation were exposed in time for me to correct them as a result of running these tests.

The application consists of a collection of endpoints, each providing read/write access to one table in the Oracle database.  Spring Data JPA really shone here.  I rarely needed to give much thought to the database access part and in the one case where I needed to do something outside of the method-name based query model (next value from an Oracle sequence), the native query escape hatch was there.  I think I like this better than GORM.  GORM tries to make everything easy for you but I always wound up spending a lot of time resolving issues with transactions, sessions and cascading... these issues just seem clearer and less hidden in Spring Data JPA.

For rendering the responses in XML, I used jackson-dataformat-xml.  See How to write an XML REST service.  At first I tried using the same classes for the database entities and the POJOs to be rendered into XML in the response, but the complexity of that soon got out of hand and I resorted to creating a separate set of DTO classes for the response POJOs.  This was the most unsatisfying part of the task for me, having this whole array of DTO classes with conversion methods ('from' and 'to')... it added a bunch of just infrastructure coding that had nothing to do with the business logic.

Going into Wildfly was mostly straightforward.  The one exception was authentication/authorization.  We had a custom Wildfly security domain which accessed LDAP for user data and Oracle for group and role data.  I was not able to find a generic way to support any Wildfly security domain with Spring Security.  I think the way to do this is with a custom authentication provider and user details service but the clock ran out on me and we went with something much simpler.

In the Wildfly environment I also needed to switch to using JNDI to resolve an Oracle data source configured in the container, but this is a breeze in Spring boot with

spring:
  datasource:
    jndi-name:    JNDI name of your data source here

in the application.yml file... this didn't take more than half an hour to figure out and make the change.

I had some difficulty at first getting the logging to work the way I wanted it to with the application deployed into a container.  I started out trying to keep the original Log4j 1.2.x configuration, but that doesn't seem to be very well supported in Spring Boot, and eventually I gave up and used Logback with a logback-spring.xml file.  This is one of those areas where there are a hundred different ways to do something and it's not always clear how to proceed.

Similarly, when it comes to configuring your Spring Boot application, there's such a wealth of features to support this, it's not always obvious what's the best way to meet a given requirement.  I wound up with two sets of application-(profile).yml files, one for running the application and one for running the integration tests

The service has been running in production for 3-4 weeks now.  This is the most ambitious Spring Boot and Spring Data project I have tackled and also marks an important milestone in our team's adoption of this technology.

Java 11

I worked through this example:

It’s time! Migrating to Java 11

It was mostly pretty straightforward.  One thing I couldn't figure out, while trying to run my modularized Spring Petclinic JAR, I was getting this error:

Two versions of module javafx.base found in target/modules (javafx.base-11.0.0-SNAPSHOT-win.jar and javafx.base-11.0.0-SNAPSHOT-linux.jar)

I looked in target/modules and discovered what looked like 3 platform-specific versions of the JavaFX JAR.  I removed them manually

rm target/modules/javafx.base-11.0.0-SNAPSHOT-win.jar
rm target/modules/javafx.base-11.0.0-SNAPSHOT-linux.jar
rm target/modules/javafx.base-11.0.0-SNAPSHOT-mac.jar 

and then I was able to run the JAR.

Googling the error yielded no matches.  Why does this non-graphical, server-side application even have any dependency on JavaFX?  What's up with 4 separate versions of this JAR?

Still breathing

I changed jobs about 13 months ago and there hasn't been much time for blogging since then but I think I'm going to start making the effort again.

The new technologies I'm learning fall into 2 categories.  On the one hand, there's Big Data stuff like Hadoop, Scala and Spark.  On the other, there's client-side web development with AngularJS.  So these things are my learning priorities for the moment.

Almost all the Java programming we do is with Spring 4.3.  I was trying really hard to get into an environment where I'd be working in Spring all the time, so that worked out good.  My employer is running a couple of years behind the latest and greatest from Pivotal so I've done some training sessions there about Spring Boot and Spring Data and I think we're getting close to the critical mass where everyone sees the value of these innovations and the team as a whole is ready to adopt them.

JHipster is a technology I've got my eye on right now, seems like it might be a good match for what we are doing at work once we've adopted Spring Boot.

Automated testing has been a focus of my career for the last 8-9 years and we're currently making a priority of improving code quality, so there has been fertile ground for these ideas.  I've been working a lot with one of the other developers who is also passionate about testing.  We have done a few presentations and I'm also exploring some techniques I hadn't found the opportunity to use such as Spring Embedded databases.

We use Wildfly, so I got to renew my love affair with JBoss, sort of.  This decision was made to take advantage of Wildfly's support for JMS, but IMO the company would be better off separating the servlet/JSP container and the messaging server.

I'm trying to get up to speed on client-side automated testing tools like Karma, Jasmine and Protractor.  I'd have to describe my progress today as 'not over the hump yet'.  I've written a few tests in both Karma and Protractor but they don't come rolling off my fingertips.  It's still a big effort.

The Angular version we use at work is AngularJS (i.e. Angular 1).  Last time I looked, Angular was on version 6, so that's a bit frustrating.  I really need to focus on mastering what we use at work first, but there's a slight sense of wasted effort coming up to speed on a legacy framework.

The IDE we use is IntelliJ IDEA.  It's been fun using this and I have to say I now feel like this is the best Java IDE I've used, but I feel a bit guilty using a non-free tool.  I've played around a bit with Visual Studio Code and could see investing more time in that.  It seems pretty fast and powerful and will also support Spring Tool Suite 4.

The cool stack for Big Data these days seems to be Scala on Spark on Hadoop.  I was briefly interested in Hadoop after an NFJS presentation a while back but haven't really kept up with it.  Our direction before this was Cassandra and I learned a bit about Cassandra before this.

I'm about 2/3 of the way through a Coursera class called Functional Programming in Scala.  I would say the Scala language itself isn't especially hard but doing absolutely everything with functional programming will tie your brain up in knots like a pretzel.  I thought I was good at this from years of being a Lisp guy but what I quickly realized is what I used to do is use recursion where recursion seemed comfortable and procedural techniques where they seemed comfortable.  Pure functional is harder.

Comparing what I'm doing now with my Learning Priorities of 2015, Cloud Foundry and Gradle have fallen by the wayside.  We use Maven at my current employer and I don't think any of the places I considered before that were looking for Gradle.

'More Spring' was a good bet.  I've always been a bit skeptical about certifications, but in restrospect I feel the Spring Certification was a good move that got me back up to speed on the Spring basics and a few new developments at a time when I was getting rusty.

We were starting to use Docker a little at my previous job and it plays a small role where I'm working.  I would say this is good stuff to know but hasn't yet been critical for me.

Jenkins and CI were important at my previous job.  Now I'm a user of the Jenkins CI server but I'm not responsible for it as before.

TextMate, meh.  It was OK.  I used it for a few months but eventually went back to The One True Editor.

Currently reading: Soft Skills.  Keep up on what I'm reading at Goodreads.

I'm generally feeling like my career is back on track and it's a great time to be an active developer.

Angular2

I'm feeling a little weak on the client side these days so I'm looking into Angular.  I watched Angular 2 Tutorial for Beginners: Learn Angular 2 from Scratch and completed the Tour of Heroes tutorial.

This is a big tutorial.  I spent something like a day on this working half an hour here and an hour there over the course of about 2 weeks.  

There was one little bug in it towards the end, unresolved dependency on angular-in-memory-web-api which can be fixed by running

npm install angular-in-memory-web-api --save

in the project directory.  See Angular2 Tutorial (Tour of Heroes): Cannot find module 'angular2-in-memory-web-api'.

There's a lot to this and so far I only scratched the surface.

Angular reminds me a bit of Tapestry with its use of components made up of templates, CSS and code.

Speaking of components, what ever happened to Web Components?  3 years ago I wrote that within a year all web development was going to be done with web components but I feel like you don't hear much about it these days.

Vaadin

They had a pretty good presentation at Boulder Java User's Group yesterday about Vaadin framework.

This reminded me of a framework I used years ago called ICEFaces with a similar approach of providing access to dynamic, client-side functionality while shielding the developer from having to work directly with JavaScript.  The architecture is very similar to the approach ICEFaces uses, with a JavaScript engine running in the client's browser receiving and processing updates to the DOM from the server-side component.

Vaadin uses GWT.  I played around with GWT a long time ago but it always struck me as sort of a crutch for the JFC/Swing developer transitioning to web development, and these days it's not like there's a lot of those.  But the code in the demo app seemed pretty reasonable.  I didn't pick up on the GWT connection until someone mentioned it in the Q&A so it wasn't a big distraction or anything.

The presenter Marcus Hellberg worked through this example in about half an hour giving a sense of how easy to use and powerful this framework is.  It was nice to see someone using Spring in one of these presentations and it is cool that Vaadin is available right there in the Spring Boot Initializr.  The demo app demonstrated some low-key dynamic effects like making DOM elements appear and disappear.

Bootstrap

So I've got this ancient relic of a web site.  I started this when I was at AT&T in 1995 and it shows.  Seriously, don't laugh at the content.  I used to try to keep it up to date but at some point other things just seemed more important.

Back about 3 years ago I read something about Bootstrap and thought, oh, that looks cool, I'll have to give that a try, then forgot all about it.  Until today.

A key feature of Bootstrap is that it's mobile-friendly, in fact, mobile-first so you should be able to resize the above page or even view it on your phone and it will stack the columns and otherwise make it look reasonable on the smaller screen.

So here is my foray into Bootstrap.  See especially the links page which uses the cool dynamic tabs feature.

Bootstrap allows you to work at a higher level, focusing more on your content and not so much on the ins and outs of HTML, CSS and JavaScript.  I'm actually looking for a way to start easing back into JavaScript but this framework really insulates you from it, at least at this simple-minded level I'm working at so far.

Spring Data REST

Last week I blogged about doing a RESTful Web Service in Spring.  Towards the end of that post I mentioned that there's another module in Spring Data which can automate some of this work called Spring Data REST.

Following through on that idea, I rewrote essentially the same service here using Spring Data REST.

From what I've seen so far, Spring Data REST helps you in two different ways.  It relieves you of having to write a controller, at least for basic CRUD methods, and it also enables Hypermedia As The Engine Of Application State (HATEOAS).

When I started trying to get my Spring MVC Test Framework integration tests working again, the first thing I noticed was the following test failing:

My test was expecting application/json but it's now returning the HATEAOS mime type application/hal+json.

Spring defines these in the MediaType class.  They don't seem to have one of those for application/hal+json.  I considered just not checking the mime type... But I googled a bit more and found Configuring Spring Data REST and Basic settings for Spring Data REST in the documentation.  I used the following configuration to change the default mime type of the JSON responses back to application/json:

The next thing that was different has to do with the argument to methods which operate on an existing entity.  My original code used a String "name" to identify the entity, but the Spring Data REST methods all use the "id" field.  Here's how I wrote the controller method to get a single entity by name:


But I'm not writing the controllers any more... Spring Data Rest creates a 'GET' endpoint taking an 'id' value, and it also creates one for my findByName() method in the Spring Data JPA repository interface.  Here's my tests for these two cases:


I rewrote the remaining tests to just use the id instead of the name.

One other change was needed... since Spring is now automatically generating a REST endpoint from the findByName() method in the repository interface, instead of me just calling it from the controller method, I needed to give that an annotation to instruct it how to handle the HTTP request parameter as its input:


Spring Data REST strikes me as a huge win for green field development situations where you just drop a dependency in the POM file (or Gradle build file) and, whoosh, instant REST endpoints for all the domain classes.  It might be more problematic to use this in a legacy app where the Spring Data REST endpoints might not match the conventions of the existing endpoints.  Also, the HATEOAS stuff, yeah, it's the semantic web wave of the future, but it is a bit different and might take some getting used to if this is your first exposure to it.

Java 8

Back around November, I started working problems at Hackerrank.  One of my goals was to learn and get better at Java 8.

Frequently in these problems I had a need to print out an array of integers.

Before we get started in Java, the programming language I've mostly used at work for the last 3 years or so is Groovy, doing application development in Grails, writing Jenkins jobs, and just for general scripting.  Here's how you print an array of integers (int[] a) in Groovy:

It's super-easy and concise.

When I was still working in Java 7, I wrote something like the following code:

Hackerrank sometimes uses huge arrays, like 50K or 200K, so I'm efficiently using StringBuilder.  But the whole thing is super-verbose.  Java 7 lacks anything like the join operator so I'm reduced to implementing it myself, more or less.  There's five lines of code here devoted to adding a space before every character but the first.  The Groovy solution is so concise, it hardly makes sense to write a method for it, but here it would be crazy not to.

Java 8 String.join() finally gives us a string join method like the one I was using in Perl over 20 years ago.  Cool!  Here's the way I was doing this in Java 8 for a while:

That's not bad.  It's a big improvement over Java 7.  It's actually a one-liner, though the line is kind of a long one.  It's basically doing the same thing as the Groovy code, except I need to explicitly shepherd the output through a series of type conversions.

There's still a way to make this a little bit better, though.  In order to come up with the second argument to String.join(), I'm converting my array to a stream, then doing another conversion to get a Collection.  It turns out that the Java 8 team provided a reduce method Collectors.joining() which eliminates the need for String.join().  This makes it possible to do everything in a stream instead of the awkward series of conversions I had before.

And it's a bit more concise, too.

It might be tempting to look at this one example and say 'Groovy wins.'  Well, after all, Groovy is built on top of Java such that the new features of Java 8 are available in Groovy, so, yeah, Groovy does win 😎.

My initial reaction to Java 8 lambdas was 'oh, Java is finally catching up to something Groovy has had since, what, 2007?'  But it's important to understand that streams and the implementation of the filter-map-reduce abstraction implemented by Java 8 is a new thing that didn't exist in Groovy before.  In recent years we've seen a lot of interest in languages like Scala and Clojure which support a functional style of programming in which the creation of mutable state is avoided for the sake of concurrency.  This is what the streams feature of Java 8 is all about, parallelism.

RESTful Web Service in Spring

I've been spending a lot of time at Hackerrank lately, also getting up to speed on Java 8, but I suddenly realized I hadn't done much with Spring for a while.

So, this is maybe about an 8 hour effort over the course of 2 weeks or so.  It's a RESTful web service for Pokemons.

Writing a RESTful web service in Spring these days is not quite as easy as in Grails, but it's really pretty easy.  I just used MVC and @RestController.

For integration testing I followed the example in Building REST services with Spring.  I had glanced at the documentation for Spring MVC Test Framework before but never really used it, so this part was mostly new to me.  It lets you test MVC requests without bringing up a full Servlet container.

I'm aware of Spring Data Rest.  Here's a pretty good presentation: Advanced Spring Data REST - Oliver Gierke.  I'm getting ready to give this a try but wanted to go through the exercise of doing the REST implementation 'by hand' first.

Ubuntu MATE 16.04

I was a solid Ubuntu guy from about 2008-2011.  I walked away mostly due to the decision to replace my beloved Gnome 2 with Unity as the default desktop.  I played around with Xubuntu, Lubuntu, even Kubuntu; I tried Gnome 3, nothing satisfied.  I wound up following the same path as a lot of other disgruntled Ubuntu users by switching to Linux Mint whose MATE desktop offers an experience very like Gnome 2.

I've been happily using Mint MATE for about 4 years.  The last Mint 17 upgrade 'Rosa' (17.3) was having some issues on my laptop, though.  The display would lock up once in a while, a condition I could clear by switching to another virtual desktop and back (CTRL-ALT-F1, ALT-F8).  But it was still a nuisance.  Another little thing that was going wrong is this release didn't seem able to support an external HDMI-connected display (worked fine on the same hardware with Windows 8.1).

I heard that Ubuntu 16 was supporting MATE so I decided to give that a try.  I've been running Ubuntu MATE 16.04 for about a month now.

The 2 display issues I was having went away, screen locking up and external display not working

Weirdly, I can change my wallpaper, but user-supplied desktop backgrounds seem to be forgotten as soon as you switch away from them, so adding a bunch of my own backgrounds to the system-supplied ones and having them all available from then on as I am used to doesn't seem to work any more.  (Update: edit wallpapers from Control Center: Appearance: Background and the changes stick).

My laptop has hybrid graphics.  The way this is intended to work is a slower Intel GPU is used most of the time, saving battery life, but for graphics-intensive applications, a faster nVidia GPU is used instead.  After some considerable effort to understand and use this new thing back in 2013, I got Bumblebee configured and learned to use optirun or primusrun for games and graphical apps.

The bumblebee packages are available from the Ubuntu repos, but as far as I can tell the distro just uses the nVidia device for everything by default.  I found that I didn't care that much about the energy savings as I rarely run on the battery for more than an hour or so, so I wound up just leaving it this way.  There is a setting in NVIDIA X Server Settings (PRIME Profiles) that allows switching GPUs on a global basis that seems adequate for controlling power consumption if I needed to.

Overall I'm pretty happy with this distro, but at some point I'll probably give Mint 18 MATE a try, also.

See: Linux Mint MATE 18 Or Ubuntu MATE 16.04

Spring Boot

Last weekend I did a coding exercise creating a small Web MVC app with Spring Boot and Spring Data JPA.  Last year I worked through Spring Boot In Action but this was my first real Spring Boot app that went very far beyond Hello World.  I didn't need to write a single line of configuration.  I'm not just saying no XML configuration, no configuration at all.  Spring Data also does a great job of just getting out of the way and letting you work with your database.  I selected Spring Data JPA and HSQLDB from the start page and it just worked with no persistence.xml, no hibernate.cfg.xml or hibernate.properties, nothing.  This really is a game changer in terms of getting you out of the starting gate and focused on your application without having to go through a lot of Spring ceremony first.

Here's one other little thing I noticed.  With the introduction of @SpringBootTest in Spring Boot 1.4 the configuration for an integration test got a bit simpler.  This change is described here: Testing improvements in Spring Boot 1.4.

Spring Core 4.2 certification

I got my first certification today.  I have not been a big believer in these, but this is what my awesome employer was paying for this year so I took it.

I took the class first week in April and took the test today (5/20/2016).

Java configuration was a big part of this.  It seems like a big complaint people have had about Spring since the early years was the amount of XML needed to configure dependency injection.  The SpringSource/Pivotal people have been working for years to reduce the need for XML, first with annotations, then more recently with Java configuration.  I was fine with the XML and never really got on board with Java configuration, until now.  After learning this new style, well, now I can do it, but I'm still feeling lukewarm about it.  I feel like the motivation for this was more to silence the critics than to actually make the framework better.  I managed to struggle through a full conversion of a web flow application I wrote 2 1/2 years ago.  It's not quite a one-to-one mapping converting XML to Java as things are organized a little differently, and there are still a few things that aren't 100% comfortable to do in Java.

The MVC stuff was pretty familiar to me.

The class devoted quite a bit of time to RESTful web services.  The conventional wisdom in the Spring world is you could use JAX-RS but it's overkill, just implement REST services using MVC @RequestMapping.  I saw it done this way at some NFJS presentation a few years ago and since then they have added a bit more support (@RESTController, REST template).

AOP is a feature of Spring I have not really found much occasion to use, so this was something I needed to brush up on.

Some non-core topics covered in the class and the exam were Spring Data, Spring Security, Spring Boot and Spring Cloud.

I understand the focus on Spring Boot as this is quickly becoming the default way to work in Spring.  All the Getting Started guides are now written for Spring Boot and most of the labs for this class were also written as a @SpringBootApplication 'main' program.

I guess they chose to make Spring Security part of this because it is pretty much indispensable for doing any kind of real-world web development.  It has been around for a while, but my first exposure to this was in Spring Boot in Action last year, so this was a good refresher about 6 months later.

Spring Data simplifies the task of working with relational and NoSQL database so much, once you have started using it you will never go back.  In fact I questioned a little bit that they were spending time on JdbcTemplate but I guess it is part of Spring Core.  JPA is also covered.

Spring Cloud was the most 'out there' topic covered.  I have played around with this a little but some of the other people in the class were totally lost.

Generally I would say this was pretty hard.  The exam focuses on a lot of nit-picky details that you might not necessarily know even if you were working in Spring every day.  Before the class I was thinking 'could I schedule the exam on the last day of the class', but that would have been a disaster.  I spent about 6 weeks re-reading the 900+ page presentation from start to finish, experimenting with changes to my own code, and finally just re-reading selected parts of the reference documentation.  Think memorize-and-regurgitate.

There were about 5 people in the class, so, reasonable class size.  There was one guy who did not even have any Java experience who was slowing things down, and we were running behind right up until the end, but the stuff we didn't get to was not that important anyway (JMS, who uses that any more? and JMX).

The labs were just right for me.  I had to think but they weren't overwhelming or anything and I was mostly able to finish them during the allotted time or at least on the same day after class.

The supported platforms for the class are Windows and OSX.  In particular you do need one of these for the video chat for the online class.  I went to all the trouble of setting everything up in Windows 8.1 but afterwards was able to get Spring Tool Suite and the exercises working in Linux (my preferred environment).

One thing that confused people is the USB drive with the class materials comes with an ancient (3.6.4) version of STS.  That was about 16 months out of date at the time of the class.  So people were getting popups asking did they want to update?  Updating didn't seem to hurt anything, neither did simply importing all the exercises into an up to date STS version.  This was a bit troubling given the emphasis on Spring Boot, since the recent STS versions have added some Spring Boot support (Spring Boot Dashboard) that is not available in the older version.

In order to take the test, you must first take the (expensive!) class.  Taking the class entitles you to a voucher to take the test one time for free, after that it is $150.  Getting the voucher takes 2-3 weeks.  I can't help feeling they are being greedy here not allowing people to just come in off the street and take the test.  The instructor told us 'if you master the material in the lecture notes and the exercises, you will be able to pass the test', which I would agree with.  There were no surprises or anything I thought was unfair in the exam.

The test is fairly high-security.  I had to put my phone, watch, wallet and also empty my pockets(!) into a locker.  You need to sign a statement that you won't reveal the test contents to anyone.  It is 50 multiple-choice questions and the 90 minutes provided are really plenty to get through this.  I went through all the questions and then carefully went through them all again and finished in about 45 minutes.

As far as Spring itself, I'm still feeling very upbeat about this framework.  The future looks bright for Spring IMO.

It's a relief to finally finish and I'm looking forward to spending at least a couple of days not eating, breathing and sleeping Spring during every spare moment.

Spring Cloud, Cloud Foundry

I returned to part 6 and part 7 of Building Cloud Native Apps with Spring with the idea of deploying the app into my local Cloud Foundry instance instead of IBM Bluemix.

I have to say that once you have got through the ordeal of setting up Cloud Foundry locally, it is a lot less trouble to work with than Bluemix.  I skipped over gobs of instructions about checking things into github, writing little shell scripts and navigating around doing things in the Bluemix web GUI.

All I needed to do to get this working was to register Eureka as a CF service:

cf cups ocr-eureka -p '{"uri": "http://ocr-eureka.bosh-lite.com/eureka/"}'

 and then make the changes described to the application.yml files.

I had a problem with the Hystrix app.  It was giving me a big stack trace about Tomcat failing to start.  This turned out to be the error that was causing the problem:

    java.lang.NoSuchMethodError: javax.servlet.ServletContext.addServlet(Ljava/lang/String;Ljavax/servlet/Servlet;)Ljavax/servlet/ServletRegistration$Dynamic;

I found some discussion of this at Getting NoSuchMethodError: javax.servlet.ServletContext.addServlet in Spring Boot while running a Spring MVC application.  Checking the dependencies, I found that Turbine had a dependency on servlet-api 2.5.  I tried excluding that in the Hystrix POM:

and that fixed the problem.

Deploying an app into CF seems really slow.  It's doing some big downloads every time (Open Jdk JRE 1.8.0_65, Open JDK Like Memory Calculator 2.0.1_RELEASE, Spring Auto Reconfiguration 1.10.0_RELEASE).  I feel like it would be hard to work this way.  Wondering if there's a way to speed this up?

Setting up a local Cloud Foundry instance

My platform for all of this is Linux Mint 17.2 (rafaela).

It has been 2 or 3 years since I played around with Cloud Foundry.

The first thing I noticed is they have really raised the bar in terms of the effort required to bring up a local instance.  There used to be a thing called Micro Cloud Foundry where you just downloaded a VMware VM and you were good to go.

Today, that is gone... instead there is a whole build process to go through.  On the one hand, it is good that these tools and sources are available, but I can't help feeling like this is a fairly big hurdle for the casual developer who is trying to evaluate CF.

I mostly followed the BOSH-lite README starting here.

When you get to Deploy Cloud Foundry, there are two choices.  The first choice leads to Deploying Cloud Foundry using BOSH which I found to be a trail of tears with lots of manual steps and big, slow downloads.  In the end I was not able to get this to work.

The second choice, checking out cf-release from github and running bin/provision_cf from your checked out bosh-lite directory, did work for me and seemed to go a lot faster (relatively speaking... still have a fast network and hours to spare).  Inside of this shell script you will find some of the same steps the first method does manually, but all neatly packaged and tested.

This got me to the point where I had CF running in my Vagrant VirtualBox and I could run 'bosh vms' and see all the VMs up and running.

My next challenge was to get access to my shiny new CF instance.  Deploying Cloud Foundry using Bosh said to try

curl api.subdomain.domain/info

but of course api.subdomain.domain is not the literal value, just an example.

I found several places indicating that the correct value here should be api.10.244.0.34.xip.io.  I read up on xip.io a little, verified that my computer was resolving this name, but still couldn't get this to work.


Googling on, I found a guy who was having this exact problem and the solution that worked for him.  So, using api.bosh-lite.com I was able to access my local CF instance:


Having done all of this I was able to complete the remainder of the setup just following the documented steps with cf api, cf login, cf create-org, cf create-space, cf target... See Running Cloud Foundry Locally with BOSH Lite for example.

So, cool, sort of... I spent about a day on this and feel like I have just reached the starting line...

I configured the Eclipse integration and worked through Deploying to Cloud Foundry from STS.  Everything works.  I saw the 'Hello world'.  Ready to try this with a real app.

Here's a nice short presentation I found afterwards that sheds a little more light on Bosh: Why Must I Use Cloud Foundry's Bosh? I just Learned Chef/Puppet!  For when you want to customize your CF instance, and apparently it can do more than just manage CF.

Update 2017: Since I wrote this, Pivotal has released PCF Dev, a downloadable CF VM not unlike the old Micro Cloud Foundry VM for the person just wanting to experiment and get up to speed with a local instance of Cloud Foundry.

Lattice is a tool to bring up a local CF instance.  Here's a pretty good presentation about Lattice and CF.  But note that Lattice is no longer actively maintained and appears to be deprecated now.  I gave this a quick try but didn't get it to work.