Installing PostgreSQL 9.6 on macOS 10.13.6 via MacPorts for Development Purposes

After finally receiving a new MacBook Pro with 2.9 Ghz, 6-core i9 processor and 32gb of memory, I am reinstalling all of my development tools. Not only is the big increase in resources really useful from the development point of view, but the computer is lighter and thinner as well. That being said, there are still things I do not like, such as the keyboard, the touchbar (I miss the tactile feedback from the escape key which, as a Vim user, I use constantly) and some other minor things. For my own purposes, I usually document the steps to make it easier in the future. But I also figured it might help other folks out there to achieve the same ends.


Having documented this procedure once or twice before, I'm going to keep this brief.

	$ $ sudo port install postgresql96-server
	---> Computing dependencies for postgresql96-server
	The following dependencies will be installed:
	libxml2
	libxslt
	postgresql96
	postgresql_select
	Continue? [Y/n]:
	---> Fetching archive for libxml2
	---> Attempting to fetch libxml2-2.9.7_0.darwin_17.x86_64.tbz2 from https://packages.macports.org/libxml2
	---> Attempting to fetch libxml2-2.9.7_0.darwin_17.x86_64.tbz2.rmd160 from https://packages.macports.org/libxml2
	---> Installing libxml2 @2.9.7_0
	---> Activating libxml2 @2.9.7_0
	---> Cleaning libxml2
	---> Fetching archive for libxslt
	---> Attempting to fetch libxslt-1.1.32_0.darwin_17.x86_64.tbz2 from https://packages.macports.org/libxslt
	---> Attempting to fetch libxslt-1.1.32_0.darwin_17.x86_64.tbz2.rmd160 from https://packages.macports.org/libxslt
	---> Installing libxslt @1.1.32_0
	---> Activating libxslt @1.1.32_0
	---> Cleaning libxslt
	---> Fetching archive for postgresql_select
	---> Attempting to fetch postgresql_select-0.3_0.darwin_17.noarch.tbz2 from https://packages.macports.org/postgresql_select
	---> Attempting to fetch postgresql_select-0.3_0.darwin_17.noarch.tbz2.rmd160 from https://packages.macports.org/postgresql_select
	---> Installing postgresql_select @0.3_0
	---> Activating postgresql_select @0.3_0
	---> Cleaning postgresql_select
	---> Fetching archive for postgresql96
	---> Attempting to fetch postgresql96-9.6.10_0.darwin_17.x86_64.tbz2 from https://packages.macports.org/postgresql96
	---> Attempting to fetch postgresql96-9.6.10_0.darwin_17.x86_64.tbz2.rmd160 from https://packages.macports.org/postgresql96
	---> Installing postgresql96 @9.6.10_0
	---> Activating postgresql96 @9.6.10_0
	---> Cleaning postgresql96
	---> Fetching archive for postgresql96-server
	---> Attempting to fetch postgresql96-server-9.6.10_0.darwin_17.noarch.tbz2 from https://packages.macports.org/postgresql96-server
	---> Attempting to fetch postgresql96-server-9.6.10_0.darwin_17.noarch.tbz2.rmd160 from https://packages.macports.org/postgresql96-server
	---> Installing postgresql96-server @9.6.10_0
	---> Activating postgresql96-server @9.6.10_0
	---> Cleaning postgresql96-server
	---> Updating database of binaries
	---> Updating database of C++ stdlib usage
	---> Scanning binaries for linking errors
	---> No broken files found.
	---> No broken ports found.
	---> Some of the ports you installed have notes:
	postgresql96 has the following notes:
	To use the postgresql server, install the postgresql96-server port
	postgresql96-server has the following notes:
	To create a database instance, after install do
	sudo mkdir -p /opt/local/var/db/postgresql96/defaultdb
	sudo chown postgres:postgres /opt/local/var/db/postgresql96/defaultdb
	sudo su postgres -c 'cd /opt/local/var/db/postgresql96 && /opt/local/lib/postgresql96/bin/initdb -D /opt/local/var/db/postgresql96/defaultdb'
	A startup item has been generated that will aid in starting postgresql96-server with launchd. It is disabled by default. Execute the following command to start it, and to cause it to launch at startup:
	sudo port load postgresql96-server
	$
	$ sudo mkdir -p /opt/local/var/db/postgresql96/defaultdb
	$ sudo chown postgres:postgres /opt/local/var/db/postgresql96/defaultdb
	$ sudo su postgres -c 'cd /opt/local/var/db/postgresql96 && /opt/local/lib/postgresql96/bin/initdb -D /opt/local/var/db/postgresql96/defaultdb'
	The files belonging to this database system will be owned by user "postgres".
	This user must also own the server process.
	The database cluster will be initialized with locale "en_US.UTF-8".
	The default database encoding has accordingly been set to "UTF8".
	The default text search configuration will be set to "english".
	Data page checksums are disabled.
	fixing permissions on existing directory /opt/local/var/db/postgresql96/defaultdb ... ok
	creating subdirectories ... ok
	selecting default max_connections ... 100
	selecting default shared_buffers ... 128MB
	selecting dynamic shared memory implementation ... posix
	creating configuration files ... ok
	running bootstrap script ... ok
	performing post-bootstrap initialization ... ok
	syncing data to disk ... ok
	WARNING: enabling "trust" authentication for local connections
	You can change this by editing pg_hba.conf or using the option -A, or
	--auth-local and --auth-host, the next time you run initdb.
	Success. You can now start the database server using:
	/opt/local/lib/postgresql96/bin/pg_ctl -D /opt/local/var/db/postgresql96/defaultdb -l logfile start
	$
	$ sudo su postgres -c '/opt/local/lib/postgresql96/bin/pg_ctl -D /opt/local/var/db/postgresql96/defaultdb -l /opt/local/var/log/postgresql96/postgres.log start &'
	server starting
	$ sudo -u postgres /opt/local/lib/postgresql96/bin/createuser -d bsnyder
	$ sudo -u postgres /opt/local/lib/postgresql96/bin/createdb bsnyder
	$ sudo -u postgres /opt/local/lib/postgresql96/bin/psql
	postgres=# alter user bsnyder with encrypted password 'bsnyder';
	ALTER ROLE
	postgres=# grant all privileges on database bsnyder to bsnyder;
	GRANT
	postgres=# \q
	$ /opt/local/lib/postgresql96/bin/psql -U bsnyder
	psql (9.6.10)
	Type "help" for help.
	bsnyder=> select version();
	version
	-----------------------------------------------------------------------------------------------------------------
	PostgreSQL 9.6.10 on x86_64-apple-darwin17.7.0, compiled by Apple LLVM version 9.1.0 (clang-902.0.39.2), 64-bit
	(1 row)
	bsnyder=> \q
	$

view raw macos_install_pg_96.txt hosted with ❤ by GitHub

BTW, below are the start/stop scripts that I created for PostgreSQL:

	#!/bin/sh
	sudo su - postgres -c '/opt/local/lib/postgresql96/bin/pg_ctl -D /opt/local/var/db/postgresql96/defaultdb -l /opt/local/var/log/postgresql96/postgres.log start &'
	pg_version=$(psql --version | cut -d ' ' -f3)
	if [ $? -eq 0 ]; then
	echo "PostgreSQL $pg_version started successfully"
	exit 0
	else
	echo "PostgreSQL $pg_version did not start in a timely fashion, please see /opt/local/var/log/postgresql96/postgres.log for details"
	exit 1
	fi

view raw pg_start.sh hosted with ❤ by GitHub

	#!/bin/sh
	sudo su - postgres -c '/opt/local/lib/postgresql96/bin/pg_ctl -D /opt/local/var/db/postgresql96/defaultdb -l /opt/local/var/log/postgresql96/postgres.log stop'
	pg_version=$(psql --version | cut -d ' ' -f3)
	if [ $? -eq 0 ]; then
	echo "PostgreSQL $pg_version stopped successfully"
	exit 0
	else
	echo "PostgreSQL $pg_version did not stop in a timely fashion, please see /opt/local/var/log/postgresql96/postgres.log for details"
	exit 1
	fi

view raw pg_stop.sh hosted with ❤ by GitHub

GopherCon Denver 2018

Recently I attended GopherCon 2018 in Denver and really enjoyed the whole event. There were about 1500 people there and I really learned a lot. One of my teams writes Kubernetes operators using Golang which got me into writing some Golang myself.

On the first day, we attended a workshop titled Advanced Ultimate Go that was taught by Bill Kennedy from Ardan Labs and it was excellent. Not only is Bill incredibly knowledgeable with Golang (he has years of experience not only in Golang but also in C++) but he is also an extremely good teacher. I learned a lot in a single day and I would have loved to have this class extend for another day or even two.

Coming from a background of mainly Java for 20+ years, the Java community has an amazing piece of engineering in the JVM that handles most performance related concerns for us. We design Java classes however we please and we make use of any data structures we please and never really give it a second thought simply because the JVM handles all the real mechanics for us. With Golang, this is not the case. Although Golang has garbage collection (which is really nice, I hated using malloc() and free() in C) it's very different than what JVM does. Also, in many ways, the Go language is very succinct compared to Java. Both in terms of the amount of code you must write (Golang requires a lot less boilerplate code) to the ease of deployment with Golang (you have a single binary to deploy, no dependencies or CLASSPATH to manage). That being said, I'm still not sure that I would completely switch all web development away from Java to Golang. While the Golang learning curve is a lot easier than Scala, the issue I see are the implications on performance with the code you write. Whereas with Java, while there are performance concerns, as I said above the JVM handles a tremendous amount of things under the covers so that we don't have to care nearly as much.

Having spoken at many, many conferences over the years, I was very pleasantly surprised by the number of female attendees and speakers! I saw more talks delivered by women than I saw delivered by men which was excellent! The community seemed very open and engaging to everyone which I really liked. Being in the software industry for so long, it's still shocking to me how much of it is dominated by males. Anyway, I really enjoyed the cultural and gender diversity at Golang.

There is one minor change that I would like to suggest to the organizers of the Golang conference. Many years ago, I spoke at a conference in Denmark where I first saw this. On tables next to the doors at the back of a room where talks are taking place, there are big glass bowls with three piles of Post-It size paper, each pile a different color -- red, yellow and green. As attendees exit the room, they are asked to grab a single piece of paper in the color that represents how you felt about the presentation/talk. When the talk completes and the room is empty, the conference organizers gather the papers from the bowl, tally them up and provide the stats to the speaker. It's basically like a quick rating of what attendees thought of the talk. This small system does not replace the comment cards that organizers always ask of attendees because this is how attendees elaborate on they rated the talk the way they did. Both systems of rating are important because they deliver two different but equally important types of data to the organizers and the speakers. Anyway, this my two cents.

I really enjoyed GopherCon for a variety of reasons and I would love to attend again. Since the conference I have written a lot more Golang code and the more I write the more I like it. In my mind, Golang should be the goto language for systems programming, DevOps type stuff. While it can easily handle general web development tasks, I'm not sure yet if I would drop Java in favor of Go. I guess I need to keep coding away in both.

Vacation and Hiking in Crested Butte

Last week my family and I took a week's vacation in Crested Butte to do some hiking and relaxing in the beautiful Colorado Rocky Mountains. Although we missed the Crested Butte Wildflower Festival by one week, we still enjoyed the flowers on our hikes.

As we explored the Crested Butte area and enjoyed numerous hikes, we got to spend time together as a family which I really enjoy doing. It's hard to believe that my girls are basically adults now (they are 15 and 20 years old). Bailey even got a second week of altitude camp in for her cross country training by running every day that we were there.

My in-laws even joined us with our two nephews and we all had a very relaxing time together. We even rented some paddle boards and went to Lake Irwin one day. This wasn't something that would have been very easy for me to do, so I enjoyed watching everyone experience the paddle boards for the first time in a cold mountain lake.

Crested Butte is a sleepy little mountain town with a well-known ski resort. We enjoyed hanging out in town, checking out the restaurants and shops, and also finding trails to explore all around the area, including on the ski mountain. Unfortunately, the Crested Butte ski resort was recently acquired by Vail Resorts. This means that it will be completely over-developed soon enough, everything will become too pricy and too crowded with traffic just like all the other resorts it owns. What a drag.

Later in the week, we visited a friend in Lake City and hiked Spring Creek Pass, a high mountain pass with an average grade of 7.5% at 10,901 feet. My phone told me that I climbed 50 flights of stairs that day which was quite an achievement for my withered legs. When we returned to the car, we took a photo by the trailhead sign and Janene had her arm around me and said, 'Your legs are shaking, you need to sit down.' I just chuckled and said, 'Yeah, I know.' Even so, I loved it! I am so thankful to be able to feel that exhaustion from completing a difficult hike.

Forming Scrum Teams

The tongue-in-cheek fable of the chicken and the pig is frequently referenced in Scrum literature to distinguish between people who are committed to the Scrum (the pigs) and people who are only involved (chickens). The reason pigs are committed is they must be sacrificed for the meal whereas chickens are simply involved. When I was software engineer, we used to say, 'It's just us pigs here' when it was a group of software engineers together. Having transitioned to management several years ago, now I say, 'It's just us chickens here.' But I digress.

In this post, I want to discuss my logic for forming Scrum teams, a topic that I have hashed and re-hashed many times throughout my career. So, here I will explain the logic behind how I form software engineering teams.

The Agile Manifesto

The Agile Manifesto lists the following 12 principles:

1. The highest priority is to satisfy the customer via early and continuous delivery of software
2. Welcome changing requirements
3. Deliver working software frequently with a preference to the shorter timescale
4. Business people and developers must work together daily throughout the project
5. Build projects around motivated individuals
6. The most efficient and effective method of conveying information to and within a development team is face-to-face conversation
7. Working software is the primary measure of progress
8. Agile processes promote sustainable development
9. Continuous attention to technical excellence and good design enhances agility
10. Simplicity is essential
11. The best architectures, requirements, and designs emerge from self-organizing teams
12. At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly

While these 12 principles define much of Agile in the broadest sense, there are variations in different Agile methods such as XP, Scrum, Lean, Kanban, etc. These methods are not meant to be followed dogmatically, instead practices within a particular method are meant to be guidelines. In this post, I will be focused on Scrum, specifically on some of the aspects of team formation. But keep these principles in mind as you read this post.

What Does Scrum Recommend?

The Scrum literature discusses team formation a lot. From this literature, it's easy to learn that there are  some key principles in Scrum focused around forming small, cross-functional teams. But Scrum best practices don't end there. There are many other aspects to building Scrum teams. In this post, I will elaborate on the following:

• Autonomous and self-organizing
• Team composition must be cross-functional
• Ideal team size

Autonomous and Self-Organizing

As stated in many Scrum publications including the Scrum Guide, a Scrum team should consist of a Scrum Master (SM), a Product Owner (PO) and Developers. Scrum teams should be cross-functional and self-organizing, i.e., they must be capable of delivering value each sprint and they are responsible for deciding how they work. This is what I refer to as setting up a team for success. It's the job of the leader to set up the team for success, not only in the structure of the team but also in communicating the expectations for the team. I have always defined expectations in black and white for everyone to see so there is no question about their responsibilities. (In fact, I have even implemented this model across all engineering teams for a product line by working very closely with my management colleagues. There must be consistency of expectations across all the product engineering teams. I cannot stress this enough.)

It is critical that the team members understand and embody their autonomy and empowerment to manage their own work. Typically an engineering manager is involved and accountable, but the team is responsible (from a RACI matrix point of view). In my opinion, the best case scenario is that the engineering manager is more of a coach and not micro-managing everything a team does. Team members must be coached to step up whenever they can and remember to bring everyone along on the journey. This requires well-structured and open communication across the team. Oftentimes, the Scrum Master is the person who guides the team most in terms of self-organization and responsibilities.  It is also important that team members understand the role that everyone plays on the team. Like a set of gears moving together as one, it only works if all cogs are operating correctly and together.

Self-organization is an important aspect for a team to operate well together. Team members cannot operate together without a shared understanding of what this means. In Scrum, this means the PO defines user stores for team and the SM leads the team members in breaking down those user stories into subsequent tasks for estimation and execution. The SM oversees all of this activity and works closely with the PO to determine priorities for the sprint. While people not deeply experienced in Scrum think this is too much for a team to handle or feel that the manager should drive all team activities, this couldn't be further from the truth. As with many things in software, concrete experience teaches a tremendous amount very quickly. As a people leader, it's very easy to see the differences between a mature Scrum team and an immature Scrum team. Like many things in life, reaching a level of maturity as a Scrum team requires proficiency and time, especially if new team members are added over time. As the Agile Manifesto mentions: 

Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.

I work very hard to build trust with my engineering teams as I believe this is the basis for building great teams. I always say that I hire talented individuals who are experts and then I give them room to get the job done. The bottom line is the team is accountable for delivering value every sprint and I have found that building trust is a requirement. I extend trust to them to get the job done and they teach me to trust them by consistently delivering value every sprint.

Team Composition Must Be Cross-Functional

As mentioned above, a Scrum team needs to be cross-functional so as to deliver value every sprint. As much as delivering value is a generic term, when developing microservices it means that the team can deliver, own and operate those microservices itself. The team should not need to rely on other teams for different job functions as doing so amounts to dividing up teams based on function. This type of team organization was an idea from the 1990s and Waterfall methodology that divided up teams based on job function, e.g., front end engineers in one team, back end engineers in another team, QA in it's own team, tech writers in their own team, etc. I've seen this before and it really falls apart quickly because departmental divisions do not work in an Agile environment. It's a mismatch of models. Agile is supposed to be iterative and incremental. When teams are defined by job function they are not set up to be incremental. A true Scrum team must be composed of the constituent parts to deliver the software owned by the team. This requires a cross-section of roles composed of all the necessary disciplines. As mentioned above, a Scrum team should consist of a Scrum Master (SM), a Product Owner (PO) and Developers:

• Scrum Master
• Product Owner
• Front end engineers (developers)
• Back end engineers (developers)
• DevOps engineers (developers)
• QA engineers (developers)
• Tech writer (developers)
• Manager

In the list above, I included a couple of roles that you don't normally see in a Scrum team -- a tech writer and a manager. I also break down the different types of engineers. So, let's discuss each one.

Tech writer: I include a tech writer in my teams whenever I can because I do not consider the product to be Done until everything is actually done including the coding, testing, documentation, performance considerations and operational requirements.

Manager: I include the manager because this person serves a vital role not only in terms of people reporting lines but also in terms of developing a contextual understanding of a team and its work. Speaking from experience, without a manager present, it becomes very easy for software engineering teams to get pulled in many directions. A manager can guard against not only this but also against members of a team reporting to numerous managers which can result in a lot of chaos. 

The other roles are simply a clarification on the types of developers. Furthermore, it illustrates the inclusion of the different types of developer disciplines -- tech writers, QA engineers, DevOps engineers, etc. None of these disciplines or roles should reside on other teams. When this happens, the team will not develop the right level of cohesion and trust necessary to become what Google identified as an effective team (this is a whole separate topic for another blog post).

Ideal Team Size

I have seen Scrum teams with three engineers and Scrum teams with 30 engineers. The goal should be to find an ideal size that works for the products being developed, but I do believe smaller teams are better. According to the experience of many Scrum practitioners, the ideal team size should be 6-8 engineers (give or take one or two) not to exceed 10. From my nearly 20 years of experience practicing Agile, the larger an engineering team grows, the more difficult it becomes to meet, to scope work, to keep track of work, etc. Basically, the larger the team the more complex and time consuming communication becomes. It's not called Agile for no reason -- software engineering teams practicing Scrum are meant to be nimble and iterate quickly. This cannot take place when too many people are involved. 

Consider Brooks' Law about software project management which says adding manpower to a software project that is behind schedule will delay it even longer. This maxim emerged from Fred Brooks eponymous book The Mythical Man-Month. Brooks explains that while cost varies with the number of men and months, progress does not:

“Hence the man-month as a unit for measuring the size of a job is a dangerous and deceptive myth. It implies that men and months are interchangeable. Men and months are interchangeable commodities only when a task can be partitioned among many workers with no communication among them… This is true of reaping wheat or picking cotton; it is not even approximately true of systems programming.”

This is further elaborated by examining the lines of communication for teams of different sizes. As explained in a blog post titled, Lines of Communication and Team Size: Applying Brooks’ Law: 

Most teams in large enterprises have between 10 and 20 members. Even at the low end of that scale, a team has so many lines of communication that progress is bound to be slow.

(source: Lighthouse)

Note the increasing number of lines of communication as a team grows. This is essentially the root cause of why teams beyond the Scrum recommended size become more and more ineffective as they grow. The drag in trying to develop a shared understanding of anything simply becomes more and more complex the larger a team grows. This is typically when people start to complain about there being too much administrative overhead. 

Conclusion

To summarize, if you want to set up your teams for success, then they must be autonomous, self-organizing, cross-functional and sized appropriately for the software being built. This is at least part of the recipe that has helped me to build what I mentioned above that Google calls effective teams. 

Resources

• The Law of Contraints and Brooks' Law
  
• What Does Scrum Mean by Cross Functional Teams?
• Cross Functional Doesn't Mean Everyone Can Do Everything

More Muscle Movement Discovered Recently

Recently I paid a visit to my physical therapist. I only see every 2-3 months to check in and guide me on the physical therapy work that I do on my own. She always checks out my progress to see how my body is doing and this time she found something new -- the tibialis muscles in both my legs are moving now!

The anterior tibialis muscles are what allow you to flex your feet upwards (this is called dorsal flexion). So far, I have only been able to press my feet down (this is called plantar flexion) because this is controlled by the calf muscles. But now that the anterior tibialis muscles are moving ever so slightly, I can begin to rebuild these as well. Just like my calves are taking time, these muscles will also take time to rebuild as well.

I am still working to rebuild my calf muscles and this is slow. My physical therapist reminded me that I'm not just strengthening my calves. I literally had no calf muscles left. So I am rebuilding my calves from nothing which is much more difficult. She also said that at this point, we have no idea if the calves are fully firing or not. In situations where there is nerve damage, you never know if you are getting a full squeeze from the muscle or if the whole muscle is firing yet. This makes the work much, much more difficult. She also told me that this is why most people give up.

My calves are increasing in size and strength, but they are very far from 100% functionality or strength. I still have a long way to go. But I still feel lucky to be where I'm at today with my body still healing.