10x

Fred Brooks started it all. In The Mythical Man Month, he quotes a study saying

individual difference between low and high performers can vary by an order of magnitude

Since then this myth of 10x productivity difference has persisted in our industry.

Nowadays it’s best seen in the use of words like rockstar, guru or wizard in job descriptions.

But is it really a myth, or reality?

It’s undeniable that individual differences exist. Not everybody can write an operating system kernel, a concurrent collection library, or cryptocurrency protocol. These achievements are examples of outstanding technical expertise.

Like in sports, the distribution of talent is skewed, and there are outliers that outperform others.

But here’s the catch: the 10x developer isn’t working 10x faster, he’s thinking differently. The 10x developer finds new way to address problems.

He doesn’t deal with complexity better. He finds way to avoid complexity altogether. Not occasionally, but systematically, as part of his work ethics.

A 10x developer is also a force multiplier. His actions make the work of several people easier. He inspires other to achieve excellence and clone his habits. The payoff can go above 10x.

So, myth or reality?

For me, reality. But such developers are very rare. Over the last 10 years I’ve only met one.

Links

A Machine for Jumping to Conclusion

The brain is a wonderful machine with an impressive computing power. We can make sense of complex information effortlessly and almost instantly. But it has one big flaw: it does not understand probabilities.

When presented with information, the brain tries to explain it by building a coherent story out of it. To do so quickly, it relies on some shortcuts, which largely ignore probabilities. So the story you get isn’t necessary the most probable, but instead the cheapest it could construct, as long as the story remains plausible.

One of the shortcuts is to trade availability of information for probability (availability heuristic): the information you can recall quickly is deemed more probable than other information. As a result, the probability of sensational events inflates while the probability of mundane events shrinks.

Another of the shortcuts is to only consider what is visible and extrapolate from there (all there is is what you see): only the visible information is considered, without even considering that something could be missing from the picture. Somebody looking nice will be considered a nice person, unless additional negative information about him is given.

The brain tries so hard to build a story that it will see patterns even in random data. It will infer causality very quickly, and with very little. As Kahneman puts it, the brain is “a machine for jumping to conclusion”.

It important to remember the dinstinction between plausible and probable when it comes to judgment, because we’re taking decisions all day long.

You hear a project was using a new methodology and was very successful with it so you want to use it as well? Beware the survivorship bias. You don’t know how many projects used the methodology and failed…

You think the biggest risk in your project is a distributed attack from China? Beware the availability heuristic. Your biggest risk might be to not have proper input validation…

You see bug reports for your teams and start detecting a pattern? Beware the law of small number. Your sample size might be too small…

Our brain is hardwired to tell us stories. So it’s very hard to improve on our handling of probabilities. Often, we’re simply not aware that probabilities are at play. And even when we are, it’s really hard to change our instinct in some cases: if you flip 9 times a coin and  got 9 times head, the probability to get tail on the 10th flip is higher, right? Well, actually not. But it’s hard to not feel otherwise.

So, the best remedy is to default to a healthy skepticism and accept that the outcome of many situations in life is simply the result of chance. It might sound like fatalism but it isn’t at all. Your actions will influence the outcome of many situations, but you don’t know how. Don’t buy the first plausible explanation your brain tries to sell you, it’s probably not the right one.

References 

Things You Can’t Abstract

The art of programming is to a large extent the art of devising abstractions. Some might be very general and reusable in many contexts, some will be more specialized and applicable only in some domains.

The purpose of abstraction is to hide complexity so that we don’t need to care about details. Using abstractions, we can “raise the abstraction level”.

Data structures, relational databases, file systems or garbage collection are all examples of common programming abstractions. There are of course many more.

Abstracting is not unique to programming. For instance, the DNA, the cell, the organ and the organism are different abstraction levels in biology.

An abstraction defines a contract between a user and a provider. The less constraints there are in the contract the more freedom there is in the implementation possibilities. It’s tempting to abstract away all non functional aspects, but it’s actually a bad idea: you will need to understand them to use the abstraction correctly.

First, you can not abstract performance. Wether an operation takes O(1) or O(n) is not something you can ignore. Eventually, at some point you will have to care about the implementation of the abstraction to understand its performance characteristics. Abstracting performance and letting the runtime figure out the best optimization strategy look nice on paper but is the source of many headache. You will need to know how your data structure performs, how your database fetches data, how many files can reasonably exist in a folder, and when your garbage collection kicks in.

Second, you can not abstract failure modes. If something can fail, you can not ignore it. This is especially true of the network: if something is remote it can be inaccessible. Attempts to abstract the network as if everything were local simply do not work. An abstraction can have few failure modes, but there is no abstraction that never fails. You will need to understand how your data structure reacts when it can’t expand, how you database reacts when your commit is so big that its transaction log is full, when your file system is not reachable, and when the garbage collector can’t reclaim space.

And third, you can not abstract the consumption of shared resources. Since shared resources are finite and common to the whole system, every component is indirectly related to every other components. You will need to understand how much memory you data structure takes, how much of your data fit in the database cache, how much disk space your system consumes, how much clock cycles are eaten by garbage collection runs.

That makes a lot of aspects we can’t abstract. Joel Spolsky was right. All non-trivial abstractions eventually “leak“. Barbara Liskov was wrong. In practice, two abstractions with the same functionality cannot be “substituted“, unless they also have the same non-functional characteristics.

It is discouraging to realize we can’t abstract as much as we want, but it doesn’t mean abstraction doesn’t work. You will need to know a bit more about data structure, database, file systems, and garbage collection than you thought to use them correctly, but you can still ignore a lot of the internal details. The goal of hiding some complexity is achieved, but not of hiding all complexity.

More

Become a Domain Expert

It is only possible to design a feature correctly if you have enough understanding of the underlying business domain. Large software systems may have complex domain models with many special situations that aren’t immediately obvious. Taking the time to understand the domain is crucial for making the right assumptions during development.

Failing to understand the domain correctly will result in features that work “most of the time” but break in special occasions. The detection of the these problems will happen late in the project, cost time to analyse and debug (see the famous estimate), und may in the worst case require a redesign of the whole feature if the domain model was used inappropriately.

A little example of domain knowledge in the project I’m working with is the following: we have train timetable modelled as sequences of stops at trains stations. Using the identifier of the station to uniquely identify a stop in the timetable is not enough since a train might pass the same train station twice in some case. Instead, it is needed to work with pairs of train stations to correctly identify the stops. It’s easy to do if you know it, but if don’t consider this correctly early on, you’re good to rewrite your code later on.

Without enough domain knowledge, the intent of the existing code remain obscure. Sure, you will understand what the goal was technically, but the problem it solves on the business side will escape you. This is unfortunate, because it limits your range of action. If you understand the problem on the business side as well you can put things in context and come up with more technical solutions.

Each minute spent better understanding the domain is worth it. If you want to become a better engineer, become a domain expert.

Lateral Thinking

Lateral thinking is a term coined by Edward De Bono to characterize the generation of alternative ideas, as opposed to vertical thinking, which generates ideas based on logic and stepwise refinements. Another way to explain lateral thinking in a much common way is “thinking out of the box.”

Often finding the best solution to a problem requires a creative move to go away from the existing solution and start with a new angle. This is were lateral thinking can help.

As a reminder of the power of lateral thinking, let us take an egg and a spoon. You are doing a brunch. How do you provide assistance to help cut the egg?

With vertical thinking you might come up with this solution:


With lateral thinking, maybe with this one:


I was absolutely amazed the first time I saw this device in action. The cut is perfect. Also, I would probably never have come to this solution, no matter how long I stared at my egg.

Each time I discuss a design issue I remember my last brunch and try to take some distance with the situation to go back to the root of the problem to solve and ask: could we do this completely differently?

Sometimes the best way to cut an egg is to not cut it actually.

Making Progress

The essence of Scrum is to ensure progress. The formal elements of the framework –the restrospective, reviews, daily standup, etc.– are not ends in themselves but ways to ensure that progress happens.

It may seem simplistic to reduce Scrum to the mere fact of ensuring progress, but ensuring progress is not that easy, and Scrum is an effective tool to do it.

To prove this point, just think of what the opposite of progress means: to be stuck. A project can stuck for many reasons. Some symptoms include:

  • Work is half done or needs to be redone
  • Work is unclear and time is spent discussing it rather than doing it
  • Work wasn’t needed (people work on the wrong stuff)
  • Work can’t be done (because of dependencies, knowledge, etc.)

When a project is stucked, people work, but the overall project doesn’t move forward. The time is wasted.

Scrum prevent waisting time by maintaining a constant pressure on delivery and keeping the amount of work in progress low (“start finishing and stop starting”). It doesn’t matter how small the work item is. Actually the smaller the better, since it favors focus and quality.

Scrum is a framework for micromanagement, but without a micromanager. The team micromanages itself (i.e. “self-organisation”) and decides itself of the tasks to perform. Taskification happens mostly during the Scrum planning but then throughout the entire Sprint as the team actualises and refines the tasks to be done. And then does them.

The goal is to move forward, to overcome difficulties, to get concrete results, to make progress. For this you want the whole team to engage and people to help each other. You want your team to be more than the sum of its individuals.

I want teams emerging from the daily standup saying things like, “Let’s nail this. Let’s do this.”       — The Origins of the Daily Standup, Jeff Sutherland

People want to make progress fast, but software development is so complex that the risk is not to make progress too slowly but no progress at all. As long as you can ensure that some progress happens and you’re not compromising quality, you’re on a good track.

Gall’s Law

Gall’s law states that complex systems can only be the result of an evolutionary process, and not the result of a design from scratch:

A complex system that works is invariably found to have evolved from a simple system that worked. A complex system designed from scratch never works and cannot be patched up to make it work. You have to start over with a working simple system.  – John Gall (1975, p.71)

A complex system evolves from simpler systems by adding successive deltas of complexity. The only way to build a complex a system is through iteration. That’s what evoluation is about.

Iterations enable us to get feedback, correct and improve the system. See what works and what doesn’t. Fix mistakes.

The system must be working after each iteration. You can add new features, as long as it refines the existing system and keeps it running.

A tadpole becomes a frog by developing its legs, then its arms, and finally shrinking its tail. The frog’s legs, arms and body aren’t developed individually and assembled at the end. That’s not how evolution works.

last_thumb1367178271

Also, you can not evolve everything at once, since in the meantime the system might not work. A tadpole develops its legs, then its arms, and finally shrink its tail. Each iteration needs focus.

Gall’s law is relieving. It’s OK to not be able to handle all the complexity at once. And it’s not only you–it’s everybody.

A complex system can not be built using only theory and first principles, because there will always be details of the environment that we were not aware of. The only way to make sure something will work is to test it for real. Practice trumps theory.

Obsessing with getting it right the first time is counter productive. Just start somewhere and iterate. Too much unknown blocks our creativity. But once we have something concrete, ideas to improve come easily.

The tadpole also teaches us a lesson here: it first develops a tail, which then disappears later on. The tail is a good idea in the water, but not so much on the ground. You will have to reinvent yourself occasionally.

Scala: An Ambitious Language

In the object paradigm, a system consists of objects with mutable state, whereas in the functional paradigm, it consists of functions and immutable values. At first, these two worlds seem incompatible.

But not so for Odersky. In 2004 he released the first version of Scala, a language that combines both.

Scala’s roots are object-oriented, sharing the same basic constructs as Java, with whom it is fully compatible. Its functional flavor comes from several features borrowed or transposed from concepts in functional languages like Haskell.  This includes first-class and higher-order functions, including currying, but also pattern matching with case classes, and the support for monads and tail recursions.

The mariage is suprisingly elegant. Maybe the two worlds are compatible after all.

But the ambitions of Scala do not stop here. It also aims at beeing scaleable, both in terms of modularity and in terms of expressivity. Scala should support the modularisation of small and large components, and help reduce the gap between the code and the domain concepts.

The many features of the Scala’s type system enables scalability along both axes. Traits enable for instance a fined grained modularisation of object behaviors. Implicit conversions on the other hand enable existing types in libraries to be extended to express code more clearly.

But more importantly, features of the language create synergies. Abstract type members combined with type nesting enable the cake pattern, a form of dependency injection, or family polymorphism, a way to type check constellation of multiple related classes. The support of call-by-name combined with implicits enable the definition of domain specific languages.

You can’t but be amazed by how features sometimes combine. It is for instance possible to map a collection and convert its type at the same time using the special breakout object. You can even pattern match regular expressions!

Such synergies are possible because the foundations of Scala are principled.

  • First, everything is an object. There is no primitive types. Instead, the type hierarchy has two main roots, one for mutable objects (with reference semantics) and one for immutable objects (with value semantics).
  • Second, you can abstract over types, values, and functions using parametrization or abstract members. The three constructs support both forms of abstractions consistently.
  • Third, any object that defines an apply() function can be used as a function. This closes the gap between functions and objects. The inverse of apply() is unapply(). Any object that defines unapply() can be used as an extractor for pattern matching.

Take the expression “val l = List(1,2,3)”. This is not native syntax for list construction, but actually the evaluation of the function “apply” on the singleton object “List” with the arguments “1,2,3”. Or take the expression “val (x,y) = (1,2)”. This is not native syntax for multiple assignments, but tuple unpacking using extractors. These principles enable nice extensions of the language.

The flexibility of Scala has a price though: it is easy to learn Scala on the surface, but mastering its intricacies is challenging.

Also, Scala comes with many additional features that seem to exist more for convenience than necessity, making it even harder to master. It is for instance questionable wether structural typing or default parameter values, to name a few, should really have made it into the language. Clearly they are usefull and alleviate some pain points of Java, but they also distract from the essence of the language. Scala might at times appear to lack focus.

The richness of the language is acknowledged by the Scala community itself. To quote Odersky, “Scala is a bit of a chameleon. It makes many programming tasks refreshingly easy and at the same time contains some pretty intricate constructs that allow experts to design truly advanced typesafe libraries.”

Scala is a language with many very powerful features and with many ways to do things. It’s up to the developers to use the features well and enforce a consistent programming style. For corporations, these two aspects could be a barrier to adoption. In comparison, a language like Kotlin offers the same basic ingredients but is a lot more simple.

The long bet of Odersky seems to pay off though. Scala has found its audience and made its way to the industry, including top players like Twitter or LinkedIn. It has established itself as a viable alternative.

Scala is a source of innovation and inspiration. While functions were already in object-oriented languages like Smalltalk in the 80s, Scala showed that object-orientation doesn’t mean mutability. The resulting programming style “OO in the large, FP in the small” is gaining traction. Having shown that the combination works, other languages will certainly follow this path.

Ten years after its inception, Scala has a mature and vivid community of users. To gain further adoption, it must now consolidate its foundation and keep it stable across releases. Fortunately, we can still count on Odersky to continue to innovate at the same time. At the recent ScalaDays 2015, he unveiled his plan to better control mutations of state, not with monads, but implicit conversions. That is yet another ambitious challenge.

Feedback

Making sure that your project will be successful really boils down to two things: 1) get things done, 2) get feedback.

It’s obvious why getting things done matters: if you want to move forward, you need to get things done. Getting things done is however not a sufficient condition to be successful. You could be moving in the wrong direction! To be successful you need to constantly get feedback and steer the ongoing progress towards the goal.

These two principles are the heart of the agile manifesto: Move one step forward, adjust, and repeat. That’s the best strategy to ensure that what’s produced is really helpful for the project.

A step can be small or big. It can be the implementation of a single method with a peer review as feedback. It can be a refactoring with the automated execution of unit tests as feedback. It can be the implementation of a feature with the customer demo as feedback.

Scrum and XP are very different but are both considered “implementations” of the agile manifesto, since both promote moving forward and getting feedback in their own way.

Scrum is technology-agnostic process to get things done. The work is split in stories and tasks, which are small actionable items. To keep the momentum high, team members should focus on one task at a time. Feedback is obtained during the daily standup, the sprint review and sprint retrospective. You can use Scrum to conduct any project, not just software development.

XP on the hand is organised around technical software practices. It emphasizes pair programming, unit testing, continuous integration, refactoring, collective code ownership. The first three practices are nothing else than ways to get feedback about the code. Refactoring and collective code ownership are ways to guarantee that the team can always move forward.

With little surprise, XP and Scrum are good complements to each other. But they can also be complemented with other elements of your own. If something works for you to improve getting things done or getting feedback, add it. 

Make sure that feedback doesn’t turn into noise though. If feedback is not actionable it’s not truly feedback. What you want is feedback that help you get your next thing done in a better way. That’s the loop.

Unit Testing Matters

Unit testing is a simple practice that can be explained in one sentence: each method should have an associated test that verifies its correctness. This idea is very simple. What is amazing with unit testing is how powerful this simple practice actually is. At first, unit testing seems like a simple approach to prevent coding mistakes. Its main benefit seems obvious:

Unit testing guarantees that the code does what it should.

This is actually very good, since it’s remarkably easy to make programming mistakes: typo in SQL statements, improper boundary conditions, unreachable code, etc. Unit tests will detect these flaws. Shortly after, you will realize that it’s way easier to test methods that are short and simple. This confers to unit testing a second benefit:

Unit testing favors clean code.

This is also very good. Unit testing forces developers to name things and break down code with more care. This will increase the readability of the code base. Now, armed with a growing suite of tests, you will feel more secure to change business logic, at least when the change has local effects. This is a third benefit of unit testing:

Unit testing provides the safety net that enables changes

This is excellent. Fear is one of the prime factor that leads to code rot. With unit tests, you can ensure that you don’t break existing behavior, and can cleanly refactor or extend the code base. You might object that many changes are not always localized, and that unit tests don’t help in such case. But remember: a non-local changes is nothing more than a sequence of local changes. Changes at the local level represent maybe 80% of the work; the remaining 20% is about making sure that the local changes fit together. Unit tests help for the 80% of the work. Integration tests and careful thinking will do for the other 20%. As you become enamoured with unit testing, you will try to cover every line you write with unit tests. You will make it a personal challenge to achieve full coverage every time. This isn’t always easy. You will embrace dependency inversion to decouple objects, and become proficient with mocks to abstract dependencies. You will systematically separate infrastructure code from business logic. With time, your production code will be organized so that your unit tests can always obtain an instance of the object to test easily. Along the way, you will have noticed that the classes you write are more focused and easier to understand. This is the fourth benefit of unit testing:

    Unit testing improves software design

This is amazing! Unit testing will literally highlight design smells. If writing unit tests for a class is painful, your code is waiting to be refactored. Maybe it depends on global state (Yes, I look at you Singleton), maybe it depends on the environment (Yes, I look at you java.lang.System), maybe it does too much (Yes, I look at you Blob), maybe it relies too much on other classes (Yes, I look at you Feature Envy). Unit testing is “a microscope for object interactions.” Unit testing will force you to think very carefully about your dependencies and minimize them as much as possible. It will naturally promote the SOLID principles, and lead to better a decomposition of the software.

Honestly, I find it amazing that such a simple practice can lead to so many benefits. There are many practices out there that improve software development in some way. What makes unit testing special is the ridiculous asymmetry between its simplicity and its outcome.

More