Lateral Thinking

Lateral thinking is a term coined by Edward De Bono to characterize the generation of alternative ideas, as opposed to vertical thinking, which generates ideas based on logic and stepwise refinements. Another way to explain lateral thinking in a much common way is “thinking out of the box.”

Lateral thinking is great to improve problem solving. Indeed, often finding the best solution requires a creative move to go away from the existing solution and start with a new angle.

As a reminder of the power of lateral thinking, let us take an egg and a spoon. You are doing a brunch. How do you provide assistance to help cut the egg?

With vertical thinking you might come up with this solution:


With lateral thinking, maybe with this one:


I was absolutely amazed the first time I saw this device in action. The cut is perfect. Also, I would probably never have come to this solution, no matter how long I stared at my egg.

Each time I discuss a design issue I remember my last brunch and try to take some distance with the situation to go back to the root of the problem to solve and ask: could we do this completely differently?

Sometimes the best way to cut an egg is to not cut it actually.

Making Progress

The essence of Scrum is to ensure progress. The formal elements of the framework –the restrospective, reviews, daily standup, etc.– are not ends in themselves but ways to ensure that progress happens.

It may seem simplistic to reduce Scrum to the mere fact of ensuring progress, but ensuring progress is not that easy, and Scrum is an effective tool to do it.

To prove this point, just think of what the opposite of progress means: to be stuck. A project can stuck for many reasons. Some symptoms include:

  • Work is half done or needs to be redone
  • Work is unclear and time is spent discussing it rather than doing it
  • Work wasn’t needed (people work on the wrong stuff)
  • Work can’t be done (because of dependencies, knowledge, etc.)

When a project is stucked, people work, but the overall project doesn’t move forward. The time is wasted.

Scrum prevent waisting time by maintaining a constant pressure on delivery and keeping the amount of work in progress low (“start finishing and stop starting”). It doesn’t matter how small the work item is. Actually the smaller the better, since it favors focus and quality.

Scrum is a framework for micromanagement, but without a micromanager. The team micromanages itself (i.e. “self-organisation”) and decides itself of the tasks to perform. Taskification happens mostly during the Scrum planning but then throughout the entire Sprint as the team actualises and refines the tasks to be done. And then does them.

The goal is to move forward, to overcome difficulties, to get concrete results, to make progress. For this you want the whole team to engage and people to help each other. You want your team to be more than the sum of its individuals.

I want teams emerging from the daily standup saying things like, “Let’s nail this. Let’s do this.”       — The Origins of the Daily Standup, Jeff Sutherland

People want to make progress fast, but software development is so complex that the risk is not to make progress too slowly but no progress at all. As long as you can ensure that some progress happens and you’re not compromising quality, you’re on a good track.

Gall’s Law

Gall’s law states that complex systems can only be the result of an evolutionary process, and not the result of a design from scratch:

A complex system that works is invariably found to have evolved from a simple system that worked. A complex system designed from scratch never works and cannot be patched up to make it work. You have to start over with a working simple system.  – John Gall (1975, p.71)

A complex system evolves from simpler systems by adding successive deltas of complexity. The only way to build a complex a system is through iteration. That’s what evoluation is about.

Iterations enable us to get feedback, correct and improve the system. See what works and what doesn’t. Fix mistakes.

The system must be working after each iteration. You can add new features, as long as it refines the existing system and keeps it running.

A tadpole becomes a frog by developing its legs, then its arms, and finally shrinking its tail. The frog’s legs, arms and body aren’t developed individually and assembled at the end. That’s not how evolution works.

last_thumb1367178271

Also, you can not evolve everything at once, since in the meantime the system might not work. A tadpole develops its legs, then its arms, and finally shrink its tail. Each iteration needs focus.

Gall’s law is relieving. It’s OK to not be able to handle all the complexity at once. And it’s not only you–it’s everybody.

A complex system can not be built using only theory and first principles, because there will always be details of the environment that we were not aware of. The only way to make sure something will work is to test it for real. Practice trumps theory.

Obsessing with getting it right the first time is counter productive. Just start somewhere and iterate. Too much unknown blocks our creativity. But once we have something concrete, ideas to improve come easily.

The tadpole also teaches us a lesson here: it first develops a tail, which then disappears later on. The tail is a good idea in the water, but not so much on the ground. You will have to reinvent yourself occasionally.

Unit Testing Matters

Unit testing is a simple practice that can be explained in one sentence: each method should have an associated test that verifies its correctness. This idea is very simple. What is amazing with unit testing is how powerful this simple practice actually is. At first, unit testing seems like a simple approach to prevent coding mistakes. Its main benefit seems obvious:

Unit testing guarantees that the code does what it should.

This is actually very good, since it’s remarkably easy to make programming mistakes: typo in SQL statements, improper boundary conditions, unreachable code, etc. Unit tests will detect these flaws. Shortly after, you will realize that it’s way easier to test methods that are short and simple. This confers to unit testing a second benefit:

Unit testing favors clean code.

This is also very good. Unit testing forces developers to name things and break down code with more care. This will increase the readability of the code base. Now, armed with a growing suite of tests, you will feel more secure to change business logic, at least when the change has local effects. This is a third benefit of unit testing:

Unit testing provides the safety net that enables changes

This is excellent. Fear is one of the prime factor that leads to code rot. With unit tests, you can ensure that you don’t break existing behavior, and can cleanly refactor or extend the code base. You might object that many changes are not always localized, and that unit tests don’t help in such case. But remember: a non-local changes is nothing more than a sequence of local changes. Changes at the local level represent maybe 80% of the work; the remaining 20% is about making sure that the local changes fit together. Unit tests help for the 80% of the work. Integration tests and careful thinking will do for the other 20%. As you become enamoured with unit testing, you will try to cover every line you write with unit tests. You will make it a personal challenge to achieve full coverage every time. This isn’t always easy. You will embrace dependency inversion to decouple objects, and become proficient with mocks to abstract dependencies. You will systematically separate infrastructure code from business logic. With time, your production code will be organized so that your unit tests can always obtain an instance of the object to test easily. Along the way, you will have noticed that the classes you write are more focused and easier to understand. This is the fourth benefit of unit testing:

    Unit testing improves software design

This is amazing! Unit testing will literally highlight design smells. If writing unit tests for a class is painful, your code is waiting to be refactored. Maybe it depends on global state (Yes, I look at you Singleton), maybe it depends on the environment (Yes, I look at you java.lang.System), maybe it does too much (Yes, I look at you Blob), maybe it relies too much on other classes (Yes, I look at you Feature Envy). Unit testing is “a microscope for object interactions.” Unit testing will force you to think very carefully about your dependencies and minimize them as much as possible. It will naturally promote the SOLID principles, and lead to better a decomposition of the software.

Honestly, I find it amazing that such a simple practice can lead to so many benefits. There are many practices out there that improve software development in some way. What makes unit testing special is the ridiculous asymmetry between its simplicity and its outcome.

More

Lines Spent

Studies have shown that the productivity of a developer is about 10 LOC/day. Considering that modern software consists in millions of lines of code, this number is appalling.

Measuring productivity with LOC is of course a dangerous thing to do. With programming, quality does not correlate with quantity, and it is wise to remember the words of Dijkstra:

If we wish to count lines of code, we should not regard them as “lines produced” but as “lines spent”. — Dijkstra

Indeed, programming is not a production process but a design process. Before a piece of code reaches maturity, various directions might first need to be explored, refined, and analyzed. Qualities like readability, performance, or reliability are competing dimensions in the design space. There is rarely “one way” to program something. Effective programming is about finding the best possible tradeoff in the shortest timeframe.

Writing high-quality code requires also a high level of rigor. Code must obey the established naming conventions, idioms and patterns of the system; it must be systematically refactored to prevent technical debt to accumulate; and it must be exhaustively tested. This level of discipline pays off in the long-term, but requires more time in the short-term.

Also, as a system grows, it becomes progressively harder to maintain an accurate mental model of the system. Considerable time is thus spent assessing the existing system to temporarily reconstruct enough knowledge for the task at hand. During this time, no code is written.

One could argue that counting total lines of code is useless; what is relevant is the number of lines of code modified per commit. Lots of modifications but no additions would suggest a modular system where features can be adapted declaratively without “writing new code”. Unfortunately, metrics tools do not follow this view and treat modifications as second-class citizens.

So, is a low number of LOC/day per day good or bad? For an optimist, it might indicate a sign of code quality; the team designs carefully and takes care to write the minimum amount of code needed. For a pessimist, if might indicate that the project is stalled; you’re not delivering enough code to make the deadline. For a realist, one thing is sure: software engineering is a very expensive activity.

More

Software As Liability

Software And Tactics

The image of a software engineer is that of a quiet and analytical guy working in isolation on some green-on-black code. There is some truth in this image. Studies have shown that interruptions are bad for programming, and that engineers need long streaks of uninterrupted time to fully immerse into a development activity. Projects are frequently structured in modules that are owned by individual programmers. In this solo view of software engineering, the less communication, the better.

In the agile view of software engineering, people and communication are at the center. You succeed as a team, or fail as a team. The code is not owned by individuals, but collectively by the team (chapter 10, Extreme Programming Explained). Work is organized into short actionable tasks, which prevents multi-tasking, ensures high focus, and high reactivity. These strict rules are the key ingredients to hyper-productivity.

Agility is defined as the combination of several elements, such as unit-testing, continuous integration, refactoring, etc. Amongst these elements, collective ownership is one of the hardest to implement. In contrast to the other elements, collective ownership requires a change of attitude, not just a change of technical practices. It requires moving from a solo mindset to a collective mindset.

To better understand how a collective mindset can be implemented, one should look at other professions were a collective mindset is critical. This is the case for instance for sport teams, firefighters, police officers, or SWAT team (Special Weapons And Tactics). A SWAT team’s effectiveness depends on its excellence in several practices:

Communication. SWAT team members communicate the action they engage in, the risks, and impediments. Communication must be concise, and adhere to a common vocabulary. The team lead oversees and coordinates the activities if necessary.

These considerations apply pretty much as is for software engineering:

“I’m about to launch the stress test of the web portal. Do you copy?”
“Copy that. I’m monitoring the logs.”

Execution.Only Practice makes perfect. SWAT team members train together standard practices and procedures to improve execution, such as the manipulation of weapon or hardware.

Software engineers should similarly train standard practices to improve execution, and master their tools. Sample practices to train include:

  • Navigating in the IDE
  • Synchronizing and merging code
  • Updating database (scripts, data, etc.)
  • Deploying software
  • Running various kinds of tests
  • Assessing code quality
  • Gathering performance metrics
  • Keeping the wiki up-to-date
  • Organizing release notes

Pairing. SWAT teams operate in dangerous environments. Mistakes are usually fatal and threats abound. By working in pairs, members can watch one others to prevent mistakes and protect themselves.

Pairing is great for software engineering, too. It reduces the risk of mistakes during coding, deployment, database updates. “given enough eyeballs, all bugs are shallow” as Linus’s Law says. While there are no external threats to software development, pairing favors knowledge transfer, and if a member is sick or leaves the team, the work can still go on smoothly.

The highly dynamic view of collective software engineering might appear as a complete clash against the highly analytical view of solo software engineering.

There are definitively parts of software engineering, such a design, that require quietness and thinking. But a large part of daily software engineering activities aren’t so: small refactorings, writing unit tests, fixing integration issues, measuring load and response times, etc. do not involve much thinking. They just need to be done.

There is scientific evidence that 80% of what a software developer does in a day—different steps and small microsteps— is not brain work. They do what they have done 50, 100, 1,000 times before. They just apply a pattern to new situations. — Mastermind of Programming, p.336

Lastly, collective software engineering requires redefining working time. In most working environments, individuals can work with their own schedule (hours, rhythm, pace). This is perfectly fine in the solo view of software engineering; however, it breaks the dynamics in collective software engineering. Ideally, team members always work together towards the team’s objective.

Software engineering is not always a creative endeavour. It is a fight against time and code rot. To win this fight, you need clever tactics. The challenge is to work as a an effective SWAT task force — where SWAT stands for Software And Tactics.

Simplicity Prevails

We engineers are masters of self-deception when it comes to our aptitudes to handle complexity. We believe we are way better at handling complexity than we actually are. In practice the level of complexity that people can master (me including) is disappointing low.

Instead of self-deceiving ourselves, we should embrace our limitations and aim for simplicity. This is what all geniuses of our time have been preaching for. Simplicity prevails.

“Simplicity is the ultimate sophistication.” — Leonardo da Vinci

It is common for tools to be too complex. We all know that only a tiny fraction of the features of Word are used, still the usual trend for a product is towards adding more and more features. Often, the best thing for a product is taking something away from it. Only simple tools prevail.

One problem with simplicity is that it is often confused with easiness or triviality. Easiness and triviallity are subjective properties relative to a user. They refer to the sense of familiarity or ordinarity. Simplicity is an objective property that refers to purity, the absence of mixing distinct elements.

Let us look at some simple features that are clear wins.

Using Venn Diagrams for Access Control

Have you ever been confused by the security settings of an application, not knowing what would be accessible or not to other users? Venn diagrams are simple and can be used to make an interface intuitive. Win!

Using Pictures instead of Texts to Log

Have you ever found yourself overwhelmed by the difficutly to compare tens of print statements to understand state mutations over time? Comparing texts is hard; comparing images is easy. Using a visual log is simple and supports better debugging. Win!

Using Examples to Test Programs

To test your code you exercise it with chosen inputs that serve as examples? Well, you could have invented unit testing. As Fowler says: “Kent’s framework had a nice combination of absurd simplicity and just the right features for me”. Win!

Using Live Code instead of Static Code to Understand Programs

Have you ever felts exhausted trying to mentally run code in your head when inspecting static sources? The underlying question is: what prevented you from runinng it and see it live? Finding a suitable unit test and breaking at the start of the method you are inspecting can be automated to become a one-click operation. Win!

Optionless search

Have you ever been repelled by the sheer amount of options in a search form? For instance this one. That’s tackling the problem of search wrongly. Google, Airbnb, and Facebook got it right, offering essentially one unique text input and hiding the magic of relevance matching. Win!

These examples show what I consider to be the level of complexity we can handle, and should aim at. These features are so simple to use they will immediately look easy and trivial. This is a good thing.

The Cost of Volatile

Assessing the scalability of programs and algorithms on multicore is critical. There is an important literature on locks and locking schemes, but the exact cost of volatile is less clear.

For the software composition seminar, I proposed this year a small project on the topic. The project was realized by Stefan Nüsch. He did a nice job and his results shed some light on the matter.

Essentially, we devised a benchmark where multiple threads would access objects within a pool. Each thread has a set of objects to work with. To generate contention, the sets could be fully disjointed, have partial overlap, of have a full overlap. The ratio of reads and writes per thread was also configurable.

On an AMD 64 Dual Core, the graph looks as follows:

bench_amd64On a i7 Quad Core, the graph looks as follows:

bench_i7We clearly see that different architectures have different performance profiles.

In future work, we could try to reproduce false sharing and assess the impact of other forms of data locality.

More details about the benchmark and methodology can be found in his presentation. The code in on github.

Here a some links about the semantics of volatile, and mememory management in general.

Debuggers: Theory and Practice

In “Not on the shelves“, Greg Wilson writes reviews about non-existing books, including one called “Debuggers: Theory and Practice”. It’s an entertaining and insightful read.

It made me curious about how such a book could look like and I gathered a selection of interesting articles about debugging and tried to organize them in a few sections. I wish I were more knowledable on the topic and had enough references to split section III into one section “exploring executions” and another one “strategies for debugging”.

Part I: Foundation

Chapter 1: Failures, Errors, Faults
Chapter 2: What is Debugging?
Chapter 3: Debuggers are Reflective Programs

Part II: Reproducing Bugs (Reproducing Failures)

Chapter 1: Every bug is a test not yet written
Chapter 2: Record and Replay
Chapter 3: Execution Synthesis
Chapter 4: Reproducing Crashes with Recrash

Part III: Fixing Bugs (Identifying Faults)

Chapter 1: The Art of System.out
Chapter 2: Asking Why Questions with the Whyline
Chapter 3: Scriptable Time-Travel Debugging with First-Class Trace
Chapter 4: Back-in-Time Alias Debugging
Chapter 5: Object-centric Debugger
Chapter 6: Debugging Concurrent Programs

Why Smalltalk?

This is a question is occasionally asked by students and here is the answer.

We are not religious. This choice is not dogmatic. We do both research in programming languages and tool support for software evolution. In both cases Smalltalk is handy:

  • Programming language — Smalltalk is extermely uniform. Experimenting with a language change is faster in Smalltak, than say, Java or Ruby. Their syntax and sets of rules are bigger, which implies more work.
  • Tool Support — If you want to extend the environment with more browsers/views/features, you can do it easily. Also, tool support and meta-programming go well together. You don’t have a separation between application code and environment code. This makes the whole system very malleable to experiement with.

There exist other research platforms out there to ease experimentations in either category. They don’t match however with the versatility of Smalltalk, which remains thus a very competitive choice to consider. Usually, we mature our project to Java or Eclipse only after initial success in Smalltalk.

Smalltalk Anthology

  • Smalltalk 80: the Language and its Implementation, by Adele Goldberg
  • Smalltalk-80: Bits of History, Words of Advice, by Glenn Krasner
  • Design Principles Behind Smalltalk, by Daniel H. Ingalls
  • The early history of Smalltalk, by Alan Kay
  • Smalltalk Best Practice Patterns, by Kent Beck
  • Smalltalk: a Reflective Language, by F. Rivard
  • Back to the future: the story of Squeak, a practical Smalltalk written in itself, by Daniel H. Ingalls et al.
  • Special issue on Smalltalk, BYTE magazine 1981