Tuesday, August 19, 2014

Takeaways from the Continuous Automated Testing Tutorial at CAST2014

I had the opportunity to attend Noah Sussman's tutorial on Continuous Automated Testing last week as part of CAST2014. It was a great tutorial, with most of the morning spent on the theory and concepts behind continuous automated testing, and the afternoon spent with some hands-on exercises. I think that Noah really understands the problems associated with test automation in an agile environment, and the solutions that he presented in his tutorial show the true depth of his understanding of, and insight into, those problems. Here are some of the main highlights and takeaways that I got from his tutorial at CAST2014.

Key Concepts

  • Design Tools – QA and testing are design tools, and the purpose of software testing is to design systems that are deterministic
  • Efficiency-to-Thoroughness-Trade-Offs – (ETTO) We do not always pick the best option, we pick the one that best meets the immediate needs
  • Ironies of automation – Automation makes things more complex and, while tools can make the process safer or faster, they cannot make things simpler
  • Hawthorne Effect -  Productivity (temporarily) goes up when you get a new process or tool
  • Goodhart’s Law – Simplified for the tutorial, the law states that people will game the system. Period.
  • Diseconomies of scale – The opposite of economies of scale, producing services at an increased unit cost
  • Conway’s Law  –  Simplified for the tutorial, the law states that software looks like your organization
  • Bikeshedding – It’s hard to build a complex, multipart system, but building a bike shed is easy, so organizations tend to spend too much time on trivial items

Automated Monitoring

In 2007 it was proposed by an engineer at Google that sufficiently advanced monitoring is indistinguishable from testing. This statement highlights the relationship that exists between monitoring and testing, and we can certainly use advanced monitoring to help us in our testing efforts. For example, we can use statsd as a means of instrumenting production code to gather high-volume data with minimal or no performance impact. The statement also highlights the issue of monitoring vs. testing. Noah provided a list of four things we should be doing as part of our monitoring efforts:

  • We should monitor all things
  • We should build real-time dashboards
  • We should deploy continuously
  • We should fix production bugs on the fly 

We should perform these four things, keeping in mind that while monitoring does provide visibility into implementation, it has nothing to do with design, and so does not replace QA and testing because they are design tools. Thus, while monitoring and testing are both necessary, it is only when practiced jointly that they are sufficient.

The Problem of Abstraction

We use abstractions as a means of hiding information, and we layer abstractions on top of the universe around us in an attempt to make things appear simpler than they really are. Eventually, however, we reach a point of complexity at which, even with multiple layers of abstractions to hide the information from us, our brains cannot process any more information. The Law of Leaky Abstractions, by Joel Spolsky, basically says that shifting layers of abstraction leak and, when this happens with software it results in bugs, commonly at the points where the abstractions integrate with each other.

Conway's Game of Life

One approach to addressing the risk introduced into a system by leaky abstractions is to limit complexity. Limiting complexity was something that Noah stressed several times, saying that “systems are safer if people keep the system under control” and “simple rules take you a lot further.” He also said that safety is derived from being able to predict system behavior, and suggested using Conway’s Game of Life as a learning environment, especially Golly. Golly serves as a good tool for learning and predicting system behavior because it can be used as a Read-Eval-Print loop (REPL), allowing you to predict what the output of the next step will be and executing that step while still maintaining the program state so that you can toggle back and forth to better understand and refine your predictions.

Jenkins for Testing and Monitoring

The hands-on portion of the tutorial walked through setting up Jenkins on your local machine, taking it beyond just a tool used for continuous integration to show how it could be used as a sort of “fancy cron” for scheduling test execution and other tasks. This is especially useful in continuous automated testing as Jenkins can be set up as a Read-Eval-Print loop for use in the rapid development of automation scripts. The tutorial continued by showing how to take advantage of other useful aspects of Jenkins, such as manipulating the URL to access the API documentation, using JSON and manipulating the URL to create real-time dashboards, and using Jenkins as a database for historical records of test executions.

Lightweight Automation

Automation in an agile environment is often too cumbersome, too brittle, and created too late in the sprint to be of any benefit to development activities in the current iteration. But it doesn’t have to be. Automated monitoring, REPLs, and less complex automated scripts can be utilized as a lightweight automation “framework” for continuous automated testing without the overhead typically associated with traditional automation techniques. Implementing an automation strategy in this way allows not only for much more agility in our automation efforts, but it also allows us to use automation as a design tool alongside our other testing activities.

Thursday, March 13, 2014

The Anomic Potential of #NoEstimates

I was looking for a bit of clarification on #NoEstimates, and I remembered reading one of Neil Killick’s posts about it that I thought would be relevant. As I was combing through his blog, looking for the post I had in mind, I stumbled across People Need Estimates. What caught my attention was not so much the title of the post, but the image of a red umbrella that went along with the post. That image of the umbrella, along with the title, had me making the connection between #NoEstimates and the work of Peter L. Berger because Berger says that society creates a sacred canopy (or umbrella) to help us relate to the world in a consistent way, and if we are forced to move from under that canopy we face chaos and fear.

I began to get excited as I read the article because, whether he knew it or not, and reference to the umbrella aside, many of the points that Neil was making were resonating with what I remembered reading in Berger’s book The Sacred Canopy: Elements of a Sociology of Religion. The thing that struck me the most was that #NoEstimates was not in the nomos of our software development methodologies, and thus contains a great deal of anomic potential, making widespread adoption an uphill battle.

Let me give a little background information before I talk more about nomos and anomic potential. It starts with what Berger termed Externalization, Objectivation, and Internalization.

Berger starts by saying that people feel out of kilter or off balance because the way we relate to our environment changes the environment (he calls this externalization), and so our relationship with the world is always changing. He also says that what people really want is to feel in balance, and to be able to predict how they will need to relate to the world. Society then, says Berger, gives us that feeling of being in balance with the world by making things familiar and providing us with an environment in which we can predict how we will need to relate to the world.

To help us feel like we are in balance society, according to Berger, teaches us to make the same choices in how we relate to the world. Repeatedly. This has the effect of making us think that the world is more objective and less likely to change, and so how we relate to the world will not change (he calls this objectivation). Moreover, it makes us forget that the choices we make as we relate to the world are really choices at all, and we begin to think of them as the way things are supposed to be.

Berger says that society does its job so thoroughly that as we become more socialized we begin to internalize the objective world that it creates (which he calls internalization). So, not only do we come to think that the way things are done is the only way that they could be done, but we start to associate our own identity with the way these things are done. This, in turn, leads us to feel out of balance with the world if we do things any other way than society tells us they should be done.

Society has created an entire, illusory world view, and this is what Berger calls the nomos. Nomos is everything that society has taught us, including everything that we think about the way the world really is and how we relate to that world. It’s everything that keeps us feeling like we are in balance and, just like creating an objective world that we can relate to, society’s job is to make us think that the nomos it provides is objective and unchangeable.

So, when Neil says that people need an umbrella, that’s true from the point of view of the nomos that was created by society. When he says that people don’t need umbrellas, they need a way to stay dry on a rainy day, that’s from a point of view outside of that nomos. But this can threaten the legitimacy of the established nomos. It threatens the stability of the nomos, possibly to the point that the nomos is destroyed.

This is what Berger calls anomy, and things that are outside of the nomos and have the potential to disrupt it have “anomic potential.” Since the nomos provides a sense of stability, comfort, and balance to the world, anything outside of that is seen as chaotic and terrifying; it’s the fear of the unknown, and people use the nomos as a sort of shield against it.

So how does that tie in with #NoEstimates? As I said above, it’s fear of the unknown. Estimates are a part of the nomos created by the “society” of software development. So, even though estimates can be useful at times, the current nomos says that we are to use estimates to get a price and a date for developing software. To obtain a price or date without using an estimate is outside of the nomos. #NoEstimates is scary; it’s anomy, and represents a certain amount of anomic potential.

And that’s why #NoEstimates faces an uphill battle; it’s a social change, and social change is hard because you really have to change the nomos before you can realize the change in society. It’s not that people don’t think it’s a good idea or that they can’t do it; everyone, whether they are aware of it or not, has the ability to act independently of the nomos. Instead, it’s more a fear that doing so will highlight the sense of being out of balance with the world, and we don’t like that.

I suspect that’s also a big factor as to why Woody Zuill says in his blog post on Life, Liberty, and the Pursuit of Agility that few people are willing to give an outright answer when he asks them "if you found estimates bring no value – what would you do?" It’s like looking death in the face. It’s unsettling because it represents so much anomic potential, and so we prefer not to do it.

Wednesday, March 5, 2014

Book Review - The Shape of Actions: What Humans and Machines Can Do

If you’re a tester and you’ve been around social media, attended a conference, watched a webinar, read blog posts, or watched videos of other testers speaking on YouTube, you may have heard at least one mention of polimorphic and/or mimeomorphic actions. But what does it mean when someone says that an action is polimorphic or mimeomorphic? Where do these ideas come from, and why, as testers, do we care?

The concepts of polimorphic and mimeomorphic actions come from the book The Shape of Actions: What Humans and Machines Can Do, by Harry Collins and Martin Kusch. In the book the authors develop a new theory about what they call the shape of actions. I’ve attempted to cover the highlights and general topics of discussion, or at least what I found most interesting, from each chapter in the summary below.

Chapter 1 – Humans and Machines

In Chapter 1, Collins and Kusch introduce the reader to their theory which basically states that humans can do three things – they can do polimorphic actions (actions that draw on an understanding derived from a sociological structure), they can do mimeomorphic actions (actions that are performed like machines, and do not require an understanding derived from sociological structure), and they can merely behave. The authors are concerned with how actions look from the point of view of an observer; they say they are concerned with the “shape” of actions, or what they call action morphicity. The authors conclude their introduction to the shape of actions by discussing the two key entities of this theory – the humans, which are all entities that can perform polimorphic actions, and the machines, which are all entities that cannot do polimorphic actions, and how the boundaries between these two entities is permeable (after all, humans do act like machines in many instances), and how the theory of actions may establish new boundaries between humans and machines.

Chapter 2 – Methods and Principles

This chapter begins the discussion of what a theory of action is by suggesting that actions may be classified as one of two types, depending on the behavior that instantiates them. The authors also make the distinction between actions (the things that we can do in a society, that get their sense from taking place in that society) and behaviors (physical movements, including those that we use to execute the actions we intend), with the primary differentiator being that actions are intentional. In particular, the theory of actions is concerned with what are called formative actions, those actions which not only get their sense from taking place in a society, but that also make that society what it is and distinguish it from other societies. The authors then go on to describe how actions and institutions are what they term “social kinds” because they exist only as long as those who are in the society/institution act as though they exist (for example, money), and “natural kinds,” which continue to exist regardless of how those taking part in the society choose to act.

Chapter 3 – Morphicity: What the Action Is

Chapter 3 examines the classification of polimorphic and mimeomorphic actions before introducing a means of representing actions as diagrams. In the case of polimorphic actions, the authors take advantage of the fact that polimorphic actions typically use different behaviors to carry out the same action in different contexts, and identify three subtypes of polimorphic actions based on context and behavioral response:

  • Open Polimorphic actions are polimorphic actions in which both context and response are open (for example, love letter writing)
  • Occasioned Polimorphic are polimorphic actions in which the context is open but the response is not (for example, voting)
  • Playful polimorphic actions are polimorphic actions in which the context is closed but the response is open (for example, relieving tedium or expressing individuality)

Mimeomorphic actions, on the other hand, lack the association with context that is a characteristic of polimorphic actions, and are classified by preference as too how they are carried out (either casual, in which case the action is marked by our indifference to how the action is accomplished within limits – the envelope of acceptable behavior, or special, in which case the action is marked by our preference for the same behavioral instantiation each time) and plurality (either singular, in which there is only a single way in which the action may be performed, or disjunctive, in which the actor must choose from one of a set of acceptable ways in which to execute the action). This allows the authors to identify four subtypes of mimeomorphic actions:

  • Special Singular Mimeomorphic actions are actions in which the action should be performed the same way each time and a single way in which it should be performed (for example, swinging a golf club)
  • Special Disjunctive Mimeomorphic actions are actions in which the action should be performed the same way each time and the actor must choose from a set of acceptable ways to execute the action (for example, expert golf club swinging)
  • Casual Singular Mimeomorphic actions are actions in which there is an indifference in how the actions should be carried out (so long as it is within an acceptable limit) and a single way in which it should be performed (for example,  telephoning one’s mother from home)
  • Casual Disjunctive Mimeomorphic actions are actions in which there is an indifference in how the actions should be carried out (so long as it is within an acceptable limit) and the actor must choose from a set of acceptable ways to execute the action (for example, telephoning)

Chapter 3 also introduces what the authors call an action tree. Action trees are diagrams that represent the relationships between actions, and look much like decision trees with the exception that the authors adopt some conventions to indicate the morphicity of the action represented at a certain node. In an action tree, the nodes at the higher levels represent actions that are realized further down the tree by other actions that are more specifically defined.

Chapter 4 – A Theory of Interaction

The authors begin discussing interactions, both within a culture (humans to humans) and across cultures (humans to machines) in Chapter 4. The primary means of intra-cultural interactions is via action cascades, in which one actor, A, carries out an action for another actor, B, in such a way that A’s action is a sub-action of B’s action. The polimorphic/mimeomorphic dichotomy is applied to action cascades at the point where the transfer of action takes place, resulting in the identification of two types of cascades – Control Cascades, in which the action above the transfer is polimorphic and below the point of transfer is mimeomorphic, and Indifference Cascades, in which the action above the transfer is mimeomorphic and below the point of transfer is polimorphic. Since polimorphic actions derive understanding from within our sociological structure, and that structure is non-existent across cultures, the authors conclude that different cultures may only incorporate the mimeomorphic actions of other cultures into their own action cascades.

Chapter 5 – Morphicity and Human Competence

Chapter 5 begins the discussion on what is involved in human learning, and how the theory of actions applies to human competence and the transfer of sets of actions. The authors guide the reader through several examples of actions, starting with special singular mimeomorphic actions which are learned through “calculation”  that requires no need for practice or drill, through increasingly complex actions, ending with polimorphic actions that can only be learned through social skills. The authors provide two main insights through these examples. First, they show how action morphicity helps us understand how competencies are learned. Second, they demonstrate that the mimeomorphic/polimorphic dichotomy does not correspond to other ways of dividing up our abilities. For example, mimeomorphic actions may be either skilled or unskilled and they may be self-conscious or internalized. The chapter concludes with a discussion about the ability to transfer sets of actions or competencies between cultures, establishing the “irreplaceability thesis” which draws on the self-referential nature of polimorphic actions to state that they may only be learned by social interaction within the society to which they belong, whereas mimeomorphic actions can be decontextualized, and hence may be transferred between cultures.

Chapter 6 – Writing

In Chapter 6 the authors examine several different forms of writing within the context of the shape of actions to assess if the action of writing is polimorphic or mimeomorphic. Not only do different styles of writing (say script writing or calligraphy) have different morphicities, but the morphicities also differ within the same style of writing. For example, a child’s script (as they are learning) is a special other-copying singular mimeomorphic action whereas the adult’s script can be seen as casual disjunctive self-copying mimeomorphic action. Interestingly, the adult’s script can also be seen as a casual singular self-copying mimeomorphic action when using the individual as context and as casual disjunctive self-copying mimeomorphic action with respect to other individuals. This apparent duality lends much credence to the author’s caution that “one must look at each action in considerable detail, for actions are not always what they seem to the casual glance.”

Chapter 7 – Machine Behavior and Human Action

Chapter 7 covers a lot of ground on the complex task of mapping machine behavior to human action. It begins by discussing how machines are classified, saying that either we can classify them by what they do or by how they work. The authors then discuss three types of machines based on what they do:

  • Tools, which help us to improve on what we already do (a hammer)
  • Proxies, which are used to replace us by doing the things we already do (some computer programs)
  • Novelties, which allow us to do things that we could not do without them (a freezer)

The authors make an interesting observation that proves to be fundamental in determining what machines can do: the differences between the three types of machines are largely a matter of perspective based on where we are in the action tree. This leads to the general rule that we can view tools as proxies by using a vantage point lower in the action tree, and we can view proxies as tools by taking a higher vantage point in the action tree. The argument then follows that what we have been seeing as proxies should really be seen as tools because we have been inattentively “repairing” the deficiencies of the machines and attributing the machine with doing an action for us when it was really only helping us to do the action better. The authors then bring the power of a fully operational theory of actions into play by saying that machines can only act as proxies when the actions they perform are far enough down the tree as to be mimeomorphic, and that the proper attribution of agency can only be applied up to the point in the action tree where polimorphic actions begins.

Chapter 8 – Organizations and Machines

Chapter 8 applies the dichotomy to organizations, and seems to extend the findings from the previous chapter by saying that even with respect to organizations, we have had the wrong vantage point, and so even the things we thought were mimeomorphic are instead polimorphic. The rules within bureaucracies, for example, do not come with rules for their application, and thus still require contextual interpretation by the actor. The book also gives the example of “complete deskilling” at a restaurant such as McDonald’s where replacing esoteric with ubiquitous skills makes it seem like a mechanical process, but it is not deskilling because it is a transference of the responsibility of exercising the skills to another actor (the customer in the case of McDonald’s). The authors discuss how the dichotomy may be applied to the world of science, showing that what is typically presented as a field dominated by mimeomorphic actions is, in fact, heavily reliant on polimorphic actions, and has been made to appear so due to the repair that historians have made as they recorded the events.

Chapter 9 – The Automation of Air Pumps

In Chapter 9 vacuum pumps are used to examine the skills required to operate a device, and how those skills change as the device changes. Comparing the use of modern, automated vacuum pumps and Robert Boyle’s experiences with vacuum pumps in the seventeenth century, the authors found that skills don’t change much when we have active closure because those skills are still needed to operate the device, but they do change as we move towards passive closure. In the case of active closure, which says that an instrument whose use has been accepted and whose general principles of design have been settled still requires skill and agreement to make it work, we have theoretical indifference at this point (say that we are indifferent to what mixture is used to seal the leather used in Boyle’s pump) but not behavioral indifference. Skills do change once we begin moving towards passive closure, however, because we have moved into the realm of behavioral indifference, and at a certain level it becomes a matter of indifference to all parties, at which point we are talking about mimeomorphic actions.

Chapter 10 – Conclusion: Dichotomies and History

The authors begin the discussion in the final chapter of the book by pointing out that the distinction between polimorphic and mimeomorphic actions does not match other dichotomies, such as the action versus behavior dichotomy, the action versus basic action, and several others. The point the authors make between their theory and other theories of action is their theory states that, in principle, all mimeomorphic actions may be mechanized, though some of them may be too complicated to do so, because the way the engineering is done is not crucial in the theory they provide. Instead, they claim that the idea of mimeomorphic actions creates a common ground for actions and behaviors that other theories do not.


The Shape of Actions is a tough yet rewarding read. It’s rather academic, and is one of those books written for another field (the authors refer to it as sociphilosophy), but it remains very applicable to software testing, and is a book that I'll be returning to frequently.

My thoughts kept turning towards test automation as I read the book, which really makes sense given that the theory creates a common ground where actions and behaviors can cohabitate and discusses, in great detail, what machines and humans can do. In particular, I found the discussions on special and casual disjunctive mimeomorphic actions very beneficial with respect to model-based test automation. But there are many more ways that the information from the book may be applied to testing software, whether it’s during test planning, test design, test execution, or test reporting. I plan to post several follow-up posts based on ideas from the book and how I apply those to my testing activities.

My challenge to you is to read the book, apply what you learn from it to improve or expand your own testing activities, then come back here and post a comment to let me know how you have put that information to use.

Wednesday, January 1, 2014

A Year in Review

The following post came to mind as I was writing my year-end self-evaluation, and provides a brief glimpse of where I started the year and how I got to where I am today. 

This year has been filled with diverse challenges, including ongoing employee issues, the continued mindset of "get it out the door", another reorg of the IT department, and the real possibility of the commoditization of testing within IT. However, as is often the case, challenge spurs innovation.

In preparing for working on the team's seven-year strategic plan, I stepped back from the day-to-day operations of my team, and took a critical look at the work we were doing and the services we performed. What I saw was that the testing services we were providing for the company were, in many cases, nearly indistinguishable from the testing services provided by alternative sourcing strategies, with the primary differentiator being cost, not quality. Seeing the threat of the commoditization of testing as a real possibility emphasized, to me, the need to improve the quality of service the team provides.  

I began by redoubling my efforts to establish a testing organization that provides true value to the business. I knew that I needed more ideas than I could generate by myself, so I started looking for industry colleagues from outside the company to better understand the challenges I faced, the solutions that are available, and to discover new ideas to incorporate into our practices. Leveraging the power of social media, I have been able to establish relationships with peers and leaders from the international software testing community. Through these relationships I have received not only validation, correction, and extension of ideas that I have had regarding changes and improvements to the way that we test software, but also an influx of new and innovative ideas that, once implemented, will truly establish the team as the definitive source of testing expertise and services within the company.

Many of the ideas require a fundamental shift in the way we think about quality and the skills required to provide quality. If we view quality as value to some person at some time, then we realize that quality changes as the needs and priorities of the business change. It also highlights the importance context plays in delivery of a quality service. By moving towards a context-driven approach, the team has taken the first steps in establishing a position that allows us to stay more relevant to the business by asking the business "what is of value to you" and using that information to address their needs rather than what we think those needs are.

There is another benefit of the new paradigm in that it allows us to address the commoditization singularity - the point at which the value of the testing we provide becomes indistinguishable from the value of the testing provided by alternate sourcing strategies, and thus becomes a commodity. This paradigm shift allows us to address that by keeping in touch with the needs of the business, and having the skills and agility to be proactive as those needs are changing, rather than being simply reactive after the change is complete or, even worse, not changing at all.