Skip to main content

Takeaways from the Continuous Automated Testing Tutorial at CAST2014

I had the opportunity to attend Noah Sussman's tutorial on Continuous Automated Testing last week as part of CAST2014. It was a great tutorial, with most of the morning spent on the theory and concepts behind continuous automated testing, and the afternoon spent with some hands-on exercises. I think that Noah really understands the problems associated with test automation in an agile environment, and the solutions that he presented in his tutorial show the true depth of his understanding of, and insight into, those problems. Here are some of the main highlights and takeaways that I got from his tutorial at CAST2014.

Key Concepts

  • Design Tools – QA and testing are design tools, and the purpose of software testing is to design systems that are deterministic
  • Efficiency-to-Thoroughness-Trade-Offs – (ETTO) We do not always pick the best option, we pick the one that best meets the immediate needs
  • Ironies of automation – Automation makes things more complex and, while tools can make the process safer or faster, they cannot make things simpler
  • Hawthorne Effect -  Productivity (temporarily) goes up when you get a new process or tool
  • Goodhart’s Law – Simplified for the tutorial, the law states that people will game the system. Period.
  • Diseconomies of scale – The opposite of economies of scale, producing services at an increased unit cost
  • Conway’s Law  –  Simplified for the tutorial, the law states that software looks like your organization
  • Bikeshedding – It’s hard to build a complex, multipart system, but building a bike shed is easy, so organizations tend to spend too much time on trivial items

Automated Monitoring

In 2007 it was proposed by an engineer at Google that sufficiently advanced monitoring is indistinguishable from testing. This statement highlights the relationship that exists between monitoring and testing, and we can certainly use advanced monitoring to help us in our testing efforts. For example, we can use statsd as a means of instrumenting production code to gather high-volume data with minimal or no performance impact. The statement also highlights the issue of monitoring vs. testing. Noah provided a list of four things we should be doing as part of our monitoring efforts:

  • We should monitor all things
  • We should build real-time dashboards
  • We should deploy continuously
  • We should fix production bugs on the fly 

We should perform these four things, keeping in mind that while monitoring does provide visibility into implementation, it has nothing to do with design, and so does not replace QA and testing because they are design tools. Thus, while monitoring and testing are both necessary, it is only when practiced jointly that they are sufficient.

The Problem of Abstraction

We use abstractions as a means of hiding information, and we layer abstractions on top of the universe around us in an attempt to make things appear simpler than they really are. Eventually, however, we reach a point of complexity at which, even with multiple layers of abstractions to hide the information from us, our brains cannot process any more information. The Law of Leaky Abstractions, by Joel Spolsky, basically says that shifting layers of abstraction leak and, when this happens with software it results in bugs, commonly at the points where the abstractions integrate with each other.

Conway's Game of Life

One approach to addressing the risk introduced into a system by leaky abstractions is to limit complexity. Limiting complexity was something that Noah stressed several times, saying that “systems are safer if people keep the system under control” and “simple rules take you a lot further.” He also said that safety is derived from being able to predict system behavior, and suggested using Conway’s Game of Life as a learning environment, especially Golly. Golly serves as a good tool for learning and predicting system behavior because it can be used as a Read-Eval-Print loop (REPL), allowing you to predict what the output of the next step will be and executing that step while still maintaining the program state so that you can toggle back and forth to better understand and refine your predictions.

Jenkins for Testing and Monitoring

The hands-on portion of the tutorial walked through setting up Jenkins on your local machine, taking it beyond just a tool used for continuous integration to show how it could be used as a sort of “fancy cron” for scheduling test execution and other tasks. This is especially useful in continuous automated testing as Jenkins can be set up as a Read-Eval-Print loop for use in the rapid development of automation scripts. The tutorial continued by showing how to take advantage of other useful aspects of Jenkins, such as manipulating the URL to access the API documentation, using JSON and manipulating the URL to create real-time dashboards, and using Jenkins as a database for historical records of test executions.

Lightweight Automation

Automation in an agile environment is often too cumbersome, too brittle, and created too late in the sprint to be of any benefit to development activities in the current iteration. But it doesn’t have to be. Automated monitoring, REPLs, and less complex automated scripts can be utilized as a lightweight automation “framework” for continuous automated testing without the overhead typically associated with traditional automation techniques. Implementing an automation strategy in this way allows not only for much more agility in our automation efforts, but it also allows us to use automation as a design tool alongside our other testing activities.

Comments

  1. It likewise enables Normal engineers to meet the universe of information science without a profound numerical foundation, which will really make the entire procedure significantly simpler for experienced software engineers.
    machine learning certification

    ReplyDelete
  2. Giving comments on a significant blog is a form of art. I am entirely moved by this piece of writing. I wish to interpret more from you.

    Data Science training in Mumbai
    Data Science course in Mumbai
    SAP training in Mumbai

    ReplyDelete
  3. Wow what a Great Information about World Day its exceptionally pleasant educational post. a debt of gratitude is in order for the post.
    data science course in India

    ReplyDelete
  4. This article is very interesting and I got more kinds of details about this topic. Thanks for your sharing and I like more updates ...
    Reactjs Training in Chennai |
    Best Reactjs Training Institute in Chennai |
    Reactjs course in Chennai

    ReplyDelete
  5. https://www.chihuahuapuppiesforsale1.com/
    https://www.newdaypuppies.com/
    https://www.myppuphouse.com/
    https://chowchowpuppiessale.com/
    https://www.globalkittens.com/


    ReplyDelete
  6. windows 7 ultimate product key is the quantity one location to become if you would like to understand everything there is to understand about Edition. Genuine Product Key For Windows 7 Ultimate

    ReplyDelete
  7. Synthesia Crack provides you with a fresh blueprint to learn that the Piano in an amusing way. The Piano while with the note on your palms Synthesia Crack

    ReplyDelete

Post a Comment

Popular posts from this blog

Mission Statement, Definition of Software Testing, and Goals of Software Testing

Why I blog? What’s the difference between a good tester and a great tester? I think the main thing is the ability to think for yourself and to be able to incorporate your experiences as a tester back into the context of your testing practices.  I think that if you look at the software testing community and pay attention to who has good ideas and who does not, you’ll find that the vast majority of people with good ideas emphasize their experience, what they have learned from it, and how they incorporate that back into their testing. Writing about my thoughts and experiences in software testing provides an opportunity for me to take a critical look at what I thought about a subject, assess it in the context of experience and information gained since I first came to think that way, and then update or reaffirm my thoughts on the subject. It also allows me to share my thoughts, experiences, successes and failures with others, creating an additional feedback loop. That, to me, is one

Let’s Continue to Drive Software Testing Education Forward

It’s Time to Embrace the Student Who Learns Differently Last in a series. In my last two posts I’ve written about enhancing the student and instructor experience in the AST’s BBST courses by focusing on updating the Fieldstones and making BBST courses more accessible by identifying some of the more common obstacles to BBST participation, and then working collectively to find ways to lower or remove those obstacles. In this post, I want to discuss the third and final area I would like to concentrate on if elected to the Board of Directors: researching and establishing alternate approaches to teaching that better suit different learning styles. As members of the AST, we have access to some of the best information and training available in the field of software testing. The AST hosts the Conference of the Association for Software Testing (CAST) each year, providing full-day tutorials, keynotes, and track sessions. They also offer four separate BBST courses:  Foundations, Bug