Skip to main content

Takeaways from the Continuous Automated Testing Tutorial at CAST2014

I had the opportunity to attend Noah Sussman's tutorial on Continuous Automated Testing last week as part of CAST2014. It was a great tutorial, with most of the morning spent on the theory and concepts behind continuous automated testing, and the afternoon spent with some hands-on exercises. I think that Noah really understands the problems associated with test automation in an agile environment, and the solutions that he presented in his tutorial show the true depth of his understanding of, and insight into, those problems. Here are some of the main highlights and takeaways that I got from his tutorial at CAST2014.

Key Concepts

  • Design Tools – QA and testing are design tools, and the purpose of software testing is to design systems that are deterministic
  • Efficiency-to-Thoroughness-Trade-Offs – (ETTO) We do not always pick the best option, we pick the one that best meets the immediate needs
  • Ironies of automation – Automation makes things more complex and, while tools can make the process safer or faster, they cannot make things simpler
  • Hawthorne Effect -  Productivity (temporarily) goes up when you get a new process or tool
  • Goodhart’s Law – Simplified for the tutorial, the law states that people will game the system. Period.
  • Diseconomies of scale – The opposite of economies of scale, producing services at an increased unit cost
  • Conway’s Law  –  Simplified for the tutorial, the law states that software looks like your organization
  • Bikeshedding – It’s hard to build a complex, multipart system, but building a bike shed is easy, so organizations tend to spend too much time on trivial items

Automated Monitoring

In 2007 it was proposed by an engineer at Google that sufficiently advanced monitoring is indistinguishable from testing. This statement highlights the relationship that exists between monitoring and testing, and we can certainly use advanced monitoring to help us in our testing efforts. For example, we can use statsd as a means of instrumenting production code to gather high-volume data with minimal or no performance impact. The statement also highlights the issue of monitoring vs. testing. Noah provided a list of four things we should be doing as part of our monitoring efforts:

  • We should monitor all things
  • We should build real-time dashboards
  • We should deploy continuously
  • We should fix production bugs on the fly 

We should perform these four things, keeping in mind that while monitoring does provide visibility into implementation, it has nothing to do with design, and so does not replace QA and testing because they are design tools. Thus, while monitoring and testing are both necessary, it is only when practiced jointly that they are sufficient.

The Problem of Abstraction

We use abstractions as a means of hiding information, and we layer abstractions on top of the universe around us in an attempt to make things appear simpler than they really are. Eventually, however, we reach a point of complexity at which, even with multiple layers of abstractions to hide the information from us, our brains cannot process any more information. The Law of Leaky Abstractions, by Joel Spolsky, basically says that shifting layers of abstraction leak and, when this happens with software it results in bugs, commonly at the points where the abstractions integrate with each other.

Conway's Game of Life

One approach to addressing the risk introduced into a system by leaky abstractions is to limit complexity. Limiting complexity was something that Noah stressed several times, saying that “systems are safer if people keep the system under control” and “simple rules take you a lot further.” He also said that safety is derived from being able to predict system behavior, and suggested using Conway’s Game of Life as a learning environment, especially Golly. Golly serves as a good tool for learning and predicting system behavior because it can be used as a Read-Eval-Print loop (REPL), allowing you to predict what the output of the next step will be and executing that step while still maintaining the program state so that you can toggle back and forth to better understand and refine your predictions.

Jenkins for Testing and Monitoring

The hands-on portion of the tutorial walked through setting up Jenkins on your local machine, taking it beyond just a tool used for continuous integration to show how it could be used as a sort of “fancy cron” for scheduling test execution and other tasks. This is especially useful in continuous automated testing as Jenkins can be set up as a Read-Eval-Print loop for use in the rapid development of automation scripts. The tutorial continued by showing how to take advantage of other useful aspects of Jenkins, such as manipulating the URL to access the API documentation, using JSON and manipulating the URL to create real-time dashboards, and using Jenkins as a database for historical records of test executions.

Lightweight Automation

Automation in an agile environment is often too cumbersome, too brittle, and created too late in the sprint to be of any benefit to development activities in the current iteration. But it doesn’t have to be. Automated monitoring, REPLs, and less complex automated scripts can be utilized as a lightweight automation “framework” for continuous automated testing without the overhead typically associated with traditional automation techniques. Implementing an automation strategy in this way allows not only for much more agility in our automation efforts, but it also allows us to use automation as a design tool alongside our other testing activities.

Comments

  1. Replies
    1. IEEE Final Year projects Project Centers in Chennai are consistently sought after. Final Year Students Projects take a shot at them to improve their aptitudes. IEEE Final Year project centers ground for all fragments of CSE & IT engineers hoping to assemble.Final Year Projects for CSE

      Spring Framework has already made serious inroads as an integrated technology stack for building user-facing applications. Spring Framework Corporate TRaining .

      Specifically, Spring Framework provides various tasks are geared around preparing data for further analysis and visualization. Spring Training in Chennai

      The Angular Training covers a wide range of topics including Angular Directives, Angular Services, and Angular programmability.Angular Training

      Delete
  2. It likewise enables Normal engineers to meet the universe of information science without a profound numerical foundation, which will really make the entire procedure significantly simpler for experienced software engineers.
    machine learning certification

    ReplyDelete
  3. Giving comments on a significant blog is a form of art. I am entirely moved by this piece of writing. I wish to interpret more from you.

    Data Science training in Mumbai
    Data Science course in Mumbai
    SAP training in Mumbai

    ReplyDelete
  4. help in understanding the recent trends and customer demands, also to analyze customer feedbacks and most importantly handle inventory and warehousing. data science course syllabus

    ReplyDelete
  5. Wow what a Great Information about World Day its exceptionally pleasant educational post. a debt of gratitude is in order for the post.
    data science course in India

    ReplyDelete
  6. This article is very interesting and I got more kinds of details about this topic. Thanks for your sharing and I like more updates ...
    Reactjs Training in Chennai |
    Best Reactjs Training Institute in Chennai |
    Reactjs course in Chennai

    ReplyDelete
  7. Please i just took up chihuahua puppies breeding as a hobby after my mom passed away because they were her favorite PUPPIES. Despite the fact that they are very intelligent, am finding it very difficult getting them to mate yorkie puppy for sale near me, teacup puppies for sale near me
    yorkie poo for sale
    yorkies puppy for sale
    chihuahua puppies for sale near me

    ReplyDelete
  8. Greetings From New Day Yorkies puppies - Specializing
    We are small breeders whose goal is to produce healthy, high quality little teacup size Yorkie puppies for sale. In order to have puppies for sale most of the time we have teamed up with a couple of other Yorkie breeders that have the same goals in mine. In fact we have some of their Yorkie breeders and they have some of our breeders. https://www.newdaypuppies.com
    teacup yorkie for sale
    teacup chihuahua for sale
    yorkies for sale near me
    yorkie puppies for sale

    ReplyDelete
  9. Are Chihuahua puppies hard to train?
    chihuahua puppies for sale
    Chihuahuas are intelligent, strong-minded dogs that like to do their own thing. This can make them stubborn, earning them a reputation for being hard to train. However, reward-based training methods do appeal to a Chihuahua and there's no reason why they can't be trained to be obedient, just as with any dog

    Chihuahuas Have Fun, Playful Personalities
    teacup chihuahuas for sale
    Though chihuahuas are loyal and affectionate with their owners, they are anything but dull and love to play! However, no two chihuahuas are the same, so if you have a friend with a chihuahua, your pet is likely to be quite different and unique

    What food is bad for Chihuahua?
    teacup chihuahua for sale
    Foods Your Chihuahua Shouldn't Eat
    Alcohol. Alcohol (ethanol) is highly toxic to dogs. ...
    Caffeine. Coffee is one of the world's most popular beverages, with roughly 83% of the United States adult population consuming it on a daily basis. ...
    Chocolate. ...
    Some Fruit Seeds, Pits and Cores. ...
    Garlic. ...
    Grapes and Raisins. ...
    Hops. ...
    Onions.
    chihuahua puppy for sale

    The easiest way to adopt a Chihuahua would be through a rescue that specializes in Chihuahuas. A great place to start would be by starting a breed search on https://www.chihuahuapuppiesforsale1.com/chihuahua-puppies-for-sale-near-me/. The search will show you all the available Chihuahuas in your area.
    chihuahua puppies for sale near me

    ReplyDelete

  10. Are Yorkies good for first time dog owners?
    teacup yorkie for sale
    The Yorkshire terrier is a great starter dog for those who want a little lap dog. This breed is affectionate towards its owner and may even act protective around strangers.
    The Yorkie has a moderate energy level and only needs basic exercise.

    yorkie puppies for sale good for first time owners?
    yorkie puppies for sale
    It is the ideal pet for people who don't want to deal with pet fur at home or in their cars. This is the ideal pet for first time dog owners, since it trains easily and needs only moderate grooming maintenance. Yorkie poos are good with kids. It will play with them, is energetic and affectionate.

    Are Yorkie hypoallergenic? Yes
    Are Yorkie Poo good for first time owners?
    yorkie for sale
    It is the ideal pet for people who don't want to deal with pet fur at home or in their cars. This is the ideal pet for first time dog owners, since it trains easily and needs only moderate grooming maintenance. Yorkie poos are good with kids. It will play with them, is energetic and affectionate.
    teacup chihuahuas for sale

    Are Yorkie Poos good dogs?
    yorkies for sale

    https://www.newdaypuppies.com/

    ReplyDelete

  11. How long does a Yorkie live? 13 – 16 years
    Are Yorkies cuddlers?
    A lover of all things comfortable, the Yorkshire terrier enjoys cuddling with loved ones and snuggling into everything soft and fluffy. And for you, their silky coat isn't too bad for petting. https://www.newdaypuppies.com/

    Adorable Teacup and Toy AKC Yorkie Puppies
    We have adorable Yorkshire Terrier puppies male and female (she is pictured with the bow).The little girl was born on and the male was born on. yorkie teacup for sale Both puppies have started potty training and have had all sets of puppy shots and deworming. They both come with AKC papers.The girl is very petite and has an adorable baby doll face. yorkies for sale near me She is a teacup and will mature to be around just lbs. She is very gentle and easy going She is the princess in our home and loves to be held.The male is full of spunk and is super playful He will mature to be lbs and would make an awesome family dog. He has a gorgeous silky coat and adorable teddy bear face.Both puppies are AKC registered and will come with a health guarantee shot record and sample puppy food. teacup yorkies for sale, teacup yorkie puppies for sale

    Adorable Teacup and Toy AKC Yorkie Puppies is a Female, Male Yorkshire Terrier Puppy for sale in US. Adorable Teacup and Toy AKC Yorkie Puppies by newdaypuppies for Female, Male Yorkshire Terrier Puppies. yorkshire terrier for sale

    ReplyDelete
  12. https://www.chihuahuapuppiesforsale1.com/
    https://www.newdaypuppies.com/
    https://www.myppuphouse.com/
    https://chowchowpuppiessale.com/
    https://www.globalkittens.com/


    ReplyDelete
  13. Is a Yorkie a good family dog?
    Yorkies, like many other Toy breeds, make good pets for people; they're especially good for senior citizens, people with medical issues, and those who may worry about the size and strength of a larger dog. ... They're loving, devoted, and very affectionate: This makes them great personal companions and good family pets.

    Do Yorkshire terriers bark a lot?
    yorkies for sale
    Yorkshire Terriers are little dogs with huge personalities. teacup yorkies for sale With those huge personalities come a fierce territorial bark. Any time your phone rings, someone speaks or knocks on your door, or your doorbell chimes, your Yorkshire Terrier will likely bark. Outside noises aren't even required for barking for some Yorkies. yorkie puppies for sale

    How much do Yorkshire terriers cost?
    yorkie for sale near me
    Typical Yorkie prices range from $1,500 to $3,000, but the cost can fall far below or above that range. Prices will vary based on the puppy's lineage, appearance, health, and the breeder. It's also possible to adopt an older Yorkie for significantly less through a rescue shelter. yorkie for sale

    ReplyDelete
  14. https://www.newdaypuppies.com/
    https://www.newdaypuppies.com/teacup-yorkie-puppies-for-sale/
    https://www.newdaypuppies.com/tea-cup-yorkie-puppy-for-sale/
    https://www.newdaypuppies.com/yorkshire-terrier-for-sale-near-me/
    https://www.newdaypuppies.com/yorkie-terrier-puppy-for-sale-near-me/

    ReplyDelete
  15. windows 7 ultimate product key is the quantity one location to become if you would like to understand everything there is to understand about Edition. Genuine Product Key For Windows 7 Ultimate

    ReplyDelete
  16. Synthesia Crack provides you with a fresh blueprint to learn that the Piano in an amusing way. The Piano while with the note on your palms Synthesia Crack

    ReplyDelete

Post a Comment

Popular posts from this blog

A Year in Review

The following post came to mind as I was writing my year-end self-evaluation, and provides a brief glimpse of where I started the year and how I got to where I am today.  This year has been filled with diverse challenges, including ongoing employee issues, the continued mindset of "get it out the door", another reorg of the IT department, and the real possibility of the commoditization of testing within IT. However, as is often the case, challenge spurs innovation. In preparing for working on the team's seven-year strategic plan, I stepped back from the day-to-day operations of my team, and took a critical look at the work we were doing and the services we performed. What I saw was that the testing services we were providing for the company were, in many cases, nearly indistinguishable from the testing services provided by alternative sourcing strategies, with the primary differentiator being cost, not quality. Seeing the threat of the commoditization of testing

Mission Statement, Definition of Software Testing, and Goals of Software Testing

Why I blog? What’s the difference between a good tester and a great tester? I think the main thing is the ability to think for yourself and to be able to incorporate your experiences as a tester back into the context of your testing practices.  I think that if you look at the software testing community and pay attention to who has good ideas and who does not, you’ll find that the vast majority of people with good ideas emphasize their experience, what they have learned from it, and how they incorporate that back into their testing. Writing about my thoughts and experiences in software testing provides an opportunity for me to take a critical look at what I thought about a subject, assess it in the context of experience and information gained since I first came to think that way, and then update or reaffirm my thoughts on the subject. It also allows me to share my thoughts, experiences, successes and failures with others, creating an additional feedback loop. That, to me, is one