Ewert Technologies Blog

PWMinder 2.5 is to be released shortly.  PWMinder is a cross-platform Password Manager that allows you to effectively manage all of your passwords and other sensitive data in one secure place.  See PWMinder Home Page, for more details, including screen shots and screen casts.

New for version 2.5

• Encryption alogithm changed to the U.S. Government standard: Advanced Encryption Standard (AES).
• Repository file format changed, to be compatible with upcoming BlackBerry release of PWMinder.
• Changes to data are automatically saved to the Repository file.
• Several minor bug fixes.

Version 2.5 is currently in testing and will be released in the next few weeks.  Shortly afterwards, the BlackBerry version of PWMinder will be released, allowing password mangement to go mobile!

Note: this article first appeared in the May 2010 Issue (Volume 23, No. 5) of ASPects, The Monthly Newsletter of the Association of Software Professionals.

Introduction
Entering and tracking defects, is one of the main tasks for testers.  It is important that any time that a defect is found (whether it was actively being looked for or not), that it is logged.  Even if you are not sure if it really is a defect, or if you think it maybe isn't really that big of a deal; I believe it is better to log it, analyze and then if necessary close it, than it is to just ignore it.
So what exactly is a defect.  As a general definition, a defect is any aspect of the application that does not act or behave as designed or expected.  This can range from the obvious cases where the application crashes, data is lost, calculations give the wrong results, etc., to more subtle cases of useability and lack of features.  From a strictly test process point of view, defects are linked to the software's requirements.  A tester examines the requirements, and then writes test cases that test against these requirements.  If a test fails, or if in any other way it is found that a requirement is not met correctly, then it is a defect.  From a product, or company point of view, however, a defect is anything that a customer finds or would find as a problem whether there is a requirement or not.  This latter definition is often more important since it is most organizations' goal to satisfy their customers.

Entering a Defect
As mentioned, any time a defect or even a potential defect is found, it should be entered.  The exact format for entering a defect will vary between organizations, and some organizations may need more or less information.  Here I will provide the basic and most common information that should be included when creating a defect report.

ID: A unique identifier for the defect.  This ID is what is typically used, when different people are referring to the defect.  When using a defect tracking system, this ID is typically generated automatically.

Product: This is the product that the defect is for.  If your organization only has one product this may not be necessary.

Component: If your software application contains several components, e.g. database, web server, UI, etc. this field is used indicate which component the defect is in.  Sometime a tester may have to guess at this, if they aren't familiar with the architecture.

Summary: A short one-line description of the defect.  This is critical because the summary is often what is shown in summary reports and search results, so it must be descriptive enough so any team member (i.e. tester, developer, manager, etc.) can get a basic understanding of the problem.

Description: This is where more details of the defect is entered.  It is important to be as clear and complete as possible so that there is no ambiguity when reading.  It should also be concise, because if it is too verbose, the reader might get lost in it and miss the point.  When appropriate, include the steps needed to reproduce the problem, including expected and actual results.  This will make it easier for the developer to locate the problem, and easier later on for the tester to verify that the problem has been fixed.

Test Case Reference: If applicable reference the Test Case that was being executed when the defect was found.

Environment: This will vary greatly depending on the type of application being tested but should at least include the version of the software being tested, and may include things like Operating System, Firmware Version, Browser, Database version, etc.

Severity: As an organization, you should decide on a standard set of Severity levels.  For example: Critical, Major, Normal, Minor.  Some organizations may choose to have more or less, but 3 to 5 levels seems to work the best.  Assigning a severity is obviously a judgment call, but it helps to prioritize defect later on.

Priority: Again, an organization should also have pre-defined priority levels, e.g. P1 to P5.  Typically it should not be up to the tester to assign the priority, this should be done by the management team.  Often the question arises between the difference between Severity and Priority.  They are quite correlated, but it is possible to have a high severity defect with a low priority.  For example, if a certain actions causes the application to crash, then that is a very Severe defect, but if the conditions to cause this would almost never occur, it may be given a low priority since there may be other more easily seen defects to be fixed.  An opposite example, may be a typo on a screen.  This has low severity, but if the typo may confuse the user and/or make the application look unprofessional and, therefore, affect sales, the defect could be set to a high priority.  Typically priority is used for developers to determine which defects to address first.

Defect Life Cycle
Although a defect is typically entered by a tester, it cannot remain with the tester, but instead needs to be fixed and then verified.  This movement of a defect between different people and different states is often called the defect life cycle.  Each organization will probably have a slightly different set of rules for how they want defects handled, and will, therefore, have a slightly different life cycle.  Most however, will have a format similar to the following:
1.When a defect is first created, it will start off a New state
2.A triage meeting (see below) is held and the defect is prioritized and then set to an Open state and assigned to a developer.
3.The developer fixes the defect, and checks in the code.  They the set the defect to Fixed and assign it to a tester to verify it.  Alternatively, if the developer either doesn't think it is really a defect or can't reproduce it, they may set it to a different state (e.g. As Designed or Can't Reproduce)
4.The tester verifies that the bug is fixed by testing it in the next build.  The tester then marks it as Verified.  Alternatively if the defect isn't fixed they set the state back to Open and reassigns it back to the developer.

As mentioned this is a simplified life cycle and they can often get more complex with more states.  It is good to try to define this ahead of time to handle all situations you may encounter in your organization.  Below is an example of a defect life cycle, used in Bugzilla:

Defect Triage Meeting

A triage meeting is a meeting that brings all stakeholders (e.g. testers, manager, developers, etc) together to discuss defects.  Typically, a triage meeting will look at all new defects that have been logged since the last triage meeting, any hold over defects from a previous meeting, or any defects that have been re-opened.  The purpose of the meeting is to discuss the defects and clarify any details about them, and then decide which defects should be fixed, and to set the priority for fixing them.  Each stakeholder may have their own agenda as to which defects get fixed and the priority of them, but it is in this meeting that a consensus should be reached.  Once the defects have been analyzed and prioritized, they should be assigned to the development team to fix them.

Metrics
One reason for tracking defects, is that the data can provide metrics, that can be used when evaluating the quality of the application and in deciding when an application is ready to be released.  A common defect related metric to track, is the number of new defects logged over the past set period of time (e.g. day, week, etc.) and then track that over time.  Typically,  there will be spike in new defects every time a new build is given to test, but over time, the number of new defects found in each period, should decline.  The following bar graph is an example, showing the number of new defects that were entered on each day of the month:

Another common metric to track is the total number defects in new/open states vs those in a verified/closed state over the life of the project.  At the beginning of the project the new/open defects will be rising, but over time they should be decreasing to near zero, as shown in the example below:

Another useful metric is a snapshot chart showing the Total number of defects, grouped by the various states, at a specific point in time.  Near the beginning of a project, most defects will be New, but as the project comes to an end most should be closed.  Below is an example of the chart part way through a project:

With all these metrics you can further add information by separating the data by priority and/or severity, which can put a better perspective on things.  You can also separate data by components to help get a better idea which parts of an application are more stable than others.
Like all metrics, however, it is important not to read too much into them, and look at them more as guidelines when making quality decisions.

Defect Tracking Software
While it is possible to manage defects using a paper based system and/or to use a Word Processor and/or Spreadsheet, I would suggest using a 3rd party application, for tracking defects.  When looking for a good defect tracking system you should be looking at the following criteria:

web-based:  This isn't a must, especially for a small group, but it is definitely easier to administer the application if it is web-based.  The software only needs to be installed and configured in one location, and then all users can access it.

multi-user: This should be a given.  Even if you are a one-man show to start, it leaves room to expand.

search: This is one of the most important things to look at.  All systems should have some kind of searching, but check out how easy it is to use and how comprehensive it is.  Does it allow searching on any custom fields?  Also does the system allow for Saved Searches or Saved Filters?  This is critical, because over time, you will set up and save many often used searches.

reports: Check what kind of reports, if any come with the system.  Are the reports useful.  Also, check that you can generate custom reports that fit the needs of your organization.

history: The defect tracking system should track all changes made to a defect, including comments added, and state transitions.  This information allows the user to easily see the history of changes made to the defect.

configurable fields: Does the system support configuration or customization.  You may want to add fields that are unique to the type of project you are working on.  You would probably also want control of the values that certain fields can take.

configurable work flow: As mentioned earlier, each organization may have a slightly different work flow.  Does the Application allow you to add new states or customize which states transitions are allowed?

email notifications:  This is not a must, but it is nice to email notifications for users for when defects get assigned to them or when defects that are assigned to them get modified.

For a list of some of the more common defect tracking systems, see the references section below.

References
Clear Quest - http://www-01.ibm.com/software/awdtools/clearquest/ (Commercial)
Star Team - http://www.borland.com/us/products/starteam/index.html (Commercial)
Jira - http://www.atlassian.com/software/jira/ (Commercial):
Bugzilla – www.bugzilla.org (Open Source)
Mantis - http://www.mantisbt.org/ (Open Source)
Trac - http://trac.edgewall.org/ (Open Source)
Redmine - http://www.redmine.org/  (Open Source)
Jtrac - http://www.jtrac.info/ (Open Source)

Cheap airfare, hotels, car rentals, vacations and cruises at Expedia.com!

Small Business and Office Supplies!

My 10 year old son has been working hard on learning how to do stop motion animation (claymation), and has completed his first video.   I have embedded the video below, or you can go to youtube to view it by following the link: http://www.youtube.com/watch?v=QxjnMqcGvPA

Note: this article first appeared in the April 2010 Issue (Volume 23, No. 4) of ASPects, The Monthly Newsletter of the Association of Shareware Professionals.

So you have decided you want to implement a more formal approach to testing your software, but don't really know where to begin. In this article I will outline a basic structure of the test process, including some tips for test case writing, and suggestions for some tools to use in testing. There is no one set process that works for all types of projects or for all teams, so I'm hoping this will get you started, and you can adapt the process to fit your own needs.

For this article, I will assume  a generalized approach to software development in which firstly, requirements are gathered, followed the actual coding coding of the application.  Once the coding is more or less complete, the software goes to a testing phase where there are several iterations where the developers create a build, the testers test it and report defects.  The developers fix the defects, create a new build and so on, until the application is ready to be released.

Test Planning
Before embarking on any kind of project, it is important to plan ahead.  This is true when designing and coding your application and is equally true when testing it.  For any of you that have worked in a structured software development organization, you will have heard of the test plan.  A test plan is a document that contains all the efforts of the test planning, and then drives the testing effort.  You can find a lot of complex templates for how to write up a test plan, and can use them if you like, but here I will just cover some of the basic things that should be included in any test plan, and more importantly, the thinking that should be involved when creating it.

One of the first things to think about in your test planning, is the test environment.  This includes what operating systems, browsers, configurations, etc. you will want to test on or with.  You will also want to think about how much testing will be done on each environment; for example, will you want to execute all the tests on Windows XP, then again on Vista and again on Windows 7, or will it be enough to only cover some of the tests on Windows Vista and Windows 7?  There is no right answer to these types of questions, but it is something you need to consider and then document in your test plan.

You will also want to consider how much testing you want to do.  If this is the first release of your product, you will probably want to execute all of your tests, so you can thoroughly test all of its features.  On the other hand, if  you are doing a fairly minor release, you may not need to execute all of your previous test cases and rather only run tests on the new features.

Next, you need to think about how much testing you will do on a build by build basis.  During the testing phase, as defects get found and fixed, there are likely to be several builds of the software before the final release.  You need to plan whether you want all your planned tests to be executed with each build, or if they only need to be executed once during the entire test phase.  A common scheme is to execute each test case at least once during the testing phase, and for the final build, at least those test cases marked as regression tests, must be executed again.

Finally, you will want to think about what is your criteria for completing testing?  If you wanted, you could test a build forever; there is always one more test to try and one more defect to find.  At some point, however, you will actually want to release this new version of your application, and it is better to decide up front what determines that the software is ready.  A common criteria would be something like: All planned tests must have been executed at least once, all regression tests must have been executed on final build, all high priority test cases must have passed and there are no critical defects left to fix.  This may mean you are releasing your product with some minor defects, but at least you will be aware of them, and so when any support calls come in you shouldn't be taken by surprise.

Test Writing
The heart of all testing is the set of test cases.  A test case is a document that contains the details of how to execute a particular test, including the actions to take, the data to input and the expected results.  The input into designing test cases needs to be the set of requirements for the application.  It is the requirements that drive the test cases and ideally for every requirement there will be one or more test case(s) that test that requirement.  This mapping of requirements to test cases is often call the Requirements Traceability Matrix.  It is important that this matrix is complete so no requirements get missed.

To make managing the test cases easier, individual test cases should be grouped and organized.  There are several different ways to group test cases including grouping by feature, grouping by requirement and grouping by testing type (e.g. GUI testing vs. Security Testing, etc.).  Groups of tests cases are sometimes call test suites.  There is no best way to group them, the main thing is to find something that works for you.

So how do you go about actually writing a test case?  Again the exact format you use, is up to you, but there are several elements each test case should have:

Test Case ID: some kind of unique identifier for the Test Case
Title: a descriptive title
Environment: the test environment needed
Pre-conditions: Any setup (of data etc.) that needs to be done prior to executing the test.
Steps: The actual instructions of the test
Expected Results: expected results

In addition, things like author, version and date created/modified should also be included if you want to include traceability and version tracking.

When writing the actual steps, or instructions of the test, the question often arises as to how much detail to include.  One approach is to write highly detailed steps, that read almost like a user guide and include exactly what button to press, the text of each user prompt, exact warning messages, etc.  The advantage of this approach is that it makes the testing easy, especially for someone not familiar with the application.  The other approach is to simply provide general instructions, such as "Create a new user with no last name".  This assumes you know how to use the GUI to create a new user, and focuses only on what the test case is actually trying to test, which, in this case, is to check what happens if the last name is left blank.  The advantage of this approach is that test cases can be written more quickly and there is less maintenance, since when the GUI changes slightly, the test probably could stay the same, whereas in the detailed approach the test steps most like would need to be changed.

As for the tests themselves, the details will obviously be different for different applications, and should be based on the software requirements.  There are, however, several general things to test for in nearly all applications:

Boundary Conditions: Most applications will have some data entry fields and will store data in some way.  Most fields have a valid range of expected values or expect values of a certain type.  When testing these fields, make sure to test valid values, invalid values, including values on either side of the the boundary.  For example, if a field is meant to be used to input the minute part of a time field, then the valid range would be 0 to 59.  Some good values to try would include:

Maximum Characters: Most data fields have some kind of maximum length allowed.  For example, in a database application, the database implementation may have a name field may be set to have a maximum of 25 characters.  The User Interface should not allow the user to enter more than 25 characters, so when testing, it would be a good to try to enter more than 25 characters into the text field.   It is always a good idea to enter the maximum allowable characters into to each data input field and see what happens.

Special Characters: Another good test is to add special characters to each field and make sure they are handled properly.  Many applications that use other technologies such as SQL or XML and may have problems with certain characters.  The set of characters I typically use is: =-+_)(*&^%$#@!~`|}{\][":';?></.,

Test Execution
Now that you have written a complete set of test cases that cover all of the requirements, and have documented your plan on how you want to test, you are ready to execute the tests.  Test execution is basically executing the test cases according to your test plan.  For each test case you execute, you follow the test steps described in the test case and check the expected results.  When you have finished all of the steps, if the expected results were achieved the test passes, otherwise the test case fails.  Any failures should be reported as a defect; include any screen shots or other attachments that help explain the defect, if applicable.  Sometimes while executing a test, you may find a defect that is not directly related to the test you are running.  It is important to still enter it as a defect.  It is also important then, to either update that test case you were executing, update a different test case or to create a new test case, that will test for the defect you found.  Often finding a defect will trigger you to think of other related tests to try.  These new test should also be added to you set of test cases.  It is important to note that the set of test cases is never really static but is always being enhanced.

While working through the tests, the results need to be kept track of.  This can be done with a simple spreadsheet as follows:

Testing Complete?
So when is testing complete?  Without a test process, this question is difficult to answer, and quite possibly the answer would be never.  If, however, you have created a test plan and followed it carefully, then testing is done when all the criteria set, has been met.  At this point you can feel confident in knowing what has been tested, what areas may be at risk, what defects may have been left unfixed, and be ready to release the software with less chances of any major surprises coming your way.  Software development is all about balance and managing risks, and while a properly implemented test process won't eliminate risk (or defects for that matter), it should minimize risk, provide you the information you need to make wise decisions about your software application.

Tools of the Trade
To be as effective as possible when testing, it is key to have a good set of tools.  Firstly, it is important to have a computer specifically dedicated to testing.  This means the tester should have their own personal desktop computer for activities such as test planning, defect tracking, report writing, etc. as well as have access to a separate computer to test on.  This test computer (or computers) should be as clean as possible, to minimize the side affects of other software, and should be able to be base lined, so various configurations can be set up and then the system can be returned to a baseline.  For this task I recommend using virtual machines such as VMWare or Sun VirtualBox.  With these you can create different guest operating system, configurations, etc. storing each as its own virtual hard drive on a single host computer.

Test planning, test case writing and test execution tracking can all be done using standard office productivity software such as Microsoft Office or OpenOffice.  Test Plans are typically written using a word processor, while for test cases, some people prefer to using word processors, others prefer spreadsheets.  Tracking test execution and results would definitely go into a spreadsheet.  Using word processors and spreadsheets works, but they are not ideal, especially when you have more than one person on your team.  Sharing of test cases and test results is not that easy and there is also the concern of where to store all the files, how to version them and back them up.  Instead, I would highly recommend using a dedicated test management software.  Most of these programs provide the ability to write test plans and test cases, and track execution results.  Some also provide linking between requirements and test case and may also provide pretty graphs and charts that manager like to look at.  There are several commercial products available by the big names in testing such as IBM Rational, HP Mercury, Borland Sile, etc., but there are free alternatives as well.  A product I have been using is TestLink, which is an open source, web based test manager with a MySQL backend.  It is a fairly mature product that is actively being developed and enhanced.

A basic tool for testers is to have a good text editor.  This is important for inspecting any data files, creating test data, etc.  There are many text editors out there both free and commercial and I must admit, I am a bit of text editor junky, constantly trying out different editors.  Over my career I have used TextPad, JEdit, Crimson Editor, and most recently have seemed to settled on Notepad++.

Often when writing up a defect, you want to include a screen shot, that shows the error.  While using print-screen with Paint, works, I find using a dedicated screen capture product makes me much more efficient, allowing me to easily capture entire screens or only portions of a screen and then annotate and highlight different parts, making the screen capture much more effective.  A couple of commerical screen capture products include HyperSnap and SnagIt.

A final tool I'll mention is a stopwatch.  This can be either a separate stopwatch, or stopwatch software.  This is particularly useful during performance testing, when seeing how long it takes for pages to load, processing to complete, calculations to occur, etc.

References
VMWare - http://www.vmware.com/
Sun VirtualBox - http://www.virtualbox.org/
OpenOffice - http://www.openoffice.org/
IBM Rational – http://www.ibm.com/software/rational/
Mercury Interactive - https://h10078.www1.hp.com/cda/hpms/display/main/hpms_content.jsp?zn=bto&cp=1-11-127_4000_100__
Borland Silk - http://www.borland.com/us/products/silk/index.html
TestLink - http://www.teamst.org/
TextPad - http://www.textpad.com/
JEdit - http://www.jedit.org/
Crimson Editor - http://www.crimsoneditor.com/
Notepad++ - http://notepad-plus.sourceforge.net/uk/site.htm
Hypersnap -http://www.hyperionics.com/
Snagit -http://www.techsmith.com/

Cheap airfare, hotels, car rentals, vacations and cruises at Expedia.com!

Small Business and Office Supplies!

I have recently been working on a BlackBerry application, and found that I often got lost in the maze of method calls that happen was a close command is initiated for a screen.  To help me keep track of it, I have created a basic flowchart which helps me follow the bouncing ball:

1. The flow starts when the User or application initiates a close screen, i.e. the user clicks on Close.
2. This triggers the onClose() method to be called.
3. The onClose() method calls isDirty() to see if there have been any changes on the screen.  If no changes have been made, the close() method is called and the screen is closed.  If changes have been made then the onSavePrompt() method is called.
4. The onSave() method brings up a Dialog with the following options: "Save", "Discard" and "Cancel".  If the user selectes "Cancel" the Dialog is simple closed and the screen remains open.  If the user selects "Discard", then the close() method is called and the screen is closed.  If the user selects "Save", then the onSave() method is called.
5. The onSave() method calls the isDataValid() method.  If isDataValid() returns false, then nothing happens and the original screen is still displayed.  If isDataValid() returns true, the the save() method is called.
6. After the save() method has completed successfull then the close() method is called and the screen is closed.

Typically you will want to override the following:

isDataValid(): You should add you own logic for whether the data is valid or not.  Here you can check that the user entered all the necessary data and data is valid.

save(): You should implement how you want to save the data.

You may also want to override the following:

close(): if you have any special actions you want to happen when the screen closes

onSavePrompt(): if you don't want the prompt just override the method and return true.  You can also override it, if you want to implement your own dialog with different options, e.g. using Dialog.ask().  When implementing your own dialog make sure you get the return type correct, the return type boolean is for whether or not you want to close the original screen.

These methods are all part of net.rim.device.api.ui.contain.MainScreen class.

Hopefully this will be helpful to some of you.

The above Flowchart can be downloaded as a pdf: BlackBerry Save Model