Product Evaluation (Continued)
Why should ProtoGenie support product testing & evaluation? Customers should know whether a product, technology, process, or program works. Developers should know the feasibility and potential of a product, technology, process, or program before going to market. The general public should know about defects in product design and manufacture and the causes. Consumers need to know whether products comply with regulations and standards and developers and funding agencies need to know whether products meet directives, mission guides, and program objectives. Distributors need to know whether the product or brand of product is superior to others on the market. In certain market areas, it is critical that products perform as advertised. This is especially true in the growing area of Assistive Technology for people with disabilities where products can mean the difference between full or marginal participation in society and where pocket books are inherently stretched to purchase them.

There are two basic types of product evaluation: performance testing and technology evaluation.

Performance testing involves the external attributes or properties of a product. Criteria generally include adequacy, flexibility, “feel,” usability, cost, effectiveness, speed, responsiveness, reliability, support, documentation, compatibility, and other. In some cases, criteria might be subjective with qualitative measurement. In other cases, criteria might be objective with quantitative (metrics) measurement. Testing may involve one product or the comparison of two or more. Performance testing can be done by one evaluator (expert) or by a panel of experts (judges). Performance testing can be done as part of pre-release, beta, and market preparation processes or anytime in the lifetime of the product. The fundamental difference between technology evaluation and performance testing is that the focus of evaluation is on the “change” technologies behind products rather than external attributes of the products. The basic question is does this intervention work?

There are two basic types of technology evaluations, one is invasive, and the other non-invasive. Invasive means getting inside the product to establish communication with ProtoGenie. Generally speaking, getting inside hardware devices and source code requires the permission, cooperation, and assistance of the product maker - which is often difficult and costly to do. Consequently, there are three workarounds not requiring access to the product’s interior. These are non-invasive and have no communication between the product and ProtoGenie. There are three non-invasive methods for evaluating product core technologies, Inferred effects, sequential, and synchronized.

The Inferred Effects Method - Before/After Test design calls for measurement of some evaluation criterion, such as reading speed, before a product (like speed reading exercises) is used with another measurement taken after the product is used. Observed changes are attributed to underlying technologies (interventions).

In the Sequential Approach design, data are collected from a run or series of runs of the target product. Measurement is independent of ProtoGenie. Measures of performance such as words per minute are made manually. Results are punched into ProtoGenie for analysis. The only coordination between the programs is in the specification of what is going to be measured. For example, standardized reading passages could be punched into a reading assistance program and ProtoGenie could be used to measure elapsed reading times with and without the special reading technology turned on.

The Synchronized Approach is a variation of the sequential model except that ProtoGenie and the target program are synchronized and run together. Two computers are used side-by-side, one with ProtoGenie set up for the run and one with the target application set up to run. When the trial begins, the subject answers the symptom questions on the ProtoGenie machine. Then, ProtoGenie gives the instruction to start when ready, the tester clicks the start button when the subject begins the trial and clicks the finished button when the subject has completed the trial. Then the subject looks at the testing computer with ProtoGenie and answers the comprehension questions for the passage read and any follow up symptoms or preference questions.

Invasive types of technology evaluation require penetration of the hardware device or software program to install some means of establishing communication between the product and ProtoGenie. That is, once two-way communication is established, then ProtoGenie can tell the subject what and when to do it and the subject can input responses to the program. Also, the sequencing of events can be controlled from ProtoGenie. There are basically three ways that this kind of communication can be established. They are custom hooks, OS method, and non-OS method.

Embedding custom hooks in a piece of software requires the cooperation of the product maker who agrees to insert them for you. Usually there must be some form of quid pro quo or compensation for product makers to do this for you. Anticipation of a friendly and positive “Scientifically Tested” testimony for promotion might be sufficient. The OS Hooks Method sets up communication between ProtoGenie and the product through COM ActiveX features in the Windows operating system. The Non-OS Method establishes communication between ProtoGenie and a product using hacking technologies yet to be defined for this purpose. Early utilities, such as screen readers for speech access, were based on “interception” technologies that could monitor the communications between the OS and applications.

Diagnosis (Continued)
There are two basic types of diagnostic procedures: expert and discretionary. In one the author is considered to be the expert, so he/she determines for all future applications which answers take which paths. In other words, this is a basic “expert information system.” In the second major type of diagnostic procedure, it is the clinician who administers the process (not the author) who is considered to be the expert. In these discretionary diagnostic systems, a large variety of materials and lines of questioning are under control of the clinician. The difference from the pure expert system is that the clinician can stop the process at any point in the diagnosis and resume it using different materials and/or a different line of questioning.