ScreenDance - the Dynamic Graphical Display Generator

Clinical Expert & Diagnostic Systems

Virtual Focus Groups and Deliberative Polls

Why should ProtoGenie support product testing & evaluation?
Customers should know whether a product, technology, process, or program works. Developers should know the feasibility and potential of a product, technology, process, or program before going to market. The general public should know about defects in product design and manufacture and the causes. Consumers need to know whether products comply with regulations and standards and developers and funding agencies need to know whether products meet directives, mission guides, and program objectives. Distributors need to know whether the product or brand of product is superior to others on the market. In certain market areas, it is critical that products perform as advertised. This is especially true in the growing area of Assistive Technology for people with disabilities where products can mean the difference between full or marginal participation in society and where pocket books are inherently stretched to purchase them.

Types of Product Evaluation
There are two basic types of product evaluation. These are performance testing and technology evaluation.

Product Performance Testing
Performance testing involves the external attributes or properties of a product. Criteria generally include adequacy, flexibility, “feel,” usability, cost, effectiveness, speed, responsiveness, reliability, support, documentation, compatibility, and other. In some cases, criteria might be subjective with qualitative measurement. In other cases, criteria might be objective with quantitative (metrics) measurement. Testing may involve one product or the comparison of two or more. Performance testing can be done by one evaluator (expert) or by a panel of experts (judges). Performance testing can be done as part of pre-release, beta, and market preparation processes or anytime in the lifetime of the product.

Product Technology Evaluation
The fundamental difference between technology evaluation and performance testing is that the focus of evaluation is on the “change” technologies behind products rather than external attributes of the products. The basic question is does this intervention work?

Invasive Versus Non-invasive Technology Evaluation
There are two basic types of technology evaluations, one is invasive, and the other non-invasive. Invasive means getting inside the product to establish communication with ProtoGenie. Generally speaking, getting inside hardware devices and source code requires the permission, cooperation, and assistance of the product maker - which is often difficult and costly to do. Consequently, there are three workarounds not requiring access to the product’s interior. These are non-invasive and have no communication between the product and ProtoGenie.

Non-Invasive Methods
There are three non-invasive methods for evaluating product core technologies, Inferred effects, sequential, and synchronized.

Inferred Effects Method - Before/After Test
This design calls for measurement of some evaluation criterion, such as reading speed, before a product (like speed reading exercises) is used with another measurement taken after the product is used. Observed changes are attributed to underlying technologies (interventions).

Sequential Approach
In this approach, data are collected from a run or series of runs of the target product. Measurement is independent of ProtoGenie. Measures of performance such as words per minute are made manually. Results are punched into ProtoGenie for analysis. The only coordination between the programs is in the specification of what is going to be measured. For example, standardized reading passages could be punched into a reading assistance program and ProtoGenie could be used to measure elapsed reading times with and without the special reading technology turned on.

Synchronized Approach
This is a variation of the sequential model except that ProtoGenie and the target program are synchronized and run together. Two computers are used side-by-side, one with ProtoGenie set up for the run and one with the target application set up to run. When the trial begins, the subject answers the symptom questions on the ProtoGenie machine. Then, ProtoGenie gives the instruction to start when ready, the tester clicks the start button when the subject begins the trial and clicks the finished button when the subject has completed the trial. Then the subject looks at the testing computer with ProtoGenie and answers the comprehension questions for the passage read and any follow up symptoms or preference questions.

Invasive Models of Technology Evaluation
Invasive types of technology evaluation require penetration of the hardware device or software program to install some means of establishing communication between the product and ProtoGenie. That is, once two-way communication is established, then ProtoGenie can tell the subject what and when to do it and the subject can input responses to the program. Also, the sequencing of events can be controlled from ProtoGenie. There are basically three ways that this kind of communication can be established. They are custom hooks, OS method, and non-OS method.

Custom Hooks
Planting hooks in a piece of software requires the cooperation of the product maker who agrees to insert them for you. Usually there must be some form of quid pro quo or compensation for product makers to do this for you. Anticipation of a friendly and positive “Scientifically Tested” testimony for promotion might be sufficient.

OS Hooks Method
This method sets up communication between ProtoGenie and the product through COM ActiveX features in the Windows operating system.

Non-OS Method
This method establishes communication between ProtoGenie and a product using hacking technologies yet to be defined for this purpose. Early utilities, such as screen readers for speech access, were based on “interception” technologies that could monitor the communications between the OS and applications.

Field and Departmental Projects to Compile Libraries of Screen Stimuli
Student projects having the aim of creating one or more stimulus components for ProtoGenie users could nicely serve the broader aims of the students to pursue their research interests while serving the needs of others in their areas of research. Students engaging in such projects would learn from each other and components and protocols coming out of the process could be shared. Research could be replicated with strategic changes and alternative stimuli and measurements could be explored. The School of Optometry, UCB will soon have a substantial library of ProtoGenie protocols and stimulus components for on-going and future research. In time, similar libraries will be created for use in cognitive research and related studies, such as human performance research.

Development of an Authoring Tool to Create Complex Screen stimuli and Measurements
In keeping with ProtoGenie’s commitment to provide tools that enable non-programmers to create their own software including very complex and technical research protocols, a new feature of ProtoGenie is on the planning board. It is a test object generator called 𠇍ynamic Graphical Display Generator (DGDG) or simply “ScreenDance.” ScreenDance is a ProtoGenie Composer option that enables users to generate test objects on computer screens for vision and psycho-physiological and related kinds of research. This option will bring up a workspace not unlike paint and design programs and a palette of tools with which are used to create display objects, such as geometric shapes or pictures, define their static properties and their movements, including Rapid Serial Visual Presentation (RSVP), rotation, morphing, vibrating, oscillating, and pulsating. Trajectories will be set by coordinates and patterns such as elliptical, circular, sinusoidal, spiral, zigzag, and random. Controls will include control over rate, periods, increments, and intervals. Framing and background options will include dimensionality, layering, lines, horizons, grids, gratings, arrows, and control over sequencing, as in the presentation of a grating immediately before or after a visual stimulus (masking effects). Display properties will include color, contrast, and luminance.

ScreenDance will also enable users to create sets of “equivalent materials” containing display objects made by the test object generator to support repeated trial experiments typical of display-centered vision and cognitive research. Screen dance may be written in Flash MX to take advantage of the advantages of Flash in the creation of moving objects and object animation, but it can be written as well in any standard programming language.

Two types of diagnostic systems
There are two basic types of diagnostic procedures: expert and discretionary. In one the author is considered to be the expert, so he/she determines for all future applications which answers take which paths. In other words, this is a basic “expert information system.”

In the second major type of diagnostic procedure, it is the clinician who administers the process (not the author) who is considered to be the expert. In these discretionary diagnostic systems, a large variety of materials and lines of questioning are under control of the clinician. The difference from the pure expert system is that the clinician can stop the process at any point in the diagnosis and resume it using different materials and/or a different line of questioning.

ProtoGenie support is not far away
Both of these types of computer supported diagnosis can be served by ProtoGenie with modest additional features, such as the ability to pause a session and restart it in a new direction. This would make an excellent project for students and faculty in schools of medicine, optometry, psychology, and many others.

See Programmatic and Vertical Applications

Since the process or evolution of opinions themselves may be of interest, ProtoGenie observational methods can be used. If special materials are introduced or other interventions are tested for their effects on attitudes, then the design is experimental and ProtoGenie tools for experimental design can be used. Online focus groups require the integration of networking technologies with ProtoGenie technologies. Samples of respondents are recruited and coached. Sessions may or may not be led by a discussion leader. Research may be focused on consensus reaching processes - not unlike the deliberations of jurors to arrive at a verdict. In most cases, the focus is on the effects of deliberation on individual opinions. If an additional group of subjects is selected from the same sample and those subjects do not deliberate, then we have a typical ProtoGenie control group experimental design in which the effects of deliberation can be measured.

Online sessions may range from minutes in real time to weeks of intermittent communications and the same subjects may participate in subsequent sessions for follow up purposes or for studies of different issues. In political research, this device has great potential to help remedy the problems of political institutions like the primary system, which has virtually eliminated deliberation from the process of nominating candidates for office. It may also become a major tool in online public interest advocacy groups, where the deliberation networks are already in place and where objectives are to extend democratic participation.