OpenCAP

I’ve just come back from presenting on the status of Open CAP at ACTFL. Since active development has all but stopped due to funding cuts, the presentation focused primarily on the intersection of test development, curriculum, and teaching. Towards the end of the session, attendees were asked to express any issues, concerns, or feature requests that they would like to see in a tool like Open CAP. Here are some of the comments:

• Quality — Several people mentioned the need to separate the wheat from the chaff in terms of item quality. The ultimate usefulness of the tool depends on the quality of the content in addition to the quantity.
• Attribution — An interesting comment was made that many online collaborations leave little trace of who made the original contribution. This makes it hard for faculty to quantify the contributions that they have made fro things such as tenure applications. To the extent that the application can track individual contributions, this problem will be minimized.
• Reputation — Several attendees expressed a desire to have some kind of reputation system within the system in order to make it easier to determine which items has a better pedigree.
• Accessibility — Given the potential size of the item bank, the need for an extensive tagging system was voiced.
• Comparability — One attendee mentioned that it would be useful to see what types of items and tests were being used in peer institutions. This would help reduce search overload initially, as folks could start with things that were being used at similar institutions and adapt them from there.
• Security — The issue of item exposure was mentioned, as well as the potential for student cheating if the system is completely open to any user.

Overall, the session attendees were very supportive of the idea of an online collaborative tool such as Open CAP. In the run-up to ACTFL, I also became familiar with a similar tool developed for university placement, called SLUPE.

In his book, Managing the Design Factory, Donald G. Reinertsen talks about the importance of generating high quality information early and often in any process. Although he is talking about manufacturing, these ideas apply to teaching and learning as well. Consider the following quote:

The key implication for management is that we must be concerned with both the efficiency with which we generate information and the timing of that information…The pressure to control testing costs often encourages us to use a few big “killer” tests, but such tests will delay the arrival of information. (Reinertsen, 2007, p. 73)

If you wait until the end of the term or the chapter to determine whether or not the students are understanding the material, there is little chance that those results will be used to actively improve the situation. It is better to gather information throughout the term. This is often called formative assessment, since feedback from the assessment can directly influence what happens next in the teaching/learning process.

Information collection can also be made more efficient by considering carefully what type of information to target. Rather than just giving “a reading test” or “a grammar quiz” based loosely on the general theme of the class, try to relate assessments to the particular goals and objectives of that lesson. For long term objectives such as improving speaking proficiency, break things down into smaller chunks.

We have been getting a lot of “bot”-generated comments recently, so please be patient if you submitted a comment and did not see it posted right away.

When designing a new product or educational tool, it is enjoyable to think about all of the possibilities, no matter how fanciful (“Gee, wouldn’t it be cool if we could automatically generate a daily report for each study abroad student using automatic speech recognition and GPS technology to track their proficiency gains during their trek across China?”).  At some point, however, those great plans need to be tempered by the realities of available resources. It would certainly be possible to try to develop a product by working down the list of great ideas one at a time until the money runs out, but that is probably not very efficient and there is no guarantee that an amalgamation of cool features will lead to a coherent product.

One tool to help prioritize work when resources are limited is the idea of core functionality. These are the absolutely essential criteria without which you don’t really have a product. For example, there are three things that a plane absolutely must be able to do:

1. Take off
2. Fly
3. Land

No matter what else you do, if your development timeline doesn’t include the tasks necessary to make those three things a reality, you will not have a successful product at the end.

For Open CAP, the core functionality might be described as:

1. make tests
2. give tests
3. see test results

Without the ability to perform those tasks, Open CAP has no chance of being useful for assessment. Of course, there mere presence of this core functionality doesn’t mean that Open CAP will be a great tool, but at least it will be complete at a core level.