Hi all,
I really look forward to R2oSE 2010.
As an online warm-up, I would like to get some community input for the following issue: during the yearly transformation tool contest [report], we typically have an interactive session to derive a feature matrix for comparing the solutions that were submitted to our case studies. We start with some brainstorming, then cluster features on a blackboard, and finally make three feature matrices that we print on paper and hand out to all workshop participants. This enables a live peer review: solution submitters get about 15 minutes to demonstrate their transformation program and the audience fills in one matrix column in the meanwhile.
By assigning weights to the different features, we then analyze which solution satisfies the criteria that were proposed by the audience the best. The following table aggregates the data from 9 evaluations of 10 contributions: BPMN2BPEL solution evaluation results.
The advantage of this interactive approach is that the winner of our contest is usually supported by most of our workshop attendees. Currently though the manual process of having the brainstorm session, clustering on the blackboard and then creating an spreadsheet is very time-consuming and the final step (creating the actual spreadsheet, documenting the meaning of a criterium, ...) needs to be done by one of the organizers (or we simply run out of time for considering the actual solutions...) This is undesirable, since it allows (probably accidental) bias and it is too error-prone.
Therefore, I have been thinking of web 2.0 support for streamlining this process. The idea is to involve workshop participants as soon as possible in the creation of the digital evaluation form (or underlying feature model) and to enable some online conflict resolution/integration. I am currently looking for a website that provides:
- an account system (since we want to avoid anonymous rant) [example: planet-research20.org, but several alternatives are out there],
- a (semi-)structured editor for feature matrices [could be google spreadsheets, a feature diagram editor, ...]
- (optional) a mechanism for merging contributed feature matrices (reconciling different documents/models created using the editor from (2),
- a mechanism to assign weights to features (to enable the automatic computation of the "best" solution,
- (optional) a mechanism to override the weights (on a user basis) after the workshop. That would enable end-users to assign a winner according to their own preferences. Usually, they would do that for finding which solution they could use out of the box or which solution they would like to extend, or ... (whatever we currently cannot imagine yet)
I can imagine this might at first sound rather specific to our transformation tool contest (TTC workshop) but I think that such a platform would be very useful for various other research purposes (researchers often want to structure a discussion about which algorithm/tool/... performs "best" according to too implicit criteria!) Once we better understand how to use (& improve) such a platform, we have a new means to perform survey research: instead of doing a large literature review, installing some tools, creating a feature model and assessing the result based on our personal understanding, we have a more efficient (and probably more effective) community based approach.
Therefore, I would like to ask the Research 2.0 community to suggest some candidate platforms that either already support some of the above functionality or that could be used to build that functionality. Notice the popularity of light-weight platforms such as doodle (when it comes to meeting planning) so perhaps we rather need a clever integration of light-weight web applications here as well (google docs + ???) rather than an inflexible "dedicated" platform?
I welcome all your input (comments, suggestions, questions, ...), please do not hesitate to contact me.
Sincerely,
Pieter Van Gorp
r2ose2010 co-organizer
Eindhoven University of Technology
