Proposal Effort Evaluation

As part of the Profile Proposal Process, the Technical Committee estimates the effort required (in terms of TC Bandwidth) to develop each of the proposals. The effort estimate is considered by the Planning Committee when they prioritize and select which Profiles will be developed this cycle.

The TC in each domain chooses how it estimates. This page reflects some ideas being tried out in the IHE Radiology domain. The idea is to expand on the Breakdown of Tasks section in the Brief Proposal Template by estimating "effort points" for each task. It turns out Agile has already thought of everything cool :-), so the material below borrows from Story Points.

This is a pilot so all details are open to discussion/suggestions. The retrospective indicated we need to do a better job of flushing out issues early in the process and our estimates should better scope the work, but we don't want to overburden. So we'll try this but try to keep it brisk. The evaluation tcon should note Debates, not presume to resolve them. The point counts are suggestions, we'll see how it goes.

Proposer Homework

After the Brief proposal has passed the Short List vote, the proposal editors create a Detailed Proposal by copying the Delta Proposal Template into the bottom of the proposal page and editing appropriately. This includes adding a Breakdown of Tasks section.

There should be a bullet for each chunk of work for the committee.

Transactions

• Bullet each new/modified transaction

• Indicate if it will clone an existing transaction, be similar, or be new
• Identify its function and protocol/mechanism (if new or similar), and what will be changed/tweaked (if clone)
• Don't list existing transactions you plan to use unchanged (they're noted in Technical Approach)
• Transaction-specific uncertainties can go here as sub-bullets

Profile

• Bullet whether profile body will clone an existing profile, be similar, or be new
• Bullet primary use case(s)

• unless variants raise a significant issue/consideration, list them under a primary use case
• indicate whether re-used transactions will need to be reviewed in the context of the new use case(s)

• Bullet new mapping tables
• Bullet major options/functions
• Profile uncertainties mostly go in the next section

Decisions/Topics/Uncertainties

• Bullet topics that will need to be discussed/decided by the TC

• note risks, uncertainties or points that are likely to have significant debate

Indicate which bullets are not part of the MUE (using italics? "(Not MUE?)").

Maybe have an "Out of Scope" section and proactively move certain things there to limit effort?

TC Review & Effort Estimation

During the Effort Estimation meetings, the TC will review the proposal and technical approach, then review/modify the Breakdown of Tasks and add point estimates to each bullet.

Points are "awarded" for effort (basic review/revision of blocks of text), complexity (new protocol/domain learning curve, getting mapping tables right, working through large sequence diagrams), and risk/uncertainty (debating/choosing between competing protocols, alternative architectures, etc.).

Transactions

• Review the description, question assumptions/details, add uncertainty sub-bullets as needed.
• Assign points

• Effort points: 1 - "normal" size transaction; 2 - "large" transaction
• Complexity points: 0 - if cloning; 1 - if new/similar; +1 - if new API/mechanism; +1 - if new context or domain
• Uncertainty points: as appropriate for listed uncertainty topics.

• Consider if any other new transactions should be added
• Consider if any existing transactions will need cloning/modification

Profile

• Review/refine/add/regroup tasks
• Assign points

• Effort points: 2 - "normal" profile; 4 - "large" profile; +0 - if cloning; +1 - if new/similar
• Effort points: 1 - each other task bullet; can pair up if small
• Complexity points: as appropriate

Decisions/Topics

• Review/resolve/refine/regroup issues
• Assign points

• Uncertainty points: as appropriate

Result

Tally up the points. Do our usual "planning poker" to get a % estimate, but with the tally and above discussions in mind.

Next year we might have a "conversion rate" from points to %. At least, there should be a pseudo-linear relationship between the proposals. One with a significantly higher tally should have a significantly higher estimate.

Profile Draft

Migrate the revised Breakdown of Tasks with point estimates into the Profile Draft, likely in the first Open Issue row or thereabouts so we can track it and see how this works out.

Brainstorming Meeting

A key goal of the Brainstorming Meeting is to resolve the Decisions/Topics items and other uncertainty points.

The Profile Editors should work to prepare clear write ups of each of the decisions/topics (and perhaps a proposed resolution) before the meeting. Hold t-cons if you think it's warranted.

The last 15-30 minutes of time allocated to each profile at the Brainstorming meeting will be spent confirming/recording the resolutions for each. Any still unresolved are flagged as critical items to address by/during Public Comment. A couple modest unresolved items is fine. More risks failure of the Profile.

Retrospective

When we do the end-of-cycle retrospective it might be interesting for each profile to think about Open Issues that came up and topics that used a lot of time and see if we might have anticipated those during evaluation.