Proposal Effort Evaluation
As part of the Profile Proposal Process, the Technical Committee estimates the effort required (in terms of TC Bandwidth) to develop each of the proposals. The effort estimate is considered by the Planning Committee when they prioritize and select which Profiles will be developed this cycle.
The TC in each domain chooses how it estimates. This page reflects some ideas being tried out in the IHE Radiology domain. The idea is to expand on the Breakdown of Tasks section in the Brief Proposal Template by estimating "effort points" for each task. Turns out Agile has already thought of everything cool :-), so the material below borrows from Story Points.
This is a pilot so all details are open to discussion/suggestions. The retrospective indicated we need to do a better job of flushing out issues early in the process and our estimates should better scope the work, but we don't want to overburden. So we'll try this but try to keep it brisk. The evaluation call should note Debates, not presume to resolve them.
After the Brief proposal has passed the Short List vote, the proposal editors should create a Detailed Proposal by copying the Delta Proposal Template into the bottom of your proposal page and editing appropriately. This includes adding a Breakdown of Tasks section.
There should be a bullet for each chunk of work. The focus is work the committee will do.
- Bullet each new/modified transaction
- Indicate if it will clone an existing transaction, be similar, or be new
- Note the function and protocol/mechanism (if new or similar), and what will be changed/tweaked (if clone)
- Don't list existing transactions you plan to use unchanged (as noted in Technical Approach)
- Transaction-specific uncertainties can go here as sub-bullets
- Indicate if you will clone an existing profile, be similar, or be new
- Bullet main use cases
- Bullet new mapping tables
- Bullet topics that will need to be discussed/decided by the TC
- note risks, uncertainties or points that are likely to have significant debate
Indicate which bullets are not part of the MUE (using italics? "(Not MUE?)").
Maybe have an "Out of Scope" section and proactively move certain things there to limit effort?
TC Review & Effort Estimation
During the Effort Estimation meetings, the TC will review the proposal and technical approach, then review/modify the Breakdown of Tasks and add point estimates to each bullet.
Points are "awarded" for effort (basic review/revision of blocks of text), complexity (learning new protocols/domains, getting mapping tables right, working through large sequence diagrams), and risk/uncertainty (debating/choosing between competing protocols, alternative architectures, etc.).
- Review the description, question assumptions/details, add uncertainty sub-bullet as needed.
- Assign points
- Effort points: 1 - "normal" size transaction; 2 - "large" transaction
- Complexity points: 0 - if cloning; 1 - if new/similar; +1 - if new API/mechanism; +1 - if new context or domain
- Uncertainty points: as needed for uncertainty topics listed here.
- Consider if any other new transactions should be added
- Consider if any existing transactions will need cloning/modification
- Assign points
- Effort points: 1 - "normal" ; 2 - "large"
- Complexity points:
- Assign points
- Uncertainty points:
Tally up the points. Do our usual "planning poker" to get a % estimate, but with the tally and above discussions in mind.
Next year we might have a "conversion rate" from points to %. At the very least, there should be a pseudo-linear relationship between the proposals. One with a significantly higher tally should have a significantly higher estimate.
We're currently recommending that the revised Breakdown of Tasks with point estimates be migrated into the Profile Draft, likely in the first Open Issue row or thereabouts so we can track it and see how this works out.
A key goal of the Brainstorming Meeting is to resolve the Decisions/Topics items and other uncertainty points.
The Profile Editors should work to prepare clear write ups of each of the decisions/topics (and perhaps a proposed resolution) before the meeting. Hold t-cons if you think it's warranted.
The last 15-30 minutes of time allocated to each profile at the Brainstorming meeting will be spent confirming/recording the resolutions for each. Any still unresolved are flagged as critical items to address by/during Public Comment. A couple modest unresolved items is fine. More risks failure of the Profile.
When we do the end-of-cycle retrospective it might be interesting for each profile to think about Open Issues that came up and topics that used a lot of time and see if we might have anticipated those during evaluation.