Stage 2: Project Attributes Sizing


IntroductionMethodology OverviewBest Practices Project AttributesMarket CapabilitiesMarket Readiness 
       

Objective

Where Stage 1 focus is on non measurable information such as the project organisation, and process, the objective of stage two is to focus on quantitative dimensions of the project. We use quantitative data in combination with best practice results from Stage 1 to compute what we call Project Attributes.

Scope

There can be hundreds of quantitative data available for a software project. However in the OW2 MRL methodology information specifically collected in stage two is limited to data common to all projects under review. This results in 13 control points as providing information of users, bugs, licenses and general software quality.

Data Collection

Stage 2 data is automatically collected from software engineering tools such as the project's issue tracker and source code manager and the application of analysis tools such as SonarQube and ScanCode. The table below provides the details of the variables that are collected and their sources.

For bug tracker stats, a maximum of 200 bugs are analyzed (100 most recent open + 100 most recent closed). If less bugs are available, all are taken into account.

    
SourceVariablesDefinitions
Bug trackerActiveUsersNumber of users with bug(s) assigned
Bug trackerAverageResponseTimeAverage number of minutes between bug creation and update
Bug trackerBlockerIssuesSonarQube "blocker_violations" metric
Bug trackerCriticalIssuesSonarQube "critical_violations" metric
Bug trackerBugOpentimeAverage number of days between bug creation and resolution
Bug trackerUnansweredBugsUnanswered bugs are open bugs with no comment
ScanCodeScancodeRatioNoLicTotal unlicensed files / Total files count
ScanCodeScancodeTotalFilesTotal files count
ScanCodeScancodeTotalLicensedTotal files with license count
ScanCodeScancodeUniqueLicCount of all distinct licenses found (same license with N versions counts for N, and 1 license max. is counted per file)
SonarQubeQualitySonarQTestCoverageSonarQube "coverage" metric
SonarQubeQualitySonarQTestSuccessSonarQube "test_success_density" metric

Computation

First of all, all quantitative data is normalized so as to help compare projects and compute averages. The distribution of values along the standard OW2 MRL normalization for each variable is determined by the amplitude of the values as measured across all OW2 projects. In this sense, data normalisation is specific to the OW2 code base (i.e. it might not be relevant for the apache Foundation or the Eclipse Foundation code bases). Outliers, if any, are handled by hand on a case by case basis after consultation with project leaders.

Project attributes are complex characteristics resulting from the combination of results from the best practices verification (stage one data collection) and data automatically collected from the project's development environment (stage two data collection).

To compute attributes, we use an intermediary step we call aggregates. Aggregates are themselves the result of raws values. Aggregates help add meaning to raw values. Some raw values may contribute to more than one aggregate or more than one attribute. Some aggregates may be comprised on only one raw value when such raw value is sufficiently explicit. Attributes are defined by the data from which they result. Raw values are all normalised along the same value scale so as to enable their combinations.

The first table below brings out the structure of the aggregates and the second one shows how the attributes are computed.

   
  The Aggregates below are computed...  ...from the average of the following Best Practices...  ...and the following Project Values 
 Active Users (cty_activeusers)   ActiveUsers  
 Bug Handling (act_bugs)  Bug Open Time (days)
 Unanswered Bugs 
 
 Bugs Open Time (sus_bugopen)   Bug Open Time (days)  
 Community Practices (cty_omm) Project Community
 Project Documentation
 Development Infrastructure
 Project Management 
 
 License Practices (lic_omm)  Project License   
 Liveliness (act_liveliness) Project Communication
 Project Community 
 ActiveUsers  
 Quality Practices (qua_omm)  Project Documentation
 Development Infrastructure
 Development Process
 Testing Process
 Release Management
 Security and Vulnerability Mgt
 Ratio (lic_ratio)   Ratio Files_No_Lic / Number_Files  
 Reactivity (act-reactivity)   Bug Average Response Time (Cumulative) (s)  
 Serious Issues (qua_issues)   Number of Blocker Issues
 Number of Critical Issues 
 
 Sustainability Practices (sus_omm) Project Communication
 Project Community
 Project Documentation
 Project Management
 Release Management
 Security and Vulnerability Mgt 
  
 Tests (qua_test)  Testing Process  Unit Test Coverage Percentage
 Unit Test Success Percentage 
 
 Unanswered Bugs (qua_unbugs)    UnansweredBugs  
 Unique Licenses (lic_uniquelicense)    Number of Unique Licenses  

The attributes are computed by combining aggregates as explicited in the table below.

   
  The Attributes below are computed...  ...from the average of the following Aggregates  
 Activity  Liveliness
 Bug Handling
 Reactivity 
 
 Community  Community Practices
 ActiveUsers
 UnansweredBugs 
  
 Compliance  License Practices
 Unique Licenses
 Ratio 
 
 Quality  Quality Practices
 Tests
 UnansweredBugs  
 Serious Issues 
 
 Sustainability  Sustainability Practices
 Active Users
 Bugs Open Time 
 

Results

The values, between 0 and 5, of the five attributes are published in the shape of a radar graph. These graphs suggest project profiles in a simple and efficient way.

Stage 1: Best practices verification < Previous | Next > Stage 3: Market capabilities

 

..