Schema for Assessing Replicability in the FLT Literature
1. Reader*:
Publication
Populate this section from the publication information found in the DBLP (http://dblp.unitrier.de/). If the publication is not found in the DBLP, then use its original source (e.g. acm, sciencedirect, ieee) to obtain the information. If theoriginal source could not be located, use Google Scholar.
- Search the publication title in DBLP;
- Link: click on the ‘view’ icon and Right Click the ‘electronic edition via DOI’ link and choose ‘Copy Link Address’;
- Title: as given in DBLP entry;
- Authors: as given in DBLP entry (comma separated list);
- Venue: short form as given in DBLP WITHOUT page numbers and conference location;
- Page numbers: as given in DBLP;
- Year: publication year.
2. Link*:
3. Title*:
4. Authors:
5. Venue/Conference/Journal:
6. Pages:
7. Year:
8. Article type Mark only one oval.
o conference
o book chapter
o journal
o other/unknown
General Information
Consider FLT as a single unit/one application.
9. Paper introduces Check all that apply.
□ combination of FLTs
□ an entirely new FLT
□ a modification of an FLT
□ an FL tool (a research prototype or a standalone/established tool) an evaluation of FLT(s)/tool(s)
□ other:
10. Research Question(s) specified
Is there a piece of text that explicitly declares the research question(s) succinctly? Mark only one oval.
o explicit
o implicit or not specified
o cited
11. FLT goal/classification Check all that apply.
□ near complete feature location all the code associated with the feature is sought
□ foothold/seed feature location where one specific location in the code associated with the feature is sought
□ general feature location where the goal could be near complete OR foothold authors’ don’t differentiate
□ clustering/mapping where code is grouped by features (a code element may appear in number of features and/or not all features may be covered)
□ not specified
□ other:
12. Paper presents ANY FLT implementation/algorithm details * Mark only one oval.
o yes
o no, e.g. evaluation
FLT Details
13. is FL tool/algorithm named?
e.g. [tool and algorithm: SNIAFL], [tool name: FLINTS algorithm: Robillards], [algorithm: Porter’s] Check all that apply.
□ yes algorithm
□ yes tool
□ no algorithm
□ no tool
14. Supporting/Accompanying FLT tool availability * Mark only one oval.
o there is an explicitly stated web link
o tool explicitly mentioned but availability not specified
o explicitly states tool is available by contacting authors
o citation those seeking the tool are referred to a citation
o authors explicitly state it’s not available
o no tool explicitly mentioned
o other:
15. Scope of the tool
Fill this field only if a tool is available/mentioned in the paper. Mark only one oval.
| 1 | 2 | 3 | 4 | ||
| most basic implementation covers only a few basic steps in the pipeline | ○ | ○ | ○ | ○ | Complete implementation covers the entire pipeline |
16. FLT tool/implementation commercial/OS status Mark only one oval.
o open source (explicit)
o open source (implicit)
o proprietary/commercial status not specified
o no tool explicitly mentioned
Types of analysis
- DYNAMIC analyses system through execution and runtime instrumentation
- STRUCTURAL takes into account structure of code: packages/classes/functions, field accesses, method calls etc.
- TEXTUAL analyses plain text similarities between documents using NLP/IR techniques
- HISTORICAL uses history from version control systems but NOT for textual or structural analysis
- e.g. cocommit analysis
- QUERY FORMULATION core focus is on query expansion/refinement
- OTHER
17. Type of analysis Check all that apply.
□ structural implicit
□ structural explicit
□ dynamic implicit
□ dynamic explicit
□ textual/NLP/IR implicit
□ textual/NLP/IR explicit
□ historical implicit
□ historical explicit
□ query formulation/optimisation implicit
□ query formulation/optimisation explicit
□ other:
18. Supported granularity * Mark only one oval.
o specified, can be identified/guessed or cited
o not specified
Supported Granularity
19. Supported granularity * Mark only one oval per row.
| Explicit |
Implicit |
Not supported/support unknown |
|
| package | ○ | ○ | ○ |
| class | ○ | ○ | ○ |
| file | ○ | ○ | ○ |
| method/function | ○ | ○ | ○ |
| statement (e.g. terminated by ‘;’) | ○ | ○ | ○ |
| line (e.g. terminated by \r? \n) | ○ | ○ | ○ |
| variable/field | ○ | ○ | ○ |
| other | ○ | ○ | ○ |
Leveraged Data
Consider FLT as a single unit/one application.
20. Main data inputs
Main data sources used by the FLT (other than user inputs). Mark only one oval.
o specified, can be identified/guessed or cited
o not specified
Leveraged Data – Main Data Analysed
Main data analysed by FLT other than any user inputs or runtime parameters.
21. Main data source(s) *
NOTE: use ‘clear’ option only to untick erroneously ticked explicit, implicit or cited option. Mark only one oval per row.
| Explicit | implicit | cited | Not used/applicable | |
| source code | ○ | ○ | ○ | ○ |
| byte code/binaries | ○ | ○ | ○ | ○ |
| version control system | ○ | ○ | ○ | ○ |
| issue tracking system | ○ | ○ | ○ | ○ |
| documentation | ○ | ○ | ○ | ○ |
| other | ○ | ○ | ○ | ○ |
Leveraged Data – User Queries and Other Inputs
22. Whether a user query or any other inputs (other than the previously specified main data input sources) required for the algorithm/technique (i.e. not for the evaluation): *
Mark only one oval.
o yes (needs user query or other inputs)
o no user query or other inputs needed
User Queries/Inputs
Specifically, for the technique, not for the evaluation
23. Types of user query * Mark only one oval per row
|
explicit |
implicit |
cited |
not used/ applicable |
|
| free text query | ○ | ○ | ○ | ○ |
| constrained query or program recommended query | ○ | ○ | ○ | ○ |
| bug report | ○ | ○ | ○ | ○ |
| code fragment (e.g. select using an IDE) | ○ | ○ | ○ | ○ |
| code element | ○ | ○ | ○ | ○ |
| other | ○ | ○ | ○ | ○ |
Overall Process
24. Overall process presented using Check all that apply.
□ text/paragraph(s)
□ diagram(s)/flow chat(s)/graphical Illustration(s)/image(s)
□ pseudocode snippet(s)
□ example(s)
□ equation(s)
□ ordered list(s) (with text) (e.g. A. B., i., ii, iii., 1,2,3)
□ unordered list(s) (with text) (e.g. bullet points)
□ table(s)
□ none
□ other:
25. Whether preprocessing or postprocessing required (by the algorithm) * Mark only one oval.
o yes, some or all steps specified in the paper
o yes, but only cited
o no, pre/postprocessing not needed
Analysis – Pre/PostProcessing Steps
26. Paper presents sufficient implementation/algorithm details to replicate pre processing/postprocessing steps
1 nothing presented, 2 missing big blocks of information, 3 only small bits missing, 4 everything presented
Mark only one oval.
| 1 | 2 | 3 | 4 | ||
| nothing presented | ○ | ○ | ○ | ○ | everything presented |
27. Describe the nature of the missing information (if selected 1 3 for the above question)
28. Details of the preprocessing steps presented using Check all that apply.
□ separate sections (i.e. with different headings/subheadings) in the paper for different pre processing/postprocessing steps
□ text/paragraph(s)
□ diagram(s)/flow chat(s)/graphical Illustration(s)/image(s)
□ pseudocode snippet(s)
□ example(s)
□ equation(s)
□ ordered list(s) (with text) (e.g. A. B., i., ii, iii., 1,2,3)
□ unordered list(s) (with text) (e.g. bullet points) table(s)
□ none
□ other:
29. Details of the individual preprocessing/postprocessing steps Mark only one oval per row.
|
fully |
partially |
not at all |
cited |
|
| inputs discussed | ○ | ○ | ○ | ○ |
| algorithm discussed | ○ | ○ | ○ | ○ |
| parameters discussed | ○ | ○ | ○ | ○ |
| outputs discussed | ○ | ○ | ○ | ○ |
| used tools/libraries discussed | ○ | ○ | ○ | ○ |
Analysis Core Algorithm
Note: core algorithm may consist of several steps. The term algorithm implies all the steps here.
30. Core algorithm at least partially specified * Mark only one oval.
o yes, in paper
o yes, but only cited
o no
Analysis Core Algorithm in Detail
31. Paper presents sufficient implementation/algorithm details to replicate core algorithm
1 nothing presented, 2 missing big blocks of information, 3 only small bits missing, 4 everything presented
Mark only one oval.
| 1 | 2 | 3 | 4 | ||
| nothing presented | ○ | ○ | ○ | ○ | everything presented |
32. Describe the nature of the missing information (if selected 1 3 for the above question)
33. Details of the core algorithm presented using Check all that apply.
□ separate sections (with different headings/subheadings) for different core algorithm steps/processes
□ text/paragraph(s)
□ diagram(s)/flow chat(s)/graphical Illustration(s)/image(s) pseudocode snippet(s)
□ example(s) equation(s)
□ ordered list(s) (with text) (e.g. A. B., i., ii, iii., 1,2,3) unordered list(s) (with text) (e.g. bullet points) table(s)
□ none
□ other:
34. Details on individual core steps Mark only one oval per row.
| fully | partially | may applicable, but not discussed at all | cited | not applicable | |
| inputs discussed | ○ | ○ | ○ | ○ | ○ |
| algorithm discussed | ○ | ○ | ○ | ○ | ○ |
| parameters discussed | ○ | ○ | ○ | ○ | ○ |
| outputs discussed | ○ | ○ | ○ | ○ | ○ |
| tools/libraries (used) discussed | ○ | ○ | ○ | ○ | ○ |
Programming Language Support
35. Evaluated programming language(s) and specified supported programming language(s)*
Following list only include the first 7 most popular programming languages according to http://www.tiobe.com/tiobeindex/ (16/08/2016). If the language discussed in the paper is not available here, choose ‘other’ option and mention it in the next question.
Mark only one oval per row.
| evaluated (implies support) | supported | support unknown/not evaluated | |
| C# | ○ | ○ | ○ |
| Python | ○ | ○ | ○ |
| PHP | ○ | ○ | ○ |
| Java | ○ | ○ | ○ |
| Other | ○ | ○ | ○ |
| C | ○ | ○ | ○ |
| C++ | ○ | ○ | ○ |
| JavaScript | ○ | ○ | ○ |
36. Specify ‘other’ supported/evaluated programming language(s)
Data Storage and Data Structures
directly extracted data = data extracted from main data source(s); derived data = computed or derived data from directly extracted data; used data structures = e.g. graph, array, linkedlist, call graph; data retrieval queries =queries used to retrieve data from data structures (e.g. get the shortest path from node A node B in Graph G).
37. Data, storage and retrieval * Mark only one oval per row.
| fully | partially | may applicable, but not discussed at all | cited | not applicable | |
| directly extracted data discussed | ○ | ○ | ○ | ○ | ○ |
| derived data discussed | ○ | ○ | ○ | ○ | ○ |
| used data structures discussed | ○ | ○ | ○ | ○ | ○ |
| data retrieval queries discussed | ○ | ○ | ○ | ○ | ○ |
FLT Output
38. Output format *
Mark only one oval per row.
| explicit | implicit | not used/not mentioned | |
| ranked list | ○ | ○ | ○ |
| list | ○ | ○ | ○ |
| clusters/mappings | ○ | ○ | ○ |
| graph(with nodes/edges) | ○ | ○ | ○ |
| visualization (chart/image/3D graph) | ○ | ○ | ○ |
| table/matrix | ○ | ○ | ○ |
| other | ○ | ○ | ○ |
39. Specify ‘other’ type of output
40. Whether any screenshots of the tool/algorithm are included in the paper Mark only one oval.
o yes
o no
41. Types of the screenshots included in the paper * Mark only one oval per row.
| holistic and segment | holistic | segment | not available | |
| input (screenshot showing input(s)) | ○ | ○ | ○ | ○ |
| intermediate output | ○ | ○ | ○ | ○ |
| results (screenshot showing result(s)) | ○ | ○ | ○ | ○ |
Related Work
42. Related work *
Mark only one oval per row.
| yes | no | |
| Similar work discussed | ○ | ○ |
| Distinguished from similar studies | ○ | ○ |
43. Number of citations in the Related Work section Count INCLUDING the duplicates.
Testing and Evaluation
44. Evaluation performed * Mark only one oval.
o yes
o no
Evaluation Details
45. Type of the evaluation Mark only one oval.
o illustrative/example scenario(s)/walkthrough
o case study/multiple case studies
o experiment/multiple experiments
o other:
46. Objective of the evaluation Check all that apply.
□ FL performance (recall/precision etc.)
□ performance (memory/process/storage efficiency)
□ usability
□ other:
47. Characterisation of systems under study Check all that apply.
□ number
□ sampling techniques specified the for the evaluation data (e.g. which queries are taken from a bug repository)
□ identification/name
□ version
□ location (e.g. web link)
□ open source vs proprietary
□ lines of code (loc, mloc)
□ number of code elements such as methods, classes and/or files
□ other metrics
□ other:
48. Study involves human subjects * Mark only one oval.
o yes
o no
Nature of the evaluation that involves humans
49. Study involves academics or professionals Mark only one oval.
o academics
o professionals
o not specified/unknown
o not applicable
50. Human subjects sufficiently characterised Mark only one oval.
| 1 | 2 | 3 | 4 | ||
| strongly disagree | ○ | ○ | ○ | ○ | strongly agree |
51. Characterisation of human subjects Check all that apply.
o gender
o age
o profession/job title
o experience in years
o other:
Benchmarks/Gold Standards
52. Employs benchmark(s) or gold standard(s) Mark only one oval.
o yes internal benchmark(s) (i.e. data developed/derived by authors’ themselves)
o yes external benchmark(s) (used externally defined data set as the benchmark)
o yes both internal and external benchmark(s)
o cited
o no
o not applicable
53. Characterisation of the benchmark(s) Check all that apply.
□ identified/name
□ location (e.g. web link)
□ version
□ number of features
□ open source vs proprietary
□ lines of code (kloc, mloc)
□ number of code elements such as classes/files
□ (in case of authors’ own benchmark,) development/derivation process/details
□ other:
54. Approaches compared
Explicitly replicable = refer to another paper that describes the technique; NOTE: use ‘clear’ option only to untick erroneously ticked yes or no option.
Mark only one oval per row.
| yes | no | clear | |
| with different, explicitly replicable technique | ○ | ○ | ○ |
| with different parameters | ○ | ○ | ○ |
| with benchmark(s) | ○ | ○ | ○ |
Results and Conclusions
55. Types of results reported Check all that apply.
□ qualitative
□ quantitative
□ not applicable
56. Reported metrics Check all that apply.
□ precision
□ recall
□ Fmeasure
□ performance memory consumption
□ performance speed
□ false +ve/ves
□ MAP
□ MRR
□ not applicable
□ other:
57. Final evaluation conclusion(s)
One line that summarizes the relative efficiency of the technique
Validity, Limitations and Future Work
58. Discusses validity/limitations of Mark only one oval per row.
| in dedicated section(s) (e.g. validity section, or limitations section) | not at all | in text (i.e. no dedicated validity or limitations section) |
clear | |
| the research/technique |
○ | ○ | ○ | ○ |
| the evaluation | ○ | ○ | ○ | ○ |
59. Future work discussed Mark only one oval.
o yes
o no
Reader’s Remarks
60. Overall impression of the paper Mark only one oval.
| 1 | 2 | 3 | 4 | ||
| very poor | ○ | ○ | ○ | ○ | very good |