Schema for Assessing Replicability in the FLT Literature

1. Reader*:

Publication

Populate this section from the publication information found in the DBLP (http://dblp.uni­trier.de/). If the publication is not found in the DBLP, then use its original source (e.g. acm, sciencedirect, ieee) to obtain the information. If theoriginal source could not be located, use Google Scholar.

  1. Search the publication title in DBLP;
  2. Link: click on the ‘view’ icon and Right Click the ‘electronic edition via DOI’ link and choose ‘Copy Link Address’;
  3. Title: as given in DBLP entry;
  4. Authors: as given in DBLP entry (comma separated list);
  5. Venue: short form as given in DBLP WITHOUT page numbers and conference location;
  6. Page numbers: as given in DBLP;
  7. Year: publication year.

2. Link*:

3. Title*:

4. Authors:

5. Venue/Conference/Journal: 

6. Pages:

7. Year:

8. Article type Mark only one oval. 

o   conference
o   book chapter
o   journal
o   other/unknown

General Information

Consider FLT as a single unit/one application.

9. Paper introduces Check all that apply.

□   combination of FLTs
□   an entirely new FLT
□   a modification of an FLT
□   an FL tool (a research prototype or a standalone/established tool) an evaluation of FLT(s)/tool(s)
□   other:

10. Research Question(s) specified

Is there a piece of text that explicitly declares the research question(s) succinctly? Mark only one oval.
o   explicit
o   implicit or not specified
o   cited

11. FLT goal/classification Check all that apply.

□   near complete feature location ­ all the code associated with the feature is sought
□   foothold/seed feature location ­ where one specific location in the code associated with the feature is sought
□   general feature location ­ where the goal could be near complete OR foothold ­ authors’ don’t differentiate
□   clustering/mapping ­ where code is grouped by features (a code element may appear in number of features and/or not all features may be covered)
□   not specified
□   other:

12. Paper presents ANY FLT implementation/algorithm details * Mark only one oval.

o   yes
o   no, e.g. evaluation

FLT Details

 13. is FL tool/algorithm named?

e.g. [tool and algorithm: SNIAFL], [tool name: FLINTS algorithm: Robillards], [algorithm: Porter’s] Check all that apply.
□   yes ­ algorithm
□   yes ­ tool
□   no ­ algorithm
□   no ­ tool

14. Supporting/Accompanying FLT tool availability * Mark only one oval.

o   there is an explicitly stated web link
o   tool explicitly mentioned but availability not specified
o   explicitly states tool is available by contacting authors
o   citation ­ those seeking the tool are referred to a citation
o   authors explicitly state it’s not available
o   no tool explicitly mentioned
o   other:

15.   Scope of the tool

Fill this field only if a tool is available/mentioned in the paper. Mark only one oval. 

1 2 3 4
most basic implementation covers only a few basic steps in the pipeline Complete implementation covers the entire pipeline

16.   FLT tool/implementation commercial/OS status Mark only one oval.

o   open source (explicit)
o   open source (implicit)
o   proprietary/commercial status not specified
o   no tool explicitly mentioned

Types of analysis

  • DYNAMIC ­ analyses system through execution and run­time instrumentation
  • STRUCTURAL ­ takes into account structure of code: packages/classes/functions, field accesses, method calls etc.
  • TEXTUAL ­ analyses plain text similarities between documents using NLP/IR techniques
  • HISTORICAL ­ uses history from version control systems but NOT for textual or structural analysis
  • e.g. co­commit analysis
  • QUERY FORMULATION ­ core focus is on query expansion/refinement
  • OTHER

17. Type of analysis Check all that apply.

□   structural ­ implicit
□   structural ­ explicit
□   dynamic ­ implicit
□   dynamic ­ explicit
□   textual/NLP/IR ­ implicit
□   textual/NLP/IR ­ explicit
□   historical ­ implicit
□   historical ­ explicit
□   query formulation/optimisation ­ implicit
□   query formulation/optimisation ­ explicit
□   other:

18. Supported granularity * Mark only one oval.

o   specified, can be identified/guessed or cited
o   not specified

Supported Granularity

19. Supported granularity * Mark only one oval per row.

Explicit

Implicit

Not supported/support unknown

package
class
file
method/function
statement (e.g. terminated by ‘;’)
line (e.g. terminated by \r? \n)
variable/field
other

Leveraged Data

Consider FLT as a single unit/one application.

20. Main data inputs

Main data sources used by the FLT (other than user inputs). Mark only one oval.
o   specified, can be identified/guessed or cited
o   not specified

Leveraged Data ­ – Main Data Analysed

Main data analysed by FLT other than any user inputs or run­time parameters.

21. Main data source(s) *

NOTE: use ‘clear’ option only to untick erroneously ticked explicit, implicit or cited option. Mark only one oval per row.

Explicit implicit cited Not used/applicable
source code
byte code/binaries
version control system
issue tracking system
documentation
other

 Leveraged Data ­ – User Queries and Other Inputs

22. Whether a user query or any other inputs (other than the previously specified main data input sources) required for the algorithm/technique (i.e. not for the evaluation): *

Mark only one oval.
o   yes (needs user query or other inputs)
o   no user query or other inputs needed

User Queries/Inputs

Specifically, for the technique, not for the evaluation

23. Types of user query * Mark only one oval per row

explicit

implicit

cited

not used/ applicable

free text query
constrained query or program recommended query
bug report
code fragment (e.g. select using an IDE)
code element
other

Overall Process

24. Overall process presented using Check all that apply.

□   text/paragraph(s)
□   diagram(s)/flow chat(s)/graphical Illustration(s)/image(s)
□   pseudo­code snippet(s)
□   example(s)
□   equation(s)
□   ordered list(s) (with text) (e.g. A. B., i., ii, iii., 1,2,3)
□   unordered list(s) (with text) (e.g. bullet points)
□   table(s)
□   none
□   other:

25. Whether pre­processing or post­processing required (by the algorithm) * Mark only one oval.

o   yes, some or all steps specified in the paper
o   yes, but only cited
o   no, pre/post­processing not needed

 Analysis ­ – Pre/Post­Processing Steps

 26. Paper presents sufficient implementation/algorithm details to replicate pre­ processing/post­processing steps

1 ­ nothing presented, 2 ­ missing big blocks of information, 3 ­ only small bits missing, 4 ­ everything presented
Mark only one oval.

1 2 3 4
nothing presented everything presented

 27. Describe the nature of the missing information (if selected 1 ­ 3 for the above question)

 28. Details of the pre­processing steps presented using Check all that apply.

□   separate sections (i.e. with different headings/sub­headings) in the paper for different pre­ processing/post­processing  steps
□   text/paragraph(s)
□   diagram(s)/flow chat(s)/graphical Illustration(s)/image(s)
□   pseudo­code  snippet(s)
□   example(s)
□   equation(s)
□   ordered list(s) (with text) (e.g. A. B., i., ii, iii., 1,2,3)
□   unordered list(s) (with text) (e.g. bullet points) table(s)
□   none
□   other:

29. Details of the individual pre­processing/post­processing steps Mark only one oval per row.

fully

partially

not at all

cited

inputs discussed
algorithm discussed
parameters discussed
outputs discussed
used tools/libraries discussed

Analysis ­ Core Algorithm

Note: core algorithm may consist of several steps. The term algorithm implies all the steps here.

30. Core algorithm at least partially specified * Mark only one oval.

o   yes, in paper
o   yes, but only cited
o    no

Analysis ­ Core Algorithm in Detail

31. Paper presents sufficient implementation/algorithm details to replicate core algorithm

1 ­ nothing presented, 2 ­ missing big blocks of information, 3 ­ only small bits missing, 4 ­ everything presented
Mark only one oval.

1 2 3 4
nothing presented everything presented

32. Describe the nature of the missing information (if selected 1 ­ 3 for the above question) 

33. Details of the core algorithm presented using Check all that apply.

□   separate sections (with different headings/sub­headings) for different core algorithm steps/processes
□   text/paragraph(s)
□   diagram(s)/flow chat(s)/graphical Illustration(s)/image(s) pseudo­code snippet(s)
□   example(s) equation(s)
□   ordered list(s) (with text) (e.g. A. B., i., ii, iii., 1,2,3) unordered list(s) (with text) (e.g. bullet points) table(s)
□   none
□   other:

34. Details on individual core steps Mark only one oval per row.

fully partially may applicable, but not discussed at all cited not applicable
inputs discussed
algorithm discussed
parameters discussed
outputs discussed
tools/libraries (used) discussed

Programming Language Support

35. Evaluated programming language(s) and specified supported programming language(s)*

Following list only include the first 7 most popular programming languages according to http://www.tiobe.com/tiobe­index/ (16/08/2016). If the language discussed in the paper is not available here, choose ‘other’ option and mention it in the next question.
Mark only one oval per row.

evaluated (implies support) supported support unknown/not evaluated
C#
Python
PHP
Java
Other
C
C++
JavaScript

36. Specify ‘other’ supported/evaluated programming language(s)

Data Storage and Data Structures

directly extracted data = data extracted from main data source(s); derived data = computed or derived data from directly extracted data; used data structures = e.g. graph, array, linked­list, call graph; data retrieval queries =queries used to retrieve data from data structures (e.g. get the shortest path from node A ­ node B in Graph G).

37. Data, storage and retrieval * Mark only one oval per row.

fully partially may applicable, but not discussed at all cited not applicable
directly extracted data discussed
derived data discussed
used data structures discussed
data retrieval queries discussed

FLT Output

38. Output format *

Mark only one oval per row.

explicit implicit not used/not mentioned
ranked list
list
clusters/mappings
graph(with nodes/edges)
visualization (chart/image/3D graph)
table/matrix
other

39. Specify ‘other’ type of output

40. Whether any screenshots of the tool/algorithm are included in the paper Mark only one oval.

o   yes
o   no

41. Types of the screenshots included in the paper * Mark only one oval per row.

holistic and segment holistic segment not available
input (screenshot showing input(s))
intermediate output
results (screenshot showing result(s))

Related Work

42. Related work *

Mark only one oval per row.

yes no
Similar work discussed
Distinguished from similar studies

43. Number of citations in the Related Work section Count INCLUDING the duplicates.

Testing and Evaluation

44. Evaluation performed * Mark only one oval.

o   yes
o   no

Evaluation Details

45. Type of the evaluation Mark only one oval.

o   illustrative/example scenario(s)/walkthrough
o   case study/multiple case studies
o   experiment/multiple experiments
o   other:

46. Objective of the evaluation Check all that apply.

□   FL performance (recall/precision etc.)
□   performance (memory/process/storage efficiency)
□   usability
□   other:

47. Characterisation of systems under study Check all that apply.

□   number
□   sampling techniques specified the for the evaluation data (e.g. which queries are taken from a bug repository)
□   identification/name
□   version
□   location (e.g. web link)
□   open source vs proprietary
□   lines of code (loc, mloc)
□   number of code elements such as methods, classes and/or files
□   other metrics
□   other:

48. Study involves human subjects * Mark only one oval.

o   yes
o   no

Nature of the evaluation that involves humans

49. Study involves academics or professionals Mark only one oval.

o   academics
o   professionals
o   not specified/unknown
o   not applicable

50. Human subjects sufficiently characterised Mark only one oval.

1 2 3 4
strongly disagree strongly agree

51. Characterisation of human subjects Check all that apply.

o   gender
o   age
o   profession/job title
o   experience in years
o   other:

Benchmarks/Gold Standards

52. Employs benchmark(s) or gold standard(s) Mark only one oval.

o   yes ­ internal benchmark(s) (i.e. data developed/derived by authors’ themselves)
o   yes ­ external benchmark(s) (used externally defined data set as the benchmark)
o   yes ­ both internal and external benchmark(s)
o   cited
o   no
o   not applicable

53. Characterisation of the benchmark(s) Check all that apply.

□   identified/name
□   location (e.g. web link)
□   version
□   number of features
□   open source vs proprietary
□   lines of code (kloc, mloc)
□   number of code elements such as classes/files
□   (in case of authors’ own benchmark,) development/derivation process/details
□   other:

54. Approaches compared

Explicitly replicable = refer to another paper that describes the technique; NOTE: use ‘clear’ option only to untick erroneously ticked yes or no option.

Mark only one oval per row.

yes no clear
with different, explicitly replicable technique
with different parameters
with benchmark(s)

Results and Conclusions

55. Types of results reported Check all that apply.

□   qualitative
□   quantitative
□   not applicable

56. Reported metrics Check all that apply.

□   precision
□   recall
□   F­measure
□   performance ­ memory consumption
□   performance ­ speed
□   false +ve/­ves
□   MAP
□   MRR
□   not applicable
□   other:

57. Final evaluation conclusion(s)

One line that summarizes the relative efficiency of the technique

Validity, Limitations and Future Work

58. Discusses validity/limitations of Mark only one oval per row.

  in dedicated section(s) (e.g. validity section, or limitations section) not at all in text (i.e. no dedicated
validity or limitations
section)
clear
the
research/technique
the evaluation

59. Future work discussed Mark only one oval.

o   yes
o   no

Reader’s Remarks

60. Overall impression of the paper Mark only one oval.

1 2 3 4
very poor very good

61. Your thoughts on the publication ­ good points (list of bullet points)

62. Your thoughts on the publication ­ bad points (list of bullet points)