Reproducibility of experiments is crucial to foster an atmosphere of open, reusable and trustworthy research. To improve and reward reproducibility, iFM 2022 introduces an optional artifact evaluation process for accepted papers.

The goal of the artifact evaluation is manifold. We want to encourage authors to provide more substantial evidence to their papers and to reward authors who aim for reproducibility of their results, i.e., create artifacts. Also, we want to give more visibility and credit to the effort of tool developers in our community. Furthermore, we want to simplify the independent replication of results presented in the paper and to ease future comparison with existing approaches.

Artifacts of interest include (but are not limited to):

• Software, Tools, or Frameworks
• Data sets
• Test suites
• Machine-checkable proofs
• Protocols used for empirical studies
• Any combination of them
• Any other artifact described in the paper

Artifact submission is optional. Papers that are successfully evaluated will be awarded one or more artifact badges (see awarding), but the result of the artifact evaluation will not alter the paper’s acceptance decision. We aim to assess the artifacts themselves and not the quality of the research linked to the artifact, which has been assessed by the iFM 2022 program committee. The goal of our review process is to be constructive and to improve the submitted artifacts. Only, if an artifact cannot be improved to achieve sufficient quality in the given time frame or it is inconsistent with the paper, it should be rejected.

Important Dates

• 23.03.2022 - Artifact registration deadline
• 28.03.2022 - Artifact submission deadline
• 01.04.2022 - Test phase notification
• 01.04.- 05.04.2022 - Communication phase and author fixes of artifact
• 14.04.2022 - Final artifact notification

Artifact Evaluation

All artifacts are evaluated by the artifact evaluation committee. Each artifact will be reviewed by at least two committee members. Reviewers will read the accepted paper and explore the artifact to evaluate how well the artifact supports the claims and results of the paper. The evaluation is based on the following questions.

• Is the artifact well-documented?
• Is the artifact consistent with the paper and the claims made by the paper, e.g., does it significantly contribute to the generation of its main results?
• Is the artifact complete, i.e., how many of the results of the paper are replicable?
• Are the results of the paper replicable through the artifact, e.g., can the included software be used to generate the results of the paper and included data be accessed and manipulated?
• Is the artifact easy to use?
• Does the artifact provide a proper and explicitly documented license?
• Is the artifact publicly and permanently available?

Test Phase

In the test phase, reviewers check if the artifact is functional, i.e., they look for setup problems (e.g., corrupted, missing files, crashes on simple examples, etc.). If any problems are detected, the authors are informed of the outcome and asked for clarification.

To this end, a shared file will be used, where discovered problems are communicated, and a first notification is given to the authors on April 1st, 2022. Authors then have 5 days (see Important Dates) to solve the problems and answer the respective issue. Reviewers and authors are encouraged to use these 5 days for further clarifying comments. This process will take place through a shared file, which will be made available at the time of review.

Assessment Phase

In the assessment phase, reviewers will try to reproduce any experiments or activities and evaluate the artifact w.r.t. the questions detailed above. The final review is communicated using EasyChair.

Awarding

Authors may use all granted badges on the title page of the respective paper. iFM awards the evaluation and availability badges of EAPLS. Availability, evaluation, and validation badges are awarded independent of each other. However, we only expect to award badges of the available and evaluated categories, as the badges in the validated category cannot be assigned to papers published in this edition of iFM.

• The availability badge will be awarded if the artifact is made permanently and publicly available and has a DOI. We recommend services like zenodo or figshare for this. Also, the artifact needs to be relevant and add value beyond the text in the paper.

• The evaluation badge has two levels, functional and reusable. Each successfully evaluated artifact receives at least the functional badge. The reusable badge is granted to artifacts of very high quality. Detailed guidelines for both levels are described here.

Artifacts that are not exercisable, as, for example, protocols used for empirical studies, will be evaluated only according to the Available badge, as Functional and Reusable badges are not applicable.

Artifact Submission

An artifact submission consists of

• An Abstract, to be written directly in EasyChair, that:
  1. summarizes the artifact and explains its relation to the paper,
  2. describes which badge the authors submit for,
  3. mentions where in the artifact it is documented how to perform the test phase and how to reproduce the results of the paper, and
  4. includes an URL - we encourage you to provide a DOI - to a .zip file of your artifact containing
     • a license file that allows the artifact evaluation committee to evaluate the artifact,
     • a clear documentation how to perform the test phase,
     • a documentation how to reproduce the results of the paper, and
  5. the SHA256 checksum of the .zip file.
• A .pdf file of the most recent version of the accepted paper, which may differ from the submitted version to take reviewers’ comments into account.

Please also look at the Artifact Packaging Guidelines for detailed information about the contents of the artifact.

The abstract and the .pdf file of your paper must be submitted via EasyChair.

https://easychair.org/conference?conf=ifm2022

We need the checksum to ensure the integrity of your artifact. You can generate the checksum using the following command-line tools.

• Linux: sha256sum <file>
• Windows: CertUtil -hashfile <file> SHA256
• MacOS: shasum -a 256 <file>

If you cannot submit the artifact as requested or encounter any other difficulties in the submission process, please contact the artifact evaluation chairs prior to submission.

Artifact Packaging Guidelines

We expect that authors package their artifact (.zip file) and write their instructions such that the artifact evaluation committee can evaluate the artifact within the iFM 2022 virtual machine (see below). The artifact must contain all the required files to replicate your results in the iFM 2022 virtual machine. In particular, the artifact must include all additional software or libraries that are not part of the virtual machine and provide instructions on how to install and set them up. Do not submit a virtual machine image in the .zip file. AEC members will copy your .zip file into the provided virtual machine. For further information, consider our recommendations on the artifact content.

iFM 2022 Virtual Machine

The iFM 2022 virtual machine was created with VirtualBox 6.1.28 and consists of an installation of Ubuntu 20.04.03 with Linux 5.11.0-40 and the following notable packages.

• A 32bit libc

• clang 12.0.0

• gcc and g++ (version 9.3.0)

• Mono 6.12.0.122

• OCaml 4.13.1 and OPAM 2.1.0

• OpenJDK 11.0.11 (default), and OpenJDK 16.0.1

  To switch between Java runtimes, you can use sudo update-alternatives --config java. To switch between Java compilers, you can use sudo update-alternatives --config javac.

• Python 2.7.18, Python 3.8.10, and pip3 (pip 20.0.2)

• Ruby 2.7.0p0

• bash 5.0.17(1)

• cmake 3.16.3

• GNU Make 4.2.1

• BenchExec 3.9

• VIM 8.1

• Emacs 26.3

• VirtualBox guest additions 6.1.28

The login and password of the default user are: ifm2022 / ifm2022. The root user has the same password.

In order to save space, the VM does not have an active swap file. Please mention in your submission if you expect that a swap file is needed. You can activate swap for the running session using the following commands.

sudo fallocate -l 1G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile

The artifact evaluation committee will be instructed not to download software or data from external sources. Any additional software required by your artifact must be included in the .zip file and the artifact must provide instructions for the installation.

Including Ubuntu Packages

To include an Ubuntu package in your artifact submission, you can provide a .deb file with all the necessary dependencies from inside the VM. Reviewers can then install them as follows.

sudo dpkg -i <.deb file>

You can get the necessary .deb files for example as follows:

• If you have only one package without dependencies, you can use apt-get download <packagename>
• If you have only one package without dependencies but with local modifications, e.g., particular configuration files, you can use the dpkg-repack utility.
• If you have a package with multiple dependencies, you can use wget together with apt to download them all and put them into a folder:

  wget $(apt-get install --reinstall --print-uris -qq <packagename> | cut -d"'" -f2)
  

  Alternatively, you may run the following code.

  sudo apt-get update
  apt-get --print-uris install <packagename> | grep -oP "(?<=').*(?=')" > <filename>
  for i in $(cat <filename>) ; do wget -nv $i ; done
  

Including OCaml Packages

You can include required OCaml packages not present in our installation via OPAM. To this end, you may want to download the packages, e.g., using the following command.

opam install --download-only --destdir=<dir> <package>

The reviewers may then use the following command to install all packages available in <dir>.

opam install <dir>

Including Python Packages

You may include the required Python packages using pip. You can get the necessary files for example as follows:

pip3 download <package>

The downloaded package can then be installed using

pip3 install <package-file>

Including Ruby Gem Packages

To provide missing Ruby Gem packages, you may download them via

gem fetch <gem>

and let the reviewer install them, e.g., via

gem install <gem-file>

Recommendations

We recommend preparing your artifact in such a way that any computer science expert without dedicated expertise in your field can use your artifact, especially replicate your results. For example, keep the evaluation process simple, provide easy-to-use scripts and a detailed README document. Furthermore, the artifact and its documentation should be self-contained.

Next to the main artifact, i.e., data, software, libraries, scripts, etc. required to replicate the results of your paper and any additional software required by your artifact including an installation description, we recommend to include the following elements.

License File

A LICENSE.md file describing the rights. Your license needs to allow the artifact evaluation committee members to download and evaluate the artifact, e.g., download, use, execute, and modify the artifact for the purpose of artifact evaluation. Please refer to typical open source licenses. Artifacts without an open source license are also accepted, but a type of license needs to be specified, which allows the committee to assess the artifact.

README

The README.md file should introduce the artifact to the user, i.e., describe what the artifact does, and guide the user through the installation, set up tests, and replication of your results. Ideally, it should consist of the following sections.

• Artifact Name: a name for your artifact.
• Summary: brief description of the artifact goal, authors, reference to the paper, and indication on how to cite the artifact.
• Set-up: describes the steps to set up your artifact within the provided iFM 2022 VM. To simplify the reviewing process, we recommend providing an installation script (if necessary).
• Hardware Requirements: hardware requirements (RAM, number of cores, CPU frequency), which you considered to test your artifact. Your resource requirements should be modest and allow replication of results even on laptops.
• Test Instructions: document how to perform the test phase evaluation, e.g., provide instructions that allow rudimentary testing (i.e., in such a way that technical difficulties would pop up) in as little time as possible.
• Replication Instructions: clear description how to repeat/replicate/reproduce the results presented in the paper.
  • Please document which claims or results of the paper can be replicated with the artifact and how (e.g., which experiment must be performed). Please also explain which claims and results cannot be replicated and why.
  • Describe in detail how to replicate the results in the paper, especially describe the steps that need to be performed to replicate the results in the paper. To simplify the reviewing process, we recommend providing evaluation scripts (where applicable).
  • Please provide for each task/step of the replication (an estimate) how long it will take to perform it or how long it took for you and what exact machine(s) you used.
• Replication with Limited Resources: For tasks or experiments that require a large amount of resources (hardware or time), we additionally recommend to offer a possibility to replicate a subset of the results of the paper that can be reproduced in a reasonable amount of time (e.g., within 8 hours) on various hardware platforms including laptops. In this case, please also include a script to replicate only a subset of the results. If this is not possible, please contact the artifact evaluation chairs early, but no later than before submission.
• Examples of Usage: describe how to use your artifact in general accompanied by small examples.