package rotor

ROTOR: A Reliable OCaml Tool for OCaml Refactoring - Trustworthy Refactoring for OCaml

ROTOR is a tool for refactoring OCaml code, written in OCaml.

The eventual aim is for ROTOR to be able to not only perform automatic refactoring, but also integrate formal verification that the refactorings are correct. It is intended that the CakeML project will be used for this.

More details can be found at the project website.

Dependencies

• The OPAM description file ensures that the following necessary OCaml dependencies are present when installing via OPAM:

  • The cppo and its associated cppo_ocamlbuild packages.
  • The containers package (version >= 2.6).
  • The ocamlfind package
  • The ocamlgraph package (version >= 1.8.7).
  • The logs package (version >=0.6.2).
  • The mparser (version >= 1.2.3) and pcre packages.
  • The visitors package (version >= 20170317). The Visitors package itself requires ppx_tools and ppx_deriving.
  • The cmdliner package.
• The diff and filterdiff utilities; diff usually comes as standard, but filterdiff can be found in the patchutils package.
• The rlwrap utility, for running the utils/toplevel.sh script.

• The Jane Street testbed code requires the following packages:

  • ocaml-migrate-parsetree
  • octavius
  • re

  Building the testbed requires jbuilder (version <= 1.0+beta11). The test scripts assume that there is an OPAM switch named rotor-testbed with these packages installed.

  Note that the testbed can only be compiled with compiler versions 4.04.x or 4.05.x.

  The testbed tarball is versioned via Git LFS, which you will need to have installed in order to download it from the repository.

Installing ROTOR

To install from a local copy of the source via opam, run the following from the root of the directory containing the source code:

> opam pin -k path add rotor .

Running ROTOR

A man page for ROTOR explaining its commands and options can be displayed by running:

> rotor --help

To rename a value in an OCaml program, within the directory containing the modules of the program, invoke the rename command:

> rotor rename <identifier> <new-name>

The syntax for specifying identifiers is described in the subsection below.

ROTOR looks for compiled .cmt and .cmti files corresponding to the modules of a program. These can be produced by passing the -bin-annot flag to the compiler. If the .cmt and .cmti files are located in a different directory to the source files, you can pass these to ROTOR using the -I option:

> rotor rename [-I <dir>]* <identifier> <new-name>

If the program depends on any libraries (in addition to OCaml's standard Pervasives library), then the directories containing the interface files for these libraries must be passed to ROTOR using the -I option.

If .cmt and .cmti files are not present, then ROTOR will attempt to parse and type the source files itself directly.

If the program's source files are distributed across many different directories then you can pass these to ROTOR using the -i option:

> rotor rename -i <src_dir> <identifier> <new-name>

You also specify that an individual file is part of the codebase using the -i option:

> rotor rename -i <src_file> <identifier> <new-name>

If the files in a source directory, or individual source file, should be processed with a sequence of particular PPX preprocessors, this can also be specified as follows:

> rotor rename -i [ppx:<ppx_exe>,]*<src_file_or_dir> <identifier> <new-name>

Additionally, if the OCaml program has been compiled with dune, then you must specify the name of the library that each source file belongs to, as follows:

> rotor rename -i [ppx:<ppx_exe>,]*[lib:<lib_name>]?<src_file_or_dir> <identifier> <new-name>

ROTOR's output can be redirected to a file using the -o option.

> rotor rename -o <file> <identifier> <new-name>

You can display ROTOR's progress as it is computing a refactoring:

> rotor rename --show-progress <identifier> <new-name>

Debugging information can be saved to a file using the --log-file flag:

> rotor rename --log-file <file> <identifier> <new-name>

The module dependencies of a codebase can be output using the mod-deps command:

> rotor mod-deps [-I <dir>]* [-i <item_spec>]*

ROTOR's Identifier Syntax

ROTOR uses an extended syntax for OCaml identifiers. OCaml programs have a hierarchical structure, in which both modules and module types can be nested within one another. OCaml uses 'dot notation' for identifiers, in which the infix operator dot (.) indicates this hierarchical nesting. ROTOR generalises OCaml's identifier notation in two ways. Firstly, instead of treating the dot as an infix operator, it uses it as a prefix operator on names to indicate an element of a particular sort and introduces new prefix operators to express other sorts (e.g. module, module type, value). Secondly, the hierarchical structure is now represented by the sequencing of prefixed names. ROTOR currently uses the operators ., #, %, *, and : to indicate structures, functors, structure types (i.e. signatures), functor types, and values, respectively. ROTOR also uses an indexer element of the form [i], to stand for the ith parameter of a functor or functor type.

Specifically, ROTOR uses the following syntax for identifiers where the nonterminal <name> denotes a standard OCaml (short) identifier, and <number> denotes a positive integer literal.

    <signifier>  ::= '.' | '#' | '%' | '*' | ':'
    <id_link>    ::= <signifier> <name> | '[' <index> ']'
    <identifier> ::= <id_link> | <id_link> <identifier>

So, for example, to specify a function foo nested within a number of (sub)modules, you could use the identifier .A.B.Bar.Baz:foo.

To give a more complex example, .Set%S:add would refer to the add value declaration within the S module type within the Set module.

Similarly, .Set#Make[1]:compare refers to the declaration of the compare value in the first parameter of the Make functor within the Set module.

Note that when specifying the new name in the invocation of ROTOR

> rotor rename <identifier> <new-name>

you should give simply a short identifier (e.g. foo), i.e. you do not need to specify a full path; indeed doing so will cause ROTOR to raise an error.

Test Suite

The test suites can be run by calling make with targets having the prefix tests (e.g. tests.jane-street.all). See the Test Suite README for more details.

Docker Image

A docker image of the tool is available on docker hub.

> docker pull reubenrowe/ocaml-rotor

To execute the image, run

> docker run -ti reubenrowe/ocaml-rotor

Progress Log

23 August 2019

• Compiler version 4.08.x now supported.
• ROTOR reads source and build artifact directories from a .merlin file, if present; a command line flag can be used to switch this behaviour off.
• ROTOR now computes module dependencies as primary (and these can be output to a file, and read back from one), and recreates the file dependencies relative to the given codebase from using the module dependencies. This is a more robust and modular way to handle dependencies.
• Gitlab CI added to automatically test build and install via OPAM.

25 May 2019

• Change cli to add commands for renaming and outputting module dependencies.

19 January 2019

• Compiler version 4.04.x through 4.07.x now supported.

20 March 2018

• Now depends on compiler-libs, rather than a custom packaging of the compiler. Set-up should be more straightforward now, and cross-compiling with different versions of the compiler too.

• Various bug fixes, including some major functional problems:

  • Recursive definitions were not being traversed to rename recursive references to the binding.
  • An off-by-one bug causing replacements that end on the last character of a line to be rejected.
  • A bug in dependency generation meaning that structure/signature items subsequent to a parent item of a binding definition were not being traversed to look for use dependencies.

03 March 2018

• Source code extracted and replacements are applied according to line and column information instead of by character index; this was to avoid some problems caused by preprocessors in the Jane Street testbed that introduce line directives to the beginning of some files.

25 February 2018

• The two value renaming refactoring modules (rename_val_impl and rename_val_intf) have now been merged, as further development has caused them to converge to compatible implementations. Thus it makes sense to merge them in order to avoid code duplication.

  Development splits onto combine_renames branch

23 February 2018

• Basic handling of functors and functor applications implemented. Handling of functor applications is somewhat restricted to applications to module identifiers since handling anonymous modules in full generality is not yet solved.

15 January 2018

• Added 'parameter' as a logical entity; in particular this is being used to refer to functor parameters.

30 December 2017

• We model a distinction between structures and functors. Note that both may be bound to modules. This distinction is represented in the rich identifiers used by the tool.

06 November 2017

• The tool can now output the refactoring dependency set that it generates.

17 September 2017

• Created a Dockerfile for building a portable Docker image of the tool.

08 September 2017

• Presented the tool at the OCaml Users Workshop 2017.

07 August 2017

• Refined renaming of values into two distinct refactorings: renaming a value in an implementation and interface (a named module type) respectively. In fact, this dichotomy will possibly be a facet of all refactorings. Finished implementing first veresion of both kinds of value renaming. Next step is to make these refactorings take functors into account.
• Extracted all persistent identifiers used the in the Jane Street test bed. This collection can be used to stress test the value renaming refactoring.

25 June 2017

• A rich representation of identifiers implemented, allowing the language kind (e.g. value, module, module type, etc.) of each segment of a long identifier to be specified. The module also encapsulates the valid nestings of each kind (e.g. a module can contain a value, but not vice-versa).

  Currently only modules, module types, and values are supported.

24 May 2017

• Automated framework for running test suites set up. Tests can specify what the resulting diff is expected to be in order to determine success or failure of the test case.
• Tests can specify that they expect the refactoring tool to fail, and also which exit code is expected (26 May 2017).

19 May 2017

• Renaming detects when shadowing may occur.

11 May 2017

• Dependencies between source files are now computed and refactorings only run on certain files based on these dependencies. Refactoring dependencies also use these file dependencies. We now rely on the concept of a "kernel" for a refactoring: this is the set of files in a codebase for which we only need look at one level of file dependencies in order to correctly apply the refactoring.

03 April 2017

• Set up logging infrastructure using the Logs package.

30 March 2017

• Added facility to compute refactoring dependencies. Computation of dependencies for renaming of values still work in progress. Main executable now outputs diffs directly.

19 Mar 2017

• Updated to use official release of Visitors package, which now supports [@name] and [@build] attributes.

02 Mar 2017

• Added shell script to run refactorings over given directories and store diffs in files.

01 Mar 2017

• Typed ASTs are now read from .cmt files when they are present.
• Implemented basic renaming of values. The value to be renamed is specified using a long identifier (e.g. Foo.Bar.baz), and function argument and record field punning is taken into account. It is not yet correct in all cases: e.g., if the parent module of a renamed value is included in another module M, then the refactorer does not automatically rename the signature of M, nor does it yet issue a suitable warning. The aim is that such "dependencies" be calculated and checked in future.

24 Feb 2017

• Common frontend functionality, with PPX processing.

22 Feb 2017

• Packaged all modules into a library and created scripts to initiate a toplevel loop and load the library.

9 Feb 2017

• Interface for refactorings as modules created.
• Implemented first basic refactoring (rename an expression identifier)!

7 Feb 2017

• Visitors can be generated for the Parsetree and Typedtree types.

Language featuers which are not yet handled

• Functors (in progress).
• The module type of construct.
• First-class modules.
• Local module bindings (within value expressions).
• Not quite sure what the Tmty_alias case in module types represents.