package alcotest

  1. Overview
  2. Docs
Alcotest is a lightweight and colourful test framework

Install

Dune Dependency

github.com Readme Changelog ISC License Edit opam file Versions (5)

Authors

  1. T
    Thomas Gazagnaire

Maintainers

  1. T
    thomas@gazagnaire.org

Sources

alcotest-1.0.1.tbz
sha256=0c8748838a89df6dee4850aa7ef5e46c573265a9bf1589dec255bd8156a793f6
sha512=f5f52dea5bb143e7001b8d0ac6131f8851389b080f46b9ad1ccacb95cc31a38143dd7122ccba59bb190abe559dbf81f33cc4dc3401ed95772d59be75fa566f19

Description

Alcotest exposes simple interface to perform unit tests. It exposes a simple TESTABLE module type, a check function to assert test predicates and a run function to perform a list of unit -> unit test callbacks.

Alcotest provides a quiet and colorful output where only faulty runs are fully displayed at the end of the run (with the full logs ready to inspect), with a simple (yet expressive) query language to select the tests to run.

Published: 13 Feb 2020

README

Alcotest logo"

Alcotest is a lightweight and colourful test framework.

Alcotest exposes simple interface to perform unit tests. It exposes a simple TESTABLE module type, a check function to assert test predicates and a run function to perform a list of unit -> unit test callbacks.

Alcotest provides a quiet and colorful output where only faulty runs are fully displayed at the end of the run (with the full logs ready to inspect), with a simple (yet expressive) query language to select the tests to run.

OCaml-CI Build Status" TravisCI Build Status" docs"

Examples

A simple example (taken from examples/simple.ml):

(* Build with `ocamlbuild -pkg alcotest simple.byte` *)

(* A module with functions to test *)
module To_test = struct
  let lowercase = String.lowercase_ascii
  let capitalize = String.capitalize_ascii
  let str_concat = String.concat ""
  let list_concat = List.append
end

(* The tests *)
let test_lowercase () =
  Alcotest.(check string) "same string" "hello!" (To_test.lowercase "hELLO!")

let test_capitalize () =
  Alcotest.(check string) "same string" "World." (To_test.capitalize "world.")

let test_str_concat () =
  Alcotest.(check string) "same string" "foobar" (To_test.str_concat ["foo"; "bar"])

let test_list_concat () =
  Alcotest.(check (list int)) "same lists" [1; 2; 3] (To_test.list_concat [1] [2; 3])

(* Run it *)
let () =
  let open Alcotest in
  run "Utils" [
      "string-case", [
          test_case "Lower case"     `Quick test_lowercase;
          test_case "Capitalization" `Quick test_capitalize;
        ];
      "string-concat", [ test_case "String mashing" `Quick test_str_concat  ];
      "list-concat",   [ test_case "List mashing"   `Slow  test_list_concat ];
    ]

The result is a self-contained binary which displays the test results. Use ./simple.byte --help to see the runtime options.

$ ./simple.native
Testing Utils.
[OK]       string-case            0   Lower case.
[OK]       string-case            1   Capitalization.
[OK]       string-concat          0   String mashing.
[OK]       list-concat            0   List mashing.
Test Successful in 0.001s. 4 tests run.

Selecting tests to execute

You can filter which tests to run by supplying a regular expression matching the names of the tests to execute, or by passing a regular expression and a comma-separated list of test numbers (or ranges of test numbers, e.g. 2,4..9):

$ ./simple.native test '.*concat*'
Testing Utils.
[SKIP]     string-case            0   Lower case.
[SKIP]     string-case            1   Capitalization.
[OK]       string-concat          0   String mashing.
[OK]       list-concat            0   List mashing.
The full test results are available in `_build/_tests`.
Test Successful in 0.000s. 2 tests run.

$ ./simple.native test 'string-case' '1..3'
Testing Utils.
[SKIP]     string-case            0   Lower case.
[OK]       string-case            1   Capitalization.
[SKIP]     string-concat          0   String mashing.
[SKIP]     list-concat            0   List mashing.
The full test results are available in `_build/_tests`.
Test Successful in 0.000s. 1 test run.

Note that you cannot filter by test case name (i.e. Lower case or Capitalization), you must filter by test name & number instead. Test names may contain only alphanumeric characters, spaces, hyphens and underscores.

See the examples folder for more examples.

Quick and Slow tests

In general you should use `Quick tests: tests that are ran on any invocations of the test suite. You should only use `Slow tests for stress tests that are ran only on occasion (typically before a release or after a major change). These slow tests can be suppressed by passing the -q flag on the command line, e.g.:

$ ./test.exe -q # run only the quick tests
$ ./test.exe    # run quick and slow tests

Passing custom options to the tests

In most cases, the base tests are unit -> unit functions. However, it is also possible to pass an extra option to all the test functions by using 'a -> unit, where 'a is the type of the extra parameter.

In order to do this, you need to specify how this extra parameter is read on the command-line, by providing a Cmdliner term for command-line arguments which explains how to parse and serialize values of type 'a (note: do not use positional arguments, only optional arguments are supported).

For instance:

let test_nice i = Alcotest.(check int) "Is it a nice integer?" i 42

let int =
  let doc = "What is your prefered number?" in
  Cmdliner.Arg.(required & opt (some int) None & info ["n"] ~doc ~docv:"NUM")

let () =
  Alcotest.run_with_args "foo" int [
    "all", ["nice", `Quick, test_nice]
  ]

Will generate test.exe such that:

$ test.exe test
test.exe: required option -n is missing

$ test.exe test -n 42
Testing foo.
[OK]                all          0   int.

Lwt

Alcotest provides an Alcotest_lwt module that you could use to wrap Lwt test cases. The basic idea is that instead of providing a test function in the form unit -> unit, you provide one with the type unit -> unit Lwt.t and alcotest-lwt calls Lwt_main.run for you.

However, there are a couple of extra features:

  • If an async exception occurs, it will cancel your test case for you and fail it (rather than exiting the process).
  • You get given a switch, which will be turned off when the test case finishes (or fails). You can use that to free up any resources.

For instance:

let free () = print_endline "freeing all resources"; Lwt.return ()

let test_lwt switch () =
  Lwt_switch.add_hook (Some switch) free;
  Lwt.async (fun () -> failwith "All is broken");
  Lwt_unix.sleep 10.

let () =
  Lwt_main.run @@ Alcotest_lwt.run "foo" [
    "all", [
      Alcotest_lwt.test_case "one" `Quick test_lwt
    ]
  ]

Will generate:

$ test.exe
Testing foo.
[ERROR]             all          0   one.
-- all.000 [one.] Failed --
in _build/_tests/all.000.output:
freeing all resources
[failure] All is broken

Screenshots

The following screenshots demonstrate the HTML testing output from the odoc project.

All tests passed

Some tests failed

Failed test with custom diffing

ok"err"diff"

Comparison with other testing frameworks

The README is pretty clear about that:

Alcotest is the only testing framework using colors!

More seriously, Alcotest is similar to ounit but it fixes a few of the problems found in that library:

  • Alcotest has a nicer output, it is easier to see what failed and what succeeded and to read the log outputs of the failed tests;
  • Alcotest uses combinators to define pretty-printers and comparators between the things to test.

Other nice tools doing different kind of testing also exist:

  • qcheck qcheck does random generation and property testing (e.g. Quick Check)
  • crowbar and bun are similar to qcheck, but use compiler-directed randomness, e.g. it takes advantage of the AFL support the OCaml compiler.
  • ppx_inline_tests allows to write tests in the same file as your source-code; they will be run only in a special mode of compilation.

Dependencies (8)

  1. stdlib-shims
  2. re >= "1.7.2"
  3. uuidm
  4. cmdliner >= "1.0.3"
  5. astring
  6. fmt >= "0.8.6"
  7. ocaml >= "4.03.0"
  8. dune >= "1.11"

Dev Dependencies

None

  1. ahrocksdb
  2. albatross >= "1.5.4"
  3. alg_structs_qcheck
  4. ambient-context
  5. ambient-context-eio
  6. angstrom >= "0.7.0"
  7. ansi >= "0.6.0"
  8. anycache >= "0.7.4"
  9. anycache-async
  10. anycache-lwt
  11. archetype >= "1.4.2"
  12. archi
  13. arp
  14. arrakis < "1.1.0"
  15. art
  16. asak >= "0.2"
  17. asli >= "0.2.0"
  18. asn1-combinators >= "0.2.5"
  19. atd >= "2.3.3"
  20. atdgen >= "2.10.0"
  21. atdpy
  22. atdts
  23. base32
  24. base64 >= "2.1.2" & < "3.2.0" | >= "3.4.0"
  25. bastet
  26. bastet_async
  27. bastet_lwt
  28. bechamel >= "0.5.0"
  29. bigarray-overlap
  30. bigstring >= "0.3"
  31. bigstring-unix
  32. bigstringaf
  33. bitlib
  34. blake2
  35. bloomf
  36. bls12-381 < "0.4.1" | >= "3.0.0" & < "18.0"
  37. bls12-381-hash
  38. bls12-381-js >= "0.4.2"
  39. bls12-381-js-gen >= "0.4.2"
  40. bls12-381-legacy
  41. bls12-381-signature
  42. bls12-381-unix
  43. blurhash
  44. brisk-reconciler
  45. builder-web < "0.2.0"
  46. bytebuffer
  47. ca-certs
  48. ca-certs-nss
  49. cactus
  50. caldav
  51. calendar >= "3.0.0"
  52. callipyge
  53. camlix
  54. camlkit
  55. camlkit-base
  56. capnp-rpc < "1.2.3"
  57. capnp-rpc-unix < "1.2.3"
  58. carray
  59. carton < "1.0.0"
  60. cborl
  61. cf-lwt
  62. chacha
  63. charrua-client
  64. checkseum >= "0.0.3"
  65. cid
  66. clarity-lang
  67. class_group_vdf
  68. cohttp < "6.0.0"
  69. cohttp-curl-async < "6.1.0"
  70. cohttp-eio = "6.0.0~beta2"
  71. colombe >= "0.2.0"
  72. color
  73. conan
  74. conan-cli
  75. conan-database
  76. conan-lwt
  77. conan-unix
  78. conex < "0.10.0"
  79. conex-mirage-crypto
  80. cookie
  81. cow >= "2.2.0"
  82. css
  83. css-parser
  84. cstruct
  85. cstruct-sexp
  86. ctypes-zarith
  87. cuid
  88. curly
  89. current_incr
  90. data-encoding
  91. dates_calc >= "0.0.10"
  92. dbase4
  93. decompress < "1.5.3"
  94. depyt
  95. digestif >= "0.9.0"
  96. dispatch >= "0.4.1"
  97. dkim
  98. dkim-bin
  99. dkim-mirage
  100. dns >= "4.4.1"
  101. dns-cli
  102. dns-client >= "4.6.3"
  103. dns-forward-lwt-unix
  104. dns-resolver
  105. dns-server
  106. dns-tsig
  107. dnssd
  108. dnssec
  109. docfd >= "2.2.0"
  110. domain-name
  111. dream
  112. dream-pure
  113. duff
  114. dune-deps >= "1.4.0"
  115. dune-release >= "1.0.0"
  116. duration
  117. echo
  118. emile
  119. encore
  120. eqaf >= "0.5"
  121. equinoxe
  122. equinoxe-cohttp
  123. equinoxe-hlc
  124. ezgzip
  125. ezjsonm < "1.3.0"
  126. ezjsonm-lwt < "1.3.0"
  127. FPauth
  128. FPauth-core
  129. FPauth-responses
  130. FPauth-strategies
  131. faraday != "0.2.0"
  132. farfadet
  133. fat-filesystem
  134. ff
  135. ff-pbt
  136. fiat-p256
  137. flex-array
  138. forester >= "5.0"
  139. fsevents-lwt
  140. functoria
  141. fungi
  142. geojson
  143. geoml >= "0.1.1"
  144. git < "3.2.0"
  145. git-kv >= "0.2.0"
  146. git-net
  147. git-split
  148. git-unix < "3.2.0"
  149. gitlab-unix
  150. glicko2
  151. gmap
  152. gobba
  153. gpt
  154. graphql
  155. graphql-async
  156. graphql-cohttp >= "0.13.0"
  157. graphql-lwt
  158. graphql_parser != "0.11.0"
  159. graphql_ppx
  160. h1_parser
  161. h2
  162. hacl
  163. hacl_func
  164. hacl_x25519
  165. highlexer
  166. hkdf
  167. hockmd
  168. html_of_jsx
  169. http < "6.0.0"
  170. http-multipart-formdata < "2.0.0"
  171. httpaf >= "0.2.0"
  172. httpun
  173. httpun-ws
  174. hugin
  175. hvsock
  176. icalendar
  177. imagelib
  178. index
  179. inferno >= "20220603"
  180. influxdb-async
  181. influxdb-lwt
  182. inquire < "0.2.0"
  183. interval-map
  184. iomux
  185. irmin < "2.7.1"
  186. irmin-bench
  187. irmin-chunk
  188. irmin-cli
  189. irmin-containers
  190. irmin-fs
  191. irmin-git
  192. irmin-pack
  193. irmin-pack-tools
  194. irmin-test < "3.6.1"
  195. irmin-tezos
  196. irmin-unix
  197. irmin-watcher
  198. jekyll-format
  199. jose
  200. json-data-encoding >= "0.9"
  201. json_decoder
  202. jsonxt
  203. junit_alcotest < "2.1.0"
  204. jwto
  205. kaun
  206. kdf
  207. ke >= "0.2"
  208. kkmarkdown
  209. lambda-runtime
  210. lambda_streams
  211. lambda_streams_async
  212. lambda_streams_lwt
  213. lambdapi
  214. ledgerwallet-tezos >= "0.2.1" & < "0.4.0"
  215. lmdb >= "1.0"
  216. logical
  217. logtk >= "1.6"
  218. lp
  219. lp-glpk
  220. lp-glpk-js < "0.5.0"
  221. lp-gurobi < "0.5.0"
  222. lru
  223. lt-code
  224. luv
  225. mbr-format
  226. mdx >= "1.6.0"
  227. mec
  228. mechaml
  229. merlin = "4.17.1-501"
  230. merlin-lib = "4.17.1-501"
  231. metrics
  232. minicaml = "0.3.1" | >= "0.4"
  233. mirage >= "4.0.0"
  234. mirage-block-partition
  235. mirage-block-ramdisk
  236. mirage-channel >= "4.0.1"
  237. mirage-channel-lwt
  238. mirage-crypto-ec
  239. mirage-flow-unix
  240. mirage-kv >= "2.0.0"
  241. mirage-kv-mem
  242. mirage-kv-unix
  243. mirage-logs
  244. mirage-nat
  245. mirage-net-unix
  246. mirage-runtime < "4.7.0"
  247. mirage-tc
  248. mjson
  249. mmdb
  250. mnd
  251. mrmime >= "0.2.0"
  252. msgpck >= "1.6"
  253. mssql >= "2.0.3"
  254. multibase
  255. multihash
  256. multihash-digestif
  257. multipart-form-data
  258. multipart_form
  259. multipart_form-eio
  260. multipart_form-lwt
  261. named-pipe
  262. nanoid
  263. nbd >= "4.0.3"
  264. nbd-tool
  265. nloge
  266. nocoiner
  267. non_empty_list
  268. nx
  269. nx-datasets
  270. nx-text
  271. OCADml >= "0.6.0"
  272. obatcher
  273. ocaml-index < "5.4.1-503"
  274. ocaml-r >= "0.5.0"
  275. ocaml-version >= "3.5.0"
  276. ocamlformat >= "0.13.0" & < "0.25.1"
  277. ocamlformat-rpc < "removed"
  278. ocamline
  279. ocluster < "0.3.0"
  280. odoc < "2.1.1"
  281. ohex
  282. oidc
  283. opam-0install
  284. opam-0install-cudf >= "0.5.0"
  285. opam-file-format >= "2.1.1"
  286. opentelemetry >= "0.6"
  287. opentelemetry-client-cohttp-lwt >= "0.6"
  288. opentelemetry-client-ocurl >= "0.6"
  289. opentelemetry-cohttp-lwt >= "0.6"
  290. opentelemetry-lwt >= "0.6"
  291. opium
  292. opium-graphql
  293. opium-testing
  294. opium_kernel
  295. orewa
  296. ortac-core
  297. ortac-wrapper
  298. osx-acl
  299. osx-attr
  300. osx-cf
  301. osx-fsevents
  302. osx-membership
  303. osx-mount
  304. osx-xattr
  305. otoggl
  306. owl >= "0.7.0" & != "0.9.0" & != "1.0.0"
  307. owl-base < "0.5.0"
  308. owl-ode >= "0.1.0" & != "0.2.0"
  309. owl-symbolic
  310. passmaker
  311. patch < "3.0.0~alpha2"
  312. pbkdf
  313. pecu >= "0.2"
  314. pf-qubes
  315. pg_query >= "0.9.6"
  316. pgx >= "1.0"
  317. pgx_unix >= "1.0"
  318. pgx_value_core
  319. pgx_value_ptime < "2.2"
  320. phylogenetics
  321. piaf
  322. polyglot
  323. polynomial
  324. ppx_blob >= "0.3.0"
  325. ppx_deriving_cmdliner
  326. ppx_deriving_ezjsonm
  327. ppx_deriving_rpc
  328. ppx_deriving_yaml
  329. ppx_inline_alcotest
  330. ppx_marshal
  331. ppx_protocol_conv >= "5.0.0"
  332. ppx_protocol_conv_json >= "5.0.0"
  333. ppx_protocol_conv_jsonm >= "5.0.0"
  334. ppx_protocol_conv_msgpack >= "5.0.0"
  335. ppx_protocol_conv_xml_light >= "5.0.0"
  336. ppx_protocol_conv_xmlm
  337. ppx_protocol_conv_yaml >= "5.0.0"
  338. ppx_subliner
  339. ppx_units
  340. ppx_yojson >= "1.1.0"
  341. pratter < "3.0.0"
  342. prc
  343. preface
  344. pretty_expressive
  345. prettym
  346. proc-smaps
  347. producer < "0.2.0"
  348. prom
  349. prometheus < "1.2"
  350. prometheus-app
  351. protocell
  352. protocol-9p < "0.11.0" | >= "0.11.2"
  353. protocol-9p-unix
  354. psq
  355. quickjs
  356. quill
  357. randii
  358. reason-standard
  359. red-black-tree
  360. reparse >= "2.0.0" & < "3.0.0"
  361. reparse-unix < "2.1.0"
  362. resp
  363. resp-unix >= "0.10.0"
  364. rfc1951 < "1.0.0"
  365. routes < "2.0.0"
  366. rpc
  367. rpclib
  368. rpclib-async
  369. rpclib-lwt
  370. rpmfile < "0.3.0"
  371. rpmfile-eio
  372. rpmfile-unix
  373. rune
  374. SZXX >= "4.0.0"
  375. salsa20
  376. salsa20-core
  377. sanddb >= "0.2"
  378. scrypt-kdf
  379. secp256k1 >= "0.4.1"
  380. secp256k1-internal
  381. semver >= "0.2.1"
  382. sendmail
  383. sendmail-lwt
  384. sendmail-miou-unix
  385. sendmail-mirage
  386. sendmsg
  387. server-reason-react
  388. session-cookie
  389. session-cookie-async
  390. session-cookie-lwt
  391. sherlodoc
  392. slug
  393. smaws-clients
  394. smaws-lib
  395. sodium-fmt
  396. sowilo
  397. spin >= "0.6.0"
  398. spurs
  399. squirrel
  400. ssh-agent
  401. ssl >= "0.6.0"
  402. stramon-lib
  403. stringx
  404. styled-ppx
  405. swapfs
  406. syslog-rfc5424
  407. tcpip < "7.1.2"
  408. tdigest < "2.1.0"
  409. term-indexing
  410. term-tools
  411. terminal_size >= "0.1.1"
  412. terminus
  413. terminus-cohttp
  414. terminus-hlc
  415. terml
  416. testo
  417. testo-lwt
  418. textrazor
  419. tezos-base-test-helpers < "16.0"
  420. tezos-test-helpers < "12.3"
  421. timedesc
  422. timere
  423. tls >= "0.12.8"
  424. toc
  425. topojson
  426. topojsone
  427. transept
  428. tsort >= "2.2.0"
  429. twostep
  430. type_eq
  431. type_id
  432. typeid >= "1.0.1"
  433. tyre >= "0.4"
  434. tyxml >= "4.2.0"
  435. tyxml-jsx
  436. tyxml-ppx >= "4.3.0"
  437. tyxml-syntax
  438. uecc
  439. ulid
  440. universal-portal
  441. unix-dirent
  442. unix-errno
  443. unix-sys-resource
  444. unix-sys-stat
  445. unix-time
  446. unstrctrd
  447. user-agent-parser
  448. uspf
  449. uspf-lwt
  450. uspf-mirage
  451. uspf-unix
  452. utop >= "2.13.0"
  453. validate
  454. validator
  455. vercel
  456. vhd-format-lwt >= "0.13.0"
  457. vpnkit
  458. wcwidth
  459. websocketaf
  460. x509 >= "0.7.0"
  461. xapi-rrd
  462. xapi-stdext-date
  463. xapi-stdext-encodings
  464. xapi-stdext-std >= "4.16.0"
  465. yaml < "3.2.0"
  466. yaml-sexp
  467. yocaml < "2.0.0"
  468. yocaml_syndication = "2.0.0"
  469. yocaml_yaml < "2.0.0"
  470. yojson >= "1.6.0"
  471. yojson-five
  472. yuscii >= "0.3.0"
  473. zar
  474. zed >= "3.2.2"
  475. zlist < "0.4.0"

Conflicts

None

OCaml

Innovation. Community. Security.

GitHub Discord Twitter Peertube RSS
About
Changelog Releases Industrial Users Academic Users Why OCaml
Ecosystem
Packages Community Events OCaml Planet Jobs
Resources
Install OCaml Get Started Platform Tools Language Manual Standard Library API Books Exercises Papers OCaml Playground Logo
Policies
Carbon Footprint Governance Privacy Code of Conduct