Module `Irmin_traces.Trace_stat_summary`Source

Conversion of a Stat_trace to a summary that is both pretty-printable and exportable to JSON.

The main type t here isn't versioned like a Stat_trace.t is.

Computing a summary may take a long time if the input Stat_trace is long. Count ~1000 commits per second.

This file is NOT meant to be used from Tezos, as opposed to some other "trace_*" files.

Sourcemodule Def = Trace_definitions.Stat_trace

Sourcemodule Conf = Trace_stat_summary_conf

Sourcemodule Utils = Trace_stat_summary_utils

Sourcemodule Vs = Utils.Variable_summary

Sourcemodule Seq = Trace_common.Seq

Sourcetype curve = Utils.curve

Sourceval curve_t : Utils.curve Repr.t

Sourcemodule Span : sig ... end

A stat trace can be chunked into blocks. A blocks is made of 2 phases, first the buildup and then the commit.

Sourcemodule Watched_node : sig ... end

Sourcetype bag_stat = {

value_before_commit : Vs.t;
value_after_commit : Vs.t;
diff_per_block : Vs.t;
diff_per_buildup : Vs.t;
diff_per_commit : Vs.t;

}

Summary of an entry contained in Def.bag_of_stat.

Properties of such a variables:

Is sampled before each commit operation.
Is sampled after each commit operation.
Is sampled in header.
Most of these entries are expected to grow linearly, it implies that no smoothing is necessary for the downsampled curve in these cases, and that the histogram is best viewed on a linear scale - as opposed to a log scale. The other entries are summarised using ~is_linearly_increasing:false.

The value_after_commit is initially fed with the value in the header (i.e. the value recorded just before the start of the play).

Sourceval bag_stat_t : bag_stat Repr.t

Sourcetype finds = {

total : bag_stat;
from_staging : bag_stat;
from_lru : bag_stat;
from_pack_direct : bag_stat;
from_pack_indexed : bag_stat;
missing : bag_stat;
cache_miss : bag_stat;

}

Sourceval finds_t : finds Repr.t

Sourcetype pack = {

finds : finds;
appended_hashes : bag_stat;
appended_offsets : bag_stat;
inode_add : bag_stat;
inode_remove : bag_stat;
inode_of_seq : bag_stat;
inode_of_raw : bag_stat;
inode_rec_add : bag_stat;
inode_rec_remove : bag_stat;
inode_to_binv : bag_stat;
inode_decode_bin : bag_stat;
inode_encode_bin : bag_stat;

}

Sourceval pack_t : pack Repr.t

Sourcetype tree = {

contents_hash : bag_stat;
contents_find : bag_stat;
contents_add : bag_stat;
node_hash : bag_stat;
node_mem : bag_stat;
node_add : bag_stat;
node_find : bag_stat;
node_val_v : bag_stat;
node_val_find : bag_stat;
node_val_list : bag_stat;

}

Sourceval tree_t : tree Repr.t

Sourcetype index = {

bytes_read : bag_stat;
nb_reads : bag_stat;
bytes_written : bag_stat;
nb_writes : bag_stat;
bytes_both : bag_stat;
nb_both : bag_stat;
nb_merge : bag_stat;
cumu_data_bytes : bag_stat;
merge_durations : float list;

}

Sourceval index_t : index Repr.t

Sourcetype gc = {

minor_words : bag_stat;
promoted_words : bag_stat;
major_words : bag_stat;
minor_collections : bag_stat;
major_collections : bag_stat;
compactions : bag_stat;
major_heap_bytes : bag_stat;
major_heap_top_bytes : curve;

}

Sourceval gc_t : gc Repr.t

Sourcetype disk = {

index_data : bag_stat;
index_log : bag_stat;
index_log_async : bag_stat;
store_dict : bag_stat;
store_pack : bag_stat;

}

Sourceval disk_t : disk Repr.t

Sourcetype store = {

watched_nodes : Watched_node.map;

}

Sourceval store_t : store Repr.t

Sourcetype t = {

summary_timeofday : float;
summary_hostname : string;
curves_sample_count : int;
moving_average_half_life_ratio : float;
config : Def.config;
hostname : string;
word_size : int;
timeofday : float;
timestamp_wall0 : float;
timestamp_cpu0 : float;
elapsed_wall : float;
elapsed_wall_over_blocks : Utils.curve;
elapsed_cpu : float;
elapsed_cpu_over_blocks : Utils.curve;
op_count : int;
span : Span.map;
block_count : int;
cpu_usage : Vs.t;
index : index;
pack : pack;
tree : tree;
gc : gc;
disk : disk;
store : store;

}

Sourceval t : t Repr.t

Source

val create_vs : 
  int ->
  evolution_smoothing:[ `Ema of float * float | `None ] ->
  scale:[ `Linear | `Log ] ->
  Vs.acc

Sourceval create_vs_exact : int -> Vs.acc

Sourceval create_vs_smooth : int -> Vs.acc

Sourceval create_vs_smooth_log : int -> Vs.acc

Sourcemodule Span_folder : sig ... end

Accumulator for the span field of t.

Sourcemodule Bag_stat_folder : sig ... end

Summary computation for statistics recorded in Def.bag_of_stat.

Sourcemodule Store_watched_nodes_folder : sig ... end

Accumulator for the store field of t.

Source

val major_heap_top_bytes_folder : 
  'a Def.header_base ->
  int ->
  ([> `Commit of 'b Def.commit_base ], Utils.Resample.acc, float list)
    Utils.Parallel_folders.folder

Build a resampled curve of gc.top_heap_words

Source

val elapsed_wall_over_blocks_folder : 
  'a Def.header_base ->
  int ->
  ([> `Commit of 'b Def.commit_base ], Utils.Resample.acc, float list)
    Utils.Parallel_folders.folder

Build a resampled curve of timestamps.

Source

val elapsed_cpu_over_blocks_folder : 
  'a Def.header_base ->
  int ->
  ([> `Commit of 'b Def.commit_base ], Utils.Resample.acc, float list)
    Utils.Parallel_folders.folder

Build a resampled curve of timestamps.

Source

val merge_durations_folder : 
  (Def.pack Def.row_base, float list, float list) Utils.Parallel_folders.folder

Build a list of all the merge durations.

Source

val cpu_usage_folder : 
  'a Def.header_base ->
  int ->
  ([> `Commit of 'b Def.commit_base ], float * float * Vs.acc, Vs.t)
    Utils.Parallel_folders.folder

Source

val misc_stats_folder : 
  'a Def.header_base ->
  ([> `Commit of 'b Def.commit_base ],
    float * float * int,
    float * float * int)
    Utils.Parallel_folders.folder

Substract the first and the last timestamps and count the number of span.

Sourceval summarise' : Def.pack Def.header_base -> int -> Def.row Seq.t -> t

Fold over row_seq and produce the summary.

Parallel Folders

Almost all entries in t require to independently fold over the rows of the stat trace, but we want:

not to fully load the trace in memory,
not to reread the trace from disk once for each entry,
this current file to be verbose and simple,
to have fun with GADTs and avoid mutability.

All the boilerplate is hidden behind Utils.Parallel_folders, a datastructure that holds all folder functions, takes care of feeding the rows to those folders, and preseves the types.

In the code below, pf0 is the initial parallel folder, before the first accumulation. Each |+ ... statement declares a acc, accumulate, finalise triplet, i.e. a folder.

val acc : acc is the initial empty accumulation of a folder.

val accumulate : acc -> row -> acc needs to be folded over all rows of the stat trace. Calling Parallel_folders.accumulate pf row will feed row to every folders.

val finalise : acc -> v has to be applied on the final acc of a folder in order to produce the final value of that folder - which value is meant to be stored in Trace_stat_summary.t. Calling Parallel_folders.finalise pf will finalise all folders and pass their result to construct.

Sourceval summarise : ?block_count:int -> string -> t

Turn a stat trace into a summary.

The number of blocks to consider may be provided in order to truncate the summary.

Sourceval save_to_json : t -> string -> unit

Install

Dune Dependency

Authors

Maintainers

Sources

doc/irmin-bench.traces/Irmin_traces/Trace_stat_summary/index.html

Module `Irmin_traces.Trace_stat_summary`Source

package irmin-bench

Install

Dune Dependency

Authors

Maintainers

Sources

doc/irmin-bench.traces/Irmin_traces/Trace_stat_summary/index.html

Module Irmin_traces.Trace_stat_summarySource

Module `Irmin_traces.Trace_stat_summary`Source