Data parallelism with Rayon
Threads showed how to split work across threads by hand. Rayon does the splitting for you. Its docs call it “a data-parallelism library that makes it easy to convert sequential computations into parallel”, and say it “guarantees data-race free executions and takes advantage of parallelism when sensible, based on work-load at runtime”.
RememberChange iter() to par_iter() and the closures run on a pool of threads. Whatever they share has to be Sync, and the compiler checks that.
parallel-sums/ ├── src/ │ └── main.rs the program on this page ├── Cargo.lock created by Cargo └── Cargo.toml dependencies go here
[package]name = "parallel-sums"version = "0.1.0"edition = "2024"
[dependencies]rayon = "1.12.0"use rayon::prelude::*;
fn main() { let numbers: Vec<u64> = (1..=1_000_000).collect();
// The same pipeline twice: only `iter` became `par_iter`. let sequential: u64 = numbers.iter().map(|n| n * n).sum(); let parallel: u64 = numbers.par_iter().map(|n| n * n).sum(); println!("sum of squares: {sequential} and {parallel}"); assert_eq!(sequential, parallel);
// `collect` into a Vec keeps the input order, even for a million items. let squares: Vec<u64> = numbers.par_iter().map(|n| n * n).collect(); let expected: Vec<u64> = numbers.iter().map(|n| n * n).collect(); println!("collected squares in input order: {}", squares == expected);
// `filter` then `count`. let sevens = numbers.par_iter().filter(|n| **n % 7 == 0).count(); println!("multiples of 7: {sevens}");
// `par_iter_mut` changes the items in place. let mut data = vec![3, 1, 2]; data.par_iter_mut().for_each(|n| *n *= 10); println!("times ten, in place: {data:?}");
// `par_sort` sorts a slice or Vec using several threads. data.par_sort(); println!("sorted: {data:?}");}sum of squares: 333333833333500000 and 333333833333500000 collected squares in input order: true multiples of 7: 142857 times ten, in place: [30, 10, 20] sorted: [10, 20, 30]
package main
import ( "fmt" "runtime" "sort" "sync")
// chunks splits 0..n into one range per CPU and runs fn on each range in its own goroutine.func chunks(n int, fn func(part, lo, hi int)) { parts := runtime.NumCPU() var wg sync.WaitGroup for p := 0; p < parts; p++ { lo, hi := n*p/parts, n*(p+1)/parts wg.Add(1) go func() { defer wg.Done() fn(p, lo, hi) }() } wg.Wait()}
func main() { numbers := make([]uint64, 1_000_000) for i := range numbers { numbers[i] = uint64(i + 1) }
// Sum of squares: each goroutine fills its own slot, then we add the slots. partial := make([]uint64, runtime.NumCPU()) chunks(len(numbers), func(part, lo, hi int) { for _, n := range numbers[lo:hi] { partial[part] += n * n } }) var total uint64 for _, s := range partial { total += s } fmt.Println("sum of squares:", total)
// Squares in input order: each goroutine writes only its own indexes. squares := make([]uint64, len(numbers)) chunks(len(numbers), func(_, lo, hi int) { for i := lo; i < hi; i++ { squares[i] = numbers[i] * numbers[i] } }) inOrder := true for i, s := range squares { inOrder = inOrder && s == uint64(i+1)*uint64(i+1) } fmt.Println("collected squares in input order:", inOrder)
// Multiples of 7: one counter per chunk again. counts := make([]int, runtime.NumCPU()) chunks(len(numbers), func(part, lo, hi int) { for _, n := range numbers[lo:hi] { if n%7 == 0 { counts[part]++ } } }) sevens := 0 for _, c := range counts { sevens += c } fmt.Println("multiples of 7:", sevens)
// In place. data := []int{3, 1, 2} chunks(len(data), func(_, lo, hi int) { for i := lo; i < hi; i++ { data[i] *= 10 } }) fmt.Println("times ten, in place:", data)
// sort.Ints is an ordinary sequential sort. sort.Ints(data) fmt.Println("sorted:", data)}sum of squares: 333333833333500000 collected squares in input order: true multiples of 7: 142857 times ten, in place: [30 10 20] sorted: [10 20 30]
par_iter(),par_iter_mut()andinto_par_iter()are the parallel versions ofiter(),iter_mut()andinto_iter(). They come fromuse rayon::prelude::*. After that you chainmap,filter,sum,countandreduceas in Iterators. The program checks that the parallel sum equals the sequential one.collectinto aVecgave the squares in input order: the program compared a million of them with the sequential result.for_eachruns the closure on several threads, so do not count on any order there (see Sharing below).par_sortsorts a slice orVecon several threads. The docs say it is stable, likesort, and that “when applicable, unstable sorting is preferred because it is generally faster than stable sorting and it doesn’t allocate auxiliary memory” (par_sort_unstable).- The Go program cuts the ranges, starts the goroutines and adds up the pieces by hand. That is the work Rayon does for you.
The pool, join and scope
Section titled “The pool, join and scope”Rayon starts a global pool of worker threads the first time you use it. Its docs say the default size is RAYON_NUM_THREADS if that is set, and otherwise the number of logical CPUs, and they warn that this default may change. If your code relies on a number, set the number. This image reported one thread per CPU, and the variable overrode it:
$ cargo run the pool has at least one thread: true the pool has one thread per CPU: true $ RAYON_NUM_THREADS=2 cargo run RAYON_NUM_THREADS=2, the pool has 2 threads
ThreadPoolBuilder sets the size of the global pool (once, before anything uses it) or builds a pool of your own. Inside pool.install(...) the parallel calls use that pool:
use rayon::prelude::*;
fn main() { // Configure the global pool once, before anything uses it. let first = rayon::ThreadPoolBuilder::new().num_threads(2).build_global(); println!("first build_global: {first:?}"); println!("global pool threads: {}", rayon::current_num_threads()); match rayon::ThreadPoolBuilder::new().num_threads(4).build_global() { Ok(()) => println!("second build_global worked"), Err(e) => println!("second build_global: {e}"), }
// A pool of your own: `install` runs the closure inside it. let pool = rayon::ThreadPoolBuilder::new().num_threads(3).build().unwrap(); let inside = pool.install(|| rayon::current_num_threads()); let total: u64 = pool.install(|| (1..=1_000u64).into_par_iter().sum()); println!("inside install: {inside} threads, sum {total}"); println!("outside install: still {} threads", rayon::current_num_threads());}first build_global: Ok(()) global pool threads: 2 second build_global: The global thread pool has already been initialized. inside install: 3 threads, sum 500500 outside install: still 2 threads
join(a, b) runs two closures, and scope lets tasks borrow local variables, like thread::scope. The docs say Rayon “attempts to only execute code in parallel when there are idle CPUs to handle it”. The cutoff in sum (it adds directly once a piece has 1,000 items or fewer) is there because a task per element would likely cost more than the addition itself. The timing section below shows what a small input costs:
use rayon::prelude::*;
// Split the slice in two and let `join` run both halves, possibly on two threads.// Below a cutoff, just add: a task per element would likely cost more than the addition.fn sum(numbers: &[u64]) -> u64 { if numbers.len() <= 1_000 { return numbers.iter().sum(); } let (left, right) = numbers.split_at(numbers.len() / 2); let (a, b) = rayon::join(|| sum(left), || sum(right)); a + b}
fn main() { let numbers: Vec<u64> = (1..=100_000).collect(); println!("join: {}", sum(&numbers)); println!("par_iter: {}", numbers.par_iter().sum::<u64>());
// `scope` lets tasks borrow local variables, like `thread::scope`. // Each task gets its own slot of `results`. let mut results = vec![0u64; 4]; rayon::scope(|s| { for (i, slot) in results.iter_mut().enumerate() { s.spawn(move |_| *slot = (i as u64 + 1) * 100); } }); println!("scope: {results:?}");}join: 5000050000 par_iter: 5000050000 scope: [100, 200, 300, 400]
Sharing, panics and grouping
Section titled “Sharing, panics and grouping”Closures that all write to one place need a lock or an atomic (Shared State). Two of the five ways below need no shared state at all, which is the better shape: each piece returns a value and Rayon combines them (sum, or fold then reduce). The mutex works, but only one closure at a time can be inside it. Pushing into a shared Vec keeps every item, but promises no order, so the program sorts before comparing:
use rayon::prelude::*;use std::sync::Mutex;use std::sync::atomic::{AtomicU64, Ordering};
fn main() { let numbers: Vec<u64> = (1..=100_000).collect();
// 1. No shared state: each piece returns a value and rayon combines them. let by_sum: u64 = numbers.par_iter().sum();
// 2. One atomic counter that every closure adds to. let atomic = AtomicU64::new(0); numbers.par_iter().for_each(|n| { atomic.fetch_add(*n, Ordering::Relaxed); });
// 3. A mutex: correct, but only one closure at a time can be inside it. let locked = Mutex::new(0u64); numbers.par_iter().for_each(|n| *locked.lock().unwrap() += n);
println!("sum {by_sum}, atomic {}, mutex {}", atomic.into_inner(), locked.into_inner().unwrap());
// 4. Pushing into a shared Vec: all items arrive, but in no promised order. let pushed = Mutex::new(Vec::new()); numbers.par_iter().for_each(|n| pushed.lock().unwrap().push(*n)); let mut pushed = pushed.into_inner().unwrap(); pushed.sort(); println!("same items after sorting: {}", pushed == numbers);
// 5. Per-piece state: `fold` gives each piece its own counts, `reduce` merges them. let by_remainder = numbers .par_iter() .fold( || [0u64; 3], |mut counts, n| { counts[(*n % 3) as usize] += 1; counts }, ) .reduce( || [0u64; 3], |a, b| [a[0] + b[0], a[1] + b[1], a[2] + b[2]], ); println!("remainders 0, 1, 2: {by_remainder:?}");}sum 5000050000, atomic 5000050000, mutex 5000050000 same items after sorting: true remainders 0, 1, 2: [33333, 33334, 33333]
A panic in a closure travels back to the code that called Rayon, here caught with catch_unwind (I silenced the default panic message so the output is the same every time). The pool kept working afterwards. The join and scope docs describe the same propagation, and say that if several closures panic, which panic value you get is not fixed for scope:
use rayon::prelude::*;use std::panic::catch_unwind;
fn main() { // Keep stderr quiet so the output is the same on every run. std::panic::set_hook(Box::new(|_| {}));
let result = catch_unwind(|| { (1..=1_000).into_par_iter().for_each(|n| { if n == 500 { panic!("bad item {n}"); } }); }); match result { Ok(()) => println!("no panic reached the caller"), Err(payload) => { let message = payload.downcast_ref::<String>().unwrap(); println!("the caller got the panic: {message}"); } }
// The pool is still usable afterwards. let total: u32 = (1..=10).into_par_iter().sum(); println!("a later par_iter: {total}");}the caller got the panic: bad item 500 a later par_iter: 55
reduce and sum may group the items any way they like. The reduce docs say that “the order in which op will be applied to reduce the result is not fully specified. So op should be associative or else the results will be non-deterministic”. The sum docs say the same about floating-point +. Subtraction is not associative, so the last line depends on how the thousand numbers were split. It can differ from run to run and from machine to machine; in this run the two results were different:
use rayon::prelude::*;
fn main() { // The same operator, grouped two ways, by hand. println!("(10 - 3) - 2 = {}", (10 - 3) - 2); println!("10 - (3 - 2) = {}", 10 - (3 - 2)); println!("(0.1 + 0.2) + 0.3 = {}", (0.1 + 0.2) + 0.3); println!("0.1 + (0.2 + 0.3) = {}", 0.1 + (0.2 + 0.3));
// `-` is not associative, so how rayon groups the items changes the answer. let numbers: Vec<i64> = (1..=1_000).collect(); let sequential = numbers.iter().fold(0, |a, b| a - b); let parallel = numbers.par_iter().cloned().reduce(|| 0, |a, b| a - b); println!("sequential {sequential}, parallel {parallel}");}(10 - 3) - 2 = 5 10 - (3 - 2) = 9 (0.1 + 0.2) + 0.3 = 0.6000000000000001 0.1 + (0.2 + 0.3) = 0.6 sequential -500500, parallel -994
This output changes from run to run. This is one real run.
When it is not worth it
Section titled “When it is not worth it”Parallel is not automatically faster: handing work to other threads has a cost, and for a small input that cost can be larger than the work. The join docs also assume the closures are CPU-bound: if one waits on the network, “the overall performance may be poor”. So: big enough CPU-bound work, and no waiting inside the closures. For async code see Rayon with Tokio, which builds on Async with Tokio.
This is the only timing on the page. It is a release build, after a warm-up call so the pool is already running, and it is one run on this VM, 8 cores (colima), varies. In this run the parallel version printed the larger time for 10 items and the smaller for a million. The program asserts that both give the same total. It says nothing about where the break-even is for your work, so measure yours:
use rayon::prelude::*;use std::time::Instant;
// The number of steps the Collatz process takes to reach 1: different work for every n.fn collatz_steps(mut n: u64) -> u64 { let mut steps = 0; while n != 1 { n = if n % 2 == 0 { n / 2 } else { 3 * n + 1 }; steps += 1; } steps}
fn compare(label: &str, count: u64) { let start = Instant::now(); let a: u64 = (1..=count).map(collatz_steps).sum(); let sequential = start.elapsed();
let start = Instant::now(); let b: u64 = (1..=count).into_par_iter().map(collatz_steps).sum(); let parallel = start.elapsed();
assert_eq!(a, b); println!("{label}: {count} items, sequential {sequential:?}, parallel {parallel:?}");}
fn main() { // Start the pool first, so its start-up is not inside the first measurement. let warm: u64 = (0..1_000u64).into_par_iter().sum(); assert_eq!(warm, 499_500); let cpus = std::thread::available_parallelism().unwrap().get(); println!("CPUs reported: {cpus}, rayon threads: {}", rayon::current_num_threads()); compare("small", 10); compare("large", 1_000_000);}CPUs reported: 8, rayon threads: 8 small: 10 items, sequential 209ns, parallel 107.968µs large: 1000000 items, sequential 91.046434ms, parallel 12.998489ms
This output changes from run to run. This is one real run.
The same job in your language
Section titled “The same job in your language”I wrote the same job, the sum of squares of 1 to 1,000,000, the way each language usually spreads work over cores, and ran it in the image. Every row printed the same total.
| Language | What you reach for | What the run showed |
|---|---|---|
| Rust | par_iter() from Rayon |
the program above |
| Go | goroutines and a WaitGroup; you cut the chunks |
the Go program above |
| Python | concurrent.futures.ProcessPoolExecutor: separate processes, because the multiprocessing docs say it side-steps the GIL by using subprocesses |
same total |
| TypeScript | worker_threads, passing messages; the total is above 253, so it uses BigInt |
same total |
| Java | .parallel() on a stream |
same total |
| Kotlin | the same Java stream (kotlinx.coroutines not run) |
same total |
| C# | PLINQ: AsParallel() |
same total |
| C | OpenMP: #pragma omp parallel for reduction(+ : total), built with gcc -fopenmp |
same total |
| C++ | OpenMP the same way. C++17 std::execution::par also compiled; the libstdc++ manual says it depends on Intel TBB, which this image does not install, and the lambda ran on 1 thread |
same total, and 1 thread |
$ go run sum.go # Go, goroutines and a WaitGroup 333333833333500000 $ java Sum # Java, parallel stream 333333833333500000 $ java -jar sum.jar # Kotlin, the same Java stream 333333833333500000 $ dotnet run # C#, PLINQ 333333833333500000 $ python3 sum.py # Python, ProcessPoolExecutor 333333833333500000 $ node sum.ts # TypeScript, worker_threads 333333833333500000 $ ./sum_c # C, OpenMP 333333833333500000 $ ./sum_cpp # C++, OpenMP 333333833333500000 $ ./par_cpp # C++17, std::execution::par 333333833333500000, threads that ran the lambda: 1
Sharing a counter between parallel closures
Looks fine: in TypeScript it runs but gives a wrong answer; in Go, Kotlin, Python, Java, C#, C and C++ it runs but has a data race. Rust won't compile it.
Broken in Go. Rust catches it.Broken in Kotlin. Rust catches it.Broken in Python. Rust catches it.Broken in TypeScript. Rust catches it.Broken in Java. Rust catches it.Broken in C#. Rust catches it.Broken in C. Rust catches it.Broken in C++. Rust catches it.use rayon::prelude::*;use std::cell::RefCell;use std::rc::Rc;
fn main() { let total = Rc::new(RefCell::new(0u64)); (1..=1_000_000u64).into_par_iter().for_each(|n| { *total.borrow_mut() += n; }); println!("{}", total.borrow());}package main
import ( "fmt" "runtime" "sync")
func main() { const n = 1_000_000 parts := runtime.NumCPU() total := 0 // shared by every goroutine var wg sync.WaitGroup for p := 0; p < parts; p++ { lo, hi := n*p/parts+1, n*(p+1)/parts wg.Add(1) go func() { defer wg.Done() for i := lo; i <= hi; i++ { total += i } }() } wg.Wait() fmt.Println("total:", total, "expected: 500000500000")}total: 252799198644 expected: 500000500000
The output changes from run to run. This is one real run.
import java.util.stream.LongStream
fun main() { val total = LongArray(1) // shared by every worker thread LongStream.rangeClosed(1, 1_000_000).parallel().forEach { n -> total[0] += n } println("total: ${total[0]} expected: 500000500000")}total: 61748989258 expected: 500000500000
The output changes from run to run. This is one real run.
import threading
N, THREADS = 8_000_000, 8 # many numbers, so each thread runs long enough to be interruptedtotal = 0 # shared by every thread
def term(i): return i
def add(lo, hi): global total for i in range(lo, hi): total = total + term(i) # read total, call a function, write total
step = N // THREADSthreads = [threading.Thread(target=add, args=(p * step + 1, (p + 1) * step + 1)) for p in range(THREADS)]for t in threads: t.start()for t in threads: t.join()print("total:", total, "expected:", N * (N + 1) // 2)total: 25424181629492 expected: 32000004000000
The output changes from run to run. This is one real run.
import { Worker, isMainThread, workerData, parentPort } from "node:worker_threads";
let total = 0; // each worker thread gets its own copy of this variable
if (isMainThread) { const workers = []; for (let p = 0; p < 8; p++) { const lo = p * 125_000 + 1; workers.push(new Promise<void>((done) => { const w = new Worker(new URL(import.meta.url), { workerData: { lo, hi: lo + 125_000 } }); w.on("exit", () => done()); })); } await Promise.all(workers); console.log("total:", total, "expected: 500000500000");} else { for (let i = workerData.lo; i < workerData.hi; i++) total += i; parentPort!.postMessage("done");}total: 0 expected: 500000500000
import java.util.stream.LongStream;
public class Race { public static void main(String[] args) { long[] total = {0}; // shared by every worker thread LongStream.rangeClosed(1, 1_000_000).parallel().forEach(n -> total[0] += n); System.out.println("total: " + total[0] + " expected: 500000500000"); }}total: 76116216758 expected: 500000500000
The output changes from run to run. This is one real run.
using System;using System.Threading.Tasks;
class Race{ static void Main() { long total = 0; // shared by every worker thread Parallel.For(1, 1_000_001, n => { total += n; }); Console.WriteLine($"total: {total} expected: 500000500000"); }}total: 97375425938 expected: 500000500000
The output changes from run to run. This is one real run.
#include <stdio.h>
int main(void) { long long total = 0; /* shared by every thread */ #pragma omp parallel for for (long long i = 1; i <= 1000000; i++) { total += i; } printf("total: %lld expected: 500000500000\n", total); return 0;}total: 100845465344 expected: 500000500000
The output changes from run to run. This is one real run.
#include <iostream>
int main() { long long total = 0; // shared by every thread #pragma omp parallel for for (long long i = 1; i <= 1000000; i++) { total += i; } std::cout << "total: " << total << " expected: 500000500000\n";}total: 64842989217 expected: 500000500000
The output changes from run to run. This is one real run.
What does the compiler say? Guess first, then open.
error[E0277]: `Rc<RefCell<u64>>` cannot be shared between threads safely --> src/main.rs:7:49 | 7 | (1..=1_000_000u64).into_par_iter().for_each(|n| { | -------- ^-- | | | | ________________________________________|________within this `{closure@src/main.rs:7:49: 7:52}` | | | | | required by a bound introduced by this call 8 | | *total.borrow_mut() += n; 9 | | }); | |_____^ `Rc<RefCell<u64>>` cannot be shared between threads safely | = help: within `{closure@src/main.rs:7:49: 7:52}`, the trait `Sync` is not implemented for `Rc<RefCell<u64>>` = note: required because it appears within the type `&Rc<RefCell<u64>>` note: required because it's used within this closure --> src/main.rs:7:49 | 7 | (1..=1_000_000u64).into_par_iter().for_each(|n| { | ^^^ ... (notes pointing into library source code trimmed)
Go compiles and runs this. One goroutine per CPU (eight on this VM) adds into one int with no lock, and total += i is a read, an add and a write, so updates get lost. The run printed a total below the expected one, and go run -race reports a data race in this program.
This uses the same Java stream as the Java version and fails the same way (kotlinx.coroutines was not run). The run printed a total below the expected one.
Python runs this. The global interpreter lock keeps the interpreter's own data consistent, but total = total + term(i) is still a read, a call and a write, so updates get lost. Whether it loses updates depends on how long each thread runs before the interpreter switches threads, so it depends on the size: the session under the gotcha runs this loop five times at each of three sizes. This tab uses 8,000,000 numbers, and its run printed a total below the expected one. Not guaranteed either way (Shared State has the details).
Node runs this, and there is no race and no sum either. Each worker thread gets its own copy of total, so the main thread printed 0. Workers can share memory by sharing a SharedArrayBuffer or by transferring an ArrayBuffer, as the Node docs say. A plain variable like total is not shared.
Java compiles and runs this. A parallel stream calls the lambda on several threads, and total[0] += n is a read, an add and a write, so updates get lost. The run printed a total below the expected one.
C# compiles and runs this. Parallel.For runs the body on several threads and total += n is not atomic (Interlocked.Add exists for this). The run printed a total below the expected one.
gcc compiles this with -fopenmp. Every thread updates total. Two sources apply, and they say different things. The OpenMP 5.2 memory model (section 1.4.1) says "If a data race occurs then the result of the program is unspecified". The C11 standard (section 5.1.2.4), for threads as the C standard defines them, says a data race is undefined behavior. Either way nothing promises the result. The run printed a total below the expected one. A reduction(+ : total) clause is the fix, as in the table below.
g++ compiles this with -fopenmp. Every thread updates total. Two sources apply, and they say different things. The OpenMP 5.2 memory model (section 1.4.1) says "If a data race occurs then the result of the program is unspecified". The C++ standard (intro.races) says a data race is undefined behavior, for the threads it defines. Either way nothing promises the result. The run printed a total below the expected one.
for_each calls your closure from several threads at once, so the closure has to be Sync and Send. This one borrows an Rc<RefCell<u64>>, and the error says Sync is not implemented for it: Rc counts its owners without atomics, and RefCell checks borrows without any locking.
The compiler is refusing a data race before it can happen. This is the check from Send and Sync applied to rayon's closures. The fix is usually to stop sharing: let each piece return a value and let rayon combine them (sum, reduce). If you do need shared state, use an atomic or an Arc<Mutex<T>> (Shared State).
The fix
Compiles and runs.
use rayon::prelude::*;
fn main() { let total: u64 = (1..=1_000_000u64).into_par_iter().sum(); println!("{total}");}500000500000
The Python tab above says whether the race shows up depends on the size of the loop. This session runs the same program five times at each of three sizes and prints exact when the total was right and lost when updates were lost. It is a count over five runs here, not a rule about Python. In the run when this page was written, 1,000,000 numbers gave the exact total every time, 2,000,000 lost updates in some runs and not in others, and 8,000,000 lost updates every time. The counts change from run to run; earlier checks of this page showed the same pattern.
$ for n in 1000000 2000000 8000000; do ...; done # race.py with N from the command line, 5 runs each
1000000 numbers: exact exact exact exact exact
2000000 numbers: exact lost lost exact lost
8000000 numbers: lost lost lost lost lost
This output changes from run to run. This is one real run.
Read more
Section titled “Read more”- Rayon on docs.rs (the version this page pins), with
ParallelIterator,ThreadPoolBuilder,joinandscope. - The Rayon FAQ: how many threads it starts and how work stealing balances the load.
- In this course: Threads, Shared State and Send and Sync.
- The Rust Book: Extensible concurrency with
SyncandSend. - Other languages: Java’s
java.util.stream(parallelism section), Introduction to PLINQ, Node’sworker_threads, Python’sconcurrent.futures, libstdc++ implementation status (see the note on the Parallel Algorithms).
The version this page is about is rayon 1.12.0, as pinned in Cargo.lock.
Learn Rust is an independent project, not affiliated with the Rust Project or the Rust Foundation. Code and text are under the MIT licence.Source on GitHub.