Development Guide
Overview
Sentinel uses a combination of the PDM package/virtual environment manager and DoIt task runnner for development.
PDM and DoIt complement each other:
In addition to managing packages and virtual environments, PDM leverages the
[tool.pdm.scripts]table ofpyproject.tomlto create shortcuts for running arbitrary commands/save typing. However, PDM by itself does not manage dependency/up-to-date information, and the command strings you can create are limited.DoIt, on the other hand, allows one to run arbitrary commands of arbitrary complexity, and manage dependency up-to-date information via a
dodo.pyfile and database files. However, DoIt by itself knows nothing about package management.
I leverage both PDM and DoIt for increased flexibility to run tests, benchmarks, and generate examples.
PDM Scripts
Scripts defined in [tool.pdm.scripts] are the main
entry points for users and developers. Type pdm run --list for a list of
scripts. Example output is below:
pdm run --list
╭─────────────────────┬───────────┬─────────────────────────────────────────────────────────────────────────╮
│ Name │ Type │ Description │
├─────────────────────┼───────────┼─────────────────────────────────────────────────────────────────────────┤
│ bench-luts │ cmd │ add stats to LUTs.csv │
│ compile-upstream │ cmd │ regnerate riscv-test binaries │
│ demo │ cmd │ create AttoSoC Sentinel demo bitstream │
│ demo-rust │ composite │ create AttoSoC Sentinel demo bitstream (Rust version) │
│ doc │ cmd │ build documentation │
│ doc-auto │ cmd │ sphinx-autobuild doc/ doc/_build/ --watch src/sentinel --watch examples │
│ doc-linkck │ cmd │ sphinx-build doc/ doc/_linkcheck/ -b linkcheck │
│ doc-test │ cmd │ sphinx-build -b doctest doc/ doc/_build │
│ doit │ cmd │ escape hatch to call doit directly │
│ gen │ call │ generate Sentinel Verilog file │
│ lint │ composite │ lint Python and Rust sources │
│ lint-fix │ composite │ fix automatically-fixable lints │
│ plot-luts │ cmd │ plot LUTs.csv │
│ riscof-all │ cmd │ run all RISCOF tests │
│ riscof-override │ cmd │ run RISCOF with custom testfile │
│ rvformal │ cmd │ run a single RISC-V Formal test │
│ rvformal-all │ composite │ run all RISC-V Formal tests │
│ rvformal-force │ composite │ force-run a single RISC-V Formal test │
│ rvformal-status │ cmd │ list a single RISC-V Formal test's status │
│ rvformal-status-all │ cmd │ list all RISC-V Formal tests' status │
│ test │ cmd │ run all pytest tests │
│ test-quick │ cmd │ run quick subset of pytest tests │
│ ucode │ cmd │ generate supplementary microcode files │
│ use-local │ cmd │ revert "use-yowasp" changes; use local binaries │
│ use-yowasp │ cmd │ prepare pdm and tools to use YoWASP binaries │
╰─────────────────────┴───────────┴─────────────────────────────────────────────────────────────────────────╯
Todo
Use sphinxcontrib.programoutput
to generate an up-to-date list of commands?
Not all scripts listed above are commonly used. In particular, the following scripts must pass as part of CI:
lint: Lint usingruff. Tested on releases only.gen,demo,demo-rust: Check that code/demo generation works. Tested with both YoWASP and OSS CAD Suite. I test the demos on:Lattice iCEstick (not required to pass, because the demo doesn’t always fit!)
test-quick,rvformal-all, andriscof-all: Run tests. Tested with OSS CAD Suite. In particular, I have issues with testing RISCOF on Windows, so CI is Linux only for now.doc,doc-test: Check that docs build and doc tests pass. Tested on releases only, mainly as a lint since ReadTheDocs builds docs automatically.
If necessary, the above PDM scripts invoke doit, which reads the dodo.py
file to find out how to do the actual work.[1]
DoIt Tasks
doit tasks are “low-level” tasks wrapped by the PDM scripts above.
They should be treated as a private and subject to change. Howevever, I provide
a doit PDM script to call doit directly if necessary. For instance, to
list available doit tasks (including private tasks),
run pdm doit list -p:
pdm doit list -p
_build_sail build SAIL RISC-V emulators in opam environment, compress
_clean_dut_ref_dirs remove dut/ref directories from last RISCOF run
_compile_rust_firmware compile Rust firmware and show size
_decompress_sail decompress previously-built SAIL emulator
_demo create a demo bitstream (for benchmarking)
_formal_gen_files copy Sentinel files and run RISC-V Formal's genchecks.py script
_formal_gen_sentinel generate Sentinel subdir and Verilog in RISC-V Formal cores dir
_git_init initialize git submodules, "doit list --all -p _git_init" for choices
_git_rev get git revision
_make_rand_firmware create a baseline gateware for firmware development
_opam extract environment vars from opam
_program_rust_firmware load Rust firmware image onto FPGA
_replace_rust_firmware replace random firmware image inside baseline gateware with Rust program
_riscof_gen run RISCOF's testlist command to prepare RISCOF files and directories
bench_luts build "pdm demo" bitstream (if out of date), record LUT usage using LogLUTs
compile_upstream compile riscv-tests tests to ELF, "doit list --all compile_upstream" for choices
list_sby_status list "run_sby" subtasks' status, "doit list --all list_sby_status" for choices
plot_luts build "pdm demo" bitstream (if out of date), plot LUT usage using LogLUTs
run_riscof run RISCOF tests against Sentinel/Sail, and report results, removes previous run's artifacts
run_sby run symbiyosys flow on Sentinel, "doit list --all run_sby" for choices
ucode assemble microcode and copy non-bin artifacts to root
I’ve documented the doit tasks (but not subtasks) as a courtesy, and to make sure
developers/users don’t get stuck. That said, prefer running pdm as a
wrapper to doit rather than running doit directly.
Note
I make heavy use of DoIt subtasks.
These are understandably excluded from default help output. Run
pdm doit list --all [task] to see all sub-tasks for a higher-level task.
For instance, the run_sby:reg_ch0 subtask runs the Register Check
for riscv-formal.
Todo
Decide whether to commit to treat all doit tasks as private/hidden or expose a set
to a user/developer (no leading underscore). Right now, I am not being consistent.
For instance, _build_sail is a command a developer might manually run in rare
cases, but is still currently private b/c of its niche use-case. Where’s the
cutoff for marking private vs public?
Todo
Use sphinxcontrib.programoutput
to generate an up-to-date list of commands?
Detailed Reference
Using PDM and DoIt to orchestrate changes, hacking on Sentinel can be divided into roughly four areas:
Internals
Microcode
Support Code
Test Code
These, plus development guidelines, all have their own sections: