Problem package
This documentation goes over how each field (or group of fields) in problem.rbx.yml
works.
Problem definition
Schema: robox.box.schema.Package
The three main required fields in a problem.rbx.yml
package are:
name
: supports anything matching the regex^[a-zA-Z0-9\-]+$
timeLimit
: the time limit of your problem, in milliseconds.memoryLimit
: the memory limit of your problem, in megabytes.
A barebones package would look something like:
Checker
Field: checker
Schema: robox.box.schema.CodeItem
Checkers are also a very important part of your problem package, and define how the output of a solution will be judged.
The checker is controlled through the top-level parameter checker
, and is optional. In the case it is not specified, robox falls back to testlib's wcmp.cpp, which is a token-based checker that compares the output of the solution with the expected answer.
robox is tightly integrated with testlib, and thus you can either specify:
-
The name of any checker in defined in testlib's checkers folder. The checker will be automatically downloaded when running solutions and building the package.
Tip
If you want to explicitly download a checker from testlib, you can run
rbx download checker [name-of-the-checker]
. -
A custom testlib-based checker of your own, placed in the folder (or any sub-folder) of your package.
Tip
robox automatically places
testlib.h
together with your code when compiling it, but you can explicitly download it withrbx download testlib
if you want.Recommended
This is usually the recommended solution when building a custom checker.
-
A custom checker (not necessarily using testlib). It can even be in other language, in which case we suggest specifying the
language
property.Note
Although this is not a testlib-based checker, we still expect checker programs to follow the same command line structure as testlib, which is receiving and reading three file paths from the command line.
Thus, your checker program will be called as
<program> <input-file> <output-file> <expected-answer-file>
.
Generators
Field: generators
Schema: List[
Generator
]
You can also specify a set of testcase generators that can be re-used in different places in your package (testgroup generation, stress tests to name a couple).
Again, we encourage the use of testlib-based generators, but give you the flexibility to develop however you like.
After implementing a generator, you can define it in problem.rbx.yml
similar to how you define a checker, but you also have to name it to be able to reference it down the line.
Notice also how the generators
field is a list, and as such you can define multiple generators.
Solutions
Field: solutions
Schema: List[
Solution
]
You can specify multiple solutions to your problem, including incorrect ones to check that your testset is strong enough.
You can define them similarly to generators and checkers, but you also have to provide an expected outcome for them.
solutions:
- path: "sols/main.cpp"
outcome: accepted
- path: "sols/slow.cpp"
outcome: tle
- path: "sols/wa.cpp"
outcome: wa
Also, you have to define an accepted solution. The first accepted solution in this list will be considered the main solution to generate answers for the testcases.
For a full list of expected outcomes, see here.
Testcase groups
Field: testcases
Schema: List[
TestcaseGroup
]
You can define multiple testgroups for you problem. For each testgroup, you can define tests for it in five (5) different ways:
-
Specifying a sequence of manually defined testcases present in your package's directory with the
testcases
field.testcases: - name: "group-1" testcases: - inputPath: "manual-tests/1.in" outputPath: "manual-tests/1.out" - inputPath: "manual-tests/1.in"
Note
Notice how the
outputPath
is optional. If it is not defined, the main solution will be used to generate an output. This is the recommended approach. -
Specifying a glob of input paths with the
testcaseGlob
field.- Pick all files ending in
.in
inside thetests/
folder as test inputs for the testgroup. The files are taken in lexicographically increasing order.
- Pick all files ending in
-
Specifying a list of generator calls with the
generators
field. -
Specifying a static generator script with the
generatorScript
field:Note
The
.txt
extension is necessary for robox to identify this is a static script.Recommended
This is usually the recommended approach to generate a few pre-defined testcases. Prefer this over the
generators
call to keepproblem.rbx.yml
cleaner. -
Specifying a dynamic generator script with the
generatorScript
field:Recommended
This is usually the recommended approach to generate multiple random testcases.
Warning
If the platform you package the problem for does not support testgroups, tests will be flattened into a single group, and the tests will be executed in the order the groups were defined.
Samples
You can also specify samples to be included in your statement by defining a testgroup named samples
. This testgroup has to be the first one defined, otherwise an error will be raised.
Variables
Field: vars
Schema: Dict[str, Union[str, int, float, bool]]
In the package definition you can define variables that can be referenced in statements, validators and stress tests.
This is useful to maintain consistency when changing constraints of your problem. A common mistake this field tries to solve is changing constraints in the statement, but not updating them in the validator.
vars:
MAX_N: 10000
MAX_A: 100000
MOD: py`10**9+7` # Backticks force the var to be evaluated as a Python expression.
Note
Variable names should be valid Python identifiers.
Tip
You can see how to include these variables in each of the possible assets in the sections below.
Validators
Field: validator
, testcases.validator
Schema: robox.box.schema.CodeItem
You can specify testlib validators that check whether both manual and generated testcases are in the correct format.
You can specify validators in two places:
-
In the package definition, through the
validator
field. -
In the testgroup definition, through the
validator
field, in which case this validator will be executed only for this group, and the package one will be ignored.
You can pass variables to validators by calling testlib's prepareOpts
function and by using its provided opt
accessor.
#include "testlib.h"
using namespace std;
int main(int argc, char *argv[]) {
registerValidation(argc, argv);
prepareOpts(argc, argv);
int MAX_N = opt<int>("MAX_N"); // Read from package vars.
inf.readInt(1, MAX_N, "N");
inf.readEoln();
inf.readEof();
}
Stress tests
Field: stresses
Schema: List[
Stress
]
You can pre-define stress tests with a few applications in mind, such as:
- Finding testcases that break incorrect solutions;
- Ensuring solutions you expect to be correct cannot be broken.
Let's break down each field in the stress test below:
stresses:
- name: "stress1"
generator:
name: "gen"
args: "[1..1000] @"
solutions: ['wa.cpp']
outcome: incorrect
name
: a name for the stress test. Useful when calling this test throughrbx stress [name]
.generator
: a generator to be repeatedly called in this stress test.name
: the name of the generatorargs
: args pattern to be passed to generator.- You can pass a random integer by writing something like
[1..1000]
. - You can pass variables defined in the
vars
section with something like<MAX_N>
. - You can pass a random choice by writing something like
(a|b|c)
. - You can pass a random hex string by passing
@
.
- You can pass a random integer by writing something like
solutions
: the solutions to stress in this test.outcome
: the outcome to look for while stress testing. Defaults toincorrect
, which captures any wrong solution.