General
-
Implement automatic multiple executions for benchmarks tests
We will need the benchmarks to be run several (sometimes hundreds or thousands of times).
Clicking NUnit's Run button for each of these iterations is unrealistic.
Will there be any way of controlling how many times the benchmark is run?
If so is there a timeframe that we can expect this feature to be introduced?14 votesunder review ·
AdminNinjaCross
(Admin, FireBenchmarks) responded
There is already and attribute (RepeatAttribute) in NUnit that allows to execute a test multiple times, but unfortunately it doesn’t work with FireBenchmarks, since it just run the body of the test without firing the events of the EventsListener interface (the connection between FireBenchmarks and the NUnit core).
Considering the described requisites, a custom runner would be the perfect solution since you can control all the execution process.
Maybe in a future release a custom runner will be implemented, but at the moment it’s not yet in the development roadmap. -
5 votesplanned ·AdminNinjaCross
(Admin, FireBenchmarks) responded
This feature is planned for release 1.2.0.0
-
Provide a callback mechanism to enable recording of code blocks within a test
Consider the scenario where I want to do performance testing for components that drive UI interaction. For example, I need to ensure that all my response times on things like dropdown box population or text highlighting happen within 100 milliseconds.
To test that, I'd like to create a unit test method where I have some setup and cleanup, and I want to collect the performance data just on the specific function call that does the UI interaction. Getting the runtime for the whole test isn't useful, because the setup and cleanup will likely dominate the <100ms for the UI interaction… more
3 votes -
External specification of OutputTarget
make it possible to specify OutputTarget outside the tests. For instance in an external config file.
That way the build server could decide where to put these output files.
2 votes -
Initialize the assembly before running the actual performance tests
Not sure how you would do this in the context of the NUnit framework, but one of the problems I notice is that when you first run the test (after say adding a test), the execution time is much longer than subsequent runs. This skews the results.
1 voteunder review ·AdminNinjaCross
(Admin, FireBenchmarks) responded
Interesting observation. Could you please provide some kind of comparision timings?
