Add additional JobRun stats for blackbox fuzzers#5238
Draft
Add additional JobRun stats for blackbox fuzzers#5238
Conversation
dylanjew
force-pushed
the
dylanj/jobrun-stats
branch
3 times, most recently
from
4c7b0a9 to
3fe165d
Compare
dylanjew
force-pushed
the
dylanj/jobrun-stats
branch
from
3fe165d to
4068b54
Compare
dylanjew
force-pushed
the
dylanj/utask-test
branch
from
cb69ba1 to
ba7b83f
Compare
part 1 of Populate job run stats for blackbox fuzzers e.g. total fuzzing hours, test cases generated, test case generation time, test case executions and test case execution time.
dylanjew
force-pushed
the
dylanj/jobrun-stats
branch
from
4068b54 to
3be8df2
Compare
dylanjew
force-pushed
the
dylanj/jobrun-stats
branch
6 times, most recently
from
03f3f24 to
33ba946
Compare
Clean up comments Fix
dylanjew
force-pushed
the
dylanj/jobrun-stats
branch
from
33ba946 to
2b259ca
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR aims to improve our ability to benchmark blackbox fuzzers effectiveness and monitor their health.
Context
When measuring fuzzer effectiveness, we’re interested in the rate at which the fuzzer can execute testcases and find bugs. This requires a baseline definition of total fuzzing hours across all types of fuzzers.
Some of these stats are already computed for Monarch monitoring in
monitoring_metrics.py, but we'd like to get them into BigQuery to have a consistent data source for analysisChanges
Adds
testcases_generated,testcase_execution_duration,testcase_generation_duration, andfuzzing_durationto the uworkerFuzzTaskOutputproto and write those stats to the JobRun BigQuery table.Notes
This only adds metrics for blackbox fuzzers. We will need to aggregate the fuzzing session hours and execution metrics we already store in the TestcaseRun tables for engine guided fuzzers if we want comparisons across fuzzer types. This either means we need to aggregate those tables and write to the JobRun tables, or do the aggregation in our plx workflows/scripts.
Testing
FuzzTaskOutputproto and written to BigQuerylogs from running a local bot: https://paste.googleplex.com/4551083230887936
local BigQuery stats: https://paste.googleplex.com/6697329928306688
Added unit tests for the stats