Accuracy

Accuracy is the quality of your verdicts over time. Higher accuracy generally improves reputation and reward outcomes, and reduces the risk of losing NCT when you bid.

What to Optimize for?

Your Engine should aim to:

• Correctly identify malicious artifacts (true positives)
• Correctly identify benign artifacts (true negatives)
• Avoid false positives (benign marked malicious)
• Avoid false negatives (malicious marked benign)
• Return UNKNOWN when your signal is weak or unsupported

Practical Accuracy Levers

1. Improve coverage and typing

• Detect artifact type early and handle each type explicitly
• For unsupported types, return UNKNOWN rather than guessing

2. Reduce false positives first

False positives hurt trust and can cause operational noise for users. Ways to reduce false positives:

• require stronger evidence before returning malicious
• use allowlists for common benign patterns when justified
• treat ambiguous cases as UNKNOWN

3. Use confidence honestly

Only emit confidence if it maps to a real signal:

• “High confidence” should be rare and repeatable
• map tool outputs and heuristics into a stable confidence scale
• keep confidence consistent across runs

4. Keep malware_family meaningful

If you provide malware_family metadata:

• avoid overly generic labels
• avoid dumping raw tool strings that are unstable
• prefer a small set of consistent family names

5. Regression testing

As you ship changes:

• keep a small local test corpus (benign + malicious)
• ensure new changes do not break previous good detections
• add tests for any bug you fix

A simple decision policy that works well

• Return MALICIOUS only when you have a strong signal
• Return BENIGN when evidence is strong that the artifact is clean

• Return UNKNOWN when:

  • the type is unsupported
  • the scan failed
  • the signal is weak
  • timeouts occur

This usually produces better long-term outcomes than guessing benign.