Validating a mobile forensics tool: NIST CFTT, SWGDE and what a test report proves
How to read a NIST CFTT mobile report, what SWGDE 18-Q-001 and the UK Forensic Science Regulator expect, and how dual-tool checks and hashes fit in.
Updated
In short
A published test report shows how one named version of a tool behaved on one named set of devices under one test plan; it is not a validation of that tool for a particular laboratory's work. SWGDE and the UK Forensic Science Regulator both allow another organisation's results to be used, but the UK code requires the receiving unit to review them against its own requirements, and SWGDE recommends baseline testing before casework. A second tool can corroborate results, and a hash can show that an evidence file has not changed since acquisition. Each has a limit the standards state: independently built tools can share an underlying problem, and hashing alone cannot establish that a mobile extraction is correct.
Verification and validation
The UK Regulator's statutory code (version 2, published 2025-06-05, covering England and Wales) defines the two words. Validation is "the process of providing objective evidence that a method or procedure is fit for the specific purpose intended", including showing that acceptance criteria derived from end user requirements are met and that the method's limitations are understood (section 24.1.2). Verification is "confirmation, through the documented assessment of existing objective evidence or through experiment, that a method is fit (or remains fit)" for that purpose (24.1.3). Where evidence comes from a manufacturer or another unit, the code requires the receiving unit to review its adequacy, reliability and relevance (24.3.2). For digital forensics, section 97.4.6 says the measurement-based validation rules apply to most methods, including extraction with automated tools.
SWGDE 18-Q-001 (version 2.1, 2024-03-07) does not define them. It says testing "is often referred to as validation or verification testing" and aims to evaluate whether a tool performs as expected and to understand its limitations. NIST SP 800-101r1 (May 2014, so dated) describes validating a tool as defining a set of test data, following acquisition procedures to recover it, and assessing the results, and says validation should be repeated when updates or new versions appear. It adds that some updates were found less capable in some respects than earlier versions.
What CFTT tests
NIST's CFTT project aims to establish a methodology for testing forensic tools through specifications, test procedures, test criteria, test sets and test hardware. The mobile specification, version 3.3 (January 2025), turns requirements into assertions checked after each test case. Its core requirements include extracting and presenting supported data artefacts from an image, rendering text correctly, not modifying the image file, and notifying the user if it has been modified. Optional requirements cover notification when a connection fails or an acquisition is interrupted, UICC access, deleted data and SQLite handling. The specification defines "supported data artifacts" as those the tool's own documentation says it can acquire, so a tool is tested against its own stated scope (this guide's inference from that definition).
The scope grows. NIST's mobile page lists timeline analysis, fitness data and Wi-Fi data in a row for 2024. Its row headed 2025 still reads "Will be adding" financial data (Google Pay, Apple Pay, Venmo, Cash App, PayPal), social media (Discord, Reddit, Telegram, Signal) and application usage logs (KnowledgeC, Biome); the page, updated July 23, 2026, does not say whether or when these were added. The specification v3.3 describes the usage logs as stored on iOS devices. A 2023 report and a 2026 report therefore do not test the same list (this guide's inference from that history).
How to read a report
The MD-LIVE report says its Section 1 summary is "sufficient for most readers" to judge suitability, then gives devices (Section 2), environment and data objects (Section 3) and per-test-case results (Section 4), scored As Expected, Partial, Not As Expected or NA (MD-LIVE report). Read these items first:
- Tool and version. Each report names a build. The Data Pilot Desktop report covers v2.0.200 (cover dated December 2024); the vendor's update post of 2026-09-09 lists version 2.0.267. See Data Pilot Desktop. The report says nothing about later builds.
- Devices and extraction type. The Cellebrite Inspector report (v10.7, February 2023) lists iPhones and an iPad on iOS 11.3.1 to 15.6, all Advanced Logical, and Android 7.1.1 to 12 phones, six of them also listed with FFS (the report does not expand the abbreviation; read here as file system extraction). Anything outside that grid is not covered by this report. See Cellebrite Inspector.
- Who ran the tests. The MD-LIVE, Inspector, XRY and Data Pilot reports say the tests were run in the NIST CFTT lab. The Secfore report says they "were performed by Secfore as part of NIST CFTT federated testing". NIST describes Federated Testing as helping investigators test and, if desired, share results, and says CFTT staff review the logs and results before shared reports are published; the Secfore report does not itself record that review. See Secfore Extractor.
- Workstation. MD-LIVE ran on Windows 10 Pro 10.0.19042.1586; Secfore on Windows 11 Pro 26200.8246.
- Not reported versus not decoded. A "not reported" entry does not by itself say whether data was missed at acquisition or at decoding. In the Secfore report a note says application data extraction is supported but not decoded for Apple Pay, Venmo, TikTok, Signal, Pinterest, Cash App, fitness apps, calendar and contact groups, while its Section 1 summary calls several of the same items "acquisition is not reported". The report does not say whether those entries mean missed acquisition or missing decoding.
- Behaviour, not only data. The XRY Office v10.9.0 report (July 2024) records that disrupting connectivity produced no error message, the tool kept trying and reported data up to the disruption point, and a "Finished with errors" message appeared on all Android phones. See MSAB XRY.
Device sets differ widely: the MD-LIVE report's Table 1 has 27 rows, the Secfore report's has 10. Anomaly lists from different reports are not comparable, and this guide does not compare tools on them.
When the narrative and the table disagree
Check the summary against the tables. The MD-LIVE v3.6.14.1440 report (cover dated February 2026) ends its Section 1 summary with a note that the tool "cannot establish connectivity with the Huawei P20 Pro and Google Pixel 4". Table 1 on pages 5 and 6 lists both phones with extraction type Logical. The same Section 1 reports Pixel 4 results (for example social media and GPS), and Section 4.1 on page 10 does too. The report does not reconcile these, so the scope of the note is unclear. This was checked on the rendered pages, not the extracted text, which scrambles the tables.
The report also exists as two files. The copy linked from NIST's CFTT page is listed there as July 9, 2026; the DHS listing links the 2026-10 file. Both carry the same version and February 2026 cover, and their wording differs in places, including the connectivity note. Record which file you relied on. See GMDSOFT MD-LIVE.
SWGDE 18-Q-001 and ISO/IEC 17025
For mobile imagers, 18-Q-001 says no tool can acquire all data from all phones and that comprehensive testing is not practical, so testing should focus on common data and phone types. It says testing must use a known dataset with known content (address book, images, messages, social media), supplemented by empirical testing, and that correctness "cannot be based solely on hashing". Testing is due before service, after repair for hardware, and after major updates; software generally needs each major version tested once. The tester may be the lab, CFTT or another third party "if the same major version". It also says testing is "not necessarily pass/fail": some faults limit how a tool is used rather than whether.
18-Q-001 does not mention ISO/IEC 17025. In the UK, the Regulator's code says UKAS assesses units against ISO/IEC 17025:2017, ISO/IEC 17020:2012 or BS EN ISO 15189:2022 where accreditation is required (3.2.5). For digital forensics, it says compliance is demonstrated by accreditation to ISO/IEC 17025:2017 for the sub-activities listed in 70.3.1 that the organisation undertakes, on its schedule of accreditation. Section 70.3.1 groups those sub-activities as "Data capture, processing and analysis from digital storage devices", performed on any storage device including mobile devices and computer drives (70.2.1, 70.3.1). Requirements elsewhere vary; check your jurisdiction's.
Dual-tool checking and its limit
SWGDE 18-F-003 (version 2.0, 2025-08-21) says a single tool may not extract or present all data, and that manually reviewing the device or using a second forensic tool can corroborate the results or provide additional data not recovered in the initial extraction. 18-Q-001 Appendix A describes comparison testing with at least two tools and warns it "may not catch as many types of errors as known dataset testing, such as when even independently developed tools have the same underlying problem". NIST SP 800-101r1 adds that hash validation across tools is hard because of proprietary reporting formats.
Two cases of shared code, one concrete and one potential. Autopsy's iOS Analyzer module runs iLEAPP, so comparing its output with iLEAPP is not an independent check of the iLEAPP parsers (this guide's inference; the Autopsy page does not state which iLEAPP version it runs). The iLEAPP README says its single-file Windows program has third-party files packed inside it, such as Mandiant's Unified Log parser. A comparison against another tool that uses the same parser would not be independent either (this guide's inference; the README does not name other users). See the LEAPP family guide, iLEAPP and ALEAPP.
A second tool that reads the first tool's extraction checks decoding only. Whatever the first tool failed to acquire is absent for both (this guide's inference). A second acquisition on a live phone is a different check, with the drift described below.
Known-data test sets
18-Q-001 says known-dataset tests for mobile imagers may use seized devices if the relevant content is documented and private information is not released, and that lab-created datasets can target the functions being tested. NIST SP 800-202 (May 2018) gives procedures for documenting and populating test phones, advises choosing data types relevant to the lab's cases, and notes that items meant to be recovered as deleted data should not be deleted until population is complete.
Public images are the other source. NIST describes CFReDS as documented sets of simulated evidence, some made by NIST and some contributed; this guide did not check which are mobile. Digital Corpora's cell phone collection includes images from Joshua Hickman, whose public images page lists Android 7 to 14 and iOS 13 to 17. The Android 7 listing includes image-creation documentation and a hashes file, and the Android 13 page names the handset (Pixel 5a) and links an encrypted solutions document. The iLEAPP and ALEAPP READMEs name "a public research image" among the data contributors may share in a pull-request fixture.
Such images let you test parsers and analysis tools offline. They cannot test a tool's acquisition from a live handset (this guide's inference).
What hash verification does and does not establish
A hash computed after acquisition and rechecked later detects changes to the evidence file; NIST SP 800-101r1 describes this as the purpose, and notes some forensic tools may not compute one, in which case compute it separately. It does not show the acquisition was complete or correct. For mobile devices 18-Q-001 says hashing alone cannot establish correctness, because mobile device hardware may change the dataset based on the underlying drive operations. SWGDE 18-F-003 (version 2.0, 2025-08-21) says hash values from multiple extractions of the same device may not match, because of the dynamic nature of storage media on mobile devices.
SP 800-101r1 explains that back-to-back acquisitions differ because the device keeps updating (the clock, for example), while hashes over individual files or directories generally stay consistent. The vendor's Secfore Extractor page states "SHA-256 hash verification built into every extraction"; the page does not say what is hashed or when. The ALEAPP and iLEAPP READMEs say each release includes SHA256SUMS.txt for checking a download, which verifies the tool you installed, not the evidence.
Limits of reproducibility on live phones
SWGDE 18-F-003 (version 2.0) says data from powered-on mobile devices is constantly changing, and that there are no write-blocking methods available for mobile devices themselves, whereas write-blocking is sometimes applicable to portable storage media. SP 800-101r1 gives a concrete case: reading unread SMS from a UICC through the handset can flip the message's status flag, so a second acquisition differs. It says hashes over selected items such as individual files and directories generally remain consistent, and that hash inconsistencies may require element-by-element verification. This guide's reading: compare repeat extractions at that level rather than by whole-image hash. 18-Q-001 adds that, given the variety of software and hardware versions, no test can ensure a tool performs as expected in all situations.
Building an in-house validation record
18-Q-001 section 6 lists what lab testing documentation should include: purpose and scope; who tested; date; procedure or scenarios; datasets and expected results; results, anomalies and their significance; and identified limitations. Third-party testing is documented by referencing the report. The UK code adds that acceptance criteria are set before testing begins (24.8.1), and that the validation report should, as a general guide and as appropriate, include or reference the name, version number and manufacturer of equipment used and expected compared with actual results (24.12.2). It says methods are validated to a specific configuration, and that a change in hardware, firmware, script or operating system, even a patch, can affect a method (97.4.10, 97.4.11). It also warns the library may be disclosable in criminal proceedings (24.14.4).
A workable record for a mobile tool, mapped to those lists:
- Tool build, installer hash, workstation OS and the CFTT report you adopted, with file name and date.
- Each test device: model, OS build, lock state (BFU or AFU), extraction type (definitions), and population record per SP 800-202.
- Expected results per artefact category, scored as the CFTT reports score them.
- Anomalies with their significance for your casework, and the limits you will state in reports.
- A trigger to re-test: major version, hardware repair, or release notes touching acquisition.
For open-source parsers, record the release tag. The iLEAPP README describes fixtures and a committed output snapshot that guards each module after merge; in this guide's reading, that is the maintainers' regression testing, not your validation. For the wider selection context, see choosing a mobile forensics tool.
Sources
- UK Forensic Science Regulator, Forensic science activities: statutory code of practice, version 2 (published 2025-06-05)
- SWGDE 18-Q-001-2.1, Minimum Requirements for Testing Tools Used in Digital and Multimedia Forensics (2024-03-07)
- SWGDE 18-F-003-2.0, Best Practices for Mobile Device Evidence Collection & Preservation Handling and Acquisition (version 2.0, 2025-08-21; supersedes v1.2 of 2020-09-17)
- NIST SP 800-101 Rev. 1, Guidelines on Mobile Device Forensics (May 2014)
- NIST SP 800-202, Quick Start Guide for Populating Mobile Test Devices (May 2018)
- NIST, Computer Forensics Tool Testing Program (CFTT), page updated 2026-05-01
- NIST CFTT, Mobile Devices (specifications, test set-up documents and DHS report list)
- NIST CFTT, Mobile Device Forensic Tool Specification, Test Assertions and Test Cases, Version 3.3 (January 2025)
- NIST CFTT, Federated Testing Project
- NIST CFTT, CFReDS (Computer Forensic Reference Data Sets), page updated 2026-05-01
- NIST/DHS, Test Results for Mobile Device Acquisition Tool: GMDSOFT MD-LIVE v3.6.14.1440 (cover dated February 2026; file linked from the DHS listing)
- NIST/DHS, Test Results for Mobile Device Acquisition Tool: GMDSOFT MD-LIVE v3.6.14.1440 (cover dated February 2026; file linked from the NIST CFTT page, listed there as July 9, 2026)
- NIST/DHS, Test Results for Mobile Device Acquisition Tool: Secfore Extractor v2.4.0.0 and Secfore Visualizer v2.4.2 (cover dated May 2026)
- NIST/DHS, Test Results for Mobile Device Acquisition Tool: Cellebrite Inspector v10.7 (cover dated February 2023)
- NIST/DHS, Test Results for Mobile Device Acquisition Tool: XRY Office v10.9.0/XAMN v7.9.0 (cover dated July 2024)
- NIST/DHS, Test Results for Mobile Device Acquisition Tool: Data Pilot Desktop v2.0.200 (cover dated December 2024)
- DATAPILOT, September DP10, DPX & DP Desktop Software Update (vendor, 2026-09-09)
- Secfore, Extractor product page (vendor, read 2026-11-01)
- Digital Corpora, Cell Phones
- Digital Corpora, Android 7 (file listing)
- Digital Corpora, Android 13 image (contributed by Joshua Hickman)
- The Binary Hick (Joshua Hickman), Public Images
- abrignoni/iLEAPP README at tag v2026.4.5 (released 2026-10-24)
- abrignoni/ALEAPP README at tag v2026.4.4 (released 2026-10-24)
- Autopsy 4.23.0 user documentation, iOS Analyzer (iLEAPP)