Nine years ago, we [1] embarked on a study of neonatal hypoglycemia using electronically extracted data from the Electronic Health Record (EHR). Within the data collection, we included an embedded validation study that compared the results of blinded manual data extraction to the electronic data extraction. Results of the validation study identified varying degrees of agreement with 43% (10/23) of extracted categorical variables having a kappa (K) coefficient for inter-user agreement of < 0.60, indicating only moderate to slight agreement. Notably, the subject of the study, “hypoglycemia,” had the lowest K coefficient, 0.16. Although all patients had laboratory-confirmed hypoglycemia, defined as a serum glucose < 40 mg/dL, only 29% were correctly identified by ICD-9 code. Of the eight categorical variables with the worst K coefficient, five showed significant improvement after modifying the definition of the ICD-9 extraction code, while three variables, including “hypoglycemia,” had K coefficients that could not be rectified. At the time of publication, we speculated as to whether the use of the new ICD-10 codes would improve extraction accuracy, since we discovered some variables that shared ICD-9 codes with other diagnoses leading to over identification.

International Classification of Diseases (ICD) codes are not the only variables that require validation, as was proven by our previous publication. It was also discovered that serum glucose measurements were recorded in three different locations within the EHR and point-of-care glucose results were, at times, duplicated with the same date and time stamp, which resulted in initial poor validity of only 33% for this variable. Notably, when these issues were identified and corrected for, the extraction code validity increased to 84%. Importantly, these issues would not have been identified without our decision to validate the dataset [1].

The integration of the EHR into most large healthcare organizations was motivated by the American Recovery and Reinvestment Act of 2009. Since then, researchers have utilized these data for secondary purposes. The recent adoption of ICD-10 and large, federated databases for research pose new challenges with data validation. Given what we know about potential errors in electronically recorded data, it is unclear if present day investigators have validated their data derived from the EHR before publishing research results and conclusions. This review explores new challenges with use of electronic health data for research and the importance of transparent reporting of data validation and quality assessment methods.

The transition from ICD-9 to ICD-10 resulted in a dramatic increase in diagnostic granularity and the number of diagnostic codes. According to Hedegaard et al. there were ~14,000 ICD-9-CM codes and ~69,000 ICD-10-CM codes in 2017, a five-fold increase [2]. Table 1 illustrates the increase in ICD-9 vs. ICD-10 codes for specific variables used in our original validation study [3,4]. While this can lead to improvement in over-identification caused by nonspecific coding definitions, there is also the potential to under-identify due to increased extraction coding complexity and thus, incomplete or inaccurate code lists. In 2022, Schaefer et al. compared ICD-10 codes for seven common comorbid conditions in the EHR for COVID-19 patients with results from a manual chart review [5]. They reported that the sensitivity for comorbidity detection via ICD-10 code to be highly variable, depending on the location within the EHR and the timing of data retrieval relative to the time of hospitalization. For instance, the ICD-10 code for chronic kidney disease via the past medical history on the day of admission only had a sensitivity of 0.3. The sensitivity of chronic kidney disease via the past medical history increased to 0.44 after removing the time constraint of the day of admission and allowing for any hospitalization day. Notably, the same diagnosis obtained from the active problem list on the day of admission had a sensitivity of 0.68, which increased to 0.77 with removal of the time constraint, further highlighting that the location and timing of data extraction within the chart can produce widely variable results, especially for ICD codes. Consequently, the authors concluded that all researchers need to validate extracted data against a manual chart review. In addition, they called for standardization in data extraction methodologies and reporting practices for studies using ICD-10-based data [5].

In 2009, Dean et al. reported that only 24% of published studies using EHR-derived data had documented validation of the EHR data with other sources of data [6]. At the time, EHR data was relatively new and the reliability questionable. By 2019, the availability of EHR-derived data was more common and, depending on the vendor, had some ability for electronic extraction. Based on our 2019 validation study [1], we also concluded that all researchers using electronically extracted data from the EHR needed to validate a subset of each variable in the dataset using manual chart review in order to identify potential sources of error and determine which variables are sufficiently accurate to remain in the dataset and those that should be removed [1]. The question today is whether validation practices have improved.

To investigate this, we examined the inclusion of validation practices by assessing 20 publications available on PubMed that utilized electronically extracted EHR data and were published in 2025 (Figure 1). Query language included “retrospective” and variations of “electronic health records” or “electronic medical records,” but excluded “machine learning”, “artificial intelligence” and “natural language processing” to focus results on clinical research and not informatics techniques. The dates of publication were from 1/1/2025–12/31/2025. A preliminary search yielded 3746 publications, illustrating the significant impact EHR data has had on the volume of clinical research in the last decade. Starting with the most recent publication, manuscripts were assessed for access to the full article and evidence of electronic data extraction in the methods section until we identified 20 inclusive manuscripts. Of note, 19 manuscripts were excluded due to: (1) inability to access the full article, (2) documentation of manual chart review instead of electronic data extraction, or (3) lack of manual vs. electronic data extraction distinction with a study population small enough to make manual extraction possible. Studies that lacked documentation distinguishing between manual vs. electronic extraction but had sufficiently large study populations to make manual data extraction infeasible (i.e., 50,000) were included. For included manuscripts, the methods section of each publication was analyzed for any documentation of data validation. Of the 20 manuscripts reviewed, only two (10%) included documented data validation. Ten (50%) were studies conducted at single centers or single healthcare organizations with access to source data but without any mention of data validation. The remaining eight (40%) utilized large, federated, de-identified databases which eliminates access to source data and thus makes source validation impossible.

Figure 1. Flowchart of article selection for review of validation documentation.

Additionally, only one of the eight reported quality control measures used by the database (Table 2) [7–26]. In summary, this descriptive analysis highlights the infrequent inclusion of data validation practices in published research using electronic health data, raising uncertainty as to whether this crucial step was performed.

Large, federated databases are increasingly available and offer unique research opportunities but also pose unique research challenges. For instance, TriNetX, the largest existing federated healthcare database, announced in June 2026 that it has surpassed 300 million individual patient files from over 240 healthcare organizations worldwide [27]. Such expansive data repositories can open doors for population-level research studies and are being utilized at an increasing rate. Notably, of the twenty manuscripts surveyed in Table 2, four of the eight (50%) that utilized large databases used data from TriNetX [19,20,25,26]. One challenge with these large databases is they often provide investigators only de-identified patient data, thus, making source data validation impossible. This, however, does not eliminate the obligation to evaluate and determine data validity. It merely changes how assessment of data quality should be performed. Kahn et al, published a Harmonized Data Quality Assessment Framework of completeness, conformance and plausibility. Completeness is the degree to which an expected data variable is available in the dataset and not missing. For example, a dataset with only 2% of the variable missing would be considered to have good completeness for that particular variable. Conformance is the degree to which the data adheres to a predefined format, data type and standard for each variable. For example, if a dataset has defined serum glucose units as mg/dL and records data as mmol/L this is considered poor conformance. Additionally, having some serum glucose values recorded as < 10 and others as actual integers (i.e. 3) is poor conformance due to lack of consistency. Lastly, plausibility is the degree to which the data is believable. For instance, a serum glucose value of 1500 mg/dL would be considered implausible [28]. This framework is the foundation for internal data quality standards claimed by both TriNetX and PCORnet (The National Patient-Centered Clinical Research Network), another large database with over 50 million patients [29,30] and should be utilized by researchers to assess EHR data quality for secondary use when access to source data is unavailable.

Understanding the internal data quality standards of large de-identified databases is essential and should be mandatory before trusting data accuracy; therefore, transparency regarding quality control measures is critical to establishing confidence in the data used. Demonstrating best practices, Cosmos, a large, federated database maintained by Epic Systems Corporation publishes completeness, conformance and plausibility metrics for variables as new data are incorporated in the database [31]. While no pre-defined acceptable metric threshold exists, this level of transparency allows researchers to determine for themselves whether each variable is fit for their own published research.

Expanding on data quality concepts, Feder described additional ways to assess data accuracy, including data triangulation with outside sources such as comparing statistical measures for research variables against the same statistical measures already published in the literature using identified patient data [32]. Large discrepancies call into question the validity of the de-identified data.

Data derived from the EHR and used for the secondary purpose of research must be held to the same high standard as any other research database. Importantly, investigators are obligated to validate all data used for clinical conclusions as they may lead to unfounded changes in clinical practice. When source data are available, a subset of each variable electronically extracted from the EHR should be compared against a manual chart review to determine potential sources of error and which variables can reliably remain in the study for analyses. Data quality frameworks offer additional tools to assess data fitness, which become increasingly more important when source data are unavailable due to use of de-identified data from large databases. As noted above, understanding internal data quality control measures used in large databases is crucial to instilling trust of the data analyzed; therefore, it is important to only use data from sources with transparency in internal data control measures, such as conformance, completeness and plausibility, as outlined by Kahn et al [28]. Importantly, validation and data quality methods used by all investigators and in all databases should be documented within published works to instill trust in the reader [32,33].

Clinical journals have both the privilege and responsibility of disseminating new and innovative research and the conclusions derived therein. As a result, it is recommended that journals should require a description of data validation or quality assessment procedures within the methods section for all studies utilizing data derived from EHR or large healthcare databases to ensure their readers that substantial effort has been made to minimize data errors.

Scientists, students, investigators, and clinicians rely on published research to make informed decisions regarding whether a change in clinical practice is warranted or should be considered. It is, therefore, important for all readers to understand the limitations and errors that can exist within data electronically extracted from the EHR and to critically appraise the quality of the data used by looking for validation processes within the methods section of published research. In the case of data used from large, federated databases, both authors and readers should familiarize themselves with the data quality assessment measures published or unpublished by the database owners and determine for themselves whether sufficient evidence exists to support confidence in the reliability of the data.

Data electronically extracted from clinical health records still require validation from source data and the details documented in published methods. When large de-identified databases are utilized, the platform’s intrinsic data quality processes should be reported along with additional efforts by researchers to validate the dataset.

The authors declare that they have no conflicts of interests.

The authors have no funding support to declare.