Liver and Gastrointestinal Transplant Division, Department of Gastroenterology, University of São Paulo School of Medicine, Brazil
Luana Regina Baratelli Carelli Mendes
Liver and Gastrointestinal Transplant Division, Department of Gastroenterology, University of São Paulo School of Medicine, Brazil
Liver and Gastrointestinal Transplant Division, Department of Gastroenterology, University of São Paulo School of Medicine, Brazil
Luiz Augusto Carneiro D'Albuquerque
Submitted date: September 22, 2017; Accepted date: October 24, 2017; Published date: October 31, 2017
Background and aim: Liver transplant is the most effective therapeutic option for patients with end-stage liver disease. The objective of this study is to develop a predictive model of costs after liver transplantation through machine learning using data obtained from the Nationwide Inpatient Sample Database.
Methods: We used the Nationwide Inpatient Sample (NIS) database, evaluating data from patients undergoing a liver transplantation procedure for the years 2011 (model training) and 2012 (model validation). Predictors of the total cost (using cost-to-charge ratios), total charges, and length of stay (LOS) were assessed using a combination of machine learning and tree regression models.
Results: A total of 2,274 individual patients met our inclusion criteria, 1,090 patients for the year 2011 and 1,184 for 2012. The most important variables predicting cost and LOS were consistent across all models and included the Charlson and Van Walraven comorbidity scores. The best performing model predicting total cost was Support Vector Machine with Linear Kernel with root mean square error (RMSE) values of 0.561 whereas for LOS was the Principal Component Analysis (RMSE=0.743). When evaluating predictors of total cost and LOS, Van Walraven score >26.5 constituted cost-drivers with an average total cost of 207,041 US dollars whereas scores ranging from 21.5-26.4 were associated with a mean increase in the LOS of 26 days.
Conclusion: Patient co-morbidities are major drivers of transplants costs, charges and LOS. Machine learning models allow for cost prediction of individual patients, thus allowing for better healthcare management and policy making.
Liver transplant; Predictive cost model; NIS database; Cost drivers
What do we know?
Liver transplant is the most effective therapeutic option for patients with end-stage liver disease. Liver transplant procedures are estimated to have an average cost of $577,100, with the costs distributed across 30 day pre-transplant procedures, procurement, hospital transplant admission, physician, procedural costs, 180 day post-transplant admission and immuno-suppressants charges. Several predictors of cost-of-liver transplant have been identified, including recipient factors such as MELD score, age, sex, Body Mass Index (BMI), pre-transplant intensive care unit status, indication for transplant, and the United Network for Organ Sharing (UNOS) status at the time of transplant. Other previously reported predictors of cost include primary procedure versus re-transplantation, liver-kidney transplantation, and laboratory parameters of both liver and kidney functions. Although many studies have evaluated predictors of cost after liver transplantation, to the best of our knowledge, no previous studies have used novel machine learning to predict these costs with increased accuracy levels, allowing for prognostic predictions that largely surpass traditional statistical models.
What does this paper add?
Patient co-morbidities are major drivers of transplant costs, charges and length of stay. The most important variables for predicting cost and length of stay across all models were the Charlson and Van Walraven comorbidity scores. High Charlson comorbidity scores increased the risk of gastrointestinal bleeding, acute respiratory failure and complications of biliary anastomosis, whereas high Van Walraven comorbidity scores increased the risk of septic or hypovolemic shock, gastrointestinal bleeding, acute respiratory failure, and hemorrhage complicating a procedure.
How this fits in with quality in primary care?
The study demonstrated that transplantation in patients with more co-morbidities and a higher risk of death or readmission represent a greater economic burden on the healthcare system. Machine learning models allow for cost prediction of individual patients, thus allowing for better healthcare management and policy making. Therefore, future liver transplantation strategies should take into account the Charlson and Van Walraven scores to yield more cost-effective outcomes.
Chronic liver disease is an important cause of morbidity and mortality worldwide. In the US, around 150,000 people were diagnosed with chronic liver disease annually between 1999 and 2001 . In 2010, liver cirrhosis alone led to over one million deaths, another million deaths caused by liver cancer and acute hepatitis . Besides its morbidity and mortality risk, chronic liver disease leads to an enormous financial burden. In 2007, the combined cost for the treatment of chronic liver disease and cirrhosis across different nations ranged from 14 million to 2 billion dollars depending on the disease etiology and the country where the treatment occurred . Also, the burden of end-stage liver disease has been projected to increase as a function of the corresponding increment in the prevalence of Hepatitis type C (HCV) infection and non-alcohol fatty liver disease (NAFLD) . This increase in burden is related to the overall increase in liver transplantations related to conditions such as non-cholestatic liver cirrhosis, cholestatic liver cirrhosis, biliary atresia, acute hepatic atresia, metabolic diseases, malignant neoplasms, among others.
Liver transplant is the most effective therapeutic option for patients with end-stage liver disease [5,6]. Physicians can stratify patients with end-stage liver disease for various interventions including liver transplant  with the introduction of the Model End State Liver Disease (MELD) score. Although expensive, liver transplantation is effective in improving the quality of life as well as the chances of survival in those with chronic liver disease [6,8]. Also, the waiting time for patients with end-stage liver disease to receive a deceased donor's liver has significantly decreased in comparison with previous years [9,10]. In 2016, a liver transplant was the second most common transplant procedure, representing 23.3% (7,481) of all transplant procedures conducted in the United States . Procedures are estimated to have an average cost of $577,100, with the costs distributed across 30-day pre-transplant procedures, procurement, hospital transplant admission, physician, procedural costs, 180-day post-transplant admission and immuno-suppressants charges .
Several predictors of cost-of-liver transplant have been identified, including recipient factors such as MELD score, age, sex, Body Mass Index (BMI), pre-transplant intensive care unit status, indication for transplant and the United Network for Organ Sharing (UNOS) status at the time of transplant [12-14]. Other previously reported predictors of cost include primary procedure versus re-transplantation, liver-kidney transplantation and laboratory parameters of both liver and kidney functions [8,13,15]. Although many studies have evaluated predictors of cost after liver transplantation, to the best of our knowledge, none of them used novel machine learning algorithms to predict these costs with increased accuracy levels, allowing for prognostic predictions that largely surpass traditional statistical models and that can be applied for the prediction of individual patient costs . Machine learning is useful in clinical decision support, predictor ranking, outcome forecasting, disease classification and preemptive complication detection [16,17].
Given this gap in the literature, the objective of this study is to develop a machine predictive cost model for liver transplantation through machine learning using data obtained from the Nationwide Inpatient Sample Database, as well as to determine the most significant predictors.
Our manuscript is reported in alignments with the recommendations of the TRIPOD Statement .
The Institutional Review Board of the University of Sao Paulo approved our study.
We used data from the Nationwide Inpatient Sample (NIS) database, containing observations for the years 2011 (model training) and 2012 (model validation). The NIS is part of the Health Cost and Utilization Project (HCUP) database, developed by the Agency for Healthcare Research and Quality (AHRQ) in collaboration with the Federal-State-Industry. These databases use a sampling design having the United States as the target population, while also stratifying the sample by geographic region, urban versus rural areas, teaching status and bed size. Patients' information is de-identified, with the procedure and diagnostic codes encoded according to the International Classification of Diseases, Ninth Edition, Clinical Modification (ICD-9-CM), which was used in the United States during the years analyzed in this study.
We included patients who had undergone a liver transplantation procedure in 2011 or 2012 (ICD-9 V42.7).
Our primary outcomes of interest were total cost and total charges. Charges represent what the hospital received for providing care, while costs represent what the hospital spent to provide that care. Costs were calculated from charges using a cost-to-charge ratio available from the Healthcare Cost and Utilization Project database. Both charges and costs were adjusted to dollars using the 2016 dollars Customer Price Index (https://www.minneapolisfed.org/community/teaching-aids/cpi-calculator-information/consumer-price-index-and-inflationrates-1913, last accessed February 2017).
Also, we included variables that are directly associated with the two primary variables, including length of hospital stay and surgical complications directly related to liver transplant: Injury to adjacent structures (998.2), hemorrhage complicating a procedure (998.11), septic or hypovolemic shock (998.0), portal vein thrombosis (452), complications of biliary anastomosis (997.4), adult respiratory distress syndrome (518.5), pulmonary edema (518.4), acute respiratory failure (518.81) and gastrointestinal bleeding (578.9).
We included the following variables as potential predictors: Age, admission type (elective or emergency), sex, quartile classification of the estimated median household income of residents based on patient's ZIP Code, admission day over the weekend, hospital bed size, race, the total number of discharges from this hospital, and co-morbidities (AIDS, alcohol consumption, anemia, arthritis, hemorrhage, congestive heart failure, chronic lung disease, coagulopathy, depression, diabetes without complication, diabetes with complications, drug addiction, hypertension, hypothyroidism, liver disease, lymphoma, electrolyte imbalance, metastatic cancer, neurological disorders, obesity, paraplegia, peripheral vascular disease, psychiatric disorders, pulmonary circulation disorders, renal failure, tumors, peptic ulcer disease excluding bleeding, valvular heart disease, weight loss) summarized by the Deyo- Charlson Comorbidity Index  and the Elixhauser-van Walraven Comorbidity Index . Both indices are validated for their ability to predict mortality [21,22]. The Charlson Comorbidity Index is a weighted score derived from the sum of the scores for each of the co-morbidities [19,20]. The Elixhauservan Walraven Comorbidity Index includes a set of 30 acute and chronic comorbidity indicators, and the index score is based on the total number of comorbidity categories required to predict in-hospital mortality [20,23]. We used a median cut point of 5 for the Charlson score and 23 for Van Walraven score to ensure an equal number of subjects within each category, following a convention from previous publications [24,25].
We started the analysis by performing a graphical exploratory analysis evaluating the frequency, percentage and near-zero variance for categorical variables, distribution for numeric variables, and missing values and patterns of all variables . Since the numeric variables total cost and length of stay did not present a normal distribution, all models were run with logtransformed variables and then subsequently exponentiated so that results could be clinically interpretable. Model training was performed using the 2011 data set, while validation was performed using the 2012 sample, thus mimicking the clinical scenario where past data are used to predict future clinical events.
The following machine-learning algorithms were used to predict numeric variables, including cost, charges, and length of stay: Linear Regression, Principal Components Analysis, Support Vector Machines, Decision Tree, and Nearest Neighbors. The following classification models were tested for the presence of complications: Regularized Least Squares, Linear Regression, Principal Components Analysis, Support Vector Machines, Decision Tree, and Nearest Neighbors. Regression models for the classification of numeric variables included Support Vector Machines, Decision Tree, and Nearest Neighbors. Comparison across models was performed using metrics for the area under the curve, sensitivity, specificity, Kappa as well as positive and negative predictive values.
For both numeric and categorical variables, we used regression trees (recursive partitioning) with the same set of outcomes and predictors. Regression trees complement the use of machine learning models as they are not only transparent (individual predictors are visible) but equally provide information on individual predictors after considering previous predictors. This sequence mimics clinical reasoning, thus adding to the overall understanding of cost driver hierarchy. Tree regression pruning was based on the following algorithm: At each pair of nodes from a common parent, we assessed the error based on the testing data, especially evaluating whether its sum of squares would decrease if both nodes were removed. In the case of a positive answer, nodes were removed; otherwise, they were left intact. Although tree regression models represent the best cut-points for values predicting outcomes, in contrast with linear regression models their results cannot be described in a single equation. However, they have a graphical representation which we present along with our results' interpretation.
All calculations were performed using the statistical language R  and packages ggplot2, caret and rpart.
In our overall description, we stratified results between 2011 and 2012, since we used the former for model training and the latter for model validation. A total of 2274 patients met our inclusion criteria, 1,090 patients for the year 2011 and 1,184 for the year 2012. Most patients were Caucasian (68%), male (64.5%), with a mean age of 51.8 (± 15.33) years. The average Charlson comorbidity score was 5.57 (± 2.04), and the average Van Walraven comorbidity score was 22.83 (± 8.13). The rate of elective hospital admissions was relatively lower in 2012 (26.5% vs. 30.2%), whereas annual total cost increased in 2012 (140151.5 (± 122394.7) vs. 138800.2 (± 129439.8). The median length of stay was 21.73 (± 26.15) days, with a higher number of hospital discharges in 2011. For the year 2011, 31.5% of patients presented a median household income in the range of 0-25th percentile whereas 27.4% of patients in 2012 were in a range of 76-100th percentile (Table 1).
|Variable [Missing]||Total (2274)||2011 (1090)||2012 (1184)||p|
|Age ||51.8 (± 15.33)||52.65 (± 13.75)||51.02 (± 16.62)||0.011|
|Female ||808 (35.5%)||388 (35.6%)||420 (35.5%)||0.986|
|- White||1447 (67.6%)||712 (68.9%)||735 (66.5%)|
|- Black||216 (10.1%)||114 (11%)||102 (9.2%)|
|- Hispanic||277 (12.9%)||128 (12.4%)||149 (13.5%)|
|- Asian or Pacific Islander||55 (2.6%)||20 (1.9%)||35 (3.2%)|
|- Native American||9 (0.4%)||2 (0.2%)||7 (0.6%)|
|- Other||136 (6.4%)||58 (5.6%)||78 (7.1%)|
|Median Household Income for ZIP Code ||<0.001|
|- 0-25th percentile||606 (27.4%)||335 (31.5%)||271 (23.5%)|
|- 26th to 50th percentile||535 (24.2%)||272 (25.6%)||263 (22.8%)|
|- 51st to 75th percentile||538 (24.3%)||236 (22.2%)||302 (26.2%)|
|- 76th to 100th percentile||535 (24.2%)||220 (20.7%)||315 (27.4%)|
|Elective Hospital Admission ||643 (28.3%)||329 (30.2%)||314 (26.5%)||0.059|
|Admission at Weekend ||528 (23.2%)||251 (23%)||277 (23.4%)||0.875|
|Hospital bed size||<0.001|
|- Small||55 (2.4%)||0 (0%)||55 (4.6%)|
|- Medium||275 (12.1%)||106 (9.7%)||169 (14.3%)|
|- Large||1944 (85.5%)||984 (90.3%)||960 (81.1%)|
|Total Hospital Discharges ||29951.8 (± 30941.48)||54112.38 (± 29151.45)||7709.38 (± 4921.22)||<0.001|
|Charlson Score ||5.57 (± 2.04)||5.54 (± 2.03)||5.59 (± 2.04)||0.574|
|Van Walraven Score ||22.83 (± 8.13)||22.54 (± 8.06)||23.09 (± 8.2)||0.106|
|Length of Stay ||21.73 (± 26.15)||22.93 (± 29.38)||20.62 (± 22.74)||0.037|
|Length of Stay Log ||2.68 (± 0.83)||2.71 (± 0.85)||2.66 (± 0.8)||0.112|
|Total Charge (US dollars) ||506753.2 (± 502706.7)||526811.3 (± 553702.8)||488322.8 (± 450245)||0.071|
|Total Charge Log ||12.86 (± 0.69)||12.87 (± 0.72)||12.85 (± 0.66)||0.489|
|Total Cost (US dollars) ||139537.2 (± 125618.8)||138800.2 (± 129439.8)||140151.5 (± 122394.7)||0.804|
|Total Cost Log ||11.62 (± 0.61)||11.61 (± 0.61)||11.63 (± 0.61)||0.457|
Table 1: Patient characteristics stratified by year.
Predicted means: In each of the subsequent analyses, we report a median cost and length of stay for each comorbidity category. The cutoffs of 5 for Charlson score and 23 for Van Walraven score were chosen to represent the median. Results are interpreted as significant when confidence intervals do not overlap. When evaluating the outcomes comparing the use of the Charlson versus Van Walraven comorbidity index scores, higher scores for both Charlson and Van Walraven comorbidity index were associated with increased total cost and length of stay (Table 2).
|Outcome variables||Charlson Score <=5||Charlson Score >5||Van Walraven Score <=23||Van Walraven Score >23|
|Average Total Cost (US dollars)||122.083 (101.573, 142.592)||167.762 (146.833, 188.691)||114.340 (94.000, 134.681)||174.106 (153.518, 194.694)|
|Length of Stay||16.1 (11.8, 20.4)||25.3 (20.9, 29.7)||14.5 (10.2, 18.7)||26.6 (22.3, 30.9)|
Table 2: Association between outcomes and comorbidity score index.
Van Walraven co-morbidity score presented a relatively stable relationship with the number of co-morbidities till 30 to 40, after which the length of stay started increasing (Figure 1). A similar relationship was found between the Van Walraven score and total cost (Figure 2).
In each of the subsequent analyses, we report odds ratio (OR) as a measure of the risk of complications. Confidence intervals are interpreted as significant when they do not exceed a value of 1.0. When evaluating the adjusted risks, we associated a high Charlson score with an enhanced risk of gastrointestinal bleeding (2.09, 95% CI; 1.04, 4.33), acute respiratory failure (1.82, 95% CI; 1.4, 2.36) and complications of biliary anastomosis (1.76, 95% CI; 1.02, 3.05). Whereas, we correlated the higher Van Walraven scores with an increased risk of septic or hypovolemic shock, gastrointestinal bleeding, acute respiratory failure and hemorrhage complicating a procedure (4.75, 95% CI 1.88, 13.7; 3.63, 95% CI 1.79, 7.84; 2.58, 95% CI 1.99, 3.36; and 1.42, 95% CI 1.08, 1.86, respectively) (Table 3).
|Complications||Charlson Score >5||Van Walraven Score >23|
|Injury to Adjacent Structures||1.06 (0.51, 2.2)||1.44 (0.7, 2.95)|
|Hemorrhage Complicating a Procedure||1.23 (0.93, 1.63)||1.42 (1.08, 1.86)|
|Septic or Hypovolemic Shock||1.3 (0.54, 3.15)||4.75 (1.88, 13.7)|
|Portal Vein Thrombosis||1.29 (0.94, 1.76)||1.34 (0.98, 1.83)|
|Complications of Biliary Anastomosis||1.76 (1.02, 3.05)||1.59 (0.93, 2.72)|
|Adult Respiratory Distress Syndrome||1.1 (0.78, 1.55)||1.29 (0.91, 1.82)|
|Pulmonary Edema||0.98 (0.49, 1.94)||1.45 (0.74, 2.86)|
|Acute Respiratory Failure||1.82 (1.4, 2.36)||2.58 (1.99, 3.36)|
|Gastrointestinal Bleed||2.09 (1.04, 4.33)||3.63 (1.79, 7.84)|
Table 3: Adjusted risk for complications.
Model performance: The model representing the lowest root mean square error (RMSE) value designates better performance. When predicting the total cost, Support Vector Machine with Linear Kernel model presented a superior performance when compared to Principal Component Analysis (PCA), Radial Basis Function Kernel Regularized Least Squares, k-Nearest Neighbors and Classification and regression trees (CART) with RMSE values of 0.561, 0.570, 0.570, 0.576 and 0.587, respectively. Finally, models with the best performance for the length of stay included Principal Component Analysis (RMSE=0.743), Radial Basis Function Kernel Regularized Least Squares (RMSE=0.747), Linear regression with a forward selection (RSME=0.748), Linear regression with a backward selection (RSME=0.748), and Support Vector Machine with Linear Kernel (RMSE=0.781). It is worth noticing that the lowest RMSE was similar for the models using Charlson and Van Walraven scores. However, the range of RMSE given by different models was wider in the case of those using Van Walraven score as a variable.
The most important variables for predicting cost were consistent across all models including Charlson and Van Walraven score, age, and the total number of discharges from this hospital whereas the most significant predictive variables for the length of stay across all models were Charlson and Van Walraven scores.
Tree regression model: Finally, a tree regression model was used to evaluate how subsequent combinations of predictors would affect the outcomes. We found that a Van Walraven score greater than 26.5 constituted cost-drivers with an average total cost of 207,041 US dollars whereas scores ranging from 21.5-26.4 were associated with a mean increase in the length of stay of 26 days (Figures 3 and 4).
While evaluating the total costs on the Charlson score, a score of less than 6.5 coupled with an age less than 16.5 years was the predictor of the highest average cost of 167,052 US dollars. In contrast, Charlson scores greater than 6.5 were associated with an increased average length of stay (20 days).
To our knowledge, this is the first report using machine learning to analyze the cost of liver transplantation based on patient baseline characteristics. We used data from a nationwide sample of 2,274 patients to predict the cost associated with liver transplant. Our results indicate that patient co-morbidities are major drivers of transplant costs, charges and LOS. Also, we found that co-morbid states heighten the risk of complications during and after the procedure. We discuss our findings in the context of recent policy changes regarding organ allocation.
Co-morbidity is a predictor of higher liver transplant costs. Increased expenses were associated with patients with higher Charlson and Van Walraven scores. Previous reports established that factors including kidney injury were even more important than the degree of liver impairment in determining the cost of liver transplant . However, the authors hypothesized that renal injury might be a surrogate for severe co-morbidities and therefore lead to higher rates of complications, which is in line with our findings. On the other hand, recent research found that patients with higher MELD scores augmented the charges of liver transplantation [8,12,14,28] because they increased morbidity , also making hospitalizations more frequent [8,30]. Interestingly, when the post-surgery period only accounted for the analysis, no increased expenses were recorded among patients in more severe stages of the disease [28,31]. This discrepancy may result from the use of charges instead of costs in the former study  and the analysis of a period after the implementation of a healthcare policy presumably leading to better pre-surgical care in the case of the latter . Since co-morbidities impact the cost of liver transplant differently depending on when the intervention was delivered, future studies should assess the cost of liver transplant before, during, and after surgery to optimize costs. The appropriate management of co-morbidities before the surgery will be critical to decreasing the costs of the liver transplant.
LOS also increased in patients with higher Charlson and Van Walraven scores, indicating that patient comorbidity is important to predict the hospitalization period. Prior studies found an association between increased MELD scores and higher LOS . Moreover, LOS has proven to raise costs of liver transplant . These results support the notion that liver transplant generates increased costs among sicker patients.
Gastrointestinal bleeding, acute respiratory failure, and hemorrhage during surgery were complications observed at increased rates among patients with greater Charlson and Van Walraven scores. Additionally, portal vein thrombosis, complications of a biliary anastomosis, and pulmonary edema were more frequent in patients with higher Van Walraven scores. Past findings support the notion that more complications after liver transplant correspond to greater costs . Moreover, complications occur more frequently in patients with higher MELD scores . Therefore, the increased incidence of complications during or following the period after surgery may account for the higher costs observed in patients with more comorbidity.
Our analysis highlights that the highest expenses are present among patients under 16.5 years old. These high costs are consistent with other reports [34,35] and may be explained by the increased risk children present to vascular and biliary complications during a liver transplant. Former studies excluded pediatric populations , making these observations of great value. Studies  demonstrated that lower costs in this population might result from factors including whole organ liver transplant and the white race while associating hepatic artery thrombosis and older age with higher expenses.
Our results represent a significant contribution to the field of liver transplant cost analysis for several reasons. First, we used a different methodology that validates previous findings of multiple regression analyses. Second, we included a nationwide sample size that is more representative of different ethnic and socioeconomic backgrounds, which is in contrast with the use of a single or just a few centers. The former is relevant in the face of previous studies demonstrating that different centers incur expenses by means independent of patient characteristics . Moreover, we were able to separately analyze total costs and total charges, a difference that has limited other reports .
Despite filling an important gap in the literature, our study does have limitations usually associated with an observational design. First, since this is a national administrative database, our cost estimates were based on cost-to-charge ratios. While this methodology has an inherent assumption that charges are related to costs based on some modifiers (academic status, rural versus urban, among others), true cost data are not unavailable but also display important differences in how they are calculated across hospitals. Second, our database did not have information on MELD and other liver-specific conditions. Another limitation is that we did not analyze the relation of donor characteristics to transplantation cost. This variable has, however, been found to be inconsistent in its contribution to higher costs. For instance, donors over the age of 40 years increased the costs of transplantation in one report , and presented no such impact in other analyses .
Another consideration for future studies is the use of data from the period after the MELD era and the implementation of healthcare policies, evaluating their impact on the economics of liver transplants. For example, although a single center study did not find an increase in the cost of care after liver transplant in the period following the implementation of these policies, authors hypothesized that this was due to adequate care in the pre-transplantation period . Studies using national samples are however required to confirm this hypothesis.
In summary, we used several statistical methods to demonstrate that liver transplant costs are higher in patients with more co-morbidity. Our findings add arguments to the discussion on how to allocate livers in transplantation programs. Recently-implemented policies prioritize more acutely and severely ill patients to receive organs first. However, we confirmed that transplantation in patients with more co-morbidities and a higher risk of death or readmission represent a greater economic burden on the healthcare system. Therefore, future liver transplantation strategies should take into account the Charlson and Van Walraven scores to yield more cost-effective outcomes.
All Published work is licensed under a Creative Commons Attribution 4.0 International License
Copyright © 2018 All rights reserved. iMedPub LTD Last revised : June 24, 2018