ESI Examples

Example 1 – Publication bias and antidepressant efficacy

Example 2 - Prednisone for vertigo in Meniere’s disease

Example 3a & Example 3b - Aripiprazole augmentation in clozapine-treated patients with refractory schizophrenia

Example 4 – Vitamin K supplementation for improved anticoagulation stabilization

Example 5 – CoEnzyme Q10 supplementation for statin-induced myopathic symptoms

Example 6 – Venlafaxine ER in Post Traumatic Stress Disorder – comparing 3 groups

 

Example 1

 

Reference article: Turner EH, Matthews AM, Linardatos E, Tell RA, Rosenthal R. Selective publication of antidepressant trials and its influence on apparent efficacy. N Engl J Med. 2008 Jan 17;358(3):252-60.

 

In January, 2008 the New England Journal of Medicine published the above article that revealed a 32% inflation of apparent antidepressant treatment effect related to publication bias. This inflation was calculated using the Hedges’ g effect size expression. The authors calculated an effect size of 0.41 based on published trials. It was 0.31 when calculated using trials submitted to the FDA. 0.41 divided by 0.31 is 1.32, indicating an inflated effect size of 32%. This and the other findings corroborating selective publication practices of the antidepressant pharmaceutical industry made headlines globally. Using the Effect Size Illustrator we can bring some clarity to the inflated effect that was not apparent in the publication.

 

 

 

 

 

 

 

 

The effect size is essentially an expression derived from comparing two Gaussian curves. Looking closer at how these curves overlap provides better insight into how the groups compare. For example, one can derive how the average person is doing in one group relative to the responses observed in the other group. An effect size of 0.41 indicates that the average person (mean=median in this case) taking an antidepressant does better (in terms of depressive symptom score) than 66% of people taking placebo. When the effect size is 0.31 this changes to 62%. As such, the inflated effect can be calculated as 0.41/0.31 = 1.32, indicating a 32% magnification of apparent effect. Or, it can be calculated as 66%/62% = 1.065, indicating a 6.5% magnification. The latter approach, which is more clinically meaningful, markedly diminishes the apparent effect of publication bias. The small difference is further clarified by a comparison of the illustrations, using Gardner’s Effect Size Illustrator:

 

 

Hedges’ g = 0.31                   Hedges’ g = 0.41

 

While it is true that the published data exaggerated the efficacy of antidepressants relative to the data submitted to the FDA, the difference was rather small. The real issue was the limited effectiveness of antidepressants and not that of publication bias.

 

Back to Top

Example 2

 

Reference article: Oral administration of prednisone to control refractory vertigo in Meniere’s disease: a pilot study. Otology & Neurotology. 2005; 26:1022-1026.

 

 

 

 

 

 

 

 

 

 

1 Data are expressed as mean and 95% confidence interval.

 

Data entry into ESI using Option 1:

 

Results from the article’s table 4 measuring the duration of patient’s vertigo attacks at 18 weeks have been used for illustration & calculation purposes.

 

Here we have used Cohen’s d for our calculation.

 

In this example, the variance around the mean is given as a 95% confidence interval (CI). To enter this into the ESI, subtract the upper limit of the CI from the lower limit giving the difference between the two limits as the number to enter in the variance field. For example, in the prednisone group, the 95% CI at 18 weeks is 31 – 11, giving a difference of 20, which is the value to be entered into the variance field. Remember to select the 95% CI option.

 

What is entered in the group name text field will appear as the labels in the illustration (see below).

 

 

Illustration Options:

Note:

 

In this example, higher values reflect worse health (longer duration of vertigo).

Here we have gone with the default Gardner’s Ornament style of illustration, side by side, and vertical layout.

The image girth has been decreased from 100% to 80% for esthetic purposes.

 

 

Data entry into Study Details using relevant information from the article:

 

Patient description: “Patients with Meniere’s disease with limited vertigo control (Class C) and severe disability (Scale 3).” (American Academy of Otolaryngology—Head and Neck Surgery criteria)

 

Study Groups/Intervention description: “Two groups (n = 8 per group) were treated orally with either diphenidol (25 mg/d) plus acetazolamide (250 mg/48 h) (control group), or the same treatment plus prednisone (0.35 mg/kg) daily for 18 weeks (prednisone group).”

 

 

Analysis & Interpretation output:

 

 

 

Illustration Output reflecting the calculated effect size (Cohen’s d) of 1.75:

 

 

 

In this example the standard deviations differ between the two groups. As such, the interpretation varies depending on which group you choose as the reference group. With prednisone as the reference group it can be seen that 91% of people taking placebo are doing less well than the average person taking prednisone. With placebo as the reference group it can be stated that 99% of people taking prednisone are doing better than the average person taking placebo. These accuracy of these interpretations need to be qualified considering that this study only compared 8 people per group were studied.

 

Back to Top

Example 3a

 

Reference articles:

 

Chang JS, et al. Aripiprazole augmentation in clozapine-treated patients with refractory schizophrenia: An 8-week, randomized, double-blind, placebo-controlled trial. J Clin Psychiatry; 2008 May;69(5):720-31.

 

Remington G. Refractory schizophrenia: adding aripiprazole to clozapine reduces negative but not overall symptoms*. Evid Based Ment Health. 2009 May;12(2):51.

* Kane et al (J Clin Psychiatry 2009), in a larger, longer RCT found no benefit of aripiprazole on negative symptoms when added to risperidone or quetiapine.

 

Data entry into ESI using Option 1:

 

In this example we used the following data (derived from reference 2) to illustrate the effect aripiprazole augmentation of clozapine on negative symptoms of schizophrenia (flat affect, poverty of speech, inability to experience pleasure, lack of desire to form relationships, lack of motivation), as measured by the Schedule for the Assessment of Negative Symptoms (SANS) at 8 weeks:

 

 

 

 

 

 

 

 

Illustration Options:

Note:

 

In this example, higher values reflect worse health, as a higher value indicates less reduction in negative symptoms (SANS).

 

Here, we have decided to use horizontal, Gaussian curves to illustrate the effect size.

 

 

Analysis & Interpretation output:

 

 

Illustration Output reflecting the calculated effect size (Cohen’s d) of 0.71:

 

 

The average person taking aripiprazole in addition to clozapine has an improvement in negative symptoms greater than 77% of people only taking clozapine. The question remains, is the difference clinically significant? See example 3b.

Back to Top

Example 3b

 

Reference articles: See example 3a

 

Data entry into ESI using Option 1:

In this example, using the same reference article as the previous example, we will create an illustration comparing the final SANS scores between the aripiprazole augmented group and the placebo augmented group, as opposed to the changes in SANS scores.

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Analysis, Interpretation and Illustration Output:

Note:

 

In this example, higher values reflect worse health (higher SANS scores reflect more negative symptoms).

 

 

The final illustration shows the differences between groups. It also indicates that both groups continue to experience a significant burden of negative symptoms. The illustration also hints that there may be a small group of patients that respond well to aripiprazole but that most will not.

 

 

Back to Top

Example 4

 

Reference article: Sconce E et al. Vitamin K supplementation can improve stability of anticoagulation for patients with unexplained variability in response to warfarin. Blood. 2007; 109: 2419-2423.

 

Data entry into ESI using Option 1:

 

This article used a randomized controlled trial to test the hypothesis that daily oral vitamin K supplementation would improve INR stability in patients with previously unstable control when taking oral anticoagulation.

 

For this example, the comparison of % time in INR range (2-3) during the intervention period (6 months) between the Vitamin K treated group and the placebo group (data from Sconce et al. Table 2) has been used for calculation and illustration purposes.

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Illustration Options:

Note:

In this example, higher values reflect better health.

 

Here, we have used the default Gardner Ornament style of illustration with the overlapping option.

 

Also, we have specified the range of values as 0% to 100%. This has resulted in a truncated illustration indicating that the data were not normally distributed.

 

 

 

Analysis, Interpretation and Illustration Output:

 

This illustration shows a common limitation of the assumption implicit in all effect size calculations, which is the assumption of normal distribution. Clearly, the upper limit in this example is 100%. It is not possible, as the illustration implies, to achieve a target INR more than 100% of the time. The cause of this is skewed or otherwise non-normal distribution of data. This limitation should be kept in mind whenever examining effect size illustrations. That is, the range of scores presented in the illustration should not be interpreted literally in many cases.

 

Back to Top

Example 5

 

Reference article: Caso G et al. Effect of Coenzyme Q10 on Myopathic Symptoms in Patients Treated With Statins. Am J Cardiol. 2007;99:1409 –1412.

 

Data entry into ESI using Option 1:

 

In this article, coenzyme Q10 supplementation was compared with Vitamin E supplementation (control) for the treatment of statin-induced myopathic pain.

 

One of the outcome measures reported was the mean change in Pain Severity Score (PSS). After 30 days, the coenzyme Q10-treated group (n=19) had a 40±11% decrease in pain intensity as measured by the PSS. Pain in the Vitamin E-treated group (n=14) intensified by 9±14%.*

* These findings were not replicated in another RCT of Coenzyme Q10 for statin-related myalgia (see Young JM, et al. Effect of coenzyme Q(10) supplementation on simvastatin-induced myalgia. Am J Cardiol. 2007 Nov 1;100(9):1400-3.)

 

Illustration output reflecting the calculated effect size (Cohen’s d) of 4.08:

 

 

The above illustration represents the unusual finding of an almost complete separation of treatment outcomes. As stated above, this result was not replicated in another trial.

 

Back to Top

Example 6

 

Reference article: Davidson J, et al. Venlafaxine extended release in posttraumatic stress disorder: a sertraline- and placebo-controlled study. J Clin Psychopharmacol. 2006 Jun;26(3):259-67.

 

Data entry into ESI using Option 1:

The objectives of this study were to determine the efficacy and tolerability of sertraline and venlafaxine for the treatment of PTSD and to compare the efficacy and tolerability of sertraline and venlafaxine with placebo.

 

The following table reports the main outcome measure:

 

 

 

 

 

 

 

 

 

 

For this example, we used the mean change score from the 17-item clinician-administered PTSD Scale (CAPS-SX17) for illustration purposes.

 

Three groups were compared. To add a group use the “+” button and to toggle between the groups use the arrow buttons on the Effect size calculation tab.

 

 

After toggling:

 

 

Analysis, Interpretation and Illustration Output:

 

Since we have three groups, the ESI will calculate and interpret an effect size comparing Group 1 to Group 2 and also comparing Group 1 to Group 3:

 

 

Illustration output reflecting the calculated effect sizes (Cohen’s d) of 0.073 and 0.26:

 

 

Back to Top

 

 

 

Effect size (Hedges’ g)

The proportion of patients receiving no treatment (placebo) that did worse in terms of depressive symptom burden compared to the average person taking an antidepressant

0.41

66%

0.31

62%

Table 4. Duration of vertigo attacks1

Week post treatment

Control (n=8), min

Prednisone (n=8), min

p value

0

69 (40-98)

63 (42-84)

NS

6

63 (48-78)

33 (20-46)

NS

12

65 (50-80)

28 (13-43)

<0.05

18

54 (37-71)

21 (11-31)

<0.05

Effect size (Hedges’ g)

The proportion of patients receiving no treatment (placebo) that did worse in terms of depressive symptom burden compared to the average person taking an antidepressant

0.41

66%

0.31

62%

Measure

Adjunctive aripiprazole (n=29)

Adjunctive placebo (n=32)

Baseline

Mean (SD)

8 weeks

Mean (SD)

Baseline

Mean (SD)

8 weeks

Mean (SD)

BPRS positive

11.2 (5.3)

10.8 (5.4)

11.4 (5.3)

10.8 (5.0)

BPRS negative

9.9 (2.2)

8.3 (2.8)

9.9 (2.6)

9.3 (2.5)

CGI-S

4.2 (0.7)

3.5 (0.9)

4.0 (0.6)

3.7 (0.7)

SANS

50.7 (15.9)

43.8 (18.0)

51.3 (13.9)

48.1 (13.0)

Y-BOCS

14.5 (10.0)

12.0 (9.8)

9.6 (11.0)

8.9 (10.5)

MADRS

14.0 (6.6)

11.8 (7.1)

14.6 (6.7)

13.4 (6.3)

BPRS, Brief Psychiatric Rating Scale; CGI-S, Clinical Global Impressions-Severity of illness scale; MADRS Montgomery Asberg Depression Rating Scale; SANS, Schedule for Assessment of Negative Symptoms; Y-BOCS, Yale–Brown Obsessive–Compulsive scale.

 

Table 2: Comparison of anticoagulation control between vitamin K- and placebo-treated groups


Vitamin K group

Placebo group


Before study

Intervention period

Difference

Before study

Intervention period

Difference

SD of INR

0.72a ± 0.11

0.47 ± 0.17

-0.24 ± 0.14

0.7 ± 0.11

0.59 ± 0.15

-0.11 ± 0.18

Time in range, %

59 ± 20

87 ± 14

28 ± 20

63 ± 18

78± 17

15 ± 20

No. of dose changes, median (range)

5 (3-7)

2 (0-5)

-2 (-5-0)

5 (3-8)

3 (1-8)

-1 (-3-3)

Table 1. Change scores from baseline


Venlafaxine ER

(n=179)

Sertraline

(n=173)

Placebo

(n=179)

CAPS-SX17

-41.5 (-45.7 to -37.4)

-39.4 (-43.7 to -35.2)

-34.2 (-38.3 to -30.0)