Guide to benchmark reports
2010 Survey of women’s experiences of maternity services

Benchmark reports are produced for most NHS national surveys to show how the survey results for each trust participating in a particular survey compares with the results from all other trusts.

This guide is divided into six sections:

Section one: provides information specific to the 2010 maternity survey
Section two: describes the benchmark reports
Section three: describes how to use the benchmark reports and the limitations of the data
Section four: describes how to understand the data
Section five: provides guidance on using the benchmark reports to make comparisons between trusts
Section six: describes how the data in the benchmark reports is calculated

1.) The 2010 survey of women’s experiences of maternity services

Over 25,000 women from 144 trusts in England responded to the survey between April 2010 and August 2010, a response rate of 52%. Women were eligible for the survey if they had a live birth in February 2010 and were aged 16 or older. Women who gave birth in a hospital, birth centre, or maternity unit, or who had a home birth, were eligible. A similar survey of women using maternity services was carried out in 2007.

The results of the survey are primarily intended to be used by NHS trusts to help them identify areas where they need to improve performance and to note where they have performed well. For this reason we have produced the benchmark reports to allow NHS acute trusts to identify where their score lies in relation to all other acute trusts’ results.

The same data in the trust benchmark reports are also available in the Care Directory in the CQC website (see Further Information section for a link to this). The scores shown there are the same as in the benchmark reports, though are displayed as scores out of 10 rather than 100 (so are the benchmark scores divided by 10). A different approach is used to categorise trusts’ performance in the Care Directory, as it identifies where a trust performed ‘better’, ‘worse’, or ‘about the same’ as expected. Please see the document containing your trust’s Care Directory data for more information, or the explanation provided on the website.

The survey asked women about care received during pregnancy, labour and birth, and in the weeks following the birth of their baby. For some of the survey results, the women may have been referring to care received from their local Primary Care Trust. We have identified 19 questions that are suitable for benchmarking where the results can be directly attributed to the acute trust, rather than PCT. These 19 questions have been scored and are contained in your benchmark report. Results for all other questions will be sent separately to you in the form of weighted percentages of respondents. These are comparable to the results that were published for your trust from the 2007 survey of women’s experiences of maternity services.

We will also be providing an online tool to trusts, PCTs and SHAs following the national publication of results, similar to that provided with the NHS Staff Survey data. This will follow the main publication of the survey, and will be sent to the survey lead at your trust. It contains data from the survey and allows for comparisons with the 2007 survey, as well as across England, SHA and other trusts.

2.) Description of the benchmark reports
The graphs included in the reports display the scores for a trust, compared with national benchmarks. Each bar represents the range of results for each question across all trusts that took part in the survey. In the graphs, the bar is divided into three sections:

• the red section (left hand end) shows the scores for the 20% of trusts with the lowest scores
• the green section (right hand end) shows the scores for the 20% of trusts with the highest scores
• the orange section (middle section) represents the range of scores for the remaining 60% of trusts

The score for a trust is represented by a white diamond. If the diamond is in the green section of the bar, for example, it means that the trust is among the top 20% of trusts in England for that question. The line on either side of the diamond shows the amount of uncertainty surrounding the trust’s score, as a result of random fluctuation. These are known as lower and upper confidence intervals. Please see section three below for more detailed information about confidence intervals.

An example of a set of graphs from a benchmark report for trust X can be seen in chart one below. For the first question (Were you given a choice of having your baby at home?) it can be seen that nationally, scores varied between 40 (the lowest score) and 96 (the highest score). Trust X scored 72 for this particular question. The lower confidence interval is 63 and the upper confidence interval is 81. The threshold score for the 20th percentile is 66 and for the 80th percentile is 84.

Chart 1

[pic]

3.) How the benchmark reports should be used
Benchmark reports should be used to identify how a trust is performing in relation to all other trusts that took part in the survey. From this, areas for improvement can be identified.

Limitations of the data
Because the average scores for each trust are estimates based on a sample of women (those giving birth in January/February 2010) rather than all women receiving care from the trust, it is very often impossible to separate the performance of trusts. That is, in many cases the differences between trusts’ mean scores will not be statistically significant. This means that if we were to repeat the survey with a different sample of women, we would not be confident that the results would show the same differences. As such, data used in the benchmark reports is fundamentally not suitable for generating league tables.

Also, it should be noted that the data only show performance relative to other trusts: there are no absolute thresholds for ‘good’ or ‘bad’ performance. Thus, a trust may score lowly relative to others on a certain question whilst still performing very well on the whole. This is particularly true on questions where the majority of trusts score very highly.

4.) Understanding the data
Since the score is based on a sample of women in a trust rather than all women, the score may not be exactly the same as if everyone had been surveyed and had responded. To account for this, a ‘confidence interval’ is calculated. For each trust, then, the benchmark report shows three values for each question – an average (‘mean’) score as well as its lower (‘lcl’) and upper (‘ucl’) confidence limits.

A confidence interval is calculated as an indication of the range within which the ‘true’ score would lie if all women had been surveyed. The confidence interval gives upper and lower limits of a range within which you have a stated level of confidence that the ‘true’ average lies. These are commonly quoted as 95% confidence intervals, which is the level used in the benchmark reports. They are constructed so that you can be 95% certain that the ‘true’ average lies between these limits.

For example, chart 2, below, show a trust’s score for a question asking if they were given a clear explanation of the reason for the 20 week scan they received. Trust X has an average score of 83, with a lower confidence limit of 77 and an upper confidence limit of 89. This means that we can be 95% confident that the ‘true’ trust score lies between 77 and 89.

Chart 2

[pic]

The width of the confidence interval gives some indication of how cautious we should be; a very wide interval may indicate that more data should be collected before any firm conclusions are made. In chart 3 below, the confidence intervals are very wide: the trust has a score of 61, with a lower confidence interval of 48 and an upper confidence interval of 73. The confidence intervals are wider here as fewer people provided a response (as it only reports the findings of those requiring stitches).

Chart 3

[pic]

When considering how a trust performs, it is very important to consider the confidence interval surrounding the score. In chart 3 above, it can be seen that the trust’s average score falls into the orange section of the graph – however, the lower confidence limit falls into the red section of the graph and the upper confidence limit exceeds the green section of the graph. This means that you should be more cautious about the trust’s result because, if the survey was repeated with a different sample of women, it is possible that their average score would be in a different place and would therefore show as a different colour.

Only if a trust’s average score and confidence intervals are completely in one section of the graph can you be confident that that is how a trust is performing. An example is below (chart 4) where it can be seen that the score for trust X and its lower and upper confidence intervals are shown within the green section or higher. This means that we can be confident that this trust’s ‘true’ score is in the top 20% of observed scores for this question.

Chart 4

[pic]

In summary: • If a trust’s average score and both of its confidence limits appear in green, you can be confident that the trust’s ‘true’ score is in the top 20% of all observed scores. • If a trust’s average score and both of its confidence limit appear in red, you can be confident that the trust’s ‘true’ score is in the bottom 20% of all observed scores. • If a trust’s average score is in one colour, but either of its confidence limits are shown as falling into another colour (for example, see chart 3 above) this means that you should be more cautious about the trust’s result because, if the survey was repeated with a different random sample of women, it is possible their average score would be in a different place and would therefore show as a different colour. • For example, if a trust’s average score is in red but their upper confidence limit is in orange or green, we cannot say that the trust’s ‘true’ score is in the bottom 20%.

This final point has important implications. By definition, one in five trusts will have an average score that is in the bottom 20% of all trusts. For many of these trusts, however, we cannot be confident that if we repeated the survey with a different sample they would again come out as being in the bottom 20%. The only instances where we can be confident that a trust’s ‘true’ average lies beneath this threshold are where both their average score and their upper confidence limit appear in red.
5.) Comparing scores between trusts
The confidence intervals make it possible to determine if the results from two trusts are significantly different. If the ranges for two trusts overlap then there is no significant difference between the trusts: we cannot be confident that the difference in the average scores does not simply result from random variation. If there is no overlap in the scores of two trusts, then we can be confident that the results for the two trusts are genuinely different.

For example, if trust A had an average score of 70 for a particular question, with a lower confidence limit of 60 and an upper confidence limit of 80; and trust B had a score of 80, with a lower confidence limit of 70 and an upper confidence limit of 90, then the two averages scores are not significantly different as the confidence intervals overlap. This is illustrated in table 1 (below).
By contrast, if trust C had an average score of 70 for a question, with a lower confidence limit of 66 and an upper confidence limit of 74; and trust D had a score of 80, with a lower confidence limit of 76 and an upper confidence limit of 84, then the two scores are significantly different as the confidence intervals do not overlap. This is illustrated in table 1.

Tables 1 (left) and 2 (right):
|Trust |qX_lcl |qX_mean |

Chart 5 (below) shows the ranges between the upper and lower confidence limits for all trusts for a question asking if they got the pain relief they wanted during labour and birth. This is shown as a line for each trust that shows the area in which we can be 95% confident that their ‘true’ score lies. As the results for this question are based only on those who did not have a planned caesarean, or who reported it was not possible to have pain relief, or did not want any, the confidence intervals are quite wide for some trusts.

The red line indicates the 20th percentile score, below which a value may be considered to be within the lowest 20% of trusts’ average scores for this question. The green line indicates the 80th percentile, above which values may be considered to be within the highest 20th percent of trusts’ average for this question.

It can be seen that when confidence intervals are considered, the scores for many trusts do not differ significantly from each other – each trust’s confidence interval overlaps with at least eight others, and the majority overlap with a great many other trusts. In other words, trusts are typically quite similar once confidence intervals are taken into account.

Where a trust’s score (including the confidence intervals) does not cross the red or green threshold lines you can be confident that that is how a trust is performing. For example, in the below chart, there are two trusts for which the score (including the confidence intervals) is over the green line indicating the highest 20% of scores, and five for which the score (including the confidence intervals) is clearly below the red line indicating the lowest 20% of scores.

Chart 5

[pic]

This strongly demonstrates why the data should not be used to construct league tables; league tables cannot fairly account for confidence intervals and, without these, differences are implied where there are none.

6.) How the data was calculated

The data in the benchmark reports is calculated by converting responses to particular questions into scores. These were calculated by converting each respondent’s answer to a question into a score (from 0 to 100) then averaging these to arrive at a single score for the trust, for each question. The higher the score, the better a trust is performing. An example of a scored question is shown below. A ‘scored’ questionnaire is available for each survey on the Care Quality Commission website which shows how each question is scored.

B17. Were the reasons for having a screening test for Down’s syndrome clearly explained to you?

1 ( Yes, definitely 100 2 ( Yes, to some extent 50 3 ( No 0 4 ( Don’t know/ Can’t remember -

In most cases, the scores are allocated such that the most positive possible response corresponds to a score of 100 and the least positive to a score of 0, with intermediary options assigned scores at equal intervals. Note that this approach is equivalent to that typically used with Likert scales.

Please also note that it is not appropriate to score all questions within the questionnaire for benchmarking purposes. This is because not all of the questions assess the trusts in any way (for example, question A4 “Roughly how many weeks pregnant were you when your baby was born?”), or they may be ‘filter questions’ designed to filter out respondents to whom following questions do not apply (for example C6 “Thinking about the birth of your baby, what kind of delivery did you have?”). As mentioned in section 1, it is also not possible to attribute responses to some of the questions to an acute trust, or to benchmark across all trusts, as services may have been delivered by the local Primary Care Trust.

Format of the Data
Results shown in the benchmark reports are based on ‘standardised’ data. We know that the views of a respondent can reflect not only their experience of NHS services, but can also relate to certain demographic characteristics, such as their age and sex. For example, older respondents tend to report more positive experiences than younger respondents. It is also likely that women who have previously given birth have different needs and expectations to those who are first time mothers. Because the mix of women varies across trusts (for example, one trust may serve a considerably younger population than another), this could potentially lead to the results for a trust appearing better or worse than they would if they had a slightly different profile of women. To account for this we ‘standardise’ the data. Standardising data adjusts for these differences and enables the results for trusts to be compared more fairly than could be achieved using non-standardised data. More detailed information for each survey is available on request to the Care Quality Commission survey team by contacting: patient.survey@cqc.org.uk

Care Quality Commission
November 2010
Further information

All NHS Trust and England results, questionnaire and scoring for the 2010 survey of women’s experiences of maternity services can be found at: http://www.cqc.org.uk/maternitysurvey2010.cfm To make the results more accessible to the general public, the results for each trust will also be available under the organisation search tool of the CQC website: http://caredirectory.cqc.org.uk/caredirectory/searchthecaredirectory.cfm (Enter a postcode or organisation name, then scroll down to ‘What people said about this trust’)

Full details of the methodology of the survey can be found at: http://www.nhssurveys.org/ More information on the programme of NHS patient surveys is available on the patient survey section of our website at: http://www.cqc.org.uk/patientsurveys.cfm Results from the 2007 survey of maternity services can be found at: www.cqc.org.uk/maternityservices2007 .