You are here: Home / Proposing / Cycle 7 Supplemental Call / Unconscious Bias

Unconscious Bias

Unconscious bias in the review process occurs when a reviewer holds a bias (of which they are often unaware) in favor of, or against, a proposal for reasons other than scientific merit. Because these biases are a result of our own culture and experiences, all reviewers are influenced by unconscious bias. Examples include gender, culture, age, prestige, language, and institutional biases. 

ALMA has studied the proposal scientific rankings in Cycles 0-6 that may portend of reviewer bias (see also Lonsdale et al. 2016, arXiv 1611.04795). Similar studies have been published by  Reid (2014, PASP, v126, p932) in analysis of Hubble Space Telescope (HST) proposals and by Patat (2016, Messenger, v165, p2) for ESO proposals. 

The ALMA proposal rankings have been correlated with the experience level of a Principal Investigator (PI) in submitting ALMA proposals, regional affiliation of the PI (Chile, East Asia, Europe, North America, or other), and gender. The analysis was conducted for both the Stage 1 proposal rankings, which are based on the preliminary scores from the reviewers, and the Stage 2 proposal rankings, which are based on the final scores from the reviewers after participating in a face-to-face panel discussion. A paper presenting the analysis has been accepted for publication in the PASP and is available on astro-ph. The main findings of the study are as follows:

  • PIs who submit an ALMA proposal in multiple cycles have systematically better proposal ranks than PIs who have submitted proposals for the first time.
  • PIs from Europe and North America have better proposal rankings than PIs from Chile and East Asia.
  • Consistent with Lonsdale et al. (2016), proposals led by men tend to have better Stage 1 rankings than proposals led by women when averaged over all cycles. This was most noticeably present in Cycle 3 but no discernible differences by gender are present in subsequent cycles. Nonetheless, in each ALMA cycle to date, women have had a lower success rate than men in terms of having their proposals added to the observing queue even after differences in demographics by gender are considered.
  • Comparison of the Stage 1 and Stage 2 rankings reveal no significant changes in the distribution of proposal ranks by experience level or gender as a result of the panel discussions. The proposal ranks for East Asian PIs show a small, but significant, improvement from Stage 1 to Stage 2 when averaged over all cycles.
  • Any systematics in the proposal rankings are introduced primarily in the Stage 1 process and not from the face-to-face discussions. 

While the origins of these systematics are uncertain, unconscious bias in the review process should not be discounted as a contributing factor.

ALMA is committed to awarding telescope time purely on the basis of the scientific merit. ALMA would like to make reviewers aware of the role that unconscious bias can play in the review process1Reviewers should also recognize that English is a second language for many, if not most, PIs. ALMA reminds the Reviewers to focus their reviews on the scientific merit of the proposals.

In Cycle 7, two changes were made to the proposal cover sheet in order to reduce potential biases in the proposal review process. First, the investigators are listed with the first letter of the first name and the full surname. Second, the list of investigators on the cover sheet have been randomized. The Science Assessors can see the team that is behind the proposal but cannot identify who is the PI.

 

 

 

1For more information on unconscious bias and how it can affect the review process, Reviewers are encouraged to read the Unconscious bias training module from the Canada Research Chairs here.