Order of Operations in Sociophonetic Analysis (NWAV49 Presentation)
Contemporary sociophonetic analysis involves a pipeline of data processing steps to transform a spreadsheet of acoustic measurements into interpretable results. Some similar techniques have been compared, like formant normalization (Adank et al. 2004, Barreda forthcoming) and vowel overlap measures (Nycz & Hall-Lew 2013, Kelley & Tucker 2020). However, the order that these steps should be applied to the data has not been discussed. This paper shows that the order of operations matters: different permutations of the same processes on the same data can result in drastic differences in the conclusions drawn from that data.
As a case study, this paper focuses on vowel changes in North American English recently dubbed the Low-Back-Merger Shift (Becker 2019), which involves merging /ɑ,ɔ/ and lowering/centralizing /ɪ,ɛ,æ/. Formant measurements from 53 Western American English speakers are analyzed with Pil-lai scores to measure vowel overlap (Hay et al. 2006) and, to measure degree of shifting in the front vowels, comparison to “benchmarks” found in Labov et al. (2006) and the Low-Back-Merger Shift Index (Becker 2019, Boberg 2019). Seven data processing steps (isolating midpoints from trajectories, removing outliers, removing stopwords, removing unstressed vowels, isolating allo-phones, vowel normalization, and removing diphthongs and /ɚ/) are applied to the data before cal-culating these three measures. Crucially, since these seven steps can be rearranged into 5,040 per-mutations, the data is processsed 5,040 times, once for each ordering. To be clear, identical formant measurements were used as input, and identical functions were applied to that data: the only modification was the order that these steps occurred.
While some pipelines produced identical results, hundreds of unique values were produced for each speaker. For the /ɑ,ɔ/ merger, some speakers’ Pillai scores were similar while others varied drastically across permutations. In an extreme case, one speaker had a score of 0.16 in one permu-tation, indicating lots of overlap, and 0.95 in another permutation, indicating virtually no overlap. When comparing normalized formant measurement to “benchmarks,” some permutations suggest that some speakers’ vowels are shifted, while other permutations do not. In fact, the proportion of speakers considered “shifted” was 58%–78%, depending on the permutation. Regarding the Low-Back-Merger Index, intra-speaker variability varied, with an extreme speaker ranging from 1.85 (suggesting no shifting) to 2.81 (on par with California English, where the shift is robust). Importantly, these pipelines produced non-negligible differences: they were similar in size to differences considered sociolinguistically meaningful.
This paper shows that the order of operations is an important piece of information in sociophonetic analysis. Because this order is typically not discussed in methods sections, any reported value is somewhat of a random draw from the range of possible values that could be taken from a particular dataset using a particular analysis. This makes comparison across studies unreliable, if not impos-sible. Fortunately, there was some patterning in the results in all three measures (e.g. isolating mid-points before removing outliers always resulted in a higher Pillai score than vice-versa), so this study concludes with recommendations on how data processing should be ordered in future sociophonetic studies.