The emergence of social media in the last 20 years has reshaped how people engage with each other.1 By enabling users to generate and share content with large audiences instantaneously, oftentimes with little monitoring or censorship, social media has fundamentally changed how people consume and communicate information.2 The shift towards large-scale public interactions provides unique opportunities for measuring and tracking changes in public opinion.3 This makes social media valuable for capturing trends in population-level attitudes that overcome some limitations of traditional survey methods, such as social desirability and acquiescence bias.4 This is especially important for public health researchers interested in how shifts in cultural norms impact health. One relevant and emerging area of research is using social media to capture population-level racial sentiment or the degree to which people view racial groups in a positive or negative way.5–7 We examined geographic differences and temporal changes in sentiment toward different racial groups in the United States (US) from 2011 to 2021. In doing so, we provide practical guidance for using social media data in public health research and offer an example of using Twitter to measure area-level racial sentiment as an indicator of cultural racism, also an emerging area of research.8

Cultural racism is the infusion of the ideology of racial hierarchy into the values, language, imagery, symbols, and unstated assumptions of the larger society.9 Racist beliefs are associated with racial discrimination at the interpersonal level and contribute to the perpetuation of structurally racist policies and practices.10,11 Systemic constraints on civil rights and access to resources and opportunities based on race further reinforce racist beliefs, thereby creating an unremitting cycle of racial inequities across various dimensions (e.g., criminal justice, education, and health). Research has shown that regional differences in self-reported experiences of racial discrimination, as well as population-level racial sentiment, are associated with negative mental and physical health outcomes for Black people in the US.10,11 Research has also demonstrated that cultural attitudes toward different social groups change alongside large-scale events.12 During World War II, the Japanese were the least favored group in the US, while Muslims became the least liked after the 9/11 attacks.13

Until recently, measuring temporal and geographic trends in cultural racism remained challenging. Common approaches for measuring cultural racism involve assessing community members’ attitudes toward minoritized racial groups using self-report or indirect behavioral measures.14,15 These approaches are expensive, time-consuming, resource-intensive, and have a time delay.3 Alternatively, the last few years have ushered in a new internet era of research using online media for investigating real-time racial disparities, including racial health disparities. For instance, research has linked the number of Google searches using the n-word to area-level disparities in birth outcomes for Black mothers.10 Other studies have turned to social media applications such as Twitter, which have become digital town squares for public discourse on a variety of issues. Importantly, the sense of anonymity provided by web-based spaces emboldens people to express views they may not express during in-person interactions.16 Thus, Twitter offers researchers real-time, large-scale public opinion data. We have used Twitter data to quantify the increase in anti-Asian sentiment following the COVID-19 pandemic5 and evaluated changes in attitudes toward Black people in the wake of the 2020 Black Lives Matter (BLM) Protests.6 Our social-media-derived racial sentiment measures were not limited to single marginalized identities, nor did our work focus on hate speech that may not be representative of population-level sentiment.17,18

Building on this prior work, we examined area-level racial sentiment across the US from 2011 to 2021. The objective is not to examine individual Twitter users’ unique experiences. Rather, we conducted descriptive analyses to examine temporal and geographic trends in online discussions referencing different racial and ethnic communities in the US. By mapping sentiment scores for different racial groups across space and time, we hope to provide a means for assessing policy-related changes as well as targets for intervention.

METHODS Step 1: Collection of Twitter Data

We collected Twitter data using Twitter’s Application Programming Interface (API) for Academic Research. More information can be found at: https://developer.twitter.com/en/use-cases/do-research/academic-research.

Data collection can be filtered using different parameters, such as specifying tweet locations, languages, or topics.

At the end of 2020, Twitter announced that it was launching its latest version of the developer’s Twitter API, Twitter API V2. The launch included a new Academic Research product tract API,19 which grants access to the V2 endpoints and full historical data of public tweets. It also expanded the monthly tweet volume cap of 500,000 under the Standard product tract API to 10 million tweets. Qualified users include research scholars and graduate students with clearly defined research objectives that are not meant for commercial use (see eMaterials; https://links.lww.com/EDE/C69 for additional details). The API for Academic Research was discontinued in May 2023.

We collected a random 1% sample of publicly available tweets from 1 January 2011 to 31 December 2021. We restricted our analyses to tweets that were in English, from the US, had location information to identify at least the state from where the tweet was sent, and used one or more of the 90 race-related keywords (eTable 1; https://links.lww.com/EDE/C69). Replies, retweets, and quoted tweets are included in our sample. Our keyword list was constructed using prior studies examining race-related online conversations,20 and an online database of racial slurs.21 For filtering tweets using keywords, the character limit is 1,024 for the Academic Research API. Our analytic sample included 55,844,310 tweets from 3,699,646 users.

Twitter’s API supports the collection of metadata such as users’ profile descriptions, users’ following and followers count, etc. We also identified the state, county, and other geographic units of tweets. Details can be found at: https://developer.twitter.com/en/docs/twitter-api/fields. An example code used to process Twitter data is included in the online eMaterials; https://links.lww.com/EDE/C69. Additional details on query parameters are included in eTable 2; https://links.lww.com/EDE/C69.

We processed all collected tweets before conducting our analyses. Irrelevant tweets can be collected if the terms are not carefully specified. For instance, using a group term such as “black girl” as a keyword in the query may pull tweets such as “the girl in black shirt” which is irrelevant to the research topic about racism. Hence, we removed tweets that did not exactly match the key terms in the list. For multiple-word phrases, we specified that the tweet had to contain all the words in the specified order. Tweets were categorized into racial and ethnic groups and were referenced according to the keywords used. Next, we created new variables for tweet location information. Collected tweets have a “place_full_name” variable that contains both city- and state-level information for each individual tweet. We extracted city and state values from the “place_full_name” and stored them separately. For those tweets that did not have any city or state information, we used users’ locations instead. For any city or state information that cannot be extracted from any of the other geographic variables, we assigned them as missing values. We added state Federal Information Processing System (FIPS) and county FIPS codes to tweets that have valid state and county information. With this geographic information, we can generate a series of maps and graphs and conduct geospatial analyses.

During this process, we also generated a cleaned version of the tweet text by making all characters lowercase and removing any links, numbers, or symbols. We removed the hashtag (#) and at (@) symbols but retained the text of the tweet so that the machine learning model could focus on the analysis of the text. This new variable could be used for subsequent natural language processing analyses.

This study was determined to be exempt by the University of Maryland College Park Institutional Review Board (1797788-1).

Step 2: Conduct Analyses with Twitter Data Sentiment Analysis

We classified sentiment using support vector machine (SVM), a supervised machine learning model. Sentiment classifiers were trained on a combination of pre-labeled data obtained through publicly available sources and manually annotated data. Publicly available training data came from Sentiment140 (n = 498), Kaggle (n = 7,086), and Sanders (n = 5,113), and an additional 6,481 tweets were labeled by our research group. Three graduate students categorized the sentiment of the tweet as negative, neutral, or positive. First, each coder independently labeled a sample of 331 tweets independently and discrepancies were resolved through team meetings. Coders then analyzed 150 more tweets and an acceptable level of agreement was reached (kappa = 90%) before coders independently labeled 2,000 tweets at a time. This yielded a total of 19,178 pre-labeled tweets that were used as training data, with 6,673 (35%) labeled as negative. The text of each tweet was cleaned and used to create a feature set via the term frequency-inverse document frequency (TF-IDF) approach, which is a popular feature extraction algorithm for text data that is implemented before training text-based machine learning models.22 TF-IDF was implemented using the TfidfVectorizer function from the SKLearn23 package for Python,24 which converted the tweets into a sparse matrix where each row represents a tweet and each column represents a unique word in the entire corpus of tweets. The TF-IDF value in each cell of the matrix is calculated by multiplying term frequency and inverse document frequency (TF * IDF), where TF is the number of times the word appears in the tweet divided by the total number of words in the tweet, and IDF is the log of the number of tweets divided by the number of tweets containing the word. Although we have published results for positive sentiment,25 we focused most of our analyses on negative sentiment for the current work. We trained separate binary classifiers for negative and positive sentiment because they require less training data and are easier to optimize than a single, multi-task classifier that could classify positive, negative, and neutral tweets. Thus, the final output from the negative sentiment model is a binary classification, with 1 indicating negative sentiment and 0 indicating non-negative sentiment.

SVM has been widely used in classification modeling. It is effective in high-dimensional spaces, maximizing predictive accuracy while avoiding overfitting.26 Some studies have compared SVM to different supervised models, such as Naïve Byes, k-nearest neighbors algorithm (kNN), Maximum Entropy on Sentiment Classification,27,28 and Valence Aware Dictionary and sEntiment Reasoner (VADER),29 and found SVM to perform better. To assess the model performance in classifying negative sentiment tweets, we used five-fold cross-validation to calculate accuracy and F1 score, and grid search was used to tune the hyperparameter when training the model.30 Accuracy is quantified as the number of posts with the correct prediction divided by the total number of tweets in the testing data set. The F1 score is quantified using precision (positive predicted value) and recall (sensitivity), and a high F1 score indicates that a model is robust in predicting posts that are labeled as 1 (1 = negative and 0 = not negative). For sentiment analysis, we compared the performance of three widely used models including (1) logistic classifier, (2) stochastic gradient descent (SGD), and (3) SVM. The accuracies and FI scores for the three respective models were: 0.90 and 0.81, (logistic); 0.91 and 0.83 (SGD), and 0.91 and 0.84 (SVM). Precision, recall/sensitivity, and specificity were also compared (Table 1). Overall, the three models had similar performance, with SVM having the highest accuracy, F1 score, and recall/sensitivity scores. We used the trained SVM model to analyze our Twitter data set for negative sentiment classification.

TABLE 1. - Comparison of Model Performance Accuracy F1 Score Precision Recall/
Sensitivity Specificity Logistic 0.8975 0.8145 0.9078 0.7387 0.9672  SDG 0.9052 0.8336 0.8965 0.7791 0.9606  SVM 0.9064 0.8381 0.8855 0.7956 0.9549

SGD indicates stochastic gradient descent; SVM, support vector machine.

We performed locally estimated scatterplot smoothing to fit smooth curves to the daily prevalence of negative sentiment. Maps of the Twitter data can help to illuminate temporal and geographic trends. We used quantum geographic information system (QGIS), a free and open-source software for geospatial data, to visualize the results with choropleth maps.31 Different colors represent different values of the geographical areas with darker red colors indicating increasing negative sentiment.

RESULTS Visualizing a Decade of Social Media Data

We did not know the racial and ethnic identity of the Twitter users, so we focused on the inclusion of racial keywords in the text of the tweet. While curating race-related keyword lists, we were mindful of contextual nuances. One prominent example of this is the different iterations of the n-word. The term has historically undergone many variations in meaning, sentiment, and even phonetics. N*gger evolved from n*gro and is generally considered derogatory, whereas n*gga is understood as a colloquial term often used in African-American Vernacular English.32 While all variations are relevant to Black or African American communities, they are not necessarily indicative of expressed negative sentiments. Therefore, sentiment classification occurred independently of race classification. For example, the tweet “my n*gga itz friday” is classified as 0 (not negative) whereas “stfu before yall get stabbed n*gga” is classified as 1 (negative sentiment). Like our previous research, “n*gga/s” consistently remained the most frequent race-related keyword during this time period with 45.2% of tweets containing the word n*gga being associated with negative sentiment. The term “n*gger” was rare, with only 137,985 (0.24% of tweets) mentions in our entire dataset, of which 87.43% were negative. “Black Lives Matter” and its acronym “BLM” entered the top 10 keywords in 2020. Immigration became a top 10 keyword in 2017–2019. Other common keywords across this period included “racism/racist,” “Black people,” “Mexican,” and “Chinese” (eTable 3; https://links.lww.com/EDE/C69).

Tweets referencing Black Americans were the most frequently occurring tweets (Figure 1). Beginning in 2016, we observed a greater proportion of tweets referencing other racial and ethnic minoritized groups. These include tweets referencing racially minoritized groups in general (captured by the “minority” category). Tweets referencing immigrants also increased in 2017–2019. In total across all years, we classified 22,514,919 (39.9%) of tweets as negative sentiment. Negative sentiment by specific racial and ethnic groups is presented in eTables 4–5; https://links.lww.com/EDE/C69. eTable 6; https://links.lww.com/EDE/C69 presents a timeline of Twitter content moderation policies during this time period.

FIGURE 1.:

Proportion of tweets referencing racial and ethnic groups over time.

Examining descriptive temporal trends, we found that negative sentiment increased by 16.5% at the national level. Tweets referencing Black and Middle Eastern people have the highest proportion of tweets with negative sentiment (Figure 2). We observed a steady increase in the proportions of tweets referencing Black people from 2011 to 2017. After a brief period of decline in 2020 corresponding to BLM protests in the summer of 2020 after the killings of Breonna Taylor, Ahmaud Arbery, and George Floyd, the proportion of negative tweets referencing Black people began to rise again.6

FIGURE 2.:

Proportion of tweets referencing specific racial and ethnic groups that are negative from 2011 to 2021. We performed locally estimated scatterplot smoothing (LOESS) to fit smooth curves to the daily prevalence of negative sentiment using a span of 5%. The shaded area represents 95% confidence bands around the smoothed trend line.

Tweets referencing Middle Eastern people that were negative steadily increased from 2011 to 2016. There was a plateauing trend for a few years and then a decline in negative sentiment in 2020–2021. American Islamophobia conflates many minority groups and has become a catch-all for anyone who appears “Muslim-like,” like Middle Easterners.33 This trend is exemplified in the rise in negative sentiment tweets referencing Middle Easterners following the San Bernardino shooting in December 2015—a terrorist attack that killed 14 people—even though it was perpetrated by individuals of Pakistani descent, which is not geographically part of the Middle East but is in South Asia.

For tweets referencing Latinx, there was an increase in negative sentiment from 2015 to 2018, peaking at the end of 2018 with the midterm elections and national discussions of the border wall and immigration.34,35 The proportion of tweets referencing Asians that were negative has steadily climbed over time since 2011. Spikes in negative sentiment were observed in March 2020 with the emergence of the COVID-19 pandemic and the use of stigmatizing language to refer to COVID-19, such as “China virus” and “Chinese virus.”5,36 Another increase in the negative sentiment of tweets referencing Asians was observed in March 2021 with the Atlanta spa shootings, when a white man shot and killed eight primarily Asian women at Atlanta-area spa and massage parlors (Figure 2).37

Overall, the US became more negative in racial sentiment, referencing racially minoritized groups (Figures 3–4) with all states experiencing an increase in negative racial sentiment from 2011 to 2021. States that experienced the greatest change toward more negative racial sentiment include Idaho, Utah, Wyoming, Vermont, and Maine. States that experienced small changes in increasing negative sentiment include Michigan, Montana, Illinois, and Maryland (Figure 5). County-level maps showing the proportion of tweets referencing minoritized groups that are negative in 2011 and 2021 are presented in eFigures 1–8; https://links.lww.com/EDE/C69.

FIGURE 3.:

State-level proportion of tweets referencing racial and ethnic minoritized groups that are negative in sentiment in 2011.

FIGURE 4.:

State-level proportion of tweets referencing racial and ethnic minoritized groups that are negative in sentiment in 2021.

FIGURE 5.:

Changes in the proportion of tweets referencing racial and ethnic minoritized groups that are negative from 2011 to 2021. Positive numbers indicate a larger proportion of negative sentiment tweets in 2021 compared with 2011.

Maps of county-level negative sentiment for specific racial and ethnic groups are presented in the online eMaterials; https://links.lww.com/EDE/C69 and show increases in the proportion of tweets referencing Asian, Black, and Latinx that are negative in sentiment from 2011 to 2021 (eFigures 1–8; https://links.lww.com/EDE/C69).

We have constructed a publicly available geoportal to allow users to interact with the derived Twitter measures. Users can navigate to the geoportal here: (https://experience.arcgis.com/experience/e3709af843fc427a9db12f6a28b12efb/?draft=true). Users are able to: (1) select the Twitter measure to display (i.e., the proportion of tweets referencing Black people that are negative) with darker colors representing a higher prevalence of that measure and (2) type a location (i.e., county name) or address in the search bar and the map will zoom to that area.

DISCUSSION

The current study explored temporal and geographic trends in sentiment toward different racial and ethnic groups by analyzing a large sample of tweets posted from 2011 to 2021. Our exploratory study found there was an overall increase in the proportion of negative tweets referencing minoritized groups over this 10-year period, followed by a slight decline in the final years we sampled. This temporal trend was observed in all states, such that there were no states showing a decline in the proportion of negative sentiment tweets referencing racially minoritized groups between 2011 and 2021. Southern states tended to have a higher proportion of negative sentiment tweets, with Alabama, Arizona, Georgia, Louisiana, and Mississippi being in the top 10 states with the highest proportion of negative sentiment tweets for both 2011 and 2021.

Stratifying temporal trends by race and ethnicity revealed unique patterns of change that reflect historical events specific to each group. For instance, spikes in sentiment aligned with notable race-related societal events such as the BLM protests in the summer of 2020, the Atlanta spa shooting, and political disputes over former President Trump’s border wall. This demonstrates the added utility of social media data for tracking attitudes toward different minoritized groups in real-time, compared with traditional racial sentiment measures that rely on expensive, time-consuming, nondynamic, and often retrospective approaches. Moreover, our findings are consistent with work showing that historical context, immigration patterns, and politics can influence cultural attitudes toward different racial groups over time.5,6,38

The current findings are exploratory and aim to provide descriptive findings about trends in racial sentiment. We provide an example of how to evaluate the causal impact of racialized historical events in the online eMaterials; https://links.lww.com/EDE/C69 (i.e., change in Black sentiment before and after the protests began in May 2020; eFigures 9–10; https://links.lww.com/EDE/C69). The ability to monitor real-time changes in racial attitudes is particularly important for informing interventions. For example, the rapid decline in negative sentiment tweets toward Black Americans following BLM protests in the summer of 2020 indicated a window of opportunity for institutions to enact policy changes aimed at increasing diversity and inclusion.6 Moreover, monitoring of racial attitudes may also be used to predict increases in hate crimes against the referenced groups. This is exemplified by the rise in anti-Asian rhetoric during COVID-19 lockdowns, which remained elevated and corresponded with a rise in violence against Asian Americans.39 Thus, our findings represent a unique approach for tracking racial sentiment that can be used to reduce the impact of racism, as well as predict the rise of hate crime incidents.

Our approach to monitoring racial sentiment through Twitter data also provides insights into the interplay of cultural and structural racism. For example, there was a sustained increase in negative sentiment toward Middle Easterners following the San Bernardino shooting. This was followed by the “Muslim ban”—the travel ban imposed on individuals coming from primarily Muslim-majority countries in 2017. Although the policy was eventually struck down, it further added to the anxieties of Muslim Americans. A qualitative study examining experiences of discrimination for Middle Eastern women revealed Middle Eastern women believed the “Muslim ban” bolstered stereotypes and negatively impacted everyday interactions, including engagement within health care settings.40 Our approach provides a means to address a major gap in the racial health disparities literature—understanding differences and similarities in the challenges faced by minoritized groups that have been relatively understudied (e.g., Asian Americans and Middle Easterners).

The current study is not without limitations. Though our keyword list is not exhaustive, we attempted to be as comprehensive as possible by balancing search constraints, which limited the number of keywords and characters. Another potential limitation is that our sentiment analysis model captures the emotional tone of the tweet as a whole. For example, statements like “I hate Asians” and “the hate of Asians is bad” may both be classified as negative. This may explain the rise in the proportion of negative sentiment tweets following the Atlanta spa shootings, which included tweets expressing outrage over the event. In a prior study, we found expressions of solidarity increased after the Atlanta spa shootings.37 The sentiment analysis model is able to analyze millions of tweets to provide a description of the overall emotional tone of tweets referencing different racial and ethnic minorities. Alternatively, qualitative research has been well suited to provide a more in-depth understanding of the themes and concepts to inform and interpret the quantitative findings. Our model comparisons were not exhaustive of all available models, and an important activity for investigators is to evaluate which models perform best for their data and research purposes. Furthermore, our models did not use emojis or symbols, and our analysis solely focused on the text data. This study paves the way for future research on racial sentiment to incorporate image and text data and build multimodal machine learning models to extract important meaning and context of the posts. Racial and ethnic minoritized populations are slightly overrepresented on Twitter compared with the US general population. For example, 17% of adult Twitter users are Latinx, compared with 15% of the US adult population.41 Twitter users skew younger than the general US population, with 42% of Twitter users being between the ages of 18 and 29,42 whereas 19- to 34-year-olds account for only 20.8% of the US population.43 Our findings may not be generalizable to the US population as a whole. Twitter users differ in their frequency of tweeting, with the majority of Twitter users being frequent users. However, in our data set of race-related tweets, tweets from users who tweeted more than 1,000 times per year represented less than 1% of all tweets. Furthermore, over the last decade, there have been increases in Twitter volume and changes in Twitter policies, including moderation policies that can impact the findings we observe.

Despite these limitations, social media data provides access to a massive sample of opinions on race-related topics with fine-grained temporal and geographic information. Improvements in machine learning models to incorporate image data are needed. Our sentiment analysis model provides a relatively simple tool to build upon. Nonetheless, the benefit of utilizing a trained sentiment model is that it can be potentially more easily adopted by other research teams and fine-tuned based on their own use cases, thus facilitating the reproducibility and sustainability of these lines of research. Sentiment models can also often be implemented on standard desktop computers, unlike the neural network alternatives, which can necessitate higher-performance computational environments and inadvertently impede research teams without access to these computational resources. In addition, qualitative data are still needed to provide greater nuance, context, and understanding of human phenomena. Future work can build on this approach to investigate the relationship between racial sentiment and racial inequities in medical, legal, and economic settings. This approach can also be extended to examine racial attitudes across the globe and tweets in other languages. In addition, data collection can be expanded to collect posts from other social media platforms. One goal of this study is to support the generation of hypotheses to guide research. For example, given that we see trends in increasing area-level negative racial sentiment, future research could explore if places with more negative expressions are more likely to enact policies that restrict rights, access, and resources to minoritized populations. This study exemplifies the use of social media data to investigate discussions related to race in the US and to facilitate future work in this area.

REFERENCES 1. Ortiz-Ospina E, Roser M. The rise of social media. Our World in Data. Published online March 20, 2023. Available at: https://ourworldindata.org/rise-of-social-media. Accessed 3 April 2023. 2. Zhong B. Social Media Communication: Trends and Theories. John Wiley & Sons; 2021. 3. Klašnja M, Barberá P, Beauchamp N, Nagler J, Tucker JA. Measuring Public Opinion with Social Media Data. In: Atkeson LR, Alvarez RM, eds. The Oxford Handbook of Polling and Survey Methods. Oxford University Press; 2018. doi:10.1093/oxfordhb/9780190213299.013.3 4. Krumpal I. Social desirability bias and context in sensitive surveys. In: Encyclopedia of quality of life and well-being research. Published online 2014:6037–6043. 5. Nguyen TT, Criss S, Dwivedi P, et al. Exploring U.S. shifts in anti-Asian sentiment with the emergence of COVID-19. Int J Environ Res Public Health. 2020;17:7032. doi:10.3390/ijerph17197032. 6. Nguyen TT, Criss S, Michaels EK, et al. Progress and push-back: how the killings of Ahmaud Arbery, Breonna Taylor, and George Floyd impacted public discourse on race and racism on Twitter. SSM Popul Health. 2021;15:100922. 7. Hswen Y, Qin Q, Williams DR, Viswanath K, Subramanian SV, Brownstein JS. Online negative sentiment towards Mexicans and Hispanics and impact on mental well-being: a time-series analysis of social media data during the 2016 United States presidential election. Heliyon. 2020;6:e04910. 8. Michaels EK, Lam-Hine T, Nguyen T, Gee G, Allen A. The water surrounding the iceberg: cultural racism and health inequities. Milbank Q. 2023;101:768–814. 9. Williams DR, Lawrence JA, Davis BA. Racism and health: evidence and needed research. Annu Rev Public Health. 2019;40:105–125. 10. Chae DH, Clouston S, Martz CD, et al. Area racism and birth outcomes among blacks in the United States. Soc Sci Med (1982). 2018;199:49–55. 11. Kim G, Parmelee P, Bryant AN, et al. Geographic region matters in the relation between perceived racial discrimination and psychiatric disorders among black older adults. Gerontologist. 2017;57:1142–1147. 12. Bogardus ES. Social distance in the city. Proc Public Am Sociol Soc. 1926;20:40–46. Available at: https://brocku.ca/MeadProject/Bogardus/Bogardus_1926.html. Accessed 28 February 2023. 13. Parrillo VN, Donoghue C. The national social distance study: ten years later. Sociol Forum. 2013;28:597–614. 14. McConnell AR, Leibold JM. Relations among the implicit association test, discriminatory behavior, and explicit measures of racial attitudes. J Exp Soc Psychol. 2001;37:435–442. 15. Axt JR. The best way to measure explicit racial attitudes is to ask about them. Soc Psychol Personal Sci. 2018;9:896–906. 16. Suler J. The online disinhibition effect. Cyberpsychol Behav. 2004;7:321–326. 17. Chae DH, Clouston S, Hatzenbuehler ML, et al. Association between an internet-based measure of area racism and black mortality. PLoS One. 2015;10:e0122963. 18. Arcila-Calderón C, Amores JJ, Sánchez-Holgado P, Vrysis L, Vryzas N, Oller Alonso M. How to detect online hate towards migrants and refugees? Developing and evaluating a classifier of racist and xenophobic hate speech using shallow and deep learning. Sustainability. 2022;14:13094. 19. Academic Research - Preparing for the Application. Twitter Development Platform. Available at: https://developer.twitter.com/en/products/twitter-api/academic-research/application-info. Accessed 5 June 2023. 20. Bartlett J, Reffin J, Rumball N, Williamson S. Anti-Social Media. Demos. 2014. Available at: https://apo.org.au/node/37598. Accessed 23 May 2023. 21. The Racial Slur Database. Available at: http://www.rsdb.org/. Accessed 28 April 2023. 22. Sammut C, Webb GI, eds. TF–IDF. In: Encyclopedia of Machine Learning. Springer US; 2010:986–987. doi:10.1007/978-0-387-30164-8_832 23. Pedregosa F, Varoquaux G, Gramfort A, et al. Scikit-learn: machine learning in Python. J Mach Learn Res. 2011;12:2825–2830. 24. Van Rossum G, Drake FL. Python 3 Reference Manual. CreateSpace; 2009. 25. Nguyen TT, Meng HW, Sandeep S, et al. Twitter-derived measures of sentiment towards minorities (2015–2016) and associations with low birth weight and preterm birth in the United States. Comput Hum Behav. 2018;89:308–315. 26. Jakkula V. (2006). Tutorial on support vector machine (svm). School of EECS, Washington State University, 37(2.5), 3. 27. Pang B, Lee L, Vaithyanathan S. Thumbs up? Sentiment Classification using Machine Learning Techniques. In: Conference on Empirical Methods in Natural Language Processing (EMNLP); 2002:79–86. Available at: https://aclanthology.org/W02-1011.pdf. 28. Ye Q, Lin B, Li YJ. Sentiment classification for Chinese reviews: a comparison between SVM and semantic approaches. In: 2005 International Conference on Machine Learning and Cybernetics. Vol 4; 2005:2341–2346 Vol. 4. doi:10.1109/ICMLC.2005.1527335 29. Borg A, Boldt M. Using VADER sentiment and SVM for predicting customer response sentiment. Expert Syst Appl. 2020;162:113746. 30. LaValle SM, Branicky MS, Lindemann SR. On the relationship between classical grid search and probabilistic roadmaps. Int J Rob Res. 2004;23:673–692. 31. QGIS Development Team. Welcome to the QGIS project! Published 2022. Available at: https://qgis.org/en/site/. Accessed 28 February 2023. 32. Rahman J. The N word: its history and use in the African American community. J Engl Linguist. 2012;40:137–171. 33. Love E. Confronting Islamophobia in the United States: framing civil rights activism among Middle Eastern Americans. Patterns Prejud. 2009;43:401–425. 34. Fleuriet KJ, Castellano M. Media, place-making, and concept-metaphors: the US-Mexico border during the rise of Donald Trump. Media Cult Soc. 2020;42:880–897. 35. Jaramillo-Dent D, Pérez-Rodríguez MA. #MigrantCaravan: the border wall and the establishment of otherness on Instagram. New Med Soc. 2021;23:121–141. 36. Hswen Y. Online hate: the new virus. Am J Public Health. 2022;112:545–547. 37. Criss S, Nguyen TT, Michaels EK, et al. Solidarity and strife after the Atlanta spa shootings: a mixed methods study characterizing Twitter discussions by qualitative analysis and machine learning. Front Public Health. 2023;11:952069. 38. Hswen Y, Zhang A, Sewalk KC, Tuli G, Brownstein JS, Hawkins JB. Investigation of geographic and macro level variations in LGBTQ patient experiences: longitudinal social media analysis. J Med Internet Res. 2020;22:e17087. 39. Findling MG, Blendon RJ, Benson J, Koh H. COVID-19 has driven racism and violence against Asian Americans: perspectives from 12 national polls. Health Affairs Forefront. 2022. 40. Wojcik S., Hughes A. (2019). Sizing up Twitter users. PEW research center, 24, 1–23. 41. Wojcik S, Hughes A. Sizing Up Twitter Users. Pew Research Center: Internet, Science & Tech. Published April 24, 2019. Available at: https://www.pewresearch.org/internet/2019/04/24/sizing-up-twitter-users/. Accessed 23 May 2023. 42. Auxier B, Anderson M. Social Media Use in 2021. Pew Research Center: Internet, Science & Tech. Published April 7, 2021. Available at: https://www.pewresearch.org/internet/2021/04/07/social-media-use-in-2021/. Accessed 23 May 2023. 43. Kaiser Family Foundation. Population Distribution by Age. KFF. Published October 28, 2022. Available at: https://www.kff.org/other/state-indicator/distribution-by-age/. Accessed 23 May 2023.