7 Identifying quality sources
Academics like to say “According to the literature…” or “A review of the literature found that…” But, what do we mean by “the literature?” The literature consists of the published works that document a scholarly conversation on a specific topic within and between disciplines. You will find in “the literature” documents that track the progression of the research and background of your topic. You will also find controversies and unresolved questions that can inspire your own project. By this point in your academic career, you’ve probably heard that you need to get “peer-reviewed journal articles.” But what are those exactly? How do they differ from news articles or encyclopedias? That is the focus of this section of the text—different types of literature.
Keeping your working question in mind, you should look at your potential sources and evaluate whether they are relevant to your inquiry. To assess the relevance of a source, ask yourself if the source will help you answer or think more deeply about your working question. Does the information help you answer this question, challenge your assumptions, or connect your question with another topic? Does the information present an opposing point of view, so you can show that you have addressed all sides of the argument in your paper? Does the article just provide a broad overview of your topic or does it have a specific focus on what you want to study? If the article isn’t helpful to you, it’s okay not to read it. No matter how good your searching skills, some articles won’t be relevant. You don’t need to read and include everything you find!
You may also want to check the relevance of the source with your professor or course syllabus. In my class, I have specific questions I will ask students to address in their literature reviews. You may want to find sources that help you answer the questions in your professor’s prompt for a literature review. For example, my rubric asks students to:
1. Identify the broader topic area
2. Review at least 14 studies
3. Identify themes found in the research
4. Identify points of conflict in the research, as appropriate
5. Identify points of agreement in the research, as appropriate
6. Identify gaps in the research or how it relates back to the research question
Almost all databases will give you access to an article’s abstract. The abstract is a summary of the main points of an article. It will provide the purpose of the article and summarize the author’s conclusions. Once you have a few good search queries, start skimming through abstracts and find the articles that are most relevant to your working question. Soon enough, you will find articles that are so relevant that you may decide to read the full text.
Finding quality sources to support your research
In addition to relevance, you should ask yourself about the quality of the source you’ve found. Is the information outdated? Is the source more than 5 years old? If so, it will not necessarily provide what we currently know about the topic – just what we used to know. Older sources are helpful for historical information, such as how our understanding of a topic has changed over time or how the prevalence of an issue has increase or decreased. Older articles are also important to find out what we already know about the topic – once researchers are comfortable with the state of knowledge in an area, they are likely to move on to new aspects of the issue, so it’s important to at least start with a wider time-frame (i.e., foundational scholars and when the topic was first addressed in the literature to recent literature within the past 5 years). Once you have a good understanding of the history of the issue, move on to sources that are more current.
Foundational articles and scholars
Older sources are important; especially if they are foundational articles by scholars prominent in the field of study. Foundational articles are cited often in the literature. They are clearly important to a lot of scholars in the field. While not all articles are foundational, you can get a quick sense of how important an article is to the broader literature on a topic by looking at how many other sources cited it. If you search for the article on Google Scholar, you can see how many other sources cited this information. Generally, the higher the number of citations, the more important the article (or at least, the more widely read). Of course, articles that were recently published will have fewer citations than older articles, and the citation count is only one indicator of an article’s importance.
Disciplinary perspectives
Information does not exist in the environment like some kind of raw material. It is produced by individuals working within a particular field of knowledge who use specific methods for generating new information, or a discipline. Disciplines consume, produce, and share knowledge. Looking through a university’s course catalog gives clues to disciplinary structure. Fields such as political science, biology, history, and mathematics are unique disciplines, as is education. Educational researches have things we study often, like educational policy, pedagogy, leadership, or developmental psychology, and a particular theoretical and ethical framework through which we view the world.
Educational researchers must take an interdisciplinary perspective, engaging with not just the education literature but literature from many disciplines to gain a comprehensive and accurate understanding of a topic. Consider how nursing and education would differ in studying sports in schools. A nursing researcher might study how sports affect individuals’ health and well-being, how to assess and treat sports injuries, or the physical conditioning required for athletics. An education researcher might study how schools privilege or punish student athletes, how athletics impact social relationships and hierarchies, or the differences in participation among children of different gender identities. Sometimes disciplines overlap in their focus, such as career counseling and counseling psychology, or urban planning and public administration.
You will need to become comfortable with identifying the disciplines that might contribute information to any literature search. When you do this, you will also learn how to decode the way people talk about a topic within a discipline. This will be useful to you when you begin a review of the literature in your area of study.
Activity: Think about your research topic and working question. What disciplines are likely to publish about your topic? For each discipline, consider:
- What is important about the topic to scholars in that discipline?
- What is most likely to be the focus of their study about the topic?
- What perspective are they most likely to have on the topic?
Types of articles
Literature searching is about finding evidence to inform your ideas, supporting or refuting what you think about a topic. A good place to start your literature search is with review articles—meta-analyses, meta-syntheses, and systematic reviews. These types of articles give you a birds-eye view of the literature in a topic area, and with meta-analyses and meta-syntheses, conduct empirical analyses on enormous datasets comprised of the raw data from multiple studies. As a result, their conclusions represent what is broadly true about the topic area. They also have comprehensive reference lists that you can browse for sources relevant to your topic. Finally, they tend to focus on resolving conflicts in the field, so you are likely to get a broad overview of the field and its tensions in the introductory parts of the reviews.
While review articles are the best place to start for any literature search, you will also need to include articles based on your working question, in particular research-based articles. Different types of sources are useful depending on what your question is (and the outcome you want). Table 4.1 contains a suggested starting point for evaluating what types of literature will provide relevant evidence for your project.
What you want know about: | Relevant evidence will be found in a: |
General knowledge about a topic | Systematic review, meta-analysis literature review, textbook, encyclopedia |
Intervention (treatment, policy, program) | Randomized controlled trial (RCT), meta-analysis, systematic review, cohort study, case-control study, case series, clinical practice guidelines |
Lived experience & sociocultural context | Qualitative study, participatory and action research, humanities and cultural studies |
Theory and practice models | Theoretical and non-empirical article, textbook, manual for an evidence-based treatment, book or edited volume |
Prevalence of a diagnosis or social problem | Survey research, government and nonprofit reports |
How practitioners think about a topic | Practice note, survey or qualitative study of practitioners, reports from professional organizations |
Qualitative studies are not designed to provide information about a broader population. As a result, you should treat their results as related to the specific context (i.e., time and place in which they occurred). If a study’s context is similar to the one you plan to research, then you might expect similar results to emerge in your research project. Qualitative studies provide the lived experience and personal reflections of people knowledgeable about your topic.
Primary sources
Periodicals may contain primary or secondary literature depending on the article in question. An article that is a primary source gathers information as an event happens (e.g. an interview with a victim of a local fire), or it may relay original research reported on by the journalists (e.g. the Guardian newspaper’s The Counted webpage which tracked how many people were killed by police officers in the United States from 2015-2016).[1] Primary sources are based on raw data.
Is it okay to use a magazine or newspaper as a source in your research methods class? If you were in my class, the answer is “probably not.” There are some exceptions, such as the Guardian page mentioned above (which is a great example of data journalism) or breaking news about a policy or community, but most of what newspapers and magazines publish is secondary information. Researchers should look for primary sources—the original source of information—whenever possible.
Secondary sources
Secondary sources interpret, discuss, and summarize primary sources. Examples of secondary sources include literature reviews written by researchers, book reviews, as well as news articles or videos about recently published research. Ideally, you should read the original source of the information, the primary source. When you read a secondary source, you are relying on another’s interpretation of the primary source. That person might be wrong in how they interpreted the primary source, or they may simply focus on a small part of what the primary source says.
Most often, I see the distinction between primary and secondary sources in the references pages students submit for the first few assignments in my research class. Students are used to citing the summaries of scientific findings from periodicals like the Vancouver Sun, Globe and Mail, or National Post. Instead, students should read the original journal article written by the researchers, as it is the primary source. Journalists are not scientists. If you have seen articles about how chocolate cures cancer or how drinking whiskey can extend your life, you should understand how journalists can exaggerate or misinterpret scientific results. Even when news outlets do a very good job of interpreting scientific findings, and that is certainly what most journalistic outlets strive for, you are still likely to only get a short summary of the key results.
When you are getting started with research, newspaper and magazine articles are excellent places to commence your journey into the literature, as they do not require specialized knowledge and may inspire deeper inquiry. Just make sure you are reading news, not opinion pieces, which may exclude facts that contradict the author’s viewpoint. The one secondary source that is highly valuable for student researchers are review articles published in academic journals, as they summarize much of the relevant literature on a topic.
Trade publications
Unlike magazines and newspapers, trade publications may take some specialized knowledge to understand. Trade publications or trade journals are periodicals directed to members of a specific profession. They often include information about industry trends and practical information for people working in the field. Because of this, trade publications are somewhat more reputable than newspapers or magazines, as the authors are specialists in their field.
Kappan Magazine and Educational Leadership are good examples of trade publications in education. The intended audiences are teachers, teacher leaders, and administrators who want to learn about important practice issues. They report news and trends in a particular field but not necessarily original scholarly research. They may also provide product or service reviews, job listings, and advertisements.
Can you use trade publications in a formal research proposal? Again, if you’re in my class, the answer would be “probably not.” A main shortcoming of trade publications is the lack of peer review. Peer review refers to a formal process in which other esteemed researchers and experts ensure your work meets the standards and expectations of the professional field. Peer review is part of the cycle of publication for journal articles and provides a gatekeeper service, so to speak, which (in theory) ensures that only top-quality articles are published.
In reality, there are a number of problems with our present system of peer review. Briefly, critiques of the current peer review system include unreliable feedback across reviewers; inconsistent error and fraud detection; prolonged evaluation periods; and biases against non-dominant identities, perspectives, and methods. It can take years for a manuscript to complete peer review, likely with one or more rounds of revisions between the author and the reviewers or editors at the journal. See Kelly and colleagues (2014)[2] for an overview of the strengths and limitations of peer review.
A trade publication isn’t as reputable as a journal article because it does not include expert peer review. While trade publications do contain a staff of editors, the level of review is not as stringent as with academic journals. On the other hand, if you are completing a study about practitioners, then trade publications may be highly relevant sources of information for your proposal.
In summary, newspapers and other popular press publications are useful for getting general topic ideas. Trade publications are useful for practical application in a profession and may also be a good source of keywords for future searching.
Journal articles
As you’ve probably heard by now, academic journal articles are considered to be the most reputable sources of information, particularly in research methods courses. Journal articles are written by scholars with the intended audience of other scholars (like you!). The articles are often long and contain extensive references that support the author’s arguments. The journals themselves are often dedicated to a single topic, like leadership, policy or curriculum theory, and include articles that seek to advance the body of knowledge about a particular topic.
Peer review
Most journals are peer-reviewed (also known as refereed), which means a panel of scholars reviews the articles to provide a recommendation on whether they should be accepted into that specific journal. Scholarly journals make available articles of interest to experts or researchers. An editorial board of respected scholars (i.e., peers) reviews all articles submitted to a journal. Editors and volunteer reviewers decide if the article provides a noteworthy contribution to the field and should be published. For this reason, journal articles are the main source of information for researchers as well as for literature reviews. Usually, peer review is done confidentially, with the reviewer’s name hidden from the author and vice versa. This confidentiality is mediated by the editorial staff at an academic journal and is termed blinded review, though this ableist language should be revised to confidential review (Ades, 2020).[3]
You can usually tell whether a journal is peer reviewed by going to its website. Under the “About Us” or “Author Submission” sections, the website should list the procedures for peer review. If a journal does not provide such information, you may have found a “predatory journal.” You may want to check with your professor or a librarian, as the websites for some reputable journals are not straightforward. It is important not to use articles from predatory journals. These journals will publish any article—no matter how bad it is—as long as the author pays them. If a journal appears suspicious and may be a predatory publication, search for it in the COPE member list. Another predatory publishing resource, Beall’s List, has been critiqued as discriminatory towards some open access publishing models and non-Western publishers so we do not recommend its use (Berger & Cirasella, 2015).[4]
Even reputable, non-predatory journals publish articles that are later shown to be incorrect, unethically manipulated, or otherwise faulty. It is important to note that peer review is not a perfect process, and that it is subject to error. There is a growing movement to make the peer review process faster and more transparent. For example, 17 life science journals moved to a model of publishing the unreviewed draft, the reviewer’s comments, and how they informed the final draft (Brainard, 2019).[5]
Foundational articles
A foundational article is “a classic work of research literature that is more than 5 years old and is marked by its uniqueness and contribution to professional knowledge” (Houser, 2018, p. 112).[6] Basically, it is a really important article.
How do you know if you are looking at a foundational article? Foundational articles are cited a lot in the literature. You can see how many authors have cited an article using Google Scholar’s citation count feature when you search for the article, as depicted in figure below. Generally speaking, articles that have been cited more often are considered more influential in the field although there is nothing wrong with citing an article with a low citation count.
There is no exact number of citations at which an article is considered foundational. A low citation count for an article published last year is not a reliable indicator of the importance of this article to the literature, as it takes time for people to read articles and incorporate them into their scholarship. For example, if you are looking at an article from five years ago with 20 citations, that’s still a good quality resource to use. The purpose here is to recognize when articles (and authors) are clearly influential in the literature, and are therefore more likely to be important to your review of the literature in that topic area.
Empirical articles
Journal articles can fall into one of several categories. Empirical articles report the results of a quantitative or qualitative analysis of raw data conducted by the author. Empirical articles also integrate theory and previous research. However, just because an article includes quantitative or qualitative results does not necessarily mean it is an empirical journal article. Since most articles contain a literature review with findings from empirical studies, you need to make sure the findings reported in the study stem from the author’s analysis of raw data.
Fortunately, empirical articles follow a similar structure, and include the following sections: introduction, methods, results, and discussion. While the exact wording in the headings may differ slightly from publication to publication and other sections may be added, this general structure applies to nearly all empirical journal articles. If you see this general structure, you are reading an empirical journal article. You can see this structure in a recent article from the open access education journal, the International Journal of Education Policy and Leadership (IJEPL). Walker & Kutsyuruba (2019)[7] conducted a mixed-methods pan-Canadian study that examined the differential impact of teacher induction and mentorship programs on the retention of early career teachers. The manuscript begins with an “Introduction” heading on page one, Review of the Literature on page two, Research Methodology on page three, Findings on page six and Discussion on page 13, with a Conclusion on page 16. Because this is a mixed-methods empirical article, the results section is full of themes and quotes based on what participants said in their interviews with researchers as well as a reporting out of survey results. If it were a quantitative empirical article, like this one from Kelly, Kubart, and Freed (2020)[8], the results section would be limited to statistical data, tables, and figures that indicated mathematical relationships between the key variables in the research question.
While this article uses the traditional headings, you may see other journal articles that use headings like “Background” instead of “Introduction” or “Findings” instead of “Results”. You may also find headings like “Conclusions”, “Implications”, “Limitations”, and so forth in other journal articles. Regardless of the specific wording, empirical journal articles follow the structure of introduction, methods, results, and discussion. If it has a methods and results section, it’s almost certainly an empirical journal article. However, if the article does not have these specific sections, it’s likely you’ve found a non-empirical article because the author did not analyze any raw data.
Review articles
Along with empirical articles, review articles are the most important type of article you will read as part of conducting your review of the literature. Review articles are journal articles that summarize the findings of other researchers and establish the state of the literature in a given topic area. They are so helpful because they synthesize a wide body of research information in a (relatively) short journal article. Review articles include literature reviews, like this one from Reynolds & Bacon (2018)[9] which summarizes the literature on integrating refugee children in schools. You may find literature reviews with a special focus like a critical review of the literature, which may apply a perspective like critical theory or feminist theory while surveying the scholarly literature on a given topic.
Review articles also include systematic reviews, which are like literature reviews but more clearly specify the criteria by which the authors search for, include, and analyze the content of the scholarly literature in the review (Uman, 2011).[10] For example, Laitsch, Nguyen and Younghusband conducted a scoping review of the literature on class size in schools as part of the BCTF’s 2010 Supreme Court challenge.[11]. We synthesized the results of 112 studies of class size to answer the question, “What is the current state of knowledge regarding the impact of class size on learning conditions of students in elementary and secondary schools?” As you can see in Figure 1 of this article, systematic reviews are very specific about which articles they include in their analysis. Because systematic reviews try to address all scholarly literature published on a given topic, researchers specify how the literature search was conducted, how many articles were included or excluded, and the reasoning for their decisions. This way, researchers make sure there is no relevant source excluded from their analysis.
The two final kinds of review articles, meta-analysis and meta-synthesis go even further than systematic reviews in that they analyze the raw data from all of the articles published in a given topic area, not just the published results. A meta-analysis is a study that combines raw data from multiple sources and analyzes the pooled data using statistics. Meta-analyses are particularly helpful for intervention studies, as it will pool together the raw data from multiple samples and studies to create a super-study of thousands of people, which has greater explanatory power. For example, this recent meta-analysis by Graham and colleagues (2016)[12] analyzes pooled data from 47 separate studies on balanced literacy instruction to see if it is effective (spoiler alert, it is).
A meta-synthesis is similar to a meta-analysis but it pools qualitative results from multiple studies for analysis. For example, this recent meta-synthesis by Pidgeon and Riley (2021)[13] explores how principles of Indigenous research methodologies informed research relationships with Indigenous communities by qualitatively analyzing and synthesizing the results of 79 relevant research studies. While meta-analyses and meta-syntheses are the most methodologically robust type of review article, any recently published review article that is highly relevant to your topic area is a good place to start reading literature. Because review articles synthesize the results of multiple studies, they can give you a broad sense of the overall literature on your topic. We’ll review a few kinds of non-empirical articles next.
Theoretical articles
Theoretical articles discuss a theory, conceptual model, or framework for understanding a problem. They may delve into philosophical or ethical analysis as well. For example, this theoretical article by Ryan and Deci (2020)[14] discusses intrinsic and extrinsic motivation and the importance of understanding motivation across educational levels and cultural contexts. While most students know they need to have statistics to back up what they think is true, it’s also important to use theory to inform what you think about your topic. Theoretical articles can help you understand how to think about a topic and may help you make sense of the results of empirical studies.
Practical articles
Practical articles describe “how things are done” from the perspective of on-the-ground practitioners. They are usually shorter than other types of articles and are intended to inform practitioners of a discipline on current issues. They may also reflect on a “hot topic” in the practice domain, a complex client situation, or an issue that may affect the profession as a whole. This practical article by Kajander and colleagues (2019)[15] describes the work a one high school team in designing, developing, supporting, and field-testing a numeracy and general learning skills course.
Type of journal article | How do you know if you’re looking at one? | Why is this type of article useful? |
Peer-reviewed | Go to the journal’s website, and look for information that describes peer review. This may be in the instructions for authors or about the journal sections. | To ensure other respected scholars have reviewed, provided feedback, and approved this article as containing honest and important scientific scholarship. |
Article in a predatory journal | Use the SIFT method. Search for the publisher and journal on Wikipedia and in a search engine. Check the COPE member list to see if your journal appears there. | They are not useful. Articles in predatory journals are published for a fee and have not undergone serious review. They should not be cited. |
*Not an journal article* | Does not provide the name of the journal it is published in. You may want to google the name of the article or report and its author to see if you can find a published version. | Journal articles aren’t everything, but if your instructor asks for one, make sure you haven’t mistakenly use dissertations, theses, and government reports. |
Empirical article | Most likely, if it has a methods and results section, it is an empirical article. | Once you have a more specific idea in mind for your study, you can adapt the measures, design, and sampling approach of empirical articles (quantitative or qualitative). The introduction and results section can also provide important information for your literature review. Make sure not to rely too heavily on a single empirical study, though, and try to build connections across studies you read. |
Quantitative article | The methods section contains numerical measures and instruments, and the results section provides outcomes of statistical analyses. | |
Qualitative article | The methods section describes interviews, focus groups, or other qualitative techniques. The results section has a lot of quotes from study participants. | |
Review article | The title or abstract includes terms like literature review, systematic review, meta-analysis, or meta-synthesis. | Getting a comprehensive but generalized understanding of the topic and identifying common and controversial findings across studies. |
Theoretical article | The articles does not have a methods or results section. It talks about theory a lot. | Developing your theoretical understanding of your topic. |
Practical articles | Usually in a section at the front or back of the journal, separate from full articles. It addresses hot topics in the profession from the standpoint of a practitioner. | Identifying how practitioners in the real world think about your topic. |
Book reviews, notes, and commentary | Much shorter, usually only a page or two. Should state clearly what kind of note or review they are in the title, abstract, or keywords. | Can point towards sources that provide more substance on the topic you are studying or point out emerging or controversial aspects of your topic. |
What kind of articles should you read?
No one type of article is better than the other, as each serves a different purpose. Foundational articles relevant to your topic area are important to read because the ideas in them are deeply influential in the literature. Theoretical articles will help you understand the social theory behind your topic. Empirical articles should test those theories quantitatively or create those theories qualitatively. Review articles help you get a broad survey of the literature by summarizing the results of multiple studies, which is particularly important at the beginning of a literature search. And finally, practical articles will help you understand a practitioner’s on-the-ground experience. Pick the kind of article that gives you the kind of information you need.
Other sources of information
Newspaper and magazine articles are good places to start your search (though they should not be the end of your search!). Another source students go to almost immediately is Wikipedia. Wikipedia is a marvel of human knowledge. It is a digital encyclopedia to which anyone can contribute. The entries for each Wikipedia article are overseen by skilled and specialized editors who volunteer their time and knowledge to making sure their articles are correct and up to date. That said, Wikipedia has been criticized for gender and racial bias as well as inequities in the treatment of the Global South (Montgomery et al., 2021).[16]
For those conducting action research that engages more with communities and target populations, researchers are responsible not only for reviewing the scientific literature on their topic but also the literature that communities find important. If there are local newspapers, television shows, religious services or events, community meetings, or other sources of knowledge that your target population finds important, you should engage with those resources as well. Understanding these information sources builds empathetic understanding of participating groups and can help inform your research study. Moreover, they are likely to contain knowledge that is not a part of the scientific literature but is nevertheless crucial for conducting scientific research appropriately and effectively in a community.
Encyclopedias
Wikipedia is an example of a tertiary source. Tertiary sources review primary and secondary sources. Examples of tertiary sources include encyclopedias, directories, dictionaries, and textbooks like this one. Tertiary sources are also an excellent place to start (but also not a good place to end) your search. A student might consult Wikipedia or the International Encyclopedia of Education to get a general idea of an education topic. Encyclopedias can also be excellent resources for their citations, and often the citations will be to seminal articles or important primary sources.
The difference between secondary and tertiary sources is not exact, and as we’ve discussed, using one or both at the beginning of a project is a good idea. As your study of the topic progresses, you will naturally have to transition away from secondary and tertiary sources and towards primary sources. We’ve already talked about one particular kind of primary source—empirical journal articles. We will spend more time on this primary source than any other in this textbook. However, it is important to understand how other types of sources can be used as well.
Books
Books contain important scholarly information. They are particularly helpful for theoretical, philosophical, and historical inquiry. You can use books to learn key concepts, theories, and keywords you can use to find more up-to-date sources. They may also help you understand the scope and foundations of a topic and how it has changed over time.
Some books contain chapters that look like academic journal articles. These are called edited volumes. They contain chapters similar to journal articles, but the content is reviewed by an editor rather than peer reviewers. Such books often allow for a more summary overview of research, rather than a focusing on singular studies, providing a broader picture of the topic. However, because these types of summaries are not systematic, they should be used with care, as they may over-emphasize the beliefs and opinions of the book’s authors and editors.
Conference proceedings and presentations
Conferences provide a great source of information. At conferences such as the the Canadian Society for the Study of Education (CSSE) Annual Meeting or the American Education Research Association, researchers present papers on their most recent research. The papers presented at conferences are sometimes published in a volume called a conference proceeding, though these are not common in education. Conference proceedings highlight the current discourse in a discipline and can lead you to scholars who are interested in specific research areas. Individual papers are often available for download if you are a member of the organization, or if you e-mail the authors directly.
A word about conference papers: several factors may render these documents difficult to find. It is not unusual that papers delivered at professional conferences are not published in print or electronic form, although an abstract may be available. In these cases, the full paper may only be available from the author or authors. The most important thing to remember is that if you have any difficulty finding a conference proceeding or paper, ask a librarian for assistance.
Gray literature
Another source of information is the gray literature, which consist of research reports released by non-commercial publishers, such as government agencies, policy organizations, and think tanks. If you have already taken a policy class, perhaps you’ve come across the Canadian Centre for Policy Alternative (CCPABC). The CCPA is a think tank—a group of scholars that conducts research and advocates for political, economic, and social change. Similarly, students often find the Centers for Disease Control website helpful for understanding the prevalence of social problems like mental illness, school health, and child abuse. If you find fact sheets, news items, blog posts, or other content from organizations like think tanks or advocacy organizations in your topic area, look at the references cited to find the primary source, rather than citing the secondary source (the fact sheet). While you might be persuaded by their arguments, you want to make sure to account for the author’s potential bias.
Think tanks and policy organizations often have a specific viewpoint they support. For example, there are conservative, liberal, and libertarian think tanks. Policy organizations may be funded by donors to push a given message to the public. Government agencies are generally more objective, though they may be less critical of government programs than other sources might be. Similarly, agencies may be less critical in the reports they share with the public. The main shortcoming of gray literature is the lack of peer review that is found in academic journal articles. Government reports are often cited in research proposals, along with some reports from policy organizations.
Dissertations and theses
Dissertations and theses can be rich sources of information and include extensive reference lists you can use to scan for resources. However, dissertations and theses are considered gray literature because they are not independently peer reviewed. The accuracy and validity of the document itself may depend on the school that awarded the doctoral or master’s degree to the author. Having completed a dissertation, I can attest to the length of time they take to write as well as read. If you come across a relevant dissertation, it is a good idea to read the literature review and note the sources the author used so you can incorporate them into your literature review. The data analysis found in these sources may be less useful as it has not passed through independent peer review yet; however, dissertations are approved by committees of scholars, so they are considered more reputable than sources that lack review by experts prior to publication. Consider searching for journal articles by the dissertation’s author to see if any of the results have been peer reviewed and published. You will also be thankful because journal articles are much shorter than dissertations and theses!
Webpages
The final source of information we are going to talk about is webpages. Webpages will help you locate professional organizations and human service agencies that address your problem, and there are often personal web pages discussion issues of interest in education. Looking at social media feeds, reports, publications, or “news” sections on an organization’s webpage can clue you into important topics to study. Because anyone can develop their own webpage, there are also a large number of education-related websites. They are usually not considered scholarly sources to use in formal writing, as they only represent the opinions, beliefs, and values of the sponsoring individual or organization, but they may be useful as you are first learning about a topic. Additionally, many advocacy webpages will provide references for the facts they site, providing you with the primary source. Keep in mind too that webpages can be tricky to quote and cite, as they can be updated or moved at any time. While some reputable sources will include editing notes, others may not. Webpages can be updated without notice.
Use quality sources
As you access and read each source you encounter, remember:
All information sources are not created equal. Sources can vary greatly in terms of how carefully they are researched, written, edited, and reviewed for accuracy. Common sense will help you identify obviously questionable sources, such as tabloids that feature tales of alien abductions, or personal websites with glaring typos. Sometimes, however, a source’s reliability—or lack of it—is not so obvious. You should consider criteria such as the type of source, its intended purpose and audience, the author’s (or authors’) qualifications, the publication’s reputation, any indications of bias or hidden agendas, how current the source is, and the overall quality of the writing, thinking, and design. (Writing for Success, 2015, p. 448).[17]
While each of these sources provides an important facet of our learning process, your research should focus on finding academic journal articles about your topic. These are the primary sources of the research world. While it may be acceptable and necessary to use other primary sources—like books, government reports, or an investigative article by a newspaper or magazine—academic journal articles are preferred. Finding these journal articles is the topic of the next section.
Activity: Conduct a general Google Scholar search on your topic and identify whether the first 10 sources that come up are primary or secondary sources. Ask a friend to conduct the same Google search and compare your results. Google’s search engine customizes the results you see based on your browsing history and other data the company has collected about you. This is called a filter bubble, and you can learn more about it in this Ted Talk about filter bubbles.
Find a secondary or tertiary source about your topic and trace one to the original, primary source.
Find a review article in your topic area and identify which kind it is (literature review, systematic review, meta-analysis, etc.).
Find a think tank, advocacy organization, or website that addresses your topic. Using their information, identify primary sources that might be of use to you. Consider adding them to your social media feed or joining an email newsletter.
Reflect and plan for the future
As you look through abstracts and search the literature, you will learn more about your topic area. You will learn new concepts that become new keywords in new queries. You will continue to come up with search queries and download articles throughout the research process. You will return to search the literature often during the research process. As such, it is important to keep notes about what you did at each stage. I usually keep a “working notes” document in the same folder as the PDFs of articles I download. I can write down which categories different articles fall into (e.g., theoretical articles, empirical articles), reflect on how my question may need to change, or highlight important unresolved questions or gaps revealed in my search.
Creating and refining your working question will help you identify the key concepts you study will address. Once you identify those concepts, you’ll need to decide how to define them and how to measure them when it comes time to collect your data. As you are reading articles, note how other researchers who study your topic define concepts theoretically in the introduction and measure them in their methods section. Tuck these notes away for the future, when you will have to define and measure these concepts.
You also need to think about who your research participants will be and what larger group(s) they may represent. You need to be able to speak intelligently about the target population you want to study, so finding literature about their strengths, challenges, and how they have been impacted by historical and cultural oppression is a good idea.
- The Guardian (n.d.). The counted: People killed by police in the US. Retrieved from: https://www.theguardian.com/us-news/ng-interactive/2015/jun/01/the-counted-police-killings-us-database ↵ ↵
- Kelly, J., Sadeghieh, T., & Adeli, K. (2014). Peer review in scientific publications: Benefits, critiques, & a survival guide. EJIFCC, 25(3), 227–243. ↵
- Ades, R. (2020, February 20). An end to "blind review." Blog of the APA. https://blog.apaonline.org/2020/02/20/an-end-to-blind-review/ ↵
- Berger, M., & Cirasella, J. (2015). Beyond Beall’s list: Better understanding predatory publishers. College & research libraries news, 76(3), 132-135. ↵
- Brainard, J. (2019, October 10). In bid to boost transparency, bioRxiv begins posting peer reviews next to preprints. Science magazine. Retrieved from: https://www.sciencemag.org/news/2019/10/bid-boost-transparency-biorxiv-begins-posting-peer-reviews-next-preprints ↵
- Houser, J., (2018). Nursing research reading, using, and creating evidence (4th ed.). Burlington, MA: Jones & Bartlett. ↵
- Keith Walker & Benjamin Kutsyuruba (2019). The Role of School Administrators in Providing EarlyCareer Teachers’ Support: A Pan-Canadian Perspective. International Journal of Education Policy &Leadership 14(2). URL: http://journals.sfu.ca/ijepl/index.php/ijepl/article/view/862 doi: 10.22230/ijepl.2019v14n2a862.IJEPL ↵
- Kelly, M. S., Kubert, P., & Freed, H. (2020). Teen depression, stories of hope and health: A promising universal school climate intervention for middle school youth. International Journal of School Social Work, 5(1), 3. ↵
- Reynolds, A. D., & Bacon, R. (2018). Interventions supporting the social integration of refugee children and youth in school communities: A review of the literature. Advances in social work, 18(3), 745-766. ↵
- Uman, L. S. (2011). Systematic reviews and meta-analyses. Journal of the Canadian Academy of Child and Adolescent Psychiatry, 20(1), 57-59 ↵
- Laitsch, D., Nguyen, H., & Younghusband, C. (2021). Class size and teacher work: Research provided to the BCTF in their struggle to negotiate teacher working conditions. Canadian Journal for Educational Administration & Policy, 196, 83-101. https://journalhosting.ucalgary.ca/ index.php/cjeap/article/view/70670 ↵
- Graham, S., Liu, X., Aitken, A., Ng, C., Bartlett, B., Harris, K. R., & Holzapfel, J. (2018). Effectiveness of Literacy Programs Balancing Reading and Writing Instruction: A Meta-Analysis. Reading Research Quarterly, 53(3), 279–304. http://www.jstor.org/stable/26622547 ↵
- Michelle Pidgeon & Tasha Riely. (2021). Understanding the Application and Use of Indigenous Research Methodologies in the Social Sciences by Indigenous and Non-Indigenous Scholars. International Journal of Education Policy & Leadership 17(8). URL: http://journals.sfu.ca/ijepl/index.php /ijepl/article/view/1065 doi:10.22230/ijepl.2021v17n8a1065IJEPL. ↵
- Ryan, R. M., Deci, E. L. (2020) Intrinsic and extrinsic motivation from a self-determination theory perspective: Definitions, theory, practices, and future directions. Contemporary Educational Psychology, 61 ,101860. ↵
- Ann Kajander, Matt Valley, Kelly Sedor, & Taylor Murie. (2021). Curriculum for Resiliency: Supporting a Diverse Range of Students’ Needs in Grade 9 Mathematics. Canadian Journal of Action Research 22(1). pp. 69-86. DOI: https://doi.org/10.33524/cjar.v22i1.547. ↵
- Montgomery, L., Hartley, J., Nevlon, C., Gillies, M., Gray, E., Herrmann-Pillath, C., Huana, C., Leach, J., Potts, J., Ren, X., Skinner, K., Sugimoto, C., & Wilson, K. (2021). Change. In Open knowledge institutions: Reinventing universities. MIT Press. https://doi.org/10.7551/mitpress/13614.001.0001 ↵
- Writing for Success (2015). Strategies for gathering reliable information. http://open.lib.umn.edu/writingforsuccess/chapter/11-4-strategies-for-gathering-reliable-information/ ↵
published works that document a scholarly conversation on a specific topic within and between disciplines
a summary of the main points of an article
an academic field, like social work
a study that combines raw data from multiple quantitative studies and analyzes the pooled data using statistics
journal articles that identify, appraise, and synthesize all relevant studies on a particular topic (Uman, 2011, p.57)
journal articles that summarize the findings other researchers and establish the state of the literature in a given topic area
in a literature review, a source that describes primary data collected and analyzed by the author, rather than only reviewing what other researchers have found
interpret, discuss, and summarize primary sources
periodicals directed to members of a specific profession which often include information about industry trends and practical information for people working in the field
a formal process in which other esteemed researchers and experts ensure your work meets the standards and expectations of the professional field
report the results of a quantitative or qualitative data analysis conducted by the author
unprocessed data that researchers can analyze using quantitative and qualitative methods (e.g., responses to a survey or interview transcripts)
a study that combines primary data from multiple qualitative sources and analyzes the pooled data
discuss a theory, conceptual model, or framework for understanding a problem
describe “how things are done” or comment on pressing issues in practice (Wallace & Wray, 2016, p. 20)
review primary and secondary sources
research reports released by non-commercial publishers, such as government agencies, policy organizations, and think-tanks