Moderator Variable

Meta-analysis in Practice

J.B. Greenhouse , in International Encyclopedia of the Social & Behavioral Sciences, 2001

2.2 Applications of Meta-analysis in Practice: Vignettes

The following vignettes are meant to provide a flavor of some of the more contemporary applications of meta-analysis and the central role that meta-analysis plays in explanation, program evaluation, and in informing policy decisions. Additional illustrations of this kind can be found in Cook et al. (1992) which presents four detailed case studies of research syntheses along with expert commentary highlighting the value of meta-analysis for scientific explanation and the challenges of implementation.

(a)

Meta-analyses that include moderator variables provide an opportunity to investigate relations that may never have been considered in any of the primary studies. Consider, for example, a meta-analysis based on 120 studies of whether or not homework worked. Cooper ( 1989) found that students given homework assignments did better than those not given assignments, and furthermore, the more time spent on homework the better. An unexpected finding that had not been anticipated in the primary studies, however, was that although homework was very effective in high school it was almost totally ineffective in elementary school. This discovery has had important practical implications, and has stimulated the consideration of hypotheses as to why this might be, illustrating the impact of what Cooper has called 'review-generated evidence.'

(b)

'Cross-design synthesis' is a relatively new research strategy for combining the results from diverse, complementary studies, such as randomized controlled studies and administrative databases, to improve knowledge about interventions and to inform policy decisions. The goal is to capture the diverse strengths of the different designs while minimizing weaknesses. The approach is anchored in meta-analytic principles and methods, and in addition relies on methods for assessing the generalizability of data sources, and methods for statistical adjustment for potential biases in intervention comparisons (Droitcour et al. 1993).

(c)

The goal of the Agency for Health Care Policy and Research (AHCPR) was to enhance the quality, appropriateness, and effectiveness of health services and to improve access to health care. The Federal legislation authorizing the creation of the agency (P.L. 101–239, December 1989) directed the agency to issue guidelines on clinical practice and required the guidelines to be based, in part, on a systematic synthesis of research evidence. Large-scale, AHCPR multidisciplinary studies called PORTS (Patient Outcomes Research Teams) as part of this mandate carried out formal meta-analyses to determine what was known, what was not known and what needed to be learned about variations in clinical practice and outcomes for a particular disease or condition (see, for example, Lehman et al. 1995).

(d)

When individual subject-level data is available from a number of similar studies, meta-analytic techniques can be used to combine information across the studies. Not only can study specific effects be estimated but individual subject characteristics can be used as covariates. An example of the use of such a meta-analysis is the study of nearly 600 outpatients treated for acute depression, who ranged in age from 18 to 80 years, from six studies conducted during a 10-year period. The goal of this study was to investigate the differential efficacy of psychotherapy alone, in comparison with the combination of psychotherapy and pharmacotherapy (Thase et al. 1997).

(e)

Given the body of research available and the quality of that research with respect to a particular question, it may be determined that doing a meta-analysis is premature for that field. Nevertheless, a research synthesis can be extremely useful in identifying obstacles in the literature that need to be addressed and successes that can be built on to guide future studies. For example, the Committee on the Assessment of Family Violence Interventions (1998) concluded that 'The quality of the existing research base of evaluations of family violence interventions is therefore insufficient to provide confident inferences to guide policy and practice, except in a few areas that we identify in this report. Nevertheless, this pool of evaluation studies and additional review articles represents a foundation of research knowledge that will guide the next generation of family violence evaluation efforts and allows broad lessons to be derived from the research literature.'

Read full chapter

URL:

https://www.sciencedirect.com/science/article/pii/B0080430767004617

Moderator Variable: Methodology

C.M. Judd , in International Encyclopedia of the Social & Behavioral Sciences, 2001

Typically theoretical models in the behavioral sciences posit that independent variables affect dependent variables. A moderator variable is a variable, which is thought to temper or modulate the magnitude of the effect of an independent variable on a dependent one. Moderators may be characteristics of people or characteristics of situations. In either case, they affect the magnitude of the relationship between an independent variable and a dependent one. Moderation implies an interaction between the independent variable and the moderator. Analytic procedures using both analysis of variance and, more generally, multiple regression, permit the testing of such interactions. Tests of interactions with measured, as opposed to manipulated variables, may not have high statistical power. Complications also arise due to the typically arbitrary metrics of many dependent variables. Nonlinear scale transformations can eliminate interactions and affect conclusions about the presence or absence of moderation.

Read full chapter

URL:

https://www.sciencedirect.com/science/article/pii/B0080430767007336

Applying the Tools to Multivariate Data

J. Douglas Carroll , Paul E. Green , in Mathematical Tools for Applied Multivariate Analysis, 1997

6.6.2.3 Three or More Matrices

Heretofore, we have described multivariate analysis of covariance in terms of a matrix of criterion variables and a matrix of predictor variables. The latter matrix consists of a mixture of dummy variables, the design variables, and covariates, whose effect on the criterion variables we desire to remove. Alternatively, we can partition the data matrix into three matrices: criterion, design dummies, and covariates, as illustrated in Panel III of Fig. 6.8.

Problems involving three, or more, matrices fall into two major types:

1.

a multivariate analysis of covariance situation, or multiple, partial correlation (Cooley and Lohnes, 1971), in which one of the data matrices consists of a set of covariates, moderators, or contingency variables whose effect is to be removed before considering the association between the remaining matrices;

2.

a generalized canonical correlation situation where the status of all three, or more, matrices is considered to be the same (in this case, we extend two-group canonical correlation to cover three or more matrices).

Multivariate analysis of covariance problems occurs frequently in the behavioral and administrative sciences. For example, one may set up various experiments in which several response measures are sought from the subjects and, furthermore, certain covariates like task familiarity, education, and IQ level are also included in the analysis.

In multivariate analysis of covariance one matrix, the response matrix, is typically made up of continuous scores, while the design matrix is typically made up of dummy variables. The matrix of covariates is usually made up of continuous scores. However, this is not necessary. In principle, any (or all three) of the matrices could consist of continuous or binary-valued scores, or, indeed, as mixtures. In this class of problems one generally allows affine transformations to be applied to any of the three matrices, in the spirit of two-set canonical correlation.

Generalized canonical correlation, employing three or more data-based matrices of equal status, is concerned primarily with configuration matching. Horst (1961), Carroll (1968), and Kettenring (1972) have all proposed models for this type of problem. For example, in the Carroll and Chang approach, an r + 1st space is defined such that the r original spaces, each consisting of the same m observations on r sets of variables, are transformed to match it as well as possible. This procedure allows an affine transformation of each "contributing" configuration.

While generalized canonical correlation has usually been considered in the context of all scores being continuous, this, again, is not necessary provided that the researcher's interest is centered on data description and summarization, rather than on statistical inference. Binary-valued scores, or mixtures of continuous and binary valued, can be dealt with just as readily. Again, affine transformations would generally be permitted.

Read full chapter

URL:

https://www.sciencedirect.com/science/article/pii/B978012160954250007X

Self-monitoring, Psychology of

M. Snyder , in International Encyclopedia of the Social & Behavioral Sciences, 2001

1 Major Themes of Self-monitoring Theory and Research

Soon after its inception, and partially in response to critical theoretical issues of the times, self-monitoring was offered as a partial resolution of the 'traits vs. situations' and 'attitudes and behaviors' controversies in personality and social psychology. The propositions of self-monitoring theory suggested that the behavior of low self-monitors ought to be predicted readily from measures of their attitudes, traits, and dispositions whereas that of high self-monitors ought to be best predicted from knowledge of features of the situations in which they operate. Self-monitoring promised a 'moderator variable' resolution to debates concerning the relative roles of person and situation in determining behavior. These issues set the agenda for the first generation of research and theorizing on self-monitoring, designed primarily to document the relatively 'situational' orientation of high self-monitors and the comparatively 'dispositional' orientation of low self-monitors (for a review, see Snyder 1987).

In a second generation of research and theorizing, investigations moved beyond issues of dispositional and situational determination of behavior to examinations of the linkages between self-monitoring and interpersonal orientations. Perhaps the most prominent of these programs concerns the links between expressive control and interpersonal orientations, as revealed in friendships, romantic relationships, and sexual involvements (e.g., Snyder et al. 1985). Other such programs of research concern advertising, persuasion, and consumer behavior (e.g., Snyder and DeBono 1985), personnel selection (e.g., Snyder et al. 1988), organizational behavior (Caldwell and O'Reilly 1982, Kilduff 1992), socialization and developmental processes (e.g., Eisenberg et al. 1991, Graziano and Waschull 1995), cross-cultural studies (e.g., Gudykunst 1985).

Central themes in these programs of research have been that high self-monitors live in worlds of public appearances created by strategic use of impression management, and that low self-monitors live in worlds revolving around the private realities of their personal identities and the coherent expression of these identities across diverse life domains. Consistent with these themes, research on interpersonal orientations has revealed that high, relative to low, self-monitors choose as activity partners friends who will facilitate the construction of their own situationally-appropriate images and appearances (e.g., Snyder et al. 1983). Perhaps because of their concern with images and appearances, high self-monitors have romantic relationships characterized by less intimacy than those of low self-monitors. Also consistent with these themes, explorations of consumer attitudes and behavior have revealed that high self-monitors value consumer products for their strategic value in cultivating social images and public appearances, reacting positively to advertising appeals that associate products with status and prestige; by contrast, low self-monitors judge consumer products in terms of the quality of the products stripped of their image-creating and status-enhancing veneer, choosing products that they can trust to perform their intended functions well (e.g., DeBono and Packer 1991). These same orientations manifest themselves in the workplace as well, with high self-monitors preferring positions that call for the exercise of their self-presentational skills; thus, for example, high self-monitors perform particularly well in occupations that call for flexibility and adaptiveness in dealings with diverse constituencies (e.g., Caldwell and O'Reilly 1982) whereas low self-monitors appear to function best in dealing with relatively homogeneous work groups.

It should be recognized that, although these programs of research, for the most part, have not grounded their hypotheses or interpretations in self-monitoring's traditionally fertile ground—issues concerning the dispositional vs. situational control of behavior—they do nevertheless reflect the spirit of the self-monitoring construct. That is, their guiding themes represent clear expressions of self-monitoring theory's defining concerns with the worlds of public appearances and social images, and the processes by which appearances and images are constructed and sustained. However, it should also be recognized that these lines of research go beyond showing that individual differences, in concern for cultivating public appearances, affect self-presentational behaviors. These programs of research have demonstrated that these concerns, and their manifestations in expressive control, permeate the very fabric of individuals' lives, affecting their friendship worlds, their romantic lives, their interactions with the consumer marketplace, and their work worlds.

Read full chapter

URL:

https://www.sciencedirect.com/science/article/pii/B0080430767017290

Unemployment and Mental Health

D. Fryer , in International Encyclopedia of the Social & Behavioral Sciences, 2001

3 The Relationship Between Unemployment and Mental Health

There have been valid grounds for concern about the social, physical, and mental health consequences of unemployment for at least 200 years, and this has been beyond reasonable doubt since the 1930s (Eisenberg and Lazarsfeld 1938, Lazarsfeld-Jahoda and Zeisel 1933). In the early days, fear, frustration, irritability, declining self-respect, decreasingly meaningful sense of time, loneliness, apathy and resignation, and poorer physical health were noted. These were broadly confirmed by substantial empirical research and scholarship in the 1980s but standardized measures checked for reliability and validity were preferred. Anxiety, depression, positive and negative self-esteem, and a variety of measures of affect were studied repeatedly. Rather than 'demoralization,' researchers wrote of 'psychological well-being' and the General Health Questionnaire was used widely. That there are negative mental health consequences of unemployment is being reiterated currently by sophisticated reviews and meta-reviews (see Depression, Hopelessness, Optimism, and Health .

Research clarifying the relationship between unemployment and mental health has been carried out in Australia, New Zealand, USA, and many European countries over many decades. The resulting large literature has operationalized mental health in a wide range of ways, has been carried out using just about every viable research method, has been supported by diverse funding arrangements, and has been undertaken from a variety of ideological starting positions. Of course the impact of unemployment is not homogenous: some are affected more negatively than others, and a number of moderator variables have been identified. Effective coping with unemployment is also widespread. However, the degree of consensus is startling. Groups of unemployed people almost always have poorer mental health than groups of otherwise similar but employed people and unemployment is overwhelmingly found to be the cause of poor mental health rather than the result of it.

For many, the well-designed, longitudinal studies, which have tracked large, carefully matched, samples of people in and out of paid jobs using standardized measures, are the key studies. Some of the most persuasive longitudinal quantitative studies of all were done with young people. Typically, these studies measure the mental health of large groups of young people in school and follow them out into the labor market, periodically measuring the mental health of those who get jobs and those who do not, and comparing group mean scores cross-sectionally and longitudinally. Groups of unemployed youngsters repeatedly are demonstrated to have poorer mental health than their employed peers but statistically significant differences are seldom found between the scores of the same groups when at school.

Many such studies have been done, but three research programs have been particularly influential. The first was the 1980s research program based at the Social and Applied Psychology Unit in Sheffield, England (see Warr 1987 for an overview). The second program was based at The Flinders University of South Australia (see Feather 1992 for an overview). The third was based at the University of Adelaide (see Winefield et al. 1993 for an overview). In brief, such studies provide powerful evidence that unemployment causes, rather than merely results from, poor psychological health.

Recently, sophisticated meta-reviews have reiterated that social causation is involved. For example, Murphy and Athanasou (1999) conducted a meta-review of 16 longitudinal studies using valid and reliable measures and published in the last 10 years in English-language scientific journals. They concluded that 'the results from the 16 longitudinal studies suggest that unemployment has reliable (negative) effects on mental health' and that 'effect size information…suggests that moving from unemployment to employment not only produces a reliable change to mental health, but a change that is "practically significant."'

It was never remotely plausible that mass unemployment was caused by mass, sometimes organizationally confined, epidemics of mental illness, but of course, in some individual cases, pre-existing mental health problems do predispose people to loss of, and/or failure to get, a job. However, even here social causation processes may be hard to discount completely. For example, during economic recession some employers ratchet up the criteria they use to select employees with the result that people are excluded from employment on mental and physical health grounds at one stage of the economic cycle, who would have been employed in other economic circumstances. Other conceptual and methodological concerns suggesting the stark dichotomy between social causation and individual drift may be unhelpful are discussed more fully in Fryer (1997).

In conclusion, research has persuaded most researchers in the field that unemployment has mental health consequences that are negative and widespread.

Read full chapter

URL:

https://www.sciencedirect.com/science/article/pii/B0080430767038298

Generalization: Conceptions in the Social Sciences

T.D. Cook , in International Encyclopedia of the Social & Behavioral Sciences, 2001

3 Generalizing Causal Connections

Much effort goes into testing causal hypotheses—statements that variation in one construct leads to variation in another and that no alternative cause can account for their covariation. Generalizing causal statements depends on methods other than formal sampling theory, given that experiments with both random selection and random assignment are rare and surveys provide causal tests that are flawed when compared to experiments. So, how is generalization achieved when the object to be generalized is a causal claim?

One way is when a population has recently been described using measures that are also available on a purposive sample of persons or sites. Then, any causal relationships obtained at the sample level can be weighted to the population level, using the empirical relationship between sample and population for this purpose. But such weighting only approximates what the population effect size would have been if the sample had been randomly selected. Correct inference depends on the sample level effect sizes not being affected by unmeasured sample characteristics that are not fully captured by the shared, measured variables. Still, we can anticipate that weighting procedures will be used more often in the future as methods for social experiments and estimating missing data advance. After all, survey researchers now describe human populations from biased sample data using basically the same weighting technique.

Sometimes, substantive theory is specific enough to detail the conditions under which a causal relationship holds. Then, a program of research can test explicit hypotheses about specific sources of population, setting, or time variation in the strength or direction of the causal relationship. This involves deliberate attempts to falsify the hypothesis that a causal connection is general (Gadenne 1976 ). Rejecting the hypothesis entails identifying specific causal moderators; failure to reject implies that the hypothesis of a general cause–effect relationship remains viable. A less formal variant of this describes common practice in the laboratory sciences. A causal claim is made and then independently replicated, often under conditions that approximate the original ones. Successful replication leads to concluding that the causal relationship is provisionally true—that is, true until future failures to replicate indicate the need to invoke causal moderator variables. In both approaches, special emphasis is on identifying moderators that reduce the causal relationship to zero or that change its sign, as opposed to those that merely vary its size without affecting its sign.

Purposive sampling is also used in other less formal generalization probes. In clinical trials, hospitals recruit patients with a given disease and then adhere to the study's intervention and data collection protocols. Hospitals are basically chosen for their willingness to comply and for their heterogeneity with respect to, say, region of the country, predominant ethnic group served, or type of hospital sponsorship. Statistical power is often only adequate for cross-site analysis, suggesting that the goal is not to identify site-level main and interaction effects, but to see if the causal estimate exceeds zero despite heterogeneity in the hospitals studied. If it does, the causal effect is held to be general. If it doesn't, several alternative claims are viable and need to be tested, especially the following: there is no effect; the causal sign varies by hospital type and obscures a main effect; or the methods used cannot detect an effect of the size expected. Only increasing the sample of sites allows direct tests of which hospital types are involved in site-level interactions. Even so, relying on a purposive sample of sites that are heterogeneous on many attributes except for volunteering is only logically defensible if researchers know from other sources that no confounds are associated with all the hospitals being volunteers for the clinical trial.

Exploratory analyses often involve measuring many constructs from among a large array of possible causal modifiers in order to probe which ones affect the strength of a causal relationship. Without a priori hypotheses about modifiers, the approach can be time-consuming and run afoul of error rate problems due to the many tests conducted and the small sample sizes for some values on the moderator. Nonetheless, subcategory breakdowns are often used within experiments to probe how robust a relationship is across persons, settings, times, and measures. This can prevent the overgeneralization that occurs when researchers conclude that a relationship found with the entire sample holds in each subgroup within the sample. The entire sample allows one to conclude that the effect is found despite the internal heterogeneity achieved. It does not imply there is independent replication within any of the individual subgroups. The story is essentially the same when partitioning data from a single survey. With adequate sample sizes per group, one can probe the robustness of a relationship across the subgroups sampled. But only those potential conditional variables directly sampled can be assessed.

The best tests of causal conditionals come from synthesizing multiple studies on a topic rather than from subgroup breakdowns within a single study (Cooper and Hedges 1994). Experiments and surveys relevant to the same causal hypothesis accumulate and can be used in meta-analysis, the best-known form of synthesis. Meta-analysis combines data across multiple studies asking the same question, taking advantage of the between-study variation in measures, manipulations, populations of people, settings, and times. The dependent variable is each study's standardized effect size(s) and these are manipulated (a) to generate an average effect size across all the studies and all the irrelevant sources of heterogeneity they contain, and (b) to identify the specific contingency factors responsible for variation in the effect sizes.

What logic buttresses meta-analysis? It is not random sampling since individual studies are selected for their relevance to the hypothesis and the hope is to achieve a census of all published and unpublished studies. However, the more relevant target population is all studies that could be conducted on the topic, not all those done on it. In meta-analytic practice, generalization depends on the sample of achieved studies including a wide range of human populations, setting types, historical periods, and operationalizations of the cause and effect. Each study is first coded to label its sampling particulars in terms of theoretical constructs and to examine the sources of irrelevancy with which the particulars are and are not collectively imbued. Thus in a hospital experiment, hospitals might be described as being university-based, public, or private, and as being large or small. Patients might be described in terms of diagnosis, age, and gender mix, etc. The treatment might be described in terms of its components, duration, and manner of delivery; and the outcome might be described in similar terms. Then researchers typically test whether the same causal relationship holds despite these sources of sampled irrelevancy.

Sample sizes permitting, researchers also test whether the relationship holds across individually identified sources of potential irrelevancy so as to generalize to hospitals in general, people diagnosed with cancer, or whatever the target inferences might be for a particular study. Important here are the number and especially the heterogeneity of the types examined. Of course, some sources of variation might have a priori substantive meaning, as when university hospitals are a special target. Then one has to do a discriminant validity test and show that the effect size for such hospitals is different from that for other hospitals. Otherwise, there is nothing unique about university hospitals. So, meta-analysis depends on the very tests for proximal similarity, heterogeneous irrelevancy, and discriminant validity that are so central in construct validation. In addition, to achieve such tests multiple replication is needed across studies that may each be individually very narrow in the range of people, settings, times, and cause and effect operations it includes. Central is the heterogeneity across studies and the pattern of results achieved despite this sampled heterogeneity.

Read full chapter

URL:

https://www.sciencedirect.com/science/article/pii/B0080430767006987

Meta-analysis of the effects of game types and devices on older adults-video game interaction: Implications for video game training on cognition

Rita Wing Lam Yu , Alan Hoi Shou Chan , in Applied Ergonomics, 2021

3.4 Moderator analysis

The effects of potential moderator variables were calculated for overall cognition. Table 6 shows the results of the moderator analysis for continuous variables in each video game type. For session length, meta-regression indicated significant moderate effect sizes for cognitive training game, g = 0.626, 95% CI [0.050, 1.203], p < 0.05 and conventional video game, g = 0.635, 95% CI [0.032, 1.238], p < 0.05, on overall cognition. The results indicated that older adults with longer training session time on cognitive training games and conventional video games had improved overall cognition. No moderator effect was observed for age, publication year, training frequency, total training hours and training duration.

Table 6. Results of moderator analyses with meta-regression.

Moderator variable (continuous) Video game type g 95% CI Heterogeneity
Age Cognitive training game 0.008 [-0.024, 0.039] Q(1) = 0.23, p &gt; 0.05
Exergame 0.025 [-0.020, 0.069] Q(1) = 1.16, p &gt; 0.05
Conventional video game 0.027 [-0.044, 0.097] Q(1) = 0.56, p &gt; 0.05
VR/simulation game 0.026 [-0.035, 0.086] Q(1) = 0.71, p &gt; 0.05
Session length Cognitive training game 0.626 a [0.050, 1.203] Q(1) = 4.54, p &lt; 0.05
Exergame −0.088 [-1.110, 0.935] Q(1) = 0.03, p &gt; 0.05
Conventional video game 0.635 a [0.032, 1.238] Q(1) = 4.07, p &lt; 0.05
VR/simulation game −0.083 [-0.627, 0.461] Q(1) = 0.09, p &gt; 0.05
Training frequency Cognitive training game −0.073 [-0.169, 0.023] Q(1) = 3.35, p &gt; 0.05
Exergame 0.156 [-0.522, 0.833] Q(1) = 0.20, p &gt; 0.05
Conventional video game 0.032 [-0.235, 0.299] Q(1) = 0.06, p &gt; 0.05
VR/simulation game −0.105 [-0.268, 0.058] Q(1) = 1.60, p &gt; 0.05
Total training hours Cognitive training game 0.002 [-0.009, 0.013] Q(1) = 0.18, p &gt; 0.05
Exergame 0.009 [-0.019, 0.038] Q(1) = 0.42, p &gt; 0.05
Conventional video game 0.004 [-0.009, 0.018] Q(1) = 0.39, p &gt; 0.05
VR/simulation game −0.015 [-0.038, 0.009] Q(1) = 1.54, p &gt; 0.05
Training duration Cognitive training game −0.013 [-0.044, 0.018] Q(1) = 0.65, p &gt; 0.05
Exergame −0.022 [-0.112, 0.068] Q(1) = 0.23, p &gt; 0.05
Conventional video game −0.010 [-0.101, 0.082] Q(1) = 0.04, p &gt; 0.05
VR/simulation game −0.023 [-0.072, 0.027] Q(1) = 0.80, p &gt; 0.05

g: Hedge's g.

a
p &lt; 0.05.

The results of the moderator analysis for control type in each video game type and game device can be found in Table 7. Three studies without control groups were excluded (Ackerman et al., 2010; Adcock et al., 2019; Zając-Lamparska et al., 2019). The heterogeneity between groups was not significant for video game type and game device. The effect size of active control for VR/simulation game and driving simulator were higher than passive control. In contrast, the effect size of passive control for exergame, mouse/keyboard and motion controller were higher than active control.

Table 7. Results of moderator analyses for control type.

Control type n g 95% CI Z Between-group heterogeneity
Video game type
Cognitive training game Active 2 0.199 [-0.116, 0.515] 1.238 Q(1) = 3.547, p &gt; 0.05
Passive 8 0.548 b [0.369, 0.727] 6.005
Exergame Active 4 0.514 a [0.099, 0.928] 2.428 Q(1) = 0.493, p &gt; 0.05
Passive 6 0.704 b [0.373, 1.035] 4.165
Conventional video game Active – – Q(1) = 0.230, p &gt; 0.05
Passive 5 0.787 b [0.523, 1.050] 5.852
VR/simulation game Active 4 0.368 b [0.248, 0.488] 6.000
Passive 2 0.311 b [0.109, 0.512] 3.016
Video game device
Mouse/keyboard Active 3 0.241 a [0.002, 0.480] 1.979 Q(1) = 2.467, p &gt; 0.05
Passive 6 0.482 b [0.300, 0.663] 5.196
Driving simulator Active 3 0.369 b [0.237, 0.500] 5.495 Q(1) = 0.223, p &gt; 0.05
Passive 2 0.311 b [0.109, 0.512] 3.016
Handheld game console Active – – Q(2) = 0.387, p &gt; 0.05
Passive 6 0.876 b [0.713, 1.038] 10.556
Motion controller Active 4 0.511 b [0.127, 0.895] 2.610
Passive 7 0.662 b [0.382, 0.943] 4.626

g: Hedge's g.

n: Number of studies.

a
p &lt; 0.05.
b
p &lt; 0.01.

Read full article

URL:

https://www.sciencedirect.com/science/article/pii/S0003687021001241

The effect of internet use on well-being: Meta-analysis

Özkan Çikrıkci , in Computers in Human Behavior, 2016

2.4 Moderator variables

Moderator analysis is an analysis method that allows testing of the difference in mean effect size of variables (moderators) and direction of difference between subgroups. Moderator analysis in a meta-analysis study should be planned appropriate to the aim of the study and should be performed in accordance with this plan (Littel et al., 2008 ). Statistical significance of differences between moderator variables is tested with the Q statistical method developed by Hedges and Olkin (1985). In this method Q is divided in two as Qbetween (Q b ) and Q within (Q w ) and analyses are performed on these two different Q. Q w tests the homogeneity within the moderator variable used while Q b tests the homogeneity between the groups (Borenstein et al., 2009; Hedges & Olkin, 1985; Kulinskaya et al., 2008).

In this study only the statistical significance of differences between moderators was examined, so only Q b values were used. One moderator variable considered to play a role in the mean effect size in this study was determined. Considering that the well-being components (self-esteem, well-being and life satisfaction) for internet use and well-being may have affected the effect size, they were evaluated as possible moderators.

Read full article

URL:

https://www.sciencedirect.com/science/article/pii/S0747563216306483

Social desirability is the same in offline, online, and paper surveys: A meta-analysis

D. Dodou , J.C.F. de Winter , in Computers in Human Behavior, 2014

3 Results

Fig. 1 shows the flow diagram of the study selection. The literature searches yielded 1473 publications. After removing 362 duplicates between and within searches, 1111 unique publications were reviewed. Of these, 644 were excluded as irrelevant based on their title or because they were reviews, overviews, or meta-analyses or because they did not provide a comparison between computer and paper surveys. The abstracts of the 467 remaining publications were reviewed, and 222 of these were excluded as irrelevant, or because they were reviews, or because of not providing a comparison of social desirability scale scores between administration modes or computer vs. paper-and-pencil comparisons. The full texts of the remaining 245 publications were then reviewed.

Fig. 1. Flow diagram of study selection. GS: Google Scholar. WoK: Web of Knowledge.

Based on the inclusion criteria described in the methods section, 29 studies were eligible for inclusion, and 22 more studies were retrieved from the reference lists of these eligible studies and the meta-analyses mentioned in the method section, leading to a total of 51 studies being included in the meta-analysis. Of these, 15 were included in the meta-analyses of both Richman et al. (1999) and Dwight and Feigelson (2000), 4 were included only in Richman et al. (1999), 6 were included only in Dwight and Feigelson (2000), 19 were published after 1997 (year up to which the literature searches of both aforementioned meta-analyses were conducted), and 7 were published before 1997 but were not mentioned in the previous meta-analyses. The 51 studies included 62 independent samples. We were not able to retrieve the full text of three studies (Baydoun & Emperada, 1995; Dahl, 1992; Mitchell, 1993); for these studies, the sample sizes for the computer and paper surveys and the effect sizes (ds) reported in Dwight and Feigelson were used in the analysis. An overview of the studies included in the meta-analysis is provided in the Supplementary material.

3.1 Main effect and moderator analyses

Table 1 shows the results of the meta-analysis for the overall effect and for each of the moderator variables. The overall effect of administration mode on social desirability was nearly zero (fixed-effect d  =   0.00 and random-effects d  =   −0.01). The effects yielded by the moderator analyses were also small; the summary effects ranged from −0.15 to 0.10 for the fixed-effect meta-analysis and from −0.17 to 0.09 for the random-effects meta-analysis, and only few were significantly different from zero (5 and 4 of the 30 effect sizes for the fixed-effect and random-effects meta-analyses, respectively).

Table 1. Meta-analysis results.

Effect sizes Participants Fixed-effect Random-effects
d [95% CI] p-value d [95% CI] p-value I 2 (%)
All 62 16,700 0.00 [−0.03, 0.03] .787 −0.01 [−0.07, 0.04] .605 63.5
Survey characteristics
Internet connectivity a Internet 17 9057 −0.01 [−0.05, 0.04] .696 −0.03 [−0.14, 0.07] .539 78.4
Offline b 46 7693 0.01 [−0.02, 0.05] .464 0.00 [−0.07, 0.06] .936 51.5
Identity Anonymous 29 10,564 0.01 [−0.03, 0.05] .755 −0.01 [−0.08, 0.07] .897 64.8
Identifiable 24 3014 −0.02 [−0.09, 0.05] .619 −0.07 [−0.21, 0.07] .317 71.8
Unknown 17 3122 0.01 [−0.05, 0.07] .647 0.01 [−0.07, 0.09] .758 30.0
Test setting Individual/Own choice c 34 8929 −0.02 [−0.06, 0.02] .311 −0.03 [−0.10, 0.05] .448 52.5
Group 24 6285 0.02 [−0.03, 0.07] .459 −0.01 [−0.11, 0.10] .888 76.3
Unknown 6 1486 0.06 [−0.02, 0.14] .153 – – –
Skipping a Allowed 31 11,362 0.00 [−0.03, 0.04] .820 0.00 [−0.06, 0.05] .873 47.2
Not allowed 13 1297 0.10 [0.00, 0.20] .052 0.04 [−0.20, 0.28] .749 81.3
Unknown 20 4121 −0.03 [−0.09, 0.03] .303 −0.06 [−0.16, 0.04] .234 57.0
Backtracking a Allowed 30 9444 0.00 [−0.04, 0.04] .976 0.00 [−0.07, 0.08] .905 61.7
Not allowed 15 2602 0.06 [−0.01, 0.13] .093 0.04 [−0.10, 0.17] .601 62.6
Unknown 21 4898 −0.02 [−0.07, 0.04] .509 −0.07 [−0.17, 0.03] .174 62.7
Nature of questions Sensitive 4 1639 0.10 [0.01, 0.20] .032 – – –
Nonsensitive 52 13,645 0.00 [−0.03, 0.03] .908 −0.01 [−0.07, 0.05] .799 64.8
Unknown 6 1416 −0.09 [−0.20, 0.01] .077 −0.15 [−0.33, 0.03] .103 62.3
Scale types BIDR-IM 21 5253 0.05 [0.00, 0.09] .055 0.09 [0.01, 0.17] .037 61.6
BIDR-SD 16 4309 0.05 [0.00, 0.10] .039 0.06 [0.00, 0.12] .033 12.1
MMPI-L 17 1563 0.02 [−0.05, 0.10] .534 0.02 [−0.06, 0.09] .689 3.3
MMPI-K 17 1525 −0.00 [−0.08, 0.07] .972 −0.08 [−0.22, 0.06] .241 57.7
ELS 7 638 0.05 [−0.07, 0.18] .399 0.04 [−0.12, 0.20] .616 35.8
MCDS 13 4053 −0.15 [−0.22, −0.07] .000 −0.17 [−0.29, −0.04] .010 55.7
Other 11 4041 0.02 (−0.04, 0.07) .589 −0.07 (−0.21, 0.08) .388 83.8

Study characteristics
Sample type Students 42 10,440 0.02 [−0.01, 0.06] .188 0.01 [−0.06, 0.08] .790 66.6
Other 20 6260 −0.04 [−0.10, 0.01] .120 −0.07 [−0.17, 0.02] .142 52.9
Study design Within 18 2893 0.03 [−0.02, 0.08] .289 0.02 [−0.06, 0.09] .614 35.0
Between 44 13,807 −0.01 [−0.04, 0.03] .679 −0.03 [−0.10, 0.04] .432 69.2
Participant assignment Randomised/Within-subjects 49 8915 0.04 [0.00, 0.07] .041 0.03 [−0.03, 0.08] .288 42.8
Non-randomised 13 7785 −0.06 [−0.11, −0.01] .020 −0.15 [−0.29, −0.01] .031 83.7

Note: A positive effect size implies higher social desirability in the computer survey as compared to the paper survey. The total number of effect sizes is larger than 62 for some moderators because, in some of the studies, different portions of the sample were tested under different conditions (e.g., half of the sample was tested in a group setting, and the other half was tested in an individual setting).

p-values &lt; .05 are in boldface.

a
Moderator for computer administration only.
b
Computers linked in a local network were also coded as offline.
c
Own choice refers to Internet surveys that the participants completed at their convenience from any place of their choice.

The effect sizes were nearly zero for both the Internet and offline surveys (fixed-effect d  =   −0.01 and 0.01, respectively and random-effects d  =   −0.03 and 0.00, respectively). An additional exploratory meta-analysis comparing Internet surveys and paper surveys with anonymity as moderator also yielded no significant effect sizes (anonymous Internet vs. paper surveys: d  =   −0.01, 95% CIs: [−0.12, 0.10], p  =   .822 based on 13 effect sizes and identifiable Internet vs. paper surveys: d  =   −0.17, 95% CIs: [−0.63, 0.28], p  =   .455 based on 5 effect sizes). The effect size of the Internet surveys that were conducted from any place that the participants desired (i.e., the test setting was the participants' choice) vs. paper surveys was d  =   0.01 (95% CIs: [−0.09, 0.11], p  =   .876 based on 9 effect sizes).

No significant effect of administration mode was found when analysing randomised studies alone (d  =   0.03 based on random-effects analysis), whereas significantly lower social desirability was found for the computer surveys than for the paper surveys for the non-randomised studies (d  =   −0.15). The heterogeneity in the non-randomised studies was twice as large as the heterogeneity in the randomised studies (83.7% vs. 42.8%, respectively).

The summary effect across the studies that used sensitive questions indicated a significantly higher social desirability for the computer surveys than for the paper surveys. Note that this summary effect relied on 4 studies only, and only one of these (Booth-Kewley, Larson, & Miyoshi, 2007) reported significantly higher social desirability for the computer surveys than for the paper surveys (specifically, for self-deception; for impression management no statistically significant administration effect was found).

The BIDR-IM and BIDR-SD measured significantly lower social desirability for paper surveys than for computer surveys (BIDR-IM: d  =   0.05 and d  =   0.09 for fixed-effect and random-effects, respectively; BIDR-SD: d  =   0.05 and d  =   0.06 for fixed-effect and random-effects, respectively), whereas the MCSD measured higher social desirability for paper surveys than for computer surveys (d  =   −0.15 and −0.17, for fixed-effect and random-effects, respectively). The MMPI-L and ELS showed near-zero effects.

The funnel plot of the effect sizes looked symmetric, with 49 of the 62 effect sizes falling within the fixed-effect 95% confidence interval of d  =   0.00. Of the remaining 13 effects, 8 (2 of which corresponded to randomised studies) were lower than the lower 95% confidence bound, and 5 (3 of which corresponded to randomised studies) were higher than the upper 95% confidence bound (Fig. 2).

Fig. 2. Funnel plot of the effect sizes versus sample sizes. A positive effect size indicates higher social desirability in the computer survey as compared to the paper survey. A 95% confidence interval (CI) is drawn around d  =   0.00, calculated as CI   =   ±   1.96 *V 0.5, where V is the within-study variance calculated as: V  =   4/N.

3.2 Longitudinal trends

The sample-size weighted Spearman correlation coefficient between the publication year and effect size was ρ  =   .24 (p  =   .060) based on 62 effect sizes, and ρ  =   .18 (p  =   .204) based on the 49 effect sizes from the randomised and within-subjects samples alone (Fig. 3; linear regression: y  =   0.006 *x  −   11.604, where x is the publication year and y is the effect size when all 62 effect sizes were included and y  =   0.003 *x  −   6.924 when only the 49 effect sizes corresponding to randomised and within-subjects samples were included). The newer studies had larger sample sizes than did the older studies (ρ  =   .64, p  =   2.3 *  10−8, 62 effect sizes), and relied more on Internet than on offline computer surveys (ρ  =   .65, p  =   8.1 *  10−9, 62 effect sizes).

Fig. 3. Scatter plot of the 62 effect sizes included in the analysis as a function of publication year. A positive effect size indicates higher social desirability in the computer survey as compared to the paper survey. The area of the markers corresponds linearly to the sample size. The markers in the legend correspond to a sample size of 200. The horizontal line at d  =   0.00 is presented for reference purposes.

Exploratory correlational analyses were conducted between the publication year and each of the moderators. The newer studies were more likely to have been conducted in a group setting (unweighted Spearman correlation: ρ  =   .24, p  =   .077, 56 effect sizes) and to have allowed backtracking in the computer version (ρ  =   .38, p  =   .015, 41 effect sizes). The newer studies also tended to be anonymous (ρ  =   .27, p  =   .069, 45 effect sizes) and randomised/within-subjects (ρ  =   .17, p  =   .194, 62 effect sizes).

3.3 Validity of the social desirability scales

To investigate whether the social desirability scales were valid and sensitive, for the studies in which the level of anonymity was a moderator variable, we conducted an additional meta-analysis with the level of anonymity as the main effect (i.e., social desirability in the surveys with anonymous vs. identifiable participants across both administration modes). To protect the quality of the dataset, non-randomised studies were excluded from this analysis. The results revealed that the anonymous surveys generated lower desirability than did the non-anonymous surveys ( d  =   −0.23, 95% CIs [−0.34, −0.13], p  =   2.0 *  10−5 based on 7 effect sizes from 7 studies, and 1343 unique participants).

Read full article

URL:

https://www.sciencedirect.com/science/article/pii/S0747563214002143

Being a cybervictim and a cyberbully – The duality of cyberbullying: A meta-analysis

Raquel Lozano-Blasco , ... M.Pilar Latorre-Martínez , in Computers in Human Behavior, 2020

3.2 Moderator variables and meta-regression analysis

Because the existence of moderator factors (male sex, female sex, age and culture) could lead to wide variability in the results (Botella & y Sánchez, 2015 ), it was necessary to test for such effects. A meta-regression test was used, and models were compared. Four moderator variables were established: male sex, female sex, mean age and culture (North American, South American, Central European, Mediterranean, Asian and Oceanic). The meta-regression (see Table 5) yielded five models: 1. simple, 2. male sex, 3. female sex, 4. age, and 5. culture. The first model, which included no moderator variable, did not help to explain any percentage of variance, and the same applied to model 2 (male sex) and model 4 (age). Although model 3 (female sex) explained 3% of the variance, these data were non-significant (p > 0.05). Model 5 explained 66% of the variance (R 2  = 0.66), with a significance of p = 0.0000 (p<0.01). However, the meta-regression (see Table 6 and Fig. 4) allowed for a better analysis of the culture variables. The significance (p<0.05) and negative signs of the Central European culture, Mediterranean culture, Asian culture, North American culture and South America Culture coefficients indicated that adolescents within these cultures were more likely to become cybervictims-bullies. In other words, culture explained the data variability. Moreover, the heterogeneity displayed by the Q and I 2 statistics and the funnel plot graph was more easily interpretable in light of this cultural diversity.

Table 5. Model comparison: Random effects (MM), Z-distribution, Fisher's Z.

Model name TauSq R2 Q df p-value
'Model 1 SIMPLE 0.0314 0.00 0.00 0 1.0000
'Model 2 MALE 0.0318 0.00 2.90 1 0.0885
'Model 3 FEMALE 0.0305 0.03 3.04 1 0.0810
'Model 4 AGE 0.0258 0.00 0.21 1 0.6459
'Model 5 CULTURE 0.0107 0.66 33,35 5 0.0000

Table 6. Meta-regression M.5.

Meta-regression M.5
Covariate Coefficient Standard Error 95%Lower 95%Upper Z-value 2-sided p-value Q df p
Intercept 0.8310 0.0972 0.6404 1.0216 8.55 0.000 50.53 6 0.000
Asian Culture −0.3214 0.1053 −0.5277 −0.1151 −3.05 0.0023
Central European Culture −0.5870 0.1050 −0.7928 −0.3812 −5.59 0.0000
Mediterranean Culture −0.3229 0.1207 −0.5243 −0.1215 −3.14 0.0017
North American Culture −0.3351 0.1187 −0.5678 −0.1024 −2.82 0.0048
Oceanian Culture −0.2126 0.1564 −0.5155 0.0903 −1.38 0.1690
South America Culture −0.2949 0.1364 −0.5622 −0.0277 −2.16 0.0306

Fig. 4

Fig. 4. Fisher's regression for culture variables.

Read full article

URL:

https://www.sciencedirect.com/science/article/pii/S0747563220301977