Appendix C: Network Analysis
Evaluation of a Learning Collaborative's Process and Effectiveness to Reduce Health Care Disparities Among Minority Populations
A. Overview of Network Analysis1
The Collaborative aims to help the nine participating firms work together to reduce racial and ethnic disparities among their members. To understand more fully the relationships between participants in the Collaborative and learn how the Collaborative functions, we conducted network analyses of the Collaborative. The analyses provide tools for assessing whether the support organizations are offering assistance (and through what channels) and whether participating firms are interacting with one another. The analyses also provide a means of examining whether the sponsor organizations are perceived as visible and active participants in the Collaborative. In addition, the analyses offer some indication as to whether certain organizations contribute more to the Collaborative than others and/or benefit more from their participation. Specifically, the network analyses allow us to capture organizations' perceptions of contributions and benefits, and whether these contributions and benefits are equal across participating organizations.
Brief Overview of Findings. The results of the network analysis indicate that the sponsor organizations and primary support organizations play a central role in the Collaborative. They have the most contact with participating firms and form the primary pathways that link participants (including both firms and nonfirm organizations). Firm-to-firm relations are much less prevalent than firm-to-nonfirm relations. In fact, interactions and influence between firms are generally quite limited outside the Collaborative, a result that may be expected and even appropriate given the competitive environment in which participating firms operate. Nonetheless, most organizations participating in the Collaborative find each other important, and most respondents report that other organizations are carrying out their responsibilities and commitments to the Collaborative, at least to a small extent. A few of the firms—namely, one national and one regional firm—stand out as more important and influential members of the Collaborative than other firms.
Background.Network theory focuses on the relationships and ties among actors or organizational entities (Wasserman and Faust 1994). Even though network analysis may capture individual actors' attributes, its focus is on relational patterns between actors. Such analysis can be used as a purely descriptive tool, but we relied on the analysis to develop an understanding of the relationships that promote or impede the Collaborative's ability to work effectively in addressing concerns related to disparities. We applied the tools of network analysis to describe the relationships among the organizations participating in the Collaborative and to identify relevant network properties that shed light on the Collaborative's outcomes.
The network analyses for the evaluation focus on relationships and perceptions between participating organizations in the Collaborative rather than within participating organizations. While the Collaborative hopes to affect the internal workings of participating organizations, our focus on relationships between participants is appropriate given that the Collaborative is most committed to developing inter-organizational ties. Moreover, an analysis of the network structure within each participating organization was not possible given the size and complexity of participating firms (especially the national firms) and the scope of the evaluation.2
Methods. To collect information on the relationships between and perceptions of organizations participating in the Collaborative, we developed a network feedback form based roughly on an instrument developed by Van de Ven and Ferry (1980). We used questions from the instrument that were most relevant for assessing the Collaborative and modified question wording to make sense in the context of the Collaborative. The feedback form included two primary sets of questions:
- Nine questions about the Collaborative overall, such as whether the Collaborative has been worthwhile and whether it has influenced the organization's activities.
- Eight questions (including one two-part question) about relations between and assessments of other Collaborative participants, such as the extent of communication between the respondent's organization and each of the other organizations outside of Collaborative meetings and the influence of other organizations on the respondent's organization with respect to disparities.
Select for the feedback form.
We asked the lead contact at each firm, support organization, and sponsor organization to complete the feedback form in late 2005—approximately 15 months after the start of the Collaborative. Although we did not receive some forms until January 2006, we achieved a 100 percent response rate (15 organizations). Some respondents, however, did not complete certain portions of the feedback form—particularly questions that asked respondents to assess the contributions of other organizations participating in the Collaborative—leading to substantial item nonresponse on a few questions. In fact, we dropped Question 15 ("To what extent has your organization changed or influenced other organizations with respect to disparities?") from the analysis because half of the Collaborative's 15 organizations did not answer or answered "cannot assess." Question 13 ("To what extent has each organization carried out its responsibilities and commitments involving disparities in regard to the Collaborative during the past six months?") also resulted in relatively high non-response, with 6 organizations rating few or no other organizations. We still present the results of Question 13, however, noting the missing data. For other questions in which respondents were asked to rate other organizations, two organizations often did not provide ratings.
Even though we achieved full cooperation on the network feedback form among Collaborative participants, the number of respondents remains small (15). Therefore, while a systematic analysis of national versus regional firms (as well as other types of participating organizations) would be interesting, the sample size does not support it. However, we do observe some interesting differences in firms versus nonfirm organizations (i.e., sponsor and support organizations) and draw such distinctions in this chapter where appropriate. We also distinguish among key support organizations (CHCS and RAND), other support organizations (GMMB and IHI), and sponsors (AHRQ and RWJF) on certain dimensions of the analysis as appropriate as the three sets of organizations have distinct roles in the Collaborative.
Caveats. An important caveat in examining the network analyses relates to the reliance on self-reported data. While analyses of networks frequently rely on self-reports, any given organization's frame of reference varies from that of other organizations. Moreover, some organizations may be more forthcoming or unbiased in their ratings than others. In some ways, the fact that organizations are asked to rate one another for our network analyses provides a partial check on self-reports; for example, Organization A may report that it fulfilled its commitments to the Collaborative, but other organizations may report that Organization A did not fulfill its commitments. Such unreciprocated claims may provide clues about the appropriate operation (or not) of the Collaborative.
An additional caveat is that an effective collaboration can occur in many different ways. Not all participants need to be highly communicative or influential. An extremely "dense" network—in which all participants communicate with all others—is actually inefficient. Therefore, while we draw out certain findings in this chapter, our analysis is not necessarily meant to be normative but rather is descriptive of the Collaborative process and its early outcomes as far as resulting action and change.
The remainder of this chapter focuses on participating organizations' overall perceptions of the Collaborative, followed by various analyses of the Collaborative network.
B. Overall Perceptions of the Collaborative
The network feedback form first asked the lead contact from each participating organization to provide feedback on the Collaborative as a whole by responding to a core set of nine questions as follows:
- Overall, how important was the Collaborative in attaining the goals of your organization?
- To what extent has the Collaborative carried out its responsibilities and commitments?
- To what extent has your organization carried out its responsibilities and commitments to the Collaborative?
- To what extent do you feel the relationship between your organization and the Collaborative is productive?
- To what extent is the time and effort spent in developing and maintaining the relationship with the Collaborative worthwhile?
- Overall, to what extent are you satisfied with the relationship between your organization and the Collaborative?
- To what extent has your organization changed or influenced the activities of the Collaborative?
- To what extent has the Collaborative changed or influenced the activities of your organization?
- Are the payoffs of the Collaborative for your organization reasonable relative to your contribution?
As presented in Table C.1, all but one participating organization felt that the Collaborative was at least somewhat important to attaining organizational goals (Question 1). In fact, 10 of the 15 organizations in the Collaborative (6 of the 9 firms and 4 of the 6 nonfirm organizations) reported that the Collaborative was very important or crucial for achieving organizational goals with regard to reducing health disparities. On average, organizations felt that the Collaborative has carried out its responsibilities and commitments "to a considerable extent," with firms slightly more positive than other (nonfirm) organizations (Question 2).
When asked to rate their own organization on carrying out responsibilities and commitments to the Collaborative, almost all organizations were very positive (Question 3). Two of the firm representatives (both from national firms), however, indicated that their organizations carried out their responsibilities and commitments only to "a little extent" (one) or "some extent" (one). Almost all respondents reported that the relationship between their organization and the Collaborative is productive and worthwhile (Questions 4 and 5); support and sponsor organizations were somewhat more positive than firms on these dimensions. Likewise, all organizations reported that they were satisfied with the relationship between their organizations and the Collaborative to at least some extent (Question 6).
Support and sponsor organizations reported changing or influencing Collaborative activities more than firms did (Question 7), a response that is probably not surprising given these organizations' roles in shaping the Collaborative and working with firms on a one-on-one basis. In comparison, firms were more likely than other organizations, however, to say that the Collaborative had changed or influenced their organization's activities, with 6 of the 9 firms saying to a "considerable extent" or "great extent" (Question 8). All organizations said that the payoffs of the Collaborative were reasonable relative to contribution, with firms somewhat more positive on this dimension than sponsor and support organizations (Question 9).
Clearly these responses paint a positive picture of the Collaborative overall. Although there was relatively little variation in the average response across questions, respondents across all organizations were most positive about the time and effort required by the Collaborative being worthwhile and least positive about the extent to which the Collaborative influenced the respondent's own organization (the latter of which was driven by the responses of sponsor and support organizations rather than by those of firms).
C. Relationships in the Context of Collaborative Goals
The network feedback form also included a series of eight questions to assess the presence and strength of relationships between Collaborative participants, and the corresponding answers provide the data for our network analyses. These questions asked respondents to rate all other participating organizations on various dimensions, such as their influence, the extent to which they carried out their responsibilities and commitments, and their contribution of good ideas to the Collaborative. For each question, respondents assessed each of the other 14 organizations using an ordered response scale (such as to no extent, to a little extent, to some extent, and to a considerable extent).
The following section describes four categories of findings from our network analyses: (1) the strength of pre-existing ties, (2) how the Collaborative works, (3) the perceived contributions to organizational action, change, and influence, and (4) the overall standing of participating organizations. We mapped Questions 10 through 17 from the feedback form into the first three categories above (based on the dimension reflected in each question). In addition, we use information from Questions 10 through 17 to create measures of overall standing. A large amount of missing data led to the exclusion of Question 15 ("To what extent has your organization changed or influenced other organizations?") from our analyses.
1. Strength of Pre-existing Ties
The feedback form queried respondents about relationships that existed prior to the start of the Collaborative between an organization's participants in the Collaborative and those of each of the other organizations. While prior relationships are not a prerequisite for a successful collaborative, they do reveal information about the extent of existing relationships and provide information about the relations formed during the Collaborative. They suggest relationships that were needed in order to undertake joint work for the Collaborative.
A two-part question (Question 10 of the network feedback form) captured: (1) any personal acquaintance with the key Collaborative staff from each organization before formation of the Collaborative and (2) if that acquaintance existed, the extent to which the respondent's organization had an effective working relationship with the other organizations. The main findings include the following:
- The sponsor organizations reported a prior acquaintance with all or almost all other participants before the start of the Collaborative. The support organizations each had prior acquaintance with at least 8 of the other 14 organizations, though the key support organizations reported knowing fewer organizations than other support organizations prior to the start of the Collaborative.
- Conversely, at least 10 organizations (and more than 10 for some organizations) reported a prior acquaintance with the sponsor organizations and key support organizations before the start of the Collaborative.
- One national firm reported a prior acquaintance with all other members of the Collaborative. In general, however, firms varied substantially in the number of organizations with which they had a prior acquaintance, with the average firm acquainted with four other organizations before the Collaborative.
- Eleven of the 14 other organizations in the Collaborative (including 6 of the 8 other firms) indicated a prior acquaintance with one particular national firm. (This was not the same firm that reported a prior acquaintance with all members of the Collaborative.)
- In only a few instances did two firms both report mutual acquaintance with one another (firm-to-firm "reciprocated" ties), perhaps reflecting the competitive environment between firms or maybe a difference in perceptions across respondents about the definition of a prior acquaintance.
- When asked whether their organization had an effective working relationship before the Collaborative with organizations for which respondents reported an existing acquaintance, participants most commonly reported effective working relationships with the sponsor organizations. Participants also reported effective working relations with the key support organizations on a relatively frequent basis. That is, before the Collaborative began, firms were more likely to have effective working relations with sponsors or support organizations than with one another.
2. The Collaborative Process
Understanding the Collaborative process—including if and how much participants communicate and share information with one another—is essential to uncovering the ways in which the Collaborative might effect change. Three questions included in the network feedback form provide information on the Collaborative process and the way the Collaborative functions (Questions 11, 13, and 17). The questions provide some sense of the success of the Collaborative process in terms of communicating, carrying out commitments, and providing ideas. These questions are:
- During the past six months—outside of formal Collaborative meetings—how frequently have people from your organization who are involved in the Collaborative communicated or been in contact with people in the organizations listed below?
- To what extent has each organization carried out its responsibilities and commitments involving disparities in regard to the Collaborative during the past six months?
- Which organizations provide good ideas for dealing with disparities at meetings of the Collaborative?
Communication. Sponsor and support organizations reported a substantial amount of communication with each other outside of formal Collaborative meetings; in fact, communication between nonfirm entities represents the most "dense" part of the communication network. A fair amount of communication also occurs between the two key support organizations and firms. This result is not surprising and is consistent with what we understand to be the way the Collaborative works—an approach that lends itself to extensive consultation between individual support organizations and firms and then among the support organizations in order to coordinate efforts.3 A number of organizations also reported communication with GMMB, though substantially fewer than the number reporting communication with the two key support organizations. This finding is consistent with the fact that GMMB has communicated about media toolkits with firms' communications departments, particularly in late 2005 (when the feedback form was distributed).
Outside of formal Collaborative meetings, firm-to-firm communication was limited. Only a few firms reported communicating with other firms, with regional firms reporting the large share of such communications. Three firms (two national and one regional) reported no communication with any other firm outside of Collaborative meetings. It is possible that the lack of such communication explains why firms found it valuable to rely on sponsors to convene the Collaborative as a vehicle for addressing competitive or other barriers to communication.
Figure C.1 provides an illustration of communication between participants. The first six rows of the figure correspond to the 6 sponsor and support organizations (labeled s1 through s6 and listed in a random order), and the remaining nine rows correspond to the firms (labeled f1 through f9 and listed in a random order).4 The rows represent how a given organization rates each of the organizations listed in the columns (which represent the 6 sponsor and support organizations, followed by the 9 firms, from left to right). For example, boxes 2 through 15 of the first row show how support organization 1 rated all other organizations (S2, S3, and so forth) in the Collaborative. The shaded blocks in the figure represent some level of communication, with darker shades indicating more frequent communication. The black squares represent the strongest ties between Collaborative organizations. White boxes indicate no communication between organizations, and red boxes indicate missing data. (Blocks on the figure's diagonal are in light red, given that organizations did not rate themselves.) Consistent with the discussion above, the figure shows that most communication occurs between nonfirm organizations and between firms and nonfirms rather than between firms.
Carrying Out Responsibilities and Commitments. With the caveat that a substantial amount of data is missing for the question on responsibilities and commitments, most organizations that rated other organizations felt that others were carrying out their commitments and responsibilities to at least a little extent. (In terms of missing data, four of the nine firms rated few or no other organizations, and two of the support organizations did not rate other organizations.) Only one organization, a national firm, reported that four other (mostly national) firms were not carrying out their responsibilities and commitments. Generally, firms tended to rate other firms as carrying out their responsibilities "to a little extent" or "to some extent" and rated other (nonfirm) organizations more favorably. Support and sponsor organizations rated firms much more favorably on commitments and responsibilities than firms rated each other.
Figure C.2 provides an illustration of the network analysis of commitments and responsibilities. Again, darker shading indicates that the rated organization is seen as carrying out its responsibilities and commitments to a greater rather than lesser extent (with the darkest boxes indicating "to a considerable extent" and white boxes indicating "to no extent"). It is important to note the large amount of missing data, shown in red.
Providing Good Ideas. When asked to rate other organizations on whether they provided many, some, or no good ideas (Question 17), most organizations were rated by others as providing the Collaborative with at least "some good ideas" (Figure C.3). Firms and nonfirm organizations alike often rated the two key support organizations and the two sponsors as providing many good ideas. In addition, one national firm and one regional firm were identified by over half of the other organizations in the Collaborative as providing many good ideas. Conversely, two national firms were identified by nearly half of the other organizations as providing no good ideas. One firm saw only three other Collaborative organizations (all support organizations) as a source of good ideas. Six of the participating organizations saw all other organizations as sources of good ideas. The red diagonal squares reflect the undefined "self-ties" in the network.
3. Perceived Contributions of Collaborative Participants to Action and Change
While understanding the Collaborative process is important, the Collaborative's ultimate aim is to bring about outcomes, namely, organizational action and change in reducing racial and ethnic disparities among participating firms. We therefore wanted to assess whether any organizations were particularly important or influential (or not) with respect to the Collaborative, perhaps spurring others to action. Three questions in the network feedback form help provide information on the perceived contribution of other participating organizations to a given organization's actions and goals (Questions 12, 14, and 16) in terms of each organization's importance, productivity, and influence relative to the other organizations. These questions are:
- Overall, how important was each organization's work through the Collaborative in attaining the goals of your organization with respect to disparities?
- To what extent do you feel the relationship between your organization and each of the other organizations with respect to disparities is productive?
- During the past six months, to what extent has each of these other organizations changed or influenced the activities of your organization with respect to disparities?
While many other factors unrelated to the Collaborative's structure—including external and internal factors—could influence whether a given organization sees its relationships with other organizations as productive or influential, the three questions provide some information on the Collaborative's possible effects or outcomes.
Importance of Others.Firms generally rated nonfirm organizations as at least moderately important in helping the firms attain their disparities-related goals (where the scale included not at all important, somewhat important, moderately important, very important, and crucial). However, few firms identified other firms as important to their organizational goals, with the most common ratings "not at all important" and "somewhat important." Firms' ratings of nonfirms make sense given that sponsor and support organizations were directly involved with helping firms with their work on disparities. It may be that most firms did not find other firms important in meeting their organizational goals, given that firms may have different goals and may be uncomfortable sharing their goals (given the competitive environment).
Sponsor and support organizations generally rated each other as at least moderately important to attaining organizational goals. Similarly, sponsor and support organizations tended to rate firms as at least moderately important to organizational goals. Thus, all three groups perceived that sponsors and support organizations were at least moderately important to the success of the Collaborative, but firms were less likely than the other groups to perceive other firms as important to them.
Figure C.4 provides an illustration of the importance of other organizations. (Note that one nonfirm organization did not rate other organizations.) Again, boxes with darker shading indicate stronger relationships, and white boxes show cases where an organization is "not at all important." Assessments of other organizations as not important are concentrated in the firm-to-firm ties. Again, it is important to stress that this result is not necessarily an adverse finding because participants bring different objectives to their participation in the Collaborative.
Productive Relationships with Others. When asked about the productivity of relationships with other Collaborative participants, most organizations saw their relationships with others as productive at some level (i.e., at least "to a little extent"). This finding suggests that participants see value in their participation. Collaborative participants most frequently reported considerably productive relationships with the key support organizations and the sponsor organizations. All other organizations received only a few nominations (three or four at most) for providing a relationship that is productive to a considerable extent. Conversely, four organizations—three national firms and one regional firm—viewed relationships with a handful of other firms as not productive.
Figure C.5 provides a graphic representation of organizations reporting considerably productive relationships with other organizations participating in the Collaborative. Sponsor and support organizations are shown as rectangles and firms as ovals. The lines or "ties" between organizations with a single arrow reflect one of the two organizations in the pair reporting a productive relationship with the other organization (with the receiving organization being the one with which the relationship is considerably productive). Lines or ties with no arrows indicate that both organizations reported considerably productive relations with one another. The figure reflects the fact that, as described above, many organizations report considerably productive relationships with the sponsor and support organizations (shown as rectangles). Most firms are viewed as offering considerably productive relationships by only a few other organizations in the Collaborative.5
Influence of Others. Collaborative participants were asked to assess the extent to which other organizations in the Collaborative changed or influenced the activities of their own organizations relative to disparities. The sponsor and support organizations are reported to have the most influence on other Collaborative members (one of the key support organizations and one of the sponsors have the most influence). Only one organization (a regional firm) reported no external influence from any other Collaborative members. The results indicate that, with one possible exception, all of the organizations in the Collaborative have been influenced by other participants.
4. Overall Findings of Organizational Standing
To understand where participating organizations fit relative to one another, we also developed a general index of relative standing (Doreian 1986; Doreian 1987) of Collaborative members. Standing is determined by three factors. First, organizations have greater standing if they receive more "nominations," that is, many others identify them as influential or important. Second, greater standing is associated with nominations of greater strength (e.g., influence is "considerable" rather than "little"). Third, organizations have greater standing if their nominations come from other organizations with high standing. While we computed relative standing for several dimensions (importance, responsibilities and commitments, productive relationships, source of good ideas), we report only on overall findings in the interest of brevity.
Consistent with the findings above, the analysis of relative standing revealed that the two key support organizations and two sponsors generally have the highest standing in the Collaborative. (The same did not hold for the measure of standing related to source of good ideas; some firms rated higher than nonfirm organizations on this measure.) In addition, one of the national firms consistently had relatively high standing across several dimensions, as did one of the regional firms.
The analysis of relative standing also revealed some important information on the Collaborative as a whole. Most notably, there are no outliers in the distributions of relative standing.6 In other words, even though participating organizations vary in their standing, no one organization stands out as extremely important or extremely unimportant to the Collaborative. For many social networks, where choices tend to concentrate on a small number of network actors, standing measures produce skewed distributions in which a few actors are viewed as extremely important or extremely unimportant.7 The analysis of standing suggests that the National Health Plan Collaborative has good potential for collaborative learnings, given the relative equality in standing across organizations.
D. Review of Key Findings and discussion
Several important findings emerged from our network analyses as follows:
- First and foremost, the key support organizations and sponsor organizations play a central role in the Collaborative. Not only are they visible and active participants in the Collaborative process, but they also appear to act as the "glue" that holds the Collaborative together. They have the most contact with participating firms and form the primary pathways that link participants. The sponsor and support organizations also engage in a substantial amount of contact with one another. Key support and sponsor organizations also play an important role in contributing to action and change among other organizations.
- Firm-to-firm relations are much less prevalent than firm-to-nonfirm relations. In terms of process measures such as communication, interactions between firms are limited (though a few firms are seen as providing many good ideas to the Collaborative process). Regarding firm-to-firm measures of action and influence, only a few firms reported considerably productive relationships with other firms, and influence between firms is limited.
- Still, most organizations participating in the Collaborative find each other important, and most respondents report that other organizations are carrying out their responsibilities and commitments to the Collaborative, at least to a small extent.
- As suggested in the discussion of participants' overall standing, a few of the firms—namely, one national and one regional firm—stand out as more important and influential members of the Collaborative than other firms. Conversely, several firms consistently ranked toward the bottom of the measures of standing. These results likely suggest that some firms are contributing more than others. Organizations' ratings of whether the Collaborative is productive and worthwhile and whether it yields a reasonable payoff compared with the level of organizations' contributions are all fairly favorable and do not appear to vary greatly with by organizational standing (though one national firm with low standing tended to rate the Collaborative lower than other firms).
Given these findings, what are the implications for the Collaborative? And does the current structure represent a "healthy" network? While our network analyses reveal that firms rarely communicate with each other, such an approach may be completely appropriate. Contact and communication occur through other pathways, namely, the support and sponsor organizations. Interactions between organizations consume time, and it would be highly inefficient for all organizations in a network to communicate with each other. In fact, organizations interacting in a network face several strategic issues in securing resources or access to resources and obtaining favorable network locations (Burt 1990). In seeking favorable locations, no organization can afford to communicate with all other organizations unless required by its role. It follows, in general, that networks requiring substantial resources to form and maintain relationships should not be complete. In addition, given the competitive nature of the health plan industry, it is not surprising that firms generally do not communicate with one another outside of formal Collaborative meetings. Unless firms operate in different markets (as is the case with regional firms) or have some business imperative for additional collaboration, they may well limit their contact with one another. Moreover, it is probably unrealistic to expect higher levels of cross-firm contact in the future, particularly between national firms. Nonetheless, despite the possibility of logical reasons for limited firm-to-firm contact, some firms—namely, regional firms, which may be less concerned about competition—may be less satisfied than other organizations with the Collaborative's network structure.
Participating organizations came to the Collaborative with different motivations, as confirmed by the findings of the network analysis. Several firms reported in interviews that they wanted to learn what other firms were doing in the area of racial and ethnic disparities while fewer firms explicitly expressed an interest in making changes. Further, though firms did not necessarily say that they were reluctant to share information, the way they described their internal clearance processes made it clear that release of firm-specific information is an important threshold decision for a firm. These motivations and constraints therefore reveal some information as to why the Collaborative's network structure looks as it does.
1. While MPR staff drafted this chapter, Patrick Doreian, a consultant to the project based at the University of Pittsburgh, guided the overall analysis and design of the network component of the evaluation. Mr. Doreian helped identify relevant items and is responsible for most of the analysis that we considered in framing findings from the work.
2. To understand firms' internal relationships related to reducing disparities, we employed the less formal approach of structured interviews with firm participants. We had originally hoped to explore communications systematically among senior executives and various line managers in order to develop a sense of the existence and strength of relations but found that the organizational structures were so complex and our interviews too limited to fully support this. However, Chapter III provides an analysis of how the Collaborative is positioned within each firm and the degree to which the firm is known for its work related to the Collaborative or, more generally, for its work on disparities.
3. The Collaborative was structured so that RAND provided technical assistance to firms—particularly on geocoding and surname analysis activities—and CHCS organized meetings of the Collaborative and collected periodic status reports from firms. Often, staff from several support organizations participated together as a team on individual calls with a firm, and frequent conference calls (known as "operational committee" calls) were conducted to help support and sponsor organizations prepare and coordinate their activities as part of the Collaborative.
5. The absence of a line or tie between organizations indicates that a given organization either reported that its relationship with the other organization is not productive to a considerable extent or did not answer the question.
6. For this analysis, we used box plots and interquartile ranges (refer to Koopmans 1987). The interquatile range is the difference between the first and third quartiles. Any data points more than 1.5 times the interquartile range above the third quartile (or the corresponding distance below the first quartile) are sufficiently extreme to be regarded as outliers.