TI21: A Technology Enhanced Inquiry Framework for Developing and Assessing 21st Century Skills


Lei Bao, Department of Physics, The Ohio State University

Kathy Koenig, Physics & STEM Education, University of Cincinnati

Version: 2012-1201

Citation: Bao, L., & Koenig, K. (2012). TI21: A Technology Enhanced Inquiry Framework for Developing and Assessing 21st Century Skills, iSTARAssessment.org.


21st Century Skills

We live in an ever-changing world – demographic change, rise of automation and workforce structural change, globalization, and corporate change are some major driving forces that demand fundamental transformations in education and skills on an individual level.  Across the globe, work is becoming increasingly bi-polar with jobs sorting out into two clusters – a low-wage, lower-skilled, routine work cluster, going to the lowest global bidder qualified to do the work, and increasingly to automation; and a fast growing, high-paying, creative work cluster requiring a combination of complex technical skills like problem-solving and critical thinking, and strong people skills like collaboration and clear communication. In the U.S., the demand for non-routine skills (expert thinking and complex communication) is rising fast, as the need for routine and manual skills falls (1960-2002).

Advances in digital technology and telecommunications now enable companies to send works and tasks to be done wherever they can be completed best and cheapest. Meanwhile, political and economic changes in developing countries such as India, China and Mexico have freed up many more workers who can adequately perform such jobs. As a result, not only do Americans have to compete for jobs with foreigners in a rising global labor market, but increasing competition will also center on highly skilled workers for more intellectually demanding and higher paying jobs. 

Due to technology development and globalization, companies have gone through radical restructure with less hierarchy and lighter supervision where workers experience greater autonomy and personal responsibility. Work has also become much more collaborative and employees must adapt to new challenges and demands when tackling projects and solving problems. 

Consequently, a growing number of educators, business leaders and politicians have called for “21st century skills” being taught as part of everyone’s education. Global competition, increased access to technology, digital information and tools are increasing the importance of 21st century knowledge-and-skills, which are critical for a country’s economic success. Advocates base their arguments on a widening gap between the knowledge and skills acquired in school and the knowledge and skills required in 21st century workplaces. That is, today’s curricula do not adequately prepare students to live and work in a technology-based economy and globalized society. Thus, in order to successfully face career challenges and a globally competitive workforce, schools must be aligned with real world environments by infusing 21st century skills in education practices.


Skills Gap between Schools and Workplaces


Previous studies have demonstrated a huge skill gap between schools and workplace requirements. In 2005 Skills Gap Report, when asking manufacturing employers which types of skills their employees will need more of over the next three years, basic employability skills (attendance, timeliness, work ethics, etc.) and technical skills were the areas most commonly selected (53%). Following that are reading/writing/communication skills, where 51% of the respondents said they will need more of these types of skills over the next three years. Beyond these, there are a number of related skills that will be needed over the next several years that are characteristic of high-performance workforces, such as the ability to work in teams (47%), strong computer skills (40%), the ability to read and translate diagrams and flow charts (39%), strong supervisory and managerial skills (37%), and innovative/creative abilities (31%). Moreover, manufacturing employers see training as a business necessity and their spending on training is increasing – not just for executives, but across all employee groups. The types of training that most employees receive are technical and basic skills training. The next tier of trainings are for problem solving, teamwork, leadership, computer skills, basic or advanced mathematics, basic reading and writing, and interpersonal skills – all standard skills for high-performance workforces.

Another landmark 2006 research study among more than 400 employers, Are They Really Ready to Work?, (conducted by Corporate Voices for Working Families, the Conference Board, the Partnership for 21st Century Skills, and the Society for Human Resource Management), clearly spotlighted employers’ concerns about the lack of preparedness of new entrants into the workforce regardless of the level of educational attainment. More specifically, the deficiencies are greatest at the high school level, with 42.4% of employers reporting the overall preparation of high school graduates as deficient; 80.9% reporting deficiencies in written communications; 70.3% citing deficiencies in professionalism; and 69.6% reporting deficiencies in critical thinking. Although preparedness increases with educational level, employers noted significant deficiencies remaining at the four-year college level in written communication (27.8%), leadership (23.8%) and professionalism (18.6%). In addition, employers reported that top five most important skills are critical thinking and problem solving, information technology, teamwork/collaboration, creativity/innovation, and diversity.

A more recent study, “Across the Great Divide”, released March 2011, surveyed 450 businesses and 751 post-secondary educational institutions and found concerning disparities between the goals of higher education and what businesses sought in workers. The skill gap exists along the entire learning-career continuum – colleges, businesses and the students all had different expectations of what was needed to prepare a workforce for today’s and tomorrow’s jobs. According to the report, employers indicated they believed the most important goal of a four-year degree was to prepare individuals for "success in the workplace" (56%). On the other hand, educational leaders saw higher education as a way of providing individuals with "core academic knowledge and skills" (64%). The study also found that only 15% of the businesses believed hiring those with an associate degree was a good return on investment for their companies.

Both workers and employers believe that the education sector has the primary responsibility to close the workforce readiness gap. Yet, as surveys indicated, majority of companies do not believe schools are doing a good job preparing students for the workplace. Therefore, continuing contact between schools and businesses is critical to developing a prepared workforce. And it is essential for business leaders, policy makers and educators to work together to address the workforce readiness gap.


What are 21st Century Skills?


So what exactly are 21st century skills? The P21 (Partnership for 21st Century Skills – a group of corporations who partnered with the U.S. Department of Education in 2002) has created a framework that identifies the key skills for success. Based on their categorization and definition, ten skills have been identified as the 21st Century skills, in four groups:

Ways of Thinking

1.      Creativity and innovation
2.      Critical thinking, problem solving, decision making
3.      Learning to learn, Metacognition

Ways of Working

4.      Communication
5.      Collaboration (teamwork)

Tools for Working

6.      Information literacy
7.      ICT literacy

Living in the World

8.      Citizenship – local and global
9.      Life and career
10.  Personal & social responsibility – including cultural awareness and competence

The essence of these skills includes collaboration, communication, creativity and innovation and critical thinking coined the 4Cs by P21. Many other researchers and authors created lists similar to the 4Cs. For example, Tony Wagner from the Harvard Graduate School of Education interviewed more than 600 chief executive officers, and asked them the same essential question: “Which qualities will our graduates need in the 21st-century for success in college, careers and citizenship?” Wagner's subsequent Seven Survival Skills correspond to the 4Cs but also include agility and adaptability, accessing and analyzing information, as well as curiosity and imagination.

There is agreement among all researchers that these skills of collaboration, communication, creativity and critical thinking are necessary and must be integrated into the classrooms. Indeed, states are adopting new standards to ensure these skills are met. For example, Common Core State Standards have been adopted by most states and several territories in the United States. Common Core State Standards are designed to provide a national, standardized set of academic standards (organized around 21st century skills) as an alternative to those previously developed by the states on an individual basis. The Common Core Standards are sought to be more rigorous; demand higher-order thinking; introduce some concepts at an earlier age; and allow for interstate comparisons.

On the other hand, the modern workplace and lifestyle demand that students balance cognitive, personal, and interpersonal abilities, but current education policy discussions have not defined those abilities well, according to a special report released by the National Research Council of the National Academies of Science in Washington. Based on the report, 21st century skills generally fall into three categories:

  • Cognitive skills, such as critical thinking and analytic reasoning;
  • Interpersonal skills, such as teamwork and complex communication; and
  • Intrapersonal skills, such as resiliency and conscientiousness (the latter of which has also been strongly associated with good career earnings and healthy lifestyles).

A relevant concept that we often hear is the “21st century learning skills.” So what it is? Ted Lai, Director of Information Technology for the Fullerton Elementary School District puts it this way:

"In a nutshell, these are the skills that will help people be globally competitive in the 21st Century. Especially with our students, these are skills that include not only the curricular standards but also a host of other essential skills like communication, collaboration, and creativity. Literacy doesn’t merely refer to the ability to read and write but also the ability to evaluate and synthesize information, media, and other technology. At the heart of 21st Century Learning, in my opinion, is the piece on creating authentic projects and constructing knowledge… essentially making connections between learning and the real world!"

Clearly, “21st century skills” has become the lasted buzz in education, which has also re-kindled a long-standing debate about content vs. skills. Although reading, writing, mathematics and science are cornerstones of today’s education, curricula must go further to include skills such as collaboration and digital literacy that will prepare students for 21st-century employment and ensure students’ success in the real world. Establishing new forms of assessment can begin a fundamental change in how we approach education worldwide.


21st Century Skills and Scientific Reasoning

21st century skills have been widely captured as the core education goals casted in the new education standard. The consensus on the definition of these skills converges on three broad clusters including cognitive, interpersonal, and intrapersonal skills that each includes a rich set of sub-dimensions (NRC 2011, 2012). Within the cognitive domain, multiple competencies have been proposed, which include non-routine problem solving, system thinking, critical thinking, information literacy, reasoning and argumentation, and innovation (NRC 2011, 2012).

In addition, business leaders and educational organizations also have taken on effort to define their desired versions of the “21st century skills”. For example, the Partnership for 21st Century Skills proposed three broad areas of skills in Learning and Innovation; Information, Media and Technology; and Life and Career; as well as a set of core subjects and 21st Century Themes (Partnership for 21st Century Skills, 2013, p. 2). In the area of Learning and Innovation, critical thinking and problem solving are listed as one of three core sub skills.

At the international level, the Assessment and Teaching of 21st Century Skills (ATC21S) project aims to expand the teaching and learning of 21st century skills globally, especially by improving assessment of these skills. In their definition, the skills include four broad areas: Ways of Thinking, Ways of Working, Tools for Working, and Living in the World.  The strand of “Ways of Thinking” also listed multiple sub skills including creativity and innovation, critical thinking, problem solving, decision making, learning to learn, and metacognition.

Although there are multiple definitions on what count as the 21st century skills by different organizations and research committees, the notions of critical thinking, problem solving, and creativity seem to have been recognized by different researchers as a common set of sub skills within the cognitive strand of the 21st century skills. Among these, critical thinking is the most commonly noted and widely studied by educators and cognitive researchers. Since critical thinking is highly connected and often blended with the other cognitive sub skills of problem solving, decision making and creative thinking (Marzano et al., 1988; Facione, 1990; Bailin, 1996; Fisher, 2001; Lipman, 2003; Kennedy, 2009;), it becomes an important education goal since the early 1980s (NCECT, 1987; Binkley, Erstad, Herman, Raizen, Ripley, & Rumble, 2010) and plays an foundational role in defining, assessing, and developing the 21st century skills. 

There is an extensive literature about critical thinking in education (e.g., Glaser, 1941; Bangert-Drowns and Bankert, 1990; Facione, 1990). Although there are various definitions of what constitutes critical thinking, there are some underlying principles that run through the various definitions. Broadly defined, critical thinking is the use of cognitive skills or strategies that increase the probability of a desirable outcome; it is reasoned, purposeful and goal directed. It is the kind of thinking involved in solving problems, formulating inferences, calculating likelihoods, and making decisions (Halpern, 1999); and as “reasonable reflective thinking focused on deciding what to believe or do” (Ennis R. , 1993). In a way, critical thinking is about “how to think”; and it is recognized as the favorable way to understand and evaluate a subject matter, which produces reliable knowledge and improves thinking itself (Siegel, 1988; Paul, 1990).

In STEM learning, the notion of scientific reasoning is often used to label the set of basic skills that support critical thinking, problem solving, and creativity. Scientific reasoning, broadly defined, includes the thinking and reasoning skills involved in inquiry, experimentation, evidence evaluation, inference and argumentation that support the formation and modification of concepts and theories about the natural world, which include the abilities to systematically explore a problem, formulate and test hypotheses, manipulate and isolate variables, and observe and evaluate the consequences (Zimmerman, 2005; Bao et al., 2009). From their definitions, critical thinking and scientific reasoning share many common features; they both involve reaching a reliable conclusion by identifying a question, formulating a hypothesis, seeking and gathering relevant data and logically testing and evaluating the hypothesis. In a way, scientific reasoning can be viewed as a somewhat domain-specific realization of critical thinking in the context of STEM learning. Therefore, based on the vast literature of related work, we view scientific reasoning as a more concrete formulation of critical thinking and a core component of the cognitive strand of the 21st century skills in the context of STEM learning.


Education Framework for Developing and Assessing 21st Century Skills

Discussions on what it means to educate for the 21st century focus on the skills the future will demand of today’s students, the necessary content knowledge they will need, and the attributes they must have to successfully contribute to the workforce and global economy. Several themes emerge which are directly tied to this project. First, students must acquire advanced problem-solving abilities including those associated with scientific reasoning (SR); e.g., ability to analyze a problem, seek relevant data, test new and familiar concepts, and process information until a viable solution is reached. Second, there is a need for students to develop literacy in information and computer technologies. Students must be able to accept, learn and adapt to new technology so as to find and synthesize information to address challenges. Third, there is an imminent need for more students pursuing STEM (science, technology, engineering, and mathematics) education, particularly among those underrepresented in the field including students with disabilities (National Center for Technology Innovation and Center for Implementing Technology in Education, 2009). Thus, it is critical to find ways to empower underrepresented groups, including those with disabilities, to enter STEM fields in order to meet the national job crisis. Not only are graduates in STEM needed to fill jobs to sustain the economic development of the country, but a STEM education is important as most jobs of the future will require a basic understanding of math and science (NRC, 2011).

In response to these needs, science education has undergone major changes. For example, A Framework for K-12 Science Education (NRC, 2012), the basis for the Next Generation Science Standards (NGSS), suggests reforms that view science education through three dimensions: scientific practices, crosscutting concepts, and core ideas. The latter two compose the content of science, while the first dimension focuses on how scientists construct knowledge. These practices include asking questions, developing and using models, planning and carrying out investigations, analyzing and interpreting data, using mathematics and computational thinking, constructing explanations, engaging in argument from evidence, and obtaining, evaluating, and communicating information. Underlying these practices is SR, which includes the reasoning involved in science inquiry that supports the formation and modification of concepts and theories about the natural and social world (Bao et al., 2009; Zimmerman, 2005).

Although SR abilities are emphasized in the K-12 Framework as the foundation for scientific practices, research has shown that teacher candidates are severely lacking in this area and the development of SR can be difficult to achieve (Koenig et. al, in press; Lawson, 2002; Tamir & Zohar, 1991). This is problematic as it has been found that teachers that have less sophisticated understanding of science as a practice are less likely to implement the types of problem-based learning or student-led investigations suggested under the K-12 Framework (Keys & Bryan, 2001; Trumbull, Scarano, & Bonney, 2006).  In addition, NSTA (2012) issued a statement expressing concern that teachers lack the expertise necessary for students to achieve particular standards in the NGSS, especially those associated with scientific practices. This indicates the importance of teacher professional development (PD) programs that place emphasis on developing teacher understanding of scientific practice.

In our own work, we have developed a curricular framework for inquiry learning that explicitly promotes scientific reasoning and an understanding of scientific practices. Elements of the framework have been applied to multiple classroom settings and have been shown to be successful in developing student abilities in these areas, including those of pre-service teachers (Bao et al., 2009; Koenig, Schen, Edwards, & Bao, 2012; Koenig, Schen, & Bao, in press). Recently, we expanded the framework to more fully encompass the broader range of 21st century skills with an emphasis on the use of technology. We refer to this Technology-Enhanced Inquiry Framework for 21st century skills as the TI21 framework (see Table 1). This curricular framework can be used to guide the development of learning modules for use in K-12 classrooms and be integrated into PD programs to improve teachers’ capacity for providing high quality STEM education to all students with emphasis on development of 21st century skills. Inside the framework, we also address the need for the development of assessments for measuring 21st century skills as called for by ATC21S (Assessment and Teaching of 21st Century Skills) and the Partnership for 21st Century Skills.


Table 1. Technology-Enhanced Inquiry framework for development and assessment of 21st Century skills (TI21, version 2012)


Model and Methodology

Classroom Practice and Evaluation

21st Century Skill

Science Inquiry with Technology Enhanced SR Framework*

Learning and Teaching Strategies that

Support Knowledge-Building Environment


(Formative and Summative)

Ways of Thinking


1. Creativity and innovation


2. Critical thinking, problem solving, decision making


3. Learning to learn, metacognition


Scientific Reasoning (SR) Framework


·   Targeting a comprehensive set of scientific reasoning and critical thinking skills

·   Use of open-ended project- and problem-based learning

·   Emphasizing scientific methods and inquiry processes


·  Repeated emphasis on using a structured SR scaffolding framework

·  Group-based science inquiry activities providing multiple and repeated opportunities for students on problem conceptualizing, hypothesis forming, experimental design, data collection and analysis, decision making, reflection, and communication

·  Activity components including brainstorming, project design/presentation, critiquing and defending ideas, troubleshooting, data modeling and theorization, theory testing and application, and evidence-based justification and debating

·  Specific questioning embedded in activities emphasizing “Why?” and “How?” rather than “What?”

·  Repeated reflection on the process of learning to foster constructivist epistemology and metacognition

·    Quantitative assessment using iSTAR** and other instruments that target specific SR skills

·    Lab reports for analysis of levels of reasoning, evidence-based justification, decision making, and argumentation

·    Student design plans

·    Students critique a provided design or data analysis

·    Videotape student groups to document the types of questions asked by students; “Why?” or “How?” versus “What?”

·    From qualitative data identify how well students plan, conduct, and interpret evidence for solving a problem and how well students go beyond what was specified in learning activities

Ways of Working


4. Communication


5. Collaboration (teamwork)


Group-based Science Inquiry

·   Group-based collaborative inquiry activities emphasizing minds-on and hands-on learning

·   Comprehensive 5E science inquiry processes including engagement, exploration, explanation, elaboration and evaluation in a technology-facilitated group environment  for sustained learning, motivation, and performances 

·  Emphasize team work during group projects highlighting collaborative learning activities including questioning, planning, discussion, argumentation, brainstorming for alternative ideas and solutions, evaluation for methods and results, and reflection of learning

·  Provide students with a variety of mediums with which to present their argument or findings

·  Promote student discussions and argumentations around important questions such that students discuss their evidence, reasoning, and claims; specific questioning including “Why?” and “How?”

·    Videotape students working on regular classwork as well as a provided task for evaluation purposes (code transcripts)

·    Social network analysis of the types and quantity of student interactions

·    Document student use of technology to support work and research

·    Document how effectively the groups work together and manage projects; specific measures include richness of argument, project presentations, collaboration skills regarding team formation, and integration of contributions and feedback



Model and Methodology

Classroom Practice and Evaluation

21st Century Skill

Science Inquiry with Technology Enhanced SR Framework*

Learning and Teaching Strategies that

Support Knowledge-Building Environment

(formative and summative)

Tools for Working

6. Information literacy


7. ICT-Information, Communications and Technology



Technology Integration

Integrate technology into instruction as:

·   a tool for learning that helps students conduct research and supports emergence of new skills

·   a tool for assessment helping students and teachers obtain formative evaluations and feedback while inquiry learning takes place

·   a tool for sustained engagement of students in using technology, developing interest in STEM and targeting STEM careers


·  Use a variety of information access tools to locate, gather, and organize potential sources of scientific information to answer questions

·  Collect real-time observations and data to synthesize and build upon existing information (e.g., online databases noaa, ePa, uSgS) to solve problems

·  Use appropriate tools to analyze and synthesize information (e.g., diagrams, flow charts, frequency tables, bar graphs, line graphs, and stem-and-leaf plots) to draw conclusions and implications based  on investigations of an issue or question

·  Create media products for presentations and reports

·  Extend the use of learned technology effectively and creatively in other domains beyond the scope of the learning in school (fostering a culture of technology)

·   Using specially designed apps to track and monitor the ways technology is used by students including time engaged, frequency of usage, types of apps used, types of information and analysis involved, and types of communication carried out 

·   Videotape students engaged in classwork and in specially assigned projects to determine how technology is used in the process of their learning and communication

·   From the data analysis, identify how easily students use technology and for what purpose they use it, etc.

·   Do controlled study to measure how effectively technology helps generate knowledge construction

Living in World

8. Citizenship – local and global


9. Life and career

10. Personal & social responsibility

STEM Content Choice

·     Topics relate to real life and world’s future development

·     Student career development and balance of personal and social needs

·     Science ethics – balance technology with society


·  Topics are chosen that engage students in current societal issues and use 21st century skills to understand and address global issues (promote understanding of the local and global implications of certain decisions)

·  Projects are chosen that promote health and environmental literacy, etc.

·  Engage students in project planning and management

·   Provide students with a real-life scenario and ask questions involving societal ethical choices and promote ability to engage in mock decision-making

·   Assess student ability to manage goals and time, work independently yet interact effectively with others in diverse teams, and negotiate

* The main framework of teaching is project-based inquiry learning. All projects and learning activities are organized around a central research question that explicitly targets causal hypothetical reasoning (e.g. “why” types of questions) embedded in cycles of scientific investigations, reflections, and communication.

** Inventory for Scientific Thinking and Reasoning (iSTAR)


A TI21 based Exemplar Learning Module on Energy Transformation

The TI21 framework structures curriculum such that the learning of 21st century skills and content occur in parallel. This will likely be a new idea for teachers; so we give an exemplar learning module that the 7th and 8th grade teachers can conceptualize and then use in their own classrooms. We demonstrate in Table 2 how the learning module for Green Energy is supported by the TI21 framework. Learning objectives include (1) identify different forms of energy (e.g., electrical, mechanical, chemical, thermal, nuclear, radiant and acoustic), (2) explain how energy can change forms but total amount of energy remains constant, (3) design, build, and test a model windmill to study the variables impacting windmill efficiency and cost-effectiveness. The first two objectives are embedded within the project-based learning structure implemented within the third learning objective.

Under the TI21 framework a central question drives the entire instructional sequence such that activities are consistently integrated into a meaningful learning experience. Through these connections, students will be better able to synthesize the outcomes of the various activities, resulting in deeper learning of both the content and targeted skills. In addition to content questions, we will pose questions that require students to engage in advanced problem solving, such as those involved in making causal inferences within a complex system with many alternative possibilities. This will contribute to developing those reasoning and higher-level abilities necessary in answering the larger question. The driving question for this module will be “How is an efficient and cost-effective windmill made? What needs to be considered and why?” Activities will be designed to address the central question with a focus on the “why.” In this way, the module will engage students fully in the training of scientific (experimental) methods.

Table 2.  The science module goes through five phases based on the 5E inquiry model.  Note that students work in collaborative groups and often cycle back and forth through the different phases as necessary, which makes multiple pathways of learning adaptive to individual characteristics.   



Specific Tasks

Specific Task Goals

Learning and Assessment Objectives

Observation and Engagement

·  Observe and study examples of real world windmills (e.g., with videos and models)

·  Collect information through web and other public and academic sources

·  Synthesize information to identify variables, relations, models, hypotheses, and research questions.

·  Study and interact in group:

- Brainstorming to generate ideas

- Presentation for communication and exchange ideas and information

- Argumentation for causal evidence and results alignment and for thought experiments on alternative ideas

- Critiques of different designs and ideas to learn from mistakes and learning to think and argue critically

- Synthesize information and results from different parties to form more generalized understanding, questions, hypotheses, and conclusions

*Cycles of the above process are referred to as group based BPACS procedures. 

·  Group and Individual Reports

Through observing, comparing and studying real world windmill examples, students complete the following task goals:

·    Identify energy forms/types

·    Identify energy transfer processes with attention to

- energy forms involved

- real world variables involved

- possible interactions and relations among variables

- involved transfer processes

- models of energy transfer including the involved variables, relations, and processes

·    Develop hypotheses about the causal relations and models that explain energy transfers 

·    Identify and study different forms of windmill applications in real world – that is how energy is used to achieve real jobs for the different needs of society

·    Compare different designs to identify unique features and their possible use in light of energy transfer models, real applications, and impacts to the environment

The training and assessment of skills are aligned with the TI21 framework in terms of four categories of abilities:

·    iSTAR: Inquiry for Scientific Thinking and Reasoning

·    iCollab: Interactive Collaboration and Communication

·    iTech: Integration of Technology

·    iWorld: Individual and World considerations and perspective  

In the first phase of activities, all four areas of skills are addressed and assessed. The training emphasizes effective use of technology, communication skills, as well as reasoning on variable identification and hypothesis development.


The assessments are primarily formative style through group interactions, critiques, and evaluations of project reports. For example, real time peer and instructor feedback in the BPACS process can inform students about their work and teachers about student performances.

Research Planning

(Emphasize the minds-on side of inquiry learning)

·  Develop a research plan including simple to complex research questions for exploration and validation

·  Develop procedural plans to test research questions including methods and procedures in data collection, data analysis and interpretation, decisions on future steps (many what if questions and considerations)

·  Develop a scheme to implement the cyclic process of scientific inquiry

·    Start with planning on testing simple questions (hypotheses) involving 1-2 variables and relations (e.g., How do width and/or length of the windmill blades affect rotation speed?) 

·    Move on to testing more complicated multi-variable questions involving 3 + variables (e.g., shape, angle, and length of blades)

·    Integrate the individual research questions to develop an optimized plan to complete the research and final designs

Students will work in groups through the BPACS procedures and complete group and individual sectional project reports. The assessments are still primarily in the formative style through group interactions, critiques, and report evaluations.

In this phase, the training emphasizes the thinking and reasoning involved in scientific methods and hypothesis forming. 



Specific Tasks

Specific Task Goals

Learning and Assessment Objectives

Design and Implement Experiments; and Explore Experimental Options

(Integrated minds-on and hands-on inquiry learning)

·  Based on the research plan, design conduct and explore experiments:

o Design concrete procedures and methods of the experiment

o Gather materials and revise design if needed in light of the constraints on practical limitations

o Build and conduct experiments

o Collect and analyze data

o Revise experiments if needed

o Improve overall design

·  Synthesize results and decide next step

·    In the design and implementation, train students on the reality and practical sides of control of variables, causal experimental design, and data collection

·    Data analysis, interpretation, and hypothesis verification

·    Evidence based conclusions and plans for next step


Students all work in groups through the BPACS procedures and complete group and individual sectional project reports. The assessments are still in formative style through group interactions, critiques, and report evaluations.


In this phase, the training emphasizes reasoning involved in hypothesis testing control of variables, data analysis (pattern recognition), and adaptation and balancing of practical constraints.  

Synthesis and Expansion

·  The class as a whole does group presentations of different groups’ designs and experimental outcomes

·  Conduct class wide project evaluation and discussion for students to critique and debate on each other’s results

·  Identify unique features and innovative ideas among different designs and explore how such ideas and designs may be applied in other contexts

·    Train communication and team work skills at a larger scale facing more diverse ideas and events in less familiar contexts

·    Train the ability to integrate multiple strands of skills (reasoning, technology, etc.) in meaningful synthesis and generalization from different contexts as well as applications to new situations.

·    Develop more general scope understanding of the involved problems and solutions

Students all work in groups and complete sectional project reports.

The assessments are formative based on class presentations, critiques, arguments, and generalized understandings and their applications (transfers to new contexts).

The training emphasizes the integration of multiple views and results to develop more generalized understandings and apply these to new contexts.


·  Through classroom discussion (above), form consensus views on important features of the different designs and experiments as well as their cost-and-benefit in light of the practical constraints and science ethics

·  Also through class discussion and consensus forming, reflect on the process inquiry learning and scientific exploration. Develop preferred meta-cognitive stances on learning

·    Training on meta-cognitive views on inquiry learning, scientific methods such as evidence based exploration processes for hypothesis testing and decision making

·    Develop extended perspective on science and society, including both scientific and society variables into project designs

·    Understand the needs and methods for balancing constraints from difference sources to develop optimal solutions for real-world problems    

Students all work in groups and complete sectional project reports.

The assessments are formative in style through class wide group interactions, critiques, and report evaluations.

The training emphasizes the meta-cognitive aspects of learning as well as diversity and ethics so that students can develop a more holistic view about science and society rather than treating science as isolated unpractical pieces.     

Additional Assessments

·  Pre-Post testing on content knowledge, scientific reasoning, attitudes on learning, communication skills, technology skills, etc.

·    Provide baseline results and learning gains to support evidence based education practices (See the list of instruments in the narrative.)

Pre-post testing using standardized tests and or open ended reports such as thought-experiment design and critiques of given design examples or arguments



Working Summary

This is a working document that is continuously developed. As discussed in the previous sections, improving students’ scientific reasoning ability is an important goal of STEM education under the 21st century education framework. However, this topic is less studied in the education research community. The overarching goals of our work are (1) to further develop a research infrastructure to systematically study and evaluate the impacts of widely used interactive engagement and inquiry based education interventions on developing students’ ability in scientific reasoning, and (2) to develop effective curricula, which are informed and validated by research, to improve critical thinking, problem solving, and other cognitive skills of the 21st century framework.  



Assessment and Teaching of 21st Century Skills (ATC21S) project. (2010). New assessments and environments for knowledge building: A Whitepaper. Retrieved from http://atc21s.org/wp-content/uploads/2011/11/4-Environments.pdf

Bao, L., Cai, T., Koenig, K., Fang, K., Han, J., Wang, J., … Wu, N. (2009). Learning and scientific reasoning. Science, 323, 586-587. doi: 10.1126/science.1167740

Bingham, R. D., & Felbinger, C. L. (2002). Evaluation in practice: A methodological approach (2nd ed.). New York: Chatham House Publishers.

Bransford, J. (2000). How people learn: Brain, mind, experience, and school. National Academies Press.

Bybee, R. W. (2009). The BSCS 5E Instructional Model and 21st Century Skills. National Academies Board on Science Education, Washington, DC: Retrieved October 4, 2012. Retrieved from: https://www7.nationalacademies.org/bose/1Bybee_21st%20Century_Paper.pdf

Cooper, M. M., and Sandi-Urena, S. (2009). Design and validation of an instrument to assess metacognitive skillfulness in chemistry problem solving. Journal of Chemical Education 86 (2), 240-245.

Creswell, J. W. (2003). Research design: Qualitative, quantitative, and mixed method approaches. (2nd ed.). Thousand Oaks, CA: Sage Publications.

Davis, E. A. (2003). Prompting middle school science students for productive reflection: Generic and directed prompts. The Journal of the Learning Sciences, 12, 91-142. doi: 10.1207/S15327809JLS1201_4

Desimone, L. (2009). Improving impact studies of teachers’ professional development: Toward better conceptualizations and measures. Educational Researcher, 38, 181-199.  doi:10.3102/0013189X08331140

Fishman, B., & Davis, E. A. (2006). Teacher learning research and the learning sciences. The Cambridge handbook of the learning sciences, 535-550.

Fuchs, L. S., Deno, S. L., & Mirkin, P. K. (1984). The effects of frequent curriculum-based measurement and evaluation on pedagogy, student achievement, and student awareness of learning. American Educational Research Journal, 21, 449-460. doi: 10.3102/00028312021002449

Ge, X., & Land, S. M. (2004). A conceptual framework for scaffolding ill-structured problem-solving processes using question prompts and peer interactions. Educational Technology Research & Development, 52(2), 5-22. doi: 10.1007/BF02504836

Grigg, J., Kelly, K., Gamoran, A. & Borman, G. (2012). Effects of two scientific inquiry professional development interventions on teaching practice. Educational Evaluation and Policy Analysis, Analysis. doi:10.3102/0162373712461851.

Grincewicz, A., Zydney, J. M., Jones, P. & Hasselbring, T. S. (2011). The role of human-technology scaffolding in changing the dynamics of the classroom. Journal of Computers in Mathematics and Science Teaching, 30, 229-249. Retrieved from http://www.editlib.org/p/35262 

Guskey, T. R. (2002). Evaluating professional development. Thousand Oaks, CA: Corwin Press, Inc.

Hewson, P.W., & Hewson M., (1989). Analysis and Use of a Task for Identifying Conceptions of

Teaching Science, Journal of Education for Teaching: International Research and Pedagogy, 15(3), 191‐209.

Johnson, R. B., & Onwuegbuzie, A. J. (2004). Mixed methods research: A research paradigm whose time has come. Educational Researcher, 33, 14-26. doi: 10.3102/0013189X033007014.

Keys, C.W., & Bryan, A. L. (2001). Co-constructing inquiry-based science with teachers: essential research for lasting reform. Journal of Research in Science Teaching, 38, 631-645.

Kim, M. C., & Hannafin, M. J. (2011). Scaffolding Problem Solving in Technology-Enhanced Learning Environments (TELEs): Bridging Research and Theory with Practice. Computers & Education, 56, 403-417. doi: 10.1016/j.compedu.2010.08.024

Kirschner, P. A., Sweller, J., & Clark, R. E. (2006). Why minimal guidance during instruction does not work: An analysis of the failure of constructivist, discovery, problem-based, experiential, and inquiry-based teaching. Educational Psychologist, 41, 75-86. doi:10.1207/s15326985ep4102_1

Koenig, K., Schen, M., & Bao, L. (in press; Fall 2012). Explicitly targeting pre-service teacher scientific reasoning abilities and understanding of nature of science through an introductory science course. Science Educator.

Koenig, K., Schen, M., Edwards, M. & Bao, L. (2012). Addressing STEM retention through a scientific thought and methods course. Journal of College Science Teaching, 41, 23-29.

Lave, J., & Wenger, E. (1991). Situated learning: Legitimate peripheral participation. Cambridge university press.

Lawson, A. E. (2002). Sound and faulty arguments generated by preservice biology teachers when testing       hypotheses involving unobservable entities. Journal of Research in Science Teaching, 39, 237-252. doi: 10.1002/tea.10019

Lawson, A. E. (1978). Development and validation of the classroom test of formal reasoning. Journal of Research in Science Teaching, 15(1), 11-24. Retrieved from http://www.public.asu.edu/~anton1/LawsonAssessments.htm

Lee, H., Linn, M. C., Varma, K., & Liu, O. (2010). How do technology-enhanced inquiry science units impact classroom learning? Journal of Research in Science Teaching, 47, 71-90. doi: 10.1002/tea.20304

Lotter, C., Harwood, W. S., & Bonner, J. J. (2007). The influence of core teaching conceptions on teachers' use of inquiry teaching practices. Journal of Research in Science Teaching, 44, 1318-1347. doi: 10.1002/tea.20191

Loucks-Horsley, S. and Matsumoto, C. (1999), Research on Professional Development for Teachers of Mathematics and Science: The State of the Scene. School Science and Mathematics, 99: 258–271. doi: 10.1111/j.1949-8594.1999.tb17484.x

Maccini, P., Mulcahy, C. A., & Wilson, M. G. (2007). A follow-up of mathematics interventions for secondary students with learning disabilities. Learning Disabilities Research & Practice, 22, 58-74. doi: 10.1111/j.1540-5826.2007.00231.x

Mastropieri, M. A., Scruggs, T. E., Norland, J. J, Berkeley, S, McDuffie, K., Tornquist, E. H., & Conners, N. (2006). Differentiated curriculum enhancement in inclusive middle school science: Effects on classroom and high-stakes tests. The Journal of Special Education, 40, 130-137. doi: 10.1177/00224669060400030101

Messick, S. (1994). The interplay of evidence and consequences in the validation of performance assessments. Educational Researcher, 32, 13-23.

Mislevy, R. J., & Haertel, G. D. (2006). Implications of evidence-centered design for educational testing. Educational Measurement: Issues and Practice, 25(4), 6-20.

Mislevy, R. J., Chudowsky, N., Draney, K., Fried, R., Gaffney, T., Haertel, G., … & Wilson, M. (2003). Design patterns for assessing science inquiry. Principled Assessment Designs for Inquiry (PADI) Technical Report1.

Montague, M. (1992). The effects of cognitive and metacognitive strategy instruction on the mathematical problem solving of middle school students with learning disabilities. Journal of Learning Disabilities, 25, 230-248. doi: 10.1177/002221949202500404

National Center for Technology Innovation and Center for Implementing Technology in Education.  (2009). Technology supported science, technology, engineering, and math (STEM) instruction for students with disabilities. LD online. Retrieved from http://www.ldonline.org/article/35729/

National Research Council. (2012a). A framework for K-12 science education: Practices, crosscutting concepts, and core ideas. Washington, DC: National Academies Press.

National Research Council. (2012b). Education for Life and Work: Developing Transferable Knowledge and Skills in the 21st Century. Washington, DC: National Academies Press. 

National Research Council. (2011). Successful K-12 STEM Education: Identifying Effective Approaches in Science, Technology, Engineering, and Mathematics. Washington, DC: National Academies Press.  

Pellegrino, J. W., Chudowsky, N., & Glaser, R. (2001). Knowing what students know: The science and design of educational assessment. National Academies Press.

Piburn, M., Sawada, D., Turley, J., Falconer, K., Benford, R., Bloom, I., and Judson, E. (2000). Reformed Teaching Observation Protocol (RTOP) reference manual. ACEPT Technical Report No. IN00‐3. Tempe, AZ: Arizona Board of Regents [On‐line]. Available at ACEPT website http://www.public.asu.edu/~anton1/AssessArticles/Assessments/Chemistry%20Assessments/RTOP%20Reference%20Manual.pdf.

Reformed Teaching Observation Protocol (RTOP) reference manual. ACEPT Technical Report No. IN00‐3. Tempe, AZ: Arizona Board of Regents [On‐line]. Available at ACEPT website (http://www.acept.asu.edu).

Scriven, M. (1991). Evaluation Thesaurus (4th ed.), Newbury Park, CA: Sage.

Tamir, P., & Zohar, A. (1991) Anthropomorphism and teleology in reasoning about biological phenomena. Science Education, 75, 57-68.

Trumbull, D., Scarano, G., & Bonney, R. (2006). Relations among two teachers' practices and beliefs, conceptualizations of the nature of science, and their implementation of student independent inquiry projects. International Journal of Science Education, 28 (14), 1717-1750.

van Garderen, D. (2007). Teaching students with LD to use diagrams to solve mathematical word problems. Journal of Learning Disabilities, 40, 540–553. doi: 10.1177/00222194070400060501

Wholey, J. S., Hatry, H. P., and Newcomer, K. E. (1994). Handbook of Practical Program Evaluation. San Francisco, CA: Jossey-Bass Publishers.

Xin, Y. P., Si, L., Hord, C., Zhang, D., Cetintas, S, & Park, J. (2012). The effects of computer-assisted instruction in teaching conceptual model-based problem solving. Learning Disabilities: A Multidisciplinary Journal, 18, 71-85.

Zhou, S., Han, J.,  Pelz, N., Wang, X., Peng, L., Xiao, H., &  Bao, L. (2011). Inquiry style interactive virtual experiments: A case on circular motion. European Journal of Physics, 32, 1597-1606.

Zimmerman, C. (2005). The development of scientific reasoning: What psychologists contribute to an understanding of elementary science learning. Retrieved from National Research Council’s Board of Science Education website http://www7.nationalacademies.org/bose/Corinne_Zimmerman_Final_Paper.pdf

Zydney, J. M. (2010). The effect of multiple scaffolding tools on students¹ understanding, consideration of different perspectives, and misconceptions of a complex problem. Computers & Education, 54(2), 360-370. doi: 10.1016/j.compedu.2009.08.017