Edmentum's Partnership with WebbAlign, Part 3: A Conversation on DOK, Assessment, and Edtech

Friday, February 12, 2016 -- Sara Christopherson

This post is the final installment in a series with the team at WebbAlign, a leading provider of research and training in assessment and standards alignment using Webb’s depth of knowledge (DOK) framework. Today, we’re sharing a Q & A session between two of our Edmentum curriculum developers, Chrystal Finch and Shanti Flaherty, and WebbAlign’s Director, Sara Christopherson. To learn more about WebbAlign and the DOK framework, check out part one of this series, or take a look at part two for details on DOK training and professional development.

 1.       How have you seen DOK application evolve since the framework was first introduced?

 Chrystal: I was a teacher for 9 years and began using DOK several years ago. The schools I taught in encouraged teachers to use as high of a DOK level as possible. This changed over time as people were taught more about DOK. I think Dr. Webb’s DOK framework has infiltrated almost every aspect of education and is widely used today in both education products, state assessments, and classrooms across the country. There are a lot of misconceptions about DOK though, so trainings like [the WebbAlign training] can really help educators understand the framework.

 Shanti: When I was a classroom and gifted education teacher, I oftentimes referred to Bloom’s Taxonomy for developing differentiated content and tasks at varying levels of rigor. More specifically, I used “spiraled” Bloom’s – using lower level tasks to pre-assess, and then moving through tasks and content sequentially, from low to high rigor).  For me, Bloom’s served as an introduction to DOK. I learned about the differences between the two, like the idea that DOK is not an instructional approach like Bloom’s; instead it is the language we use to discuss standards. Also, DOK levels are not meant to be used in any specific order. DOK tasks should be dynamic; there is no greater value placed on a level 3 DOK task than there is on a level 1 task – they are both necessary for achieving mastery of a standard or topic.

 Sara: Dr. Norman Webb developed the DOK framework in 1997. It started as an evaluative tool, used solely in alignment studies, to help determine the degree of alignment between standards and assessments. Educators from all parts of the education system interacted with the DOK framework when contributing to alignment studies—and they found it to be a very useful and effective way to talk about content complexity. Today, school districts across the country are working to incorporate DOK into school practice, content developers and assessment developers are using DOK to inform their products, and, of course, it continues to be the language system used in alignment studies.

 2.       How do you incorporate DOK into instructional design?

 Chrystal: When we design our Technology Enhanced Items for Study Island Standards Mastery or Assessment products, we design each question to have a certain DOK level and provide a justification of why that question needs to be written to that DOK level. To bring a question to a certain DOK level, we decide what complexity is appropriate for the Common Core or state standard it is being written to. For instance, at a DOK 2, we make sure that the question does not simply rely on an algorithm or simple recall of information, but that it includes appropriate context and that the solution requires interpretation and reasoning skills.  Then, when the question is actually written, the writer has a good idea of the complexity that it needs to achieve.

 In the Assessments that we design for various states, we use the state’s recommendation of the percentages of problems that are DOK 1, 2 or 3. On the other hand, the Math Practice Activities in Study Island Standards Mastery were designed for classrooms use. These all were designed to a DOK 3 or 4 level. Overall, using DOK levels is a very important part of instructional design in our products because it drives the complexity of the final product.

 Shanti: When designing our Study Island Standards Mastery and Assessment products, we develop our products to meet state established criteria, which can include:

  • Percentage of item types (e.g., Technology Enhanced and SRMC)
  • Percentage of items that should be at a DOK 1, 2, or 3
  • The standards that should be addressed in the product: Reading, Writing, or English Language Arts

 We then make determinations about which item types and DOK levels best align to each standard, taking into account the complexity of the both the standard and the associated task.  For example, Language and Writing items related to on grade-level spelling, punctuation, conventions, and usage would be at a DOK 1 because the standard and task may demand nothing more than recall. More advanced Language, Reading, and Writing tasks tend to fall within the range of DOK 2-3, depending on the complexity of the task. For example, asking students to determine the structure of a text would be a DOK 2, while asking them to determine the author’s purpose for structuring the text in a certain way would be a DOK 3. 

 Sara: We are working in a standards-based education system. And the standards—whichever ones they may be—specify not only the content that is covered in K-12 education but also the complexity or “depth” at which students are expected to engage with the content. DOK informs instructional design because we can check that the instructional questions and formative and summative assessment items are targeting at least the level of complexity that is expressed in the target standard.

 3.       How does application of the DOK framework vary between technology companies like Edmentum and traditional educators?

 Chrystal: Using the DOK framework, we can assure that students using our products get a good distribution of complexity in the math that they are learning. Students can practice working with a topic at multiple DOK levels – some problems may require rote procedure, others may require strategic reasoning, and still others may require using more conceptual knowledge. Using Technology Enhanced Items in our Study Island Standards Mastery or Assessment products allows students to move freely between the DOK levels as they can graph functions, plot answers, type constructed responses, and complete many other kinds of tasks that might not be available in the traditional classroom. Many of our math products go hand in hand with students’ learning in the traditional math classroom, and educators can use our products as a springboard for implementation of DOK into their instruction. Using DOK levels in our Technology Enhanced items allows students to use a variety of different methods to solve a problem.

 Shanti: To ensure students engage with a wide range of standards-based tasks, at varying levels of complexity, our products are designed to include a variety of standards-based items and types, at various DOK levels.  Because of this, our products serve as engaging learning tools for students.

 Sara: Depth of Knowledge is a language system. Ideally, all stakeholders are using this language system in the same way. There are, however, some common misunderstandings about DOK and it is sometimes applied in ways that are not consistent with the original intent. Edmentum’s work with Dr. Webb and the WebbAlign team is an important step toward achieving the goal of applying the DOK framework consistently and with fidelity of implementation.

 4.       What is the value of incorporating DOK into the development of educational technology for classroom use?

 Chrystal: Using technology can help incorporate DOK in the classroom as the students can be delivered content easily, with many different modes of solving a problem. For example, compared to paper and pencil and calculator, our software can quickly deliver a quadratic equation and ask students to plot the zeros on a graph. On pencil and paper, the process may take 5 or 10 minutes, while on the computer or tablet, students may get to two or three more problems in that time frame. Students can practice concepts with virtual flashcards, read lessons, and practice more problems designed to all different DOK levels. A DOK 3 problem designed by a teacher may take half of a class period and a ton of planning, while our software provides a variety of DOK 3 level problems at their fingertips. Also, using Technology Enhanced Items allows students to order quantities, arrange a proof, plot on a number line, and many more things that would not be possible all in one class period in the traditional classroom.

 Shanti: Technology provides engaging, on-demand student tasks that teachers can use to determine if a learner is advanced, proficient, or in need of remediation in any given topic or standard.  Technology based products also have the capacity to respond to student needs, essentially providing on the spot differentiation of content or tasks. 

 5.       What are some of the challenges of incorporating DOK into edtech tools?

 Chrystal: In early development of edtech, many tools were designed at a DOK 1 level. The current challenge for our products, and I would guess many other products, is to make sure all levels of complexity are present. This starts at the top-level of design, and it is challenging to make sure we get a good variety of DOK 1, 2, 3 and even 4 into our products. With that said, I am very proud of our products and how we have incorporated DOK into them in the past several years.

 Shanti: I agree with Chrystal.  In addition, it is important to keep in mind that the depth of a task is what determines DOK, not the type of task.  For example, we may have a tendency to think a constructed response task is at a DOK 3 because it requires students to type a lengthy response to a given question.  However, what is most key to remember is that what we are asking students to write about and what type of response is considered an adequate response is what determines complexity of the task, not what we are asking them to do to complete the task.

 6.       How does application of DOK vary across different subject areas?

 Chrystal: At Edmentum, the application of DOK does vary slightly depending on the subject area. For instance, in science there aren’t as strict procedural rules as there are in math, so they must consider the process that the student will go through to answer the question when assigning a DOK level. Whereas in math, applying DOK comes down applying the DOK definitions specific for mathematics. For instance, giving a problem to a 9th grader to solve a linear equation with no real-world context would most always be a DOK 1. But in science, asking about the cell-division process depends on if the student had to classify or organize information or simply recall information.

 To help with this process, Dr. Webb provided valuable information specific to each subject area which helps fine-tune the different DOK levels that are assigned to problems. Although there are differences in the way we may apply DOK in problems in the different subject areas at Edmentum, we apply the same overall DOK ideas into the design of our products.

 Shanti: I agree with Chrystal. Thanks to the training we received from Dr. Webb, as well as internal trainings and application of DOK, Edmentum designers apply DOK similarly across subject areas, creating more consistency than variance. For ELA and Social Studies, the consistency and complexity of standards may be the biggest difference, which in turn leads to nominal differences between the percentages of DOK 1-4 items developed for our respective products.

 Sara: Four DOK levels have sufficed to describe content complexity across all content areas and there are strong similarities between and among the different disciplines. But there are different DOK definitions, specific to each discipline; it is not one-size-fits-all. It can’t be one-size-fits-all because there are inherent differences in the different disciplines. We don’t approach the teaching and learning of mathematics the same way that we approach the teaching and learning of reading (for example). What a DOK 2 “looks like” in mathematics is very different from what it “looks like” in ELA (and the same goes for all DOK levels and for other subjects, of course).

 7.       What was your biggest takeaway from the training with WebbAlign?                                                                                                                                                                             

Chrystal: In the WebbAlign training, I obtained clarity on the differences between the DOK levels and refined my perspective of DOK and complexity.  We worked in small groups going through items specific to our subject area. I was very fortunate to be in the group with Dr. Webb, which was really neat. Hearing his perspective on specific math questions and the reasons why they were a certain DOK level, I refined my own skills for determining the complexity of a question.

 Another thing that I thought was particularly interesting, was having a lower DOK level for a question wasn’t necessarily a bad thing. Dr. Webb discussed why there needs to be a distribution of the different DOK levels in our product and backed this up with a sound rationale.

 Shanti: Some of my biggest takeaways include:

  • DOK is based on the “typical” student – while at times we may think about how complex a task would be for students who are either advanced or remedial, when determining DOK, it is essential to base decisions on how complex the task would be for the typical student, since standards are based on grade-level expectations (not above or below).
  • DOK is more like categories than levels, which supports the idea that one level is not “better” or more valuable than another level – students need to engage with all levels of complexity.
  • DOK levels for items can be dynamic, meaning that they do not need to be addressed in a specific order, within one (multi-part) or a series of items.

 8.       What was something unexpected that you learned during the training?

Chrystal: Dr. Webb stressed the importance of using valid resources for DOK definitions. These were provided to us and included the detailed DOK definitions and specifics for the different subject areas. In the training, we got a very in-depth look at the DOK definitions for each individual subject area. 

Shanti: The difference between complexity and difficulty was unexpected to me.  Also, it was interesting to learn that DOK levels should not be determined based on the verb in the question. 

Sara: We often see these sort of “aha moments” when we’re working with people to help them strengthen their understanding of DOK—it’s great!

 9.       How is the Item Review conducted? What is the value of this review? 

Chrystal: For math and ELA, we selected items for WebbAlign, and they evaluated them. We got a fantastic report back highlighting the DOK levels for each item. They also provided feedback regarding standards alignment item quality.  At the same time, we also conducted a fresh internal review of these items for comparative analysis. This was a very valuable exercise as it encouraged discussion and justifications. 

Sara: Each set of items was reviewed by a team of three educators with content expertise and with extensive experience with DOK. The goal of the review was to provide feedback about the extent of agreement on DOK assignment. In other words, to what extent did the external review team agree with the DOK that the Edmentum team assigned to each item? Thorough feedback was provided for any disagreements. This process helps a team to synch and to calibrate their interpretation of DOK. The external review helped the Edmentum team advance their goals of consistent and accurate application of the DOK framework.

 10.   What advice do you have to continue DOK learning and training after the professional development session has concluded? 

Chrystal: For me, the WebbAlign training really sparked my interest in learning more about DOK. I strongly suggest reading and using the materials provided in the training to assist you whether you design education software or are a classroom teacher.  We took the DOK definitions and parsed them up so we could use them in the development of our products. We also took each and every Common Core standard and assigned a DOK level to each one. We had follow-up trainings to discuss these within our elementary, middle school and high school math teams and came to a consensus. I suggest having follow-up trainings, and implementing new procedures to incorporate DOK into your organization.

Shanti: In ELA, after the training with Dr. Webb, we worked with colleagues in grade band teams (Elementary School, Middle School, High School) to determine a DOK level for each standard within the band.  We worked independently in advance of group discussions to determine DOK for each CCSS standard then shared our thinking within groups to arrive at consensus.  This was a great follow-up, because we really dug into the resources provided at the training to make DOK determinations, support justifications, and build on one another’s ideas to gain a deeper understanding of DOK, including specific aspects of item writing that are key when trying to attain a certain DOK level; for example, the importance of strong distractors.  

Sara: Because DOK is a language system, it really requires putting time into the discussions, listening to each other’s ideas, perspectives, and expertise. Continuing the conversations, working to come to consensus about DOK assignments for materials, always using the definitions as arbiter—all of these help individuals become more familiar with the definitions and help a team hone a common understanding of the different levels. 

 11.   What advice would you give an educator just beginning to explore using DOK in their classroom? 

Chrystal: Use the resources that Dr. Webb has available and read all you can on the subject. Try to incorporate all levels of DOK in your classroom, and fine-tune your activities and lessons to work through the different levels DOK. 

Shanti: I agree with Chrystal about reading the resources created by Dr. Webb, and would add that it is essential to work with colleagues to collaborate, brainstorm, and discuss ideas for incorporating a variety of DOK levels into instruction, as well as formative and summative assessments.                                                                                                                                              

For us at Edmentum, our partnership with the WebbAlign team has been an outstanding professional development experience. We believe in the effectiveness of the DOK framework, and have loved the opportunity to deepen our understanding of it. In turn, we’re excited to be able to provide our customers with online teaching and learning solutions that are as closely aligned to state and Common Core standards as possible. We’re looking forward to continuing our partnership with WebbAlign and passing along new insights into depth of knowledge, assessment, and edtech! 

To learn more about the WebbAlign team and DOK framework, check out their professional development programs. Or, for more information on Edmentum’s built-to-standards online assessment solutions watch this video