NASA
sponsors independent peer reviews of its K-12 Earth and space science education
materials. The purpose of the reviews is to ensure that education products
distributed by NASA are of high quality and meet rigorous standards, as well as
to provide feedback to product developers from educators and scientists. The
reviews are conducted for NASA by the Institute for Global Environmental
Strategies (IGES) - a non-profit education organization.
Education products are reviewed by
panels including both scientists and educators (e.g., classroom teachers,
college faculty, NASA teacher trainers, curriculum developers, and national
education standards experts). This peer review provides NASA with information
regarding the effectiveness of specific products and recommendations for
distribution.
IGES has developed the following
guidelines to help education product developers design outstanding education
materials for K-12 education audiences.
·
General
·
Electronic Resources
·
Pilot Testing
·
Standards
·
Reading Level
·
Education Level
·
Call for Products
·
Product Submission Form
Communications
Materials
Review (CMR)................................... 9
The following is a list of some of the most common areas in which products that don’t pass the review have needed improvement. While this list is not exhaustive, we strongly believe that if you address all of these items, then you will have developed an outstanding NASA education product.
All education products should go through an appropriate formative review and testing before they are submitted to the NASA education products review. What is a reasonable review? It’s one in which the product is reviewed by appropriate science content experts, as well as field tested and/or reviewed with the target audience (or number of audiences if the product is aimed at a broad range of grade levels or audiences). Any shortcomings that are found should be corrected.
Target a Specific
Audience
Products that are developed for a very broad audience, tend to not serve any specific audience well. Often a product will be identified as K-12. Unless that product is actually broken into sections that are specifically designed for a smaller grade bands (e.g., K-1, 2-4, 5-6, 7-8, 9-10, 11-12), it is not likely to be useful to that wide an education range. Even products that are identified as appropriate for K-4 are attempting to serve students with very different capabilities and interests.
Design Products for
the Targeted Audience
We see many products that are not appropriate for the targeted audience. They are written at too high (or low) a level, the concepts are too complex (or simple), or the graphics aren’t age appropriate for the target audience. One common mistake is to pair text that is written at a higher level, with graphics that would appeal to younger children. Another problem often identified by reviewers is an inconsistent reading level. Individual paragraphs, pages or sections show a marked increase or decrease in difficulty. Watch for consistency throughout the product.
Support National
Education Standards
Identify the primary national education standards (science,
mathematics, technology, and/or geography) that the product addresses. Be selective and specific about what
standards are supported and identify how the product helps a teacher address
those standards. Rarely does a product
address several standards – list only those standards that the product supports
in a meaningful, and not superficial, way.
Incorporate Student-Centered Approach
While
some teachers will take NASA content and incorporate it into an investigative,
student-centered teaching/learning style, developers should build this kind of
strategy in from the start. Many still use multiple choice and “cookbook” style
activities; to do well in the review, developers need to incorporate activities
that provide inquiry-based, problem-solving opportunities.
Include as many
relevant, age appropriate, and engaging activities as you can in your product.
How can a teacher using this education product evaluate student performance? Does the teachers guide provide sample rubrics, grading/scoring guidelines, answer keys for specific questions, or other appropriate assessments/resources?
References and
Further Investigations
Provide accurate, accessible and age appropriate references for outside investigation. This can include books, magazine articles, journal publications, Web sites, etc. (see also “Maintain a Web” below). Exercises, labs, and slightly more advanced investigations are a plus for students willing to go further with the material.
Most teachers and students are not familiar with NASA imagery. Be sure to provide the tools that are essential for interpreting images. At a minimum, provide color keys/scales, legends, captions, and identify the units of measurement that are used. (See “Go Metric,” below)
Maintain a Web, not
Cobweb, Site
Regularly check and update WWW links. One solution is to host all critical WWW pages (so that you can ensure the links are kept active) or obtain permission to mirror critical sites.
Go Metric
Metric measurements are the agency standard and should be consistently used. Including English equivalents is also very good, as well as other comparisons that would be meaningful to the target audience (e.g., 1.35 mm or as thick as one dime). Metric units should be given first, with equivalent measurements in parentheses, e.g., 10 km (6.2 miles).
Check your Math
Does your product include math problems (or metric equivalents/conversions)? Be sure to check your math and make sure your answers and conversions are correct.
NASA electronic and information technology-based education
products (e.g., WWW, DVD, CD, videotape) must meet Section 508 requirements.
Section 508 of the Rehabilitation Act requires that Federal agencies'
electronic and information technology is accessible to people with
disabilities, including employees and members of the public. This includes WWW
sites and PDF files, as well as providing closed captioning and audio for
videotapes, DVDs, and CDs. For information on Section
508, visit http://www.section508.gov/.
For a checklist for making a site accessible, visit: http://web508.gsfc.nasa.gov/developing/checklist/index.cfm.
NASA
communications products, which include education products, should follow the
design guidelines presented in Visual
Standards for NASA Communications Products, which is available at http://communications.hq.nasa.gov. This site is only accessible within
NASA.
A copy of the guidelines is also available for those outside NASA at: http://www.strategies.org/SMD_CommunicationsReview/interim_style_guide.pdf. While this document is noted as an “interim”
guide, the final guidelines are not expected to differ from this document, but
to provide additional details that clarify information in the interim guide.
Prior
to Developing a NASA Education Product
·
Conduct Market Research.
Talk to educators and identify whether a need exists for your product. Consult
the NASA Space Science Resource Directory at http://teachspacescience.stsci.edu/ to ensure a comparable NASA product doesn't already exist.
·
Review Visual Standards for NASA Communications
Products
NASA communications products, which
include education products, should follow Visual Standards for NASA
Communications Products, which is available at http://communications.hq.nasa.gov. This site is only accessible within
NASA. A copy of the guidelines is also
available for those outside NASA at: http://www.strategies.org/SMD_CommunicationsReview/interim_style_guide.pdf. While this document is noted as an “interim”
guide, the final guidelines are not expected to differ from this document, but
to provide additional details that clarify information in the interim guide.
Prior
to the Review
General
·
Proofread and edit for
typographic and grammatical errors.
·
Provide all measurements in
metric format (e.g., Celsius, grams, liters, meters), if equivalent
measurements are included, provide in parentheses, for example: 10 km (6.2
miles). Also, providing familiar examples is also useful (e.g., 1.35 mm or as
think as one dime).
·
Check your math for all
formulas, problems, and conversions to/from metric.
·
Clearly label graphs, charts,
and images, including information for the reader on units of measurements that
are used. Color keys and legends (e.g., for imagery) are essential.
·
Include tools in your product
to help teachers and students unfamiliar with the content, such as a glossary
of terms, references for additional information, etc. Teachers' guides and
answer keys are also very valuable tools to teachers.
·
Ensure that answer keys, if
included on a Web site or CD-ROM, are password protected and that the password
is easily accessible to the teacher, and not students! For example, do not
print the password in the student materials.
·
Design your materials to be
“classroom ready” by teachers. For
example, provide blank student worksheets that are ready to use or student
editions.
·
Materials printed by NASA
must be free from copyright restrictions. Ensure that there are no copyright
restrictions or obtain permission for NASA use.
·
Design posters so that
information on the backside can be easily photocopied by teachers. This can be
done by designing information in 8 1/2" x 11" panels that will fit on
a photocopier.
Logos/Insignias
The
only permissible NASA logo to use is the official NASA "meatball"
insignia. Education products printed by NASA should include the NASA
"meatball" insignia, as well as the NASA publication number and
education product reference box. Guidelines for proper use of the NASA logo can
be found in the in Visual
Standards for NASA Communications Products, which is available at http://communications.hq.nasa.gov. This site is only accessible within
NASA. A copy of the guidelines is also
available for those outside NASA at: http://www.strategies.org/SMD_CommunicationsReview/interim_style_guide.pdf.
Guidelines for the education publication number and
education product reference box are included in the NASA Curriculum Support Product Development and Dissemination Guide,
which is available from NASA Center Education Offices.
Electronic Resources
(Web,
Video, CD, DVD, Software)
·
Test Accessibility by Disabled Users. Section 508 of the Rehabilitation Act requires that
Federal agencies' electronic and information technology is accessible to people
with disabilities, including employees and members of the public. This includes
WWW sites and PDF files, as well as closed captioning of video (on videotapes,
Web, DVD, and CD). See the following URLs for more information:
http://section508.nasa.gov/ - NASA Website for section 508. The site includes an overview of section 508 requirements and
contact information for Section 508 coordinators at NASA HQ and field centers.
Note: some sections are restricted to NASA internal use only.
http://serch.cofc.edu/special/overview.htm - Website by the NASA Southeast Regional Clearinghouse
(SERCH), which provides excellent resources regarding accessibility and Section
508, including information on best practices and designing accessible education
resources.
http://webxact.watchfire.com/ - WebXACT is a free
online service that lets you test single pages of web content for quality, accessibility, and privacy
issues.
·
Make sure all links are up-to-date. Web sites submitted to the review that are maintained
with current content and links (no broken or dead links) generally receive a
higher rating.
·
Test software on multiple platforms.
·
Produce software in a dual platform versions for use with both Macintosh and Windows computers. Use the
ISO 9660 international standard for defining CD-ROM file systems.
Pilot Testing
·
Education Review. Have
your product pilot tested by the target audience. Pilot testing activities in
the classroom is also essential.
·
Education Product Criteria. The criteria used for the space science education product
review are available at http://www.strategies.org/SpaceScienceReview.html. Developers are strongly encouraged to use these criteria
to assess their product.
Pedagogy
Most
products submitted to the OSS review contain very good resources and great
content. However, most still do not incorporate learning styles that are in
tune with the national education standards and systemic change efforts. While
some teachers will take this content and incorporate the material into an
investigative, student-centered teaching/learning style, developers should
build this kind of strategy into their education products. Many still use
multiple choice and programmed learning styles; to do well in the review,
developers need to incorporate activities that provide intuitive
problem-solving opportunities.
Standards
Curriculum materials should be
well-aligned with appropriate national education standards, such as science,
mathematics, technology, and/or geography. Other standards should be
identified as appropriate (e.g., English language arts – reading, writing, and
speaking). Content should match the standards and
specific outcomes that are identified.
National standards can be found
online at:
·
National Science Education
Standards, http://www.nap.edu/readingroom/books/nses/
·
Principles and Standards for
School Mathematics, http://standards.nctm.org/
·
Standards for Technological
Literacy, http://www.iteawww.org/TAA/TAA.html
·
Geography for Life,
http://www.nationalgeographic.com/education/standards.html
In general, it is expected that NASA education materials will support national standards; in some cases it may be appropriate to identify relevant state standards, either in addition to or in place of national standards.
Reading Level
Product
developers should ensure that the reading level of their product matches that
of their intended audience. This is especially true for products targeted for
elementary students – many K-4 materials that are submitted for review are too
sophisticated for the target audience.
There
are guidelines for assessing and validating the reading levels of products.
Many word processing applications include readability formulas. For example,
Microsoft includes the Flesch Reading Ease Score and Flesch-Kincaid Grade Level
Score in their tools. Under the “Tools” menu, select “Spelling and Grammar” and
then “Options.” You can then select
“Show readability statistics.” After
running the spell checker, the reading level will be displayed with the
results.
The Fog
Index is another method for analyzing written materials by computer or by hand
and is used primarily in education. If your grammar checker doesn't include the
reading level, let it count the number of sentences and words and plug them
into the Fog formula below:
·
Select a sample
·
Determine the average number
of words per sentence
·
Determine the percentage of
hard words
· Add the 2 factors and multiply by 0.4
· The result is an indication of the minimum grade level at which the writing is easily read.
Caveat:
Please be aware that these formulas are based on measuring words and sentences,
but they cannot measure indicators such as how complex the ideas are, whether
the content is in a logical order, and if the vocabulary is appropriate and the
writing engaging for the intended audience. Slavishly adhering to these
formulas, for example to produce text for younger readers, can result in a
monotonous writing style consisting of several short sentences.
Therefore,
these formulas should be used as a general guide and reading level/suitability
should also be determined in combination with other assessments. For example,
test student comprehension and engagement and use experts, such as teachers and
librarians, to review material and assign an appropriate grade level to the
text.
Education Level
Product
developers targeting a broad education range need to be aware of the
developmental differences in that range. This is especially true for products
targeted for K-4. For example:
·
K-1 can’t read three-place
value numbers, so one- and two-digit data need to be used.
·
Angles have no meaning to K-2
and some 3rd grade students, especially expressed in degrees. Visuals of angles
need to be used.
·
Developmentally, K-1 and most
2nd graders can compare two sets of data at a time. Activities/problems working
with three sets of data are too complex for this level.
Entering
the Earth and Space Science Education Product Review
Call for Products and Schedule—the latest schedule, guidelines, product submission form,
and review criteria are available here.
Product Submission Form. Submit your product with a copy of the Product Submission
Form, which will be available with the call for products. Send 12 copies or
sets of the best-quality print available. Some materials that are expensive to
produce (e.g., color posters) can be submitted on a Web site for reviewers to
assess.
Reviewers
receive a copy of the product submission form for every product that they
review. This form collects information
that our reviewers have consistently found to be very useful in their
assessments. Please provide as complete
information as possible. Developers are
also encouraged to include any additional information that they believe will
help reviewers evaluate the product.
For example:
·
How could a teacher use this
product (for example, to introduce a topic, as an extension or conclusion to a
unit)?
·
Are there other products that
this material should/could be used with?
· Is training or other support available for teachers using the product?
Communications
Materials Review
Education products that pass the SMD review will then need to pass the NASA HQ Communications Materials Review (CMR) to be printed and/or distributed by NASA. The CMR takes approximately 10 days, and it is a requirement that before entering the communications review, education products must have passed the SMD education product review. Products are entered into the CMR through the Website: http://communications.nasa.gov.
Theresa
Schwerin
theresa_schwerin@strategies.org
Associate
Director, Education Programs
Institute
for Global Environmental Strategies
October
2005, updated April 2007