3D printing makes virtual world more real for blind learners

Michael A. Kolitsky

Introduction

Transition of the real world into the virtual world has been ongoing in online and on-ground classrooms and labs for many years. But the more the world goes virtual, the more difficult it becomes for those who are blind or visually impaired to participate fully in the learning process. The main thrust of this report is to explore how 3D printing and the use of conductive filament can begin to reverse that process by making 3D printed talking tactile learning objects that can generate audio when touched after being placed on the surface of an iPad or other tablet computer. Examples of talking tactile learning objects from STEAM (Science, Technology, Engineering, Arts and Math) disciplines will be presented with a focus on how the design of 3D printed STEAM learning objects can lead to a common learning object template model. The template can be designed to serve as a learning tool not only for a blind or visually impaired student but also can work as well for a sighted student or a teacher, aide or parent demonstrating how universal design principles can drive the learning object production process.

Virtual reality, video and complex images all contribute to making access to the virtual digital world more difficult for students who are blind or visually impaired. Efforts to make virtual reality more accessible by using audio are ongoing as shown in the use of 3D printed touch-responsive maps that can vocalize information about buildings and directions¹. Another earlier attempt² was designed to permit an audio-augmented experience for exploration of virtual environments that simulate exploring a museum. The use of audio has also been emphasized in the design of AudioMath³, a virtual environment designed for blind children to learn mathematical concepts. Audio-enhanced learning was also utilized in a museum exhibit at Pine Knoll Shores Aquarium (Winston-Salem, North Carolina) titled Fintastic! Exhibit and was the first of its kind museum exhibit to display tactile talking fish models so that blind or low vision visitors could begin to experience by touch what sighted visitors see⁴. A white paper⁵ by RAF Models reported research on the Fintastic! Exhibit showing learning was dependent upon a tight link between the audio descriptions of fish models and the shape, form and textures on those models. In this report, 3D printed audio-enhanced tactile learning objects were made using conductive filament so that a 3D print could then be laid on top of an iPad or other tablet computer for audio production when the 3D print was touched by the learner's finger. Audio inclusion in 3D printed tactile learning objects by using commonly available iPads or tablets expands options for designing new ways to learn about virtual worlds especially for students who are blind or visually impaired.

Making tactile learning objects talk

For a 3D print to become a learning object, it must be integrated into a stand-alone, reusable chunk of learning that traditionally was also in digital format as shown by the many thousands of learning objects now available for downloading from repositories like Merlot⁶. In order to make a 3D printed talking tactile learning object, the combination of a digital component and a real 3D printed object had to be constructed. The real portion was the 3D print and the digital component required for the audio response was provided by an iPad for this report but other computers such as the Microsoft Surface or Android tablets were tested and could work as well. Conductive 3D print filament from BlackMagic3D⁷ containing graphene was also needed to carry the electrical charge from a fingertip touch on the 3D print to the iPad surface so that some areas on the 3D print that were made for touch sensitivity could transfer a strong enough fingertip electrical charge to change the capacitance of the iPad screen on which the 3D print was placed. The final piece needed to make a functional talking tactile learning object centered on an HTML5 3D graphics app named „Hype 3”⁸ by Tumult which permitted the design of iPad audio buttons that could be positioned directly underneath the conductive filament so that a fingertip touch would now cause the iPad button to generate an audio response.

Hype 3 permits the iPad screen graphic to closely replicate the 3D print so that it can also serve as a learning tool by sighted students without the 3D print meeting the goals of universal design pedagogy⁹. Although Hype 3 is designed for the Mac OS X 10.8+ operating system, the produced HTML5 content is web deliverable and runs on all modern browsers and tablet computers.

A Makerbot Replicator 2¹⁰ was used to make a 3D printed „holder” designed to hold the iPad in place so that a plastic sheet with 3D printed objects made from conductive filament attached to it could be placed on the iPad screen surface and held so that areas of conductive filament were located directly above the software audio buttons on the iPad. Some software adjustments in the HTML5 code of Hype 3 were also made to inhibit screen movement on the iPad so that audio buttons were always in a location directly underneath the conductive filament portion of the 3D print. This combination of thin plastic sheet cut to fit in the holder along with the 3D printed material that was attached to the top of the thin plastic sheet was called a template. The thin plastic sheet was cut from file folders and did not inhibit touch activated iPad audio production.

Complex image templates

The author¹¹ and others¹² have begun to show that 3D printers can make 3D prints of microscope slides, telescope photos and earth-facing satellite images that can be used as a starting point for making tactile learning objects. 3D printing is also changing the way teachers teach and students learn across many disciplines¹³. One program valuable for making 3D print stl files for 3D print production is „PhotoToMesh”¹⁴ which can make 3D prints that are in „bas relief” form with one side flat and the other side designed for providing a tactile function. In the template below in Figure 1, a bar graph comparing use of practice quizzes in an Anatomy and Physiology course was used as an example of a complex graphic. The stl file produced by PhotoToMesh was designed so that one data set had bars raised above the background level to a different level than the other comparable data set and also the data sets were designed to be different by making them with different patterns able to be discerned as different when touched. The upper region of each bar graph was made with conductive filament making it capable of carrying an electrical charge from a fingertip touch that can then cause the audio buttons on the iPad to generate audio, i.e., „talk” as in talking tactile learning object. An audio button made with Hype 3 was placed on the iPad screen just beneath the 3D printed conductive filament so that audio is produced when a finger touches the top of the bar graph. The audio presents information about that bar graph component such as its xy location, the percentage value on the vertical axis and what grade category it represents. This is also an example of a way of making a complex image more discernable via touch and audio delivery for the blind learner as well as for a sighted student fitting what is expected for designing by universal design standards.

The left side of Figure 1 below shows an iPad with a complex graphic image of a bar graph comparing the percentage of students receiving A, B, C, D and F grade when credit was given or not given for doing practice exams¹⁵. The 3D printed bar graph was designed for an invited presentation made on the main campus of Penn State University in the fall of 2015 to demonstrate the way that conductive filament could be added for an audio-enhanced tactile experience. The right side of Figure 1 shows the iPad now with a 3D printed complex graphic template on a thin blue plastic sheet added to the iPad surface being held in place by a 3D printed plastic holder. For a more complete learning experience, there would have to be additional buttons made from conductive filament such as the star shaped symbol in the upper left of the template so that information about what the graph represents could be delivered when that conductive button was touched. Also note the 3D printed triangle below the 3D printed bar graph which when touched provides an audio description of the results shown on the 3D printed bar graph. The dark rectangular areas at the top of each bar are composed of conductive filament and when touched, provide an audio response giving the y axis value in percent of students receiving a particular grade as well as whether the bar represents students who received or did not receive credit for doing the practice quizzes.

Example of video template based on NASA moon transit of Earth

During the summer of 2015, NASA published some spectacular videos showing the moon making a transit across the surface of the Earth¹⁶. The videos were made from images taken by the EPIC camera on the DSCOVER satellite parked in stationary orbit one million miles from Earth. The unique nature of this video was that the moon was being viewed from the dark side as it was making its transit and this side of the moon is never seen from the Earth. Can a blind or visually impaired viewer sense the excitement of this video when the video cannot be seen? One approach has been to make 3D prints of the moon in various stages of transit and attach these isolated 3D prints to the plastic template with audio buttons underneath on the iPad surface so that by finger touch, a blind student can sense and also hear reinforcement of what is being touched by a corresponding audio response. Discussion with a NASA engineer¹⁷ also indicated that a template could be designed to measure the speed of the moon in its orbit using the diameter of the Earth and the moon in the calculations that would add a real world math component to the moon transit template that appears below in Figure 2.

Left portion of Figure 2 below shows the iPad graphic and partial view of a 3D print on a clear thin plastic overlay. Right side of Figure 2 below shows 3D print on surface of iPad.

By touching the dark circles on right image, an audio response is generated indicating that the moon is in a particular phase of moon transit. The rounded star on upper left provides audio response describing what the template is showing and the small triangles below the three transit phases provide an audio response indicating what stage the transit is in as it passes the Earth's surface.

Virtual world template

Several decades ago, the national Library of Medicine completed the Visible Human Project¹⁸ in which a male cadaver was frozen and sectioned to produce over 1800 transverse sections that have been used by many universities and publishers to make highly interactive virtual reconstructions for analysis using imaging algorithms. But, again, how do blind students wishing to learn anatomy utilize these virtual resources? One recent example was demonstrated by the author at the regional meeting of the American Association of Anatomists Conference in London, Ontario in which a poster presentation was made showing that 3D prints of a transverse section of the head that had conductive filament embedded in the 3D print at locations of the right eye, optic nerve and optic chiasma could now generate an audio response when placed on an iPad and touched in the region of the conductive filament¹⁹. A testing component was also developed for this template so that incorrect answers generated a wrong answer response and correct answers were confirmed with an audio response.

Left side in Figure 3 below shows iPad graphic and partial view of 3D print on clear plastic overlay. Right portion of Figure 3 below shows 3D print template on surface of iPad. By touching the dark conductive filament on the complete template, an audio response is generated indicating what structure was being touched.

Reading poetry in 3D space template

An article published in the English Journal²⁰ showed how four Haiku with the same middle line and written by four different students could now be configured in 3D space so that the four Haiku could arrange themselves in a virtual cube of poetry with first and third lines of each Haiku now positioned along the exterior of a poetic cube. Sighted students in an upper division creative writing class exhibited an „ah ha” moment when for the first time the edge lines on the poetic cube formed from the first and third lines of the four Haiku could now be read to produce poems which were not able to be seen until arranged into the 3D virtual poetic cube. Figure 4 below on the left shows one view of a 3D Haiku cube and the image below on the right shows the same 3D Haiku cube rotated to the right further clarifying the position of the nine lines composing this 3D Haiku cube. The Haiku lines represented in the 3D poetic cube are listed below the 3D Haiku cube rotation views. Note the value of being able to arrange the four Haiku into a poetic cube since it is only when in 3D cube arrangement that the exterior end lines can be read as poems.

Figure 5 below on the left shows the iPad with a Hype 3 graphic based on viewing a 3D Haiku cube from the side where the nine Haiku lines would each begin. Touching a white rectangle generates audio indicating which line is represented in the 3D Haiku cube. For example, touching the upper right white rectangle would cause the audio button to respond „Haiku 1 Line 1” and touching the middle white rectangle would cause the audio button to respond „Haiku 1 Line 2”. When the white button on the lower left is touched, the audio button would respond „Haiku 1 Line 3”. The image on the right shows the Haiku reading template added on top of the iPad surface. The rectangular black buttons made with conductive filament are located directly over the white audio buttons on the iPad graphic. With a new Hype 3 page loaded onto the iPad, touching the rectangular black buttons composed of conductive filament now cause the buttons on the iPad surface to generate the Haiku line related to the position of the button in the poetic cube. For example, touching now the button in the upper right corner of the 3D Haiku cube would generate the line „Wind begats waves white caps” and touching the middle button would generate „Dreams smith beachwalk hand-in-hand”.

Value of templates in universal design

For all the templates described in this paper, the image on the iPad screen is an exact duplicate of the 3D printed template meaning that a sighted person could also utilize the learning object but without the 3D printed template. This is of importance for teachers, support staff and parents who may be assisting a blind student in the learning process and also for sighted students who are learning in the same classroom. When a learning tool can be used for learning by both sighted as well as blind or visually impaired students, one of the basic requirements has been met for applying universal design principles to the development of talking tactile learning objects.

Availability of audio-enhanced 3D printed templates

The templates described in this article can serve as examples of how images from complex graphics, videos and virtual worlds can be made into audio-enhanced tactile learning objects using entry level 3D printing skills and some knowledge of design software. Many academic institutions are now forming maker communities to assist faculty in the production of audio-enhanced 3D printed learning objects. A critical component though in the future application of audio-enhanced 3D printed learning objects is testing their effectiveness in the learning process. In this regard, the author is interested in developing collaborative opportunities that would involve the sharing of 3D print files for production of learning objects such as now exist at the NIH 3D Print Exchange²¹ and where the author has listed several 3D print stl files for the study of mitosis²². All 3D prints in this paper and ways to make audio-enhanced tactile learning objects are available without cost to colleagues for testing with the goal of sharing data for the improvement of both the design process and its impact on student learning.

References

• A 3-D talking map for the blind (and everyone else), ScienceDaily, 2014, http://www.sciencedaily.com/releases/2014/11/141119101654.htm.
• Fuller R., Watkins W.R., Exhibit Design Relating to Low Vision and Blindness (White paper), Indiana University, Bloomington 2010, http://www.ncaonline.org/docs/tactile_exhibits-fuller_watkins.pdf.
• Hasper E., et al, Methods for Creating and Evaluating3D Tactile Images To Teach STEM Courses to the Visually Impaired, „Journal of Science Teacher Education” 2014, Vol. 4, No. 6, pp. 92-99.
• Kolitsky M., 3D printing makes Visible Human cadaver sections accessible for blind students, http://www.nextgenemedia.com/NLM3Dprints/NLM3Dprints.html.
• Kolitsky M.A., 3D Printed Tactile Learning Objects: Proof of Concept, „Journal of Blindness Innovation and Research” 2014, http://dx.doi.org/10.5241/4-51.
• Kolitsky M.A., Reshaping teaching and learning with 3D printing technologies, „e-mentor” 2014, pp. 84-94, http://dx.doi.org/10.15219/em56.1130.
• Sanchez J., Flores H., AudioMath: Blind Children Learning Mathematics Through Audio, „International Journal on Disability and Human Development” 2005, Vol. 4, No. 4, pp. 311-316, http://dx.doi.org/10.1515/IJDHD.2005.4.4.311.
• Tweedie S., Kolitsky M.A., 3-D Haiku: A New Way to Teach a Traditional Form, „English Journal” 2002, Vol. 91, No. 3, pp. 84-88.
• Zimmermann A., Lorenz A., LISTEN: A User-adapted Audio-augmented Museum Guide, „The Journal of Personalized Research” 2008, Vol. 18, No. 5, pp. 389-416, http://dx.doi.org/10.1007/s11257-008-9049-x.