Instructional Design for Mediated Education feed

Last Post

Mon, 02 Jun 2014 00:00:00 -0500

The office hosting this blog is retiring some of its servers, including the one on which this blog resides. By mutual agreement, this will be the last post on the IDOS blog. It's been a good run.

Creating Accurate Surprise Eventgraphs (as Expressed on Social Media Platforms)

Sun, 01 Jun 2014 06:00:00 -0500

One way to create motivation to explore a topic more deeply is to make a public commitment to address a topic. Where this is high-risk is if one does not actually make the time to learn the topic well and there are people in the audience who are totally willing to call you on it (and there are always those).

If it were not for a double-booked conference, I would most certainly be having a discussion with an audience about how to accurately create “surprise eventgraphs” or social network graphs of surprise (unexpected) events…using low-cost (free) software tools.

As it stands, I have only just started looking at this challenge and finding that the research literature on event graphs is about a whole other thing—the (archaic) visual mapping of machine-based processes for awareness and analyses. What’s worse is that I know there has to already be designed answers to the question I’m asking…minus the “using low-cost (free) software tools”.

The Challenge

If I am to be totally honest, this is a challenge that has already been solved by others. The way such events are tracked is that there are tripwires in the “human sensor network” (created based on the cobbling of various social media platforms). These tripwires are certain terms that are being watched for. There are bursts of activities that are identified (geolocationally)… There are busts of activities in certain known networks…such as those surrounding particular “gurus” and other thought leaders. There is tracking for event announcements (meet where and when), particularly around particular locales with relevance for people culturally and politically. There are observations of gatherings. In other words, if there is an event, those who need to know will know with some lead time. Such events are studied and graphed in a variety of ways. How people express themselves individually and collectively is fairly widely known.

A more basic challenge is this. When is an unfolding or surprise discrete event deemed to have started? (The assumption of an “event” is that it has some relevance to people.) What is its trajectory? Will it have one peak or multiple peaks? How will it resolve? Who will be involved, and why? What will the various people contribute to an event? How would one know if the event is part of a larger series of events? How may instigators of events be identified? Then, based on the electronic communications element, who are the main communicators? What are the roles they’re playing? How much of an overflow is there between messaging and the event? (What is the messaging is private and hidden? What are ways to identify the messaging, the social networks, and the animating messages?)

The Non-Trivial Part

So where the challenging part is for this approach is what can be done on the cheap in terms of mapping such events? Essentially, what can the general public know by engaging in a range of data extractions across a range of tools in real time somewhat beyond what the formal media is reporting? Also, what conclusions may actually be drawn, and what advantages may be attained in knowing both early and then in real time…and then in the aftermath (and retroactively). Because there is not currently one tool or site that may enable this (no all-in-one system), one has to cobble together a range of tools and chain them into some sort of logical sequence to capture information…where time is of the essence. (The problem with manually chaining is that it is time-intensive and therefore highly limited. This method will miss a lot of data. Optimally, there would be continuous information collection through an event…and the packaging of the data into something coherent. That said, if continuous is not an option, then it helps to have critical snapshots, with some sufficient analysis that enables the filling in of gaps. This will look like a kind of un-smooth time-lapse photography.)

Pre-positioning

In a way, I think just looking at technology for this issue would be limiting. It would help to pre-position the self, based on potential would-be leaders of a particular event to be on the receiving end of any animating messages (assuming an event sparks that way—but the world is clearly complex enough that any number of unplanned events can be the spark for an out-of-control or fast-moving event, and any individual can become a leader or facilitator of some event). The best pre-positioning would be to have ties to all known leaders of a potential movement and to know that movement so well that one can tell when certain events in the world would potentially spark mass action. It would help to know the mindset and the messaging which could spark behavior. (By an optimal pre-position, I would suggest closeness to the potential leaders sufficiently so that one is included on private chats, not just the public messaging.) Ideally, a pre-positioned individual would know a field so deeply that he or she is an insider…and even knows the unspoken rules and the social codes of those relevant groups.

Another form of pre-positioning involves knowledge—the knowledge of history…the knowledge of societies…the knowledge of people. It helps to know the worldviews of the target groups and to know what might spark them to action.

Sudden Eventgraphing…even as a Laggard

But let’s say that there is an event that is sparked beyond the control of individuals per se. Let’s say that there are mindsets and attitudes that are primed to respond to particular messages, and an event occurs that just hits the right spot for mass rage (which tends to be activating) or mass grief or emotion (which tends to be activating) or mass sense of unfairness (which tends to be activating). How would one know?

In some ways, I’m thinking that any who are trying to “watch” using public tools will be caught somewhat flatfooted but can get up to speed fairly quickly by identifying relevant terms, hashtags (hopefully fairly disambiguated), leaders, and other information, and to begin mapping the surprise “eventgraph” at the moment of realization…and based on the tool APIs about a week back from the moment of the query. There can be geolocational queries based on the posted messaging. Theoretically, then, there is a way to map an event even if one is a bit of a laggard. And this is based on a microblogging site alone. There are many ways to query other social media platforms once the event is sparked…and again with a certain amount of retroactive data extraction. (There are a number of machine-ways to extract sentiment from a big data set of text-based messages.)

What if an event is occurring within another event? What if an event is part of a larger trend or sequence or supra-event? What will the nuances be understandable?

How will people understand when it looks like an event is at the peak and if that peak is the midpoint or one of many peaks? Or something else altogether? How much is actually knowable from social media platforms and the human nodes that are acting as sensors (and know only a piece of what is happening at the macro level)?

Planning Interventions

For those who actually plan interventions, how are those to be designed and executed? How can they measure the effects of the interventions (such as a timely message) on the events and on attitudes? (Theoretically, according to social network analysis, this should be knowable.)

The Accuracy of Eventgraphs

Finally, are there ways to check for the validity of the methods? Are there ways to run certain queries to see how accurate an interpretation was vs. another? How would such informal ways of knowing stack up against formal journalistic insights? In terms of culling data from electronic spaces, what are the limitations? What sorts of events happen below radar from electronic sources? What are ways of being aware of those?

This is an initial brainstorm then on a presentation that never happened…but maybe will in the next year if time allows for the actual research and the actual trial runs on this.

Crediting Broadly or Not?

Sat, 31 May 2014 06:00:00 -0500

In June, a library journal will publish a multi-page invited article that I wrote about future academic libraries and the needs that they would meet from the point-of-view of a long-time faculty member. I started work on this piece months ago when one of the marketing directors at a publishing house asked if I would write this.

What finally resulted was a piece that involved a brainstorm about the various academic library services of the future. I had one thread using game theory to show the enablements that libraries would provide. I had another thread where I conducted data extractions from various social media platforms in order to highlight the electronic social aspects of libraries and how they were hubs of connectivity among users. This was informational but also highly visual with eye-popping graphs and analyses. The marketing director provided helpful feedback during her read-through of the draft.

Then, when an InDesign draft got sent out, I noticed that under my byline were several others…members of the marketing staff who sponsored the column (by asking for the work and shepherding it through the process but not “commissioning” anything in any sense of that word)… I was reminded of the fact that a book is never a work on an individual alone. Even if there is only one byline, there are many others at the book publishers who ensure that a work is properly copyrighted and registered, published, and distributed.

When I first saw that, I thought snarkily of the saying, “Success has many fathers, but failure is an orphan.” Then, a more rational response settled in. While it was true that I’d put in the hard work and long slog, I could see that those others on the below-byline listing had also invested something of their social network and their effort. We all stood to gain by promoting the brand of the author and of the publishing house. My supervisor noted that people only get used if they allow themselves to be, and there’s truth to that. In trade for this extra work, the publishers agreed to provide a prize for a conference that I serve on a steering committee for—in a broad quid pro quo.

Warhol’s “15 Minutes”

Andy Warhol’s 15 minutes of fame has had long staying power in culture as a meme because it touches on a core truth. People care about credit; they care about social regard. People do engage in social performance for others. They do have something of a self-promotion aspect. In an academic economy, bylines matter because it’s used as a proxy for productivity and professional value.

If people are listed as co-authors on various works, it’s a very simple check to see if they have a history of research achievement. Trends (and patterns) matter. If co-authors are only one-offs, that says something about their interests (non-interests)…and work focus. Their history may / may not shed light on how they achieved co-authorship. (With stylometry, it’s somewhat hard to hide authorship, and I think every now and again that ghosted works will easily come to the fore as well.)

Controlling for Byline Inflation

At core, bylines should be used if there was actual shared co-research and co-authoring, and then the names on the list should be accurate to the investment in the work. If there is necessity to trade for particular work, then I can totally see the rationale for bylines where—especially if budgets cannot be stretched far enough to buy the work…and the team has to resort to goodwill (every now and again). Sometimes, there are benefits to listing others’ names because of their social capital and reach.

I can also brainstorm a list of lame reasons for sharing byline credit—all of which came from firsthand experiences: respect for the positions of principal investigators on grants (at their request and insistence) and a sense of sharing credit to “be nice.” Faculty sometimes assume that a research write-up is part of the instructional design skillset, and they’ll assume that their position as PI on the grant means a byline even if they’ve not contributed to the writing. (It is part of the practice in some academic areas to always include PIs.)

There is a sense of dilution with each added name. Cost-benefit considerations do apply when considering whether to add someone on. If I’ve learned nothing else, it’s to not tell PIs that there is a forthcoming work that will be published that is peripherally related to the project and which is based on my own expertise.

Never Working in Isolation

It is true that people are never working in isolation. People work on teams, and they need each other to succeed. It helps to think of names maybe as brands, and those brands are actualized through various types of supports. It also helps to realize that in case of incurred liabilities from a work, the lead is most responsible, even if lawyers will rope in anyone with even a slight interest.

Least-Cost Software Options

Thu, 29 May 2014 06:00:00 -0500

In daily work life, there is very much a real cost barrier which is prohibitive various computational work. While many business organizations have some open-source aspect, many of these endeavors are highly limited (and maybe tied more to marketing and public relations than to anything of core interest—as it probably generally should be). What this means though is that one is constantly trying to cobble together functionalities from a range of free tools (many of them created in university environments and occasionally from corporate-funded foundations).

Challenges with Least-Cost Options

There are clearly some downsides to going with the least-cost option. Sometimes, there is a sense of whiplash given changes to a tool because one is not privy to the internal conversations about the evolution of the tool, and the makers are not necessarily paying attention to users’ needs (or one falls into the minority of users who likes a particular function). Open-source tools can just disappear because if they are built on goodwill or political whimsy, either sentiment can go wobbly based on a little financial pressure or challenge. Many open source tools can be abandoned by their users after a certain period of time. One engaging agent-based modeling tool used for simulations was building up a fairly strong library of simulations based on research but has since been apparently backburnered; it did not build up a sufficient base of users (both those who can create user the tool and those who can use it to analyze and visualize).

Often, such software may be based on a limited shtick, which will not outlast the next competitor with a different shtick or additional functionalities or somewhat slicker marketing.

Said another way, the software that tends to have staying power has to be well designed and high functioning. It should function in a robust way. Its interface has to be fairly intuitive. Its labels have to be accurate and memorable. Its functionality should be clearly documented. The cost of its use should not be prohibitive (and free can be very expensive depending on the circumstance). The use of the tool should not involve unnecessary stresses like security vulnerabilities or data inaccuracies or hidden costs. The software should be professionally maintained and evolving. There should be sufficient and continuing outreach to bring the software to public attention and to keep it there. Often, it helps to have virtual communities and sites where users may share their work and insights. It helps to have charismatic users.

Degraded Free Versions (of a Commercial Product)

Some tools are degraded (with limited features) free versions of a commercial product. Others involve the creation of watermarked products. For the broad public, the degrading of a tool can be somewhat frustrating and doesn’t read well. Other commercial organizations will release full versions for free and educational release but include clauses that companies that use their products and earn a certain amount annually must pay a certain amount for the tool; in other words, they go after the “deep pockets.” In a sense, the free rollouts are to develop users who have the skillsets to use the tool to create commercializable products (such as digital games built on various game platforms).

Developer Communities

It is encouraging to dip into the online open-source developer community spaces, which have become very sophisticated. There are the spaces eliciting talent. There are the highly impressive spaces for distributing the software, with descriptions, updates, and rankings and ratings, and other features. These sites are vivid and engaging, and they very much show the “long tail” of software code (with many applications built for both broad-scale and limited-scale usage). There are clearly professional benefits in having immersive side projects that are broadly used and appreciated.

Variety

That said, there is a lot of benefit from open-source and freeware tools. There is a broad variety of such tools. Some of the tools’ clunkiness ironically encourage more assiduous fan-dom because there is a cost to entry (users have to actually invest themselves to learn the tool). Such releases can be very positive for morale for the development teams. There can be very constructive alignment with academic research and other work.

Staying on the “Straight and Narrow” with Survey Analysis

Tue, 27 May 2014 06:00:00 -0500

Having just roughed out reports linked to several surveys for in-house usage (and eventually some public acknowledgment), I realize how important it is to establish rules for objectivity in the analysis—so that I don’t slip into subjectivity (and potential political hot water). While rule-following is generally fairly boring, there is a lot to be said such care.

What are some of these rules that have applied to the work, and how are they helpful in staying true to the survey findings (and out of trouble)?

Objective Survey Design

As many researchers would likely agree, it helps to have an instrument that has been calibrated against skewed responses. On a campus where there are plenty of researchers, there is a large pool of respondents who are willing to critique the instrument and point to its shortcomings. The point is to have a tool that focuses on what the researchers want to know but which will not skew the results based on the design. This was even more challenging this year given the integration of multimedia into the survey, which could theoretically and practically skew results. Nuanced phrasing of questions or term selection or the design of selected answers in closed-ended questions may skew responses.

I have looked at several years of data, which I had to crunch for prior reports, and I’ve noted the many suggestions that were made for improving the instruments. This year, before we launched, we vetted all the suggestions and made as many changes as possible to try to accommodate the campuses.

The original survey was apparently based off an open-source one deployed by a major university. Since then, the survey has evolved over time. Because there are many stakeholders across campus who have an interest in this information, many have had a hand in evolving respective questions in the survey.

The Survey System

The reports and data output by various survey systems will vary. (Some of these outputs may be changed based on settings that may be input for various surveys, but that is not so for everything.) Some survey systems output information that is more raw (unprocessed) than others and which may require additional processing before the data may be used in a report. While systems may output their own “reports” which are just automatically created tables in PDF or Word format, none that I’ve ever seen would ever go out in that form. It’s not only that the data has to be processed, but there have to be the various report elements that have to be written, the subheadings made, the data visualizations created (in tools outside of the survey system), and so on.

Adherence to Analytical Rules

So what are some of the rules that have to be applied to keep the analyst on the straight-and-narrow? First, it helps to be accountable and responsible. By this, I mean it helps for the internal team to have access to the original data, the original survey, and the machine-created reportage. This access is a constant, and that access is never rescinded. Also, the report itself points back to the questions, with very clear labeling and citing. If anyone at any time wanted to move from the report back to the survey and the direct findings, they can because of the clean referencing.

Second, it helps to have standards for weighting commentary and insights for open-ended questions. This is done simply with the number of references by respondents as a metric for weighting the importance of the response. This is a crude metric, but it provides a kind of organizational structure (from most cited to least cited) of the mentioned issues. What else could be used? Well, there could be citations of issues based on the general order of issues raised in the survey, but that’s a mechanistic sort of structuring. There could be citations by emotional vehemence, but that’s not any systematic or logical way to organize. There could be an alphabetical approach, but that’s again…just a system…without clear reasoning.

Third, it is important to be as accurate as possible. This means that the numbers all have to be rendered accurately. Percentages have to be listed along with the numbers of respondents per question (so the percentages have an underlying context). All quotes have to be verbatim, typos and all [even if (sic) has to be used]. [This is tough to adhere to when a report goes through multiple hands, and if others go through and correct for typos. It helps to keep all versions of a report, so one can go back to a more correct version as needed.] This accuracy requires consistency to be relevant as well; every question of a type needs to be treated the same way across the board. Nothing, especially not the comments in the open-ended text-based questions, should be omitted at the discretion of the analyst. Even single-mentions of concerns should be included—even if it means challenging attention to details…and how the campus then chooses to respond is up to the various leaders and others.

Fourth, it’s important that the analyst stands apart from the findings. It helps to have a suspended sense of interests. There should be no playing favorites. (It helps to have no analyst name on the actual report. On a campus, though, word gets out…so there’s no lack of credit. It also helps to not name any names of people mentioned in the comments by survey respondents, who are not always disciplined in their responses.) Further, the report should not be politicized on the campus. Where information is relevant, it should be used to improve services. (Of course, it’s very difficult to have any apolitical data in a human organization.)

Fifth, a survey report has to be transparent, so that those who read the report are aware of the standards by which the analyst conducted the work. These standards should be clear not only in the quality of the data but also in the formal information releases around the survey.

Interpretive Restraint

It helps not to get ahead of the official data flow of the leadership in terms of processing surveys. It’s helpful to leave the draft report somewhat rough-cut.

This means not writing the executive summary unless that’s requested. It also means that none of the information in the report should be shared until the formal and final version has been approved. In other words, it’s critical not to overstep responsibilities. Where political sensibilities are may be somewhat predictable in some ways but wholly not at all in others. This restraint also means not leaping to conclusions about what actions should be taken.

In terms of statistics, it’s helpful to stay with simple descriptive statistics. Anything more than that could be confusing or offensive to some. While one may be tempted to pursue trends over time, that may be “TMI” for the leadership. The point is to be practical.

“Eyes Only”

In draft format, the draft is watermarked as such, and it is essentially embargoed from any other usage beyond that of the team. That report will go through multiple versions before it ever makes it out to in-house circulation. There are multiple other vettings before anything goes out to the public. The public generally gets the easy-to-digest article form, and they can access the report if they want to engage the information further.

It’s critical that information is vetted for political sensitivities while maintaining the integrity of the information.

Professional Learning

This is not to say that there are not opportunities to learn from the insights provided by those who’ve shared their insights on the survey. Having a breadth of feedback means that there will occasionally be some fresh ideas that may be applied to improve the quality of services on a campus.

Avoiding the Recycling of Work

Sun, 25 May 2014 06:00:00 -0500

In some ways, the Internet’s features can force a kind of honesty that people would not have otherwise. The fact that many works are becoming digitized and flooding online means that it’s hard to leave a past behind. It’s hard to be the single voice if one wants to pose or create public performances online. It means that our more immature selves will be available in some digital format over time. It also means that works that one has published becomes single-sourced online, and re-usability will be tough.

A Scrubbed Manuscript for Review

Recently, in the midst of a busy episode, I sent out a scrubbed manuscript for double-blind peer review without first reading it. A day after I had it sent out to multiple colleagues, I finally had the time to read it, and about three sentences in, I suddenly realized that the work was likely unoriginal or had been published elsewhere. The “tells” included the years of the research, the depth of the research, and the visuals. (At the moment of the realization, let me just say that it’s a sub-second tingling; it’s not as if the logical analysis as fully kicked in.)

I went online, and a few seconds later, I found a work that seemed very similar. A quick scan later (the human brain’s Cohen’s Kappa similarity analysis), I had found the first published work from the information back in 2013. The bylines were different except for one, who was the “lead author” on the work submitted to me. The similarities were obvious—the topic, the date span of the research, the verbiage (re-ordered), the organizational structure, and the visuals.

This led to a fast retraction (and apology) by me to the two reviewers, and I emailed the team that submitted the work (no response now, days later). With open spaces like the Internet, with its pseudo-transparency, it is hard to make unsupportable assertions. Further, this pseudo-transparency encourages timely due diligence. (Had I waited a day and done the work, I would have saved my reviewers precious time and myself from professional embarrassment of sorts.)

That said, if I had not caught this early and let this go through all the development phases and final acceptance (very low probability), I can guarantee that the publisher would have run this through plagiarism checks and likely would have stopped this from being published. Publishers do collaborate with each other to head off such re-publishing.

Handing Over Presentations to Conference Organizers

This event occurred during a national conference that was held locally. Before, during, and after the conference, presenters were told that they could submit their presentations to the organizers to host on a local site, or they could upload their slides on Slideshare with a particular conference hashtag. I know that once a work is uploaded anywhere, spiders / crawlers will identify it, and the slideshows will have a life of their own. They’ll be swiped by people who run various websites. The “not forgetting” aspect of the Internet will manifest. There is an automatic “reuse” function online if one’s work is passable.

The other option when deciding whether to hand over a copy of one’s work or not is to be protectionist and to not distribute anything… This would limit the benefit of the original presentation. To me, this is not only other-limiting (little can be learned in a 50-minute session) but also deeply self-limiting. Protecting an original slideshow will mean I can re-use it again and again to new audiences as if it were a new work. I would not be forcing myself to learn new things for each new presentation. I would be institutionalizing my own stagnation on the topic.

Also, it’s disrespectful to recycle work for others. Besides that, it’s somewhat mind-deadening to go with a formerly created work. There’s something to be said about using a range of different strategies to approach a presentation. Each audience has a different vibe and a different background, and it helps to rework a topic to suit that audience. It also helps to go with something new so as not to bore oneself. It helps to take a fresh angle, so one can learn something new in the preparation. Also, learning is cumulative, and ensuing works should capture that additional learning.

Using Transferable Ideas

What transfers well is not the original work (which is single-sourced) but the learning from the building of each of those works. The learning may be very obvious and explicit—such as how to use a new technology in certain contexts. Or, the learning may be incremental and implicit. Sometimes, the learning surfaces a long time later, after the project has wrapped.

Moving goalposts is helpful for the actual learning. This helps stretch one’s skill sets, and it helps trigger new insights. This makes for better outcomes overall not only for oneself but for all the other stakeholders to the various projects.

Unsolicited Advice to Customer Satisfaction Survey Takers

Fri, 23 May 2014 06:00:00 -0500

Now in my second year of analyzing information technology satisfaction survey results (and immersing in my third year of such campus data), I am realizing that there are two main ways that information from such surveys can generally lead to change. By change, I mean gaining the attention of administrators who control the resources and personnel who can make the requested changes.

One main way to be heard is by being part of a crowd. This means that it helps to be trending a certain way regarding a certain issue. If there is general satisfaction and high numbers for that, administrators will be sure to notice and to tout there. If there are low levels of satisfaction regarding certain services, they will note that and respond also. Often, dissatisfaction with one mission critical service is enough to cause all sorts of other grumblings about other services. It’s critical to get the basics right.

The other way to be heard is to be original and insightful (rare). In other words, it helps to have a unique perspective or to suggest a new way of doing things that is beneficial and practicable. In information theory, information that is novel is more valuable than what is already known or common. That is absolutely so here. People who look at the data in surveys focus on where the surprises are.

To better understand this, it may help to get a sense of what common responses look like.

Get Real

It is said that every time people interact with each other (or with each other through systems), they gain some information, and they leave some information behind. (It’s a basic rule of forensics, too, that a person cannot enter and exit a room without leaving trace of themselves somehow.) Survey takers are reading something about the organization and its people…and vice versa.

Those who put out surveys are aware that this is very much a public-facing outreach—from the random stratified sampling all the way to the publicly released reports of the findings and then the changes that come from the feedback. One of the inherent public relations (PR) messages they are sending out is that they value their customers and care about their experiences and needs. Another is that the university (in this case) is putting a lot of resources out there to support the teaching, learning, research, and intercommunications of the campus. Another is that this unit plays well with others on campus. And so on.

This care for the client is true only up to a practical point. There are very real human limits to what may be done given resources. The way publicly funded organizations work is that they work with limited budgets and with all sorts of fiscal uncertainties. They will push the limits of the human talent they have. They will use computers for longer and longer cycles to maximize their value. They struggle daily with limited and tight resources. This means that each leader is highly focused on maximizing his / her capabilities locally, and if there is any sharing going on, it’s often only quid pro quo. If administrators form a reputation of never sharing resources and only being politically self-interested, they will lose any of the general allies they may have cultivated before.

At some level, each administrator has to be about his / her own interests both strategically and tactically. People will cross lines to collaborate now and again for mutual benefit, but creating such opportunities requires a fair amount of work and coordination.

This means that while people may ask for all sorts of resources, any changes have to be feasible—politically and fiscally. Not only that, there has to be the local talent to make things real. It’s helpful to understand that there is a real price tag on every service offered.

What Doesn’t Get Heard

It doesn’t help to respond to questions with a misfocused answer. In other words, if the survey is asking about one thing, don’t opine about something else unrelated. Responses have to have informational value. It’s not a good idea to name-drop because that gets sanitized out of the report. Venting is not very helpful either—even if it’s somewhat amusing. Laying down a string of curses doesn’t help for emphasis because that gets sanitized out (even though the main idea is captured).

Humor doesn’t hurt. For example, three of the randomly invited respondents mentioned wanting higher wireless connectivity at “the spork,” a sculpture that looks like the iconic spoon-fork. There are a few benches by the said sculpture, but it’s not so common to see people sitting around there accessing their wireless. It’s not fully clear how to read that suggestion, but the impression it has left does remain. Sharing firsthand stories can be sufficiently compelling so as to be integrated into the “color” of the report from the survey results.

Going Local

To be highly effective, learners may form coalitions in their departments and colleges and request certain software in their labs or for discounted purchases. They may ask for features in their smart classrooms. They may request access to software tools on campus. Students have a lot of power in their various locales, and they really can make big changes where they are.

I hope this is one of their takeaways as they respond to the satisfaction surveys on campus. They really do have voice and power. They can make changes…but it helps to push ideas up their local chains as well. If there is a wellspring of need and interest, administrators are fairly fast to listen and respond.

A New Design

This year, our survey includes multimedia. It’s a much more user-friendly survey and somewhat shorter (even though one survey taker may not agree that it was sufficiently short). The survey took into account responses from past surveys and involved some changes to the fundamental design to minimize text responses (in exchange for more closed-choice questions). On the back end, it took just two days to analyze, create graphs, and output an initial draft report. That is down from the weeks it took last year (given a clunkier survey design).

As for the changes that will come about because of client feedback, surely there will be some. The campus is always at work to instantiate new programs and to meet ever-changing needs of the university. It’s beyond my purview to suggest what changes may come of this. Still, there is a lot to be said for making sure that one is informed about the resources on campus…and also that one is articulate about experiences and perceived needs.

Brainstorming and Creating Digital Flashcards

Tue, 20 May 2014 06:00:00 -0500

Usually, when we think of flashcards, we think of learning requiring rote memorization, whether that would be the multiplication table or foreign languages (and various forms of memorization related to symbolic reasoning). Flash cards were seen to lighten the load for learning as a tool for cognitive scaffolding.

For online learning, digital flashcards may be brought into play as an opt-in tool for memorization and practice. They enable easy ways to refresh on terminology and ideas. They provide a sense of fun even for adult learners—such as subject matter experts (SMEs) in their respective fields who are expert in their areas…but may know little about instructional design.

The Users

Before starting this build, it helps to consider who the ultimate users of the flashcards will be. What do they already know, and who do they have yet to learn? In what contexts will they use this learning object? (Of course, this will have to be mobile-friendly.) Will the flashcards need context-setting with lead-up text and lead-away text? What are critical contents that have to be defined? How in-depth should the definitions themselves be?

The Draft Form

It makes the most sense for me to start out with a table with two columns. The left column consists of the alphabetized terms, and the right consists of the definitions. The authoring tools that can create such flashcards have pretty tight limits on text length both for layout / display and for ease-of-learning. Having this table enables spell checks…and word form consistency checks…among others. It helps to have the table to run by the development team to save on development time because changes may be made early on (before work is expended on the learning object itself). Also, a table is much easier to skim, scan, and read…much more so than the digital flashcard object. A table is easier to read for number of words, so those may be addressed before going to the authoring tool.

Learning Objectives for Digital Flashcards

In terms of quality standards, the digital flashcards have to enable the learning objectives. Some real-world learning objectives related to digital flashcards that I’ve worked on include the following: Learners will

acquire some of the basic terminology in particular field
experience interactive learning of terms in a foreign language
learn some basic words, their definitions, and some underlying concepts to be able to request instructional design services (without feeling intimidated)

I would think that flashcards would be excellent also for memorizing acronyms, if needed. It can also be used for learning various types of codes—if it’s based on rote memorization alone (and not mnemonics and other memory-enhancing devices).

Quality Digital Flashcards

The third point above is one of the learning objectives for this current digital flashcard object. In other words, we are trying to train subject matter experts to learn about instructional design both for their own online work and for working with professional instructional designers. This one does not have much in the way of terms of art. There is nothing deeply abstract or complex about these words. Again, the point is to offer entrée to the field, not to be exclusionary or clubby or provincial. The learners here need the basic words to begin go function with some familiarity and some confidence.

To create a quality set, it helps to cover the important points…but no digital flashcard set is going to be fully comprehensive. A full set would be impractical to study (for most)…and without filtering, it would be highly awkward to use. The word definitions have to be original (no plagiarizing) but also fully accurate and applicable to a wide range of situations. Because digital flashcards are simply authored objects, they are eminently editable and revisable. They pretty much play on any website or LMS space.

For our project, we want to pique SME interest to continue learning…even as we’re empowering them with some basic concepts and terms to get started.

Walk-throughs

As with all learning objects, it helps to do a walk-through of each learning object to make sure that there are no typos or consistency gaffes. It helps to back all the way out to review the table to make sure that nothing is missing overall. It helps to continually evolve this learning object until the project is ready to launch.

If there are grounds to offer several digital flashcard sets, then it makes sense to go back to the drawing board…and create the digital flashcard “glossaries” in the two-column format…again, to vet the contents for spelling, consistency, depth, comprehensiveness, and so on. It helps to have team awareness of the criteria for adding new terms, and it helps to follow through on those additions at every chance (while vetting the list against over-population of terms).

Responding to User Reactions

Often, it’s difficult to encourage users to provide feedback, even if there are broad calls for feedback and maybe various incentives to provide those insights. It does help to use their comments to improve the learning object. (Sometimes, such feedback comes through innocuous emails or comments.)

Also, if there is back-end tracking of the use of the tool, it may help to see if there’s any data there that may be helpful. (Usually, the reporting granularity isn’t such that one can know when a certain word and its definition is difficult or ambiguous.)

The Broad Findability of Digital Contents

Sat, 17 May 2014 06:00:00 -0500

It used to be that I would go searching for formal informational materials on a certain topic. I would start with Google Scholar to get a broad flavor of the materials available. I would look at the names of the publications to understand who the main publishers were for the topic. I would be happily downloading resources, and then by the second web page in, I would run up against a for-pay source.

For some of those, I would jot down the title…or key words…and the publication, and I can then go into the subscription-based databases to find the materials. Or other times, I would be able to simply go to a general search and find the source hosted as a free resource. [I’m not talking about fly-by-night sites that swipe others’ materials and ignore cease-and-desist letters. I also will not go to peer-to-peer content sharing sites (which involve so much potential for information loss, network security compromise, and computer system compromises). I’m actually talking about a formal and respected publisher which has rights to the original material.]

In former days, I would run up against such sites and be frustrated at not being able to access the materials (because of an unwillingness to pay book prices for an article). Now, I have a bit more of a reasonable approach. If I need the information, I’ll do what it takes to get it…even though I’m still generally hesitant to pay sites.

An E-Text Reader for Students

For years now, I’ve used an e-text reader for my students. The readers in this listing are all professionally written ones that are anthology quality. The works assigned are actual ones that I used to assign from professionally published anthologies. What I’d learned maybe a decade or more ago was that most of the anthologized contents were available online, hosted by other universities (oftentimes). While I would have to update the list every now and again due to links decay, I have been able to maintain strong resource lists—even while watching for the quality of the URL and the side matter that might be problematic (like ads, opinions, annotations, and others’ assignments) that sometimes accompany the works. (I generally use resources that are as unmarked as possible.) It is hard to justify the extra costs for the students for broadly available contents (particularly when many readers are already in the public domain and are legally free).

Patents and Protections

At a recent presentation, a company COO spoke about a new technology she had originated and how she and a colleague ran a company evolving that technology to meet huge market demand. After her presentation, she was asked how she protected her intellectual property. Her response was that the patent system itself did not enable a lot of protection (back in the day before her company was bought out by a mega-conglomerate with strong legal counsel) in the day because of the need to use the legal system to enforce those rights. What was more important, she said, was to play offense, not defense. Defense itself was a dominated strategy. For her, playing offense meant gaining as much market share as possible and maintaining leadership by constantly innovating and adding value.

For IP that is not even at the level of patents, a lot of those same concepts apply. It seems that only a certain small amount of time passes before material protected under IP is simply leaked to the public and hosted on some site (often in countries with no enforcement of foreign IP). Cease-and-desist letters are met with silence. This is so even for places with some history of IP protections. The technological cost to cross that boundary from legal to illegal sharing is low. For many, they may feel invisible.

There is a thin space though where copyrighted works are hosted in legal ways. For example, a work may be both copyrighted and also open-access. In such cases, a publisher has a for-pay version but also an open-access version. The latter one requires some more time to find, but it’s there, and it’s as good as (the same as) the original for-pay version. In other cases, authors sometimes write works that are very close in meaning between one work to another. The works are not truly interchangeable, but the ideas are sufficiently close that one can cite one and not the other for the same effect.

(When I am on the other side and am an author, I try very hard not to repeat ideas in works. I am too impatient to repeat the ideas in one work in another, generally speaking. Also, I have decided that if I’ve gone the open-source route to publish, I’m totally okay about having that work broadly available and repurposed, which it is… When I go the for-profit publisher route, I try very hard not to undercut the publisher because their margins are thin as well…and they have invested real elbow grease and resources into my projects. While I can ask for materials to be released to the public under an open-access license, under a “fair use” release, I have found that to be somewhat less satisfying. I have a responsibility to let a work sit behind a “walled garden” or a “paywall” for at least a certain amount of time before it goes out into the wild—which is almost an inevitability, and even if it’s against the will and rights of the copyright holder (the publisher).

Interlibrary Loan

If all else fails and I am unable to acquire a work (even after some creative and thorough tries) that I think will be helpful, I will rely on the awesome professional service of the librarians who handle interlibrary loan. Without fail, they’ve acquired all electronic articles I’ve needed…and all print materials as well. Because there are costs involved to the institution, I don’t ask unless I absolutely need the materials, and without fail, I have always used and cited the contents. (I’ve also tried to be conscientious about supporting the library as an institution on campus with material and other resources.)

What’s Worth Pursuing?

Over time, I am gaining a better sense of the fairly large gap between what title piques my interest vs. the small minority of works that are citable in a published work. So much more seems promising at first blush but isn’t ultimately so (but should still be checked out anyway, generally speaking). I have a sense of the diminishing value of older works (unless I’m working on a full meta-analysis or need a thorough historical review)…but occasional older (undiscovered) gems that are rare and irreplaceable and insightful do exist.

Usually, though, I’m just reading for methods and technologies, information, thinking, and possible leads to yet other sources.

Gray Literature and Embargoed Materials

Occasionally, I read about research using informal collections (including gray literature) or works that were embargoed but only recently released. Of course, there is the actual research work of acquiring primary-sourced information.

Acquiring sources is tough, but the next steps are even tougher: the analytical work.

Non-linear Machine-Enhanced “Reading”

Wed, 14 May 2014 06:00:00 -0500

Of late, I’ve just started experimenting in the world of text corpus analysis. As part of the current fascination with big data, and the wide availability with plenty of text of all sorts, researchers have been tapping into these various collections to extract meaning. The obsession with fast processing speeds (to promote computational efficiencies and near real-time situational awareness) has meant greatly speeded-up extractions of meanings through various types of text parsing—breaking up texts into meaning-laden words (semantics) and structure-based words (syntax), and evoking what may be learned from each type of analysis.

The idea here is that the machine is used to surface data based on human-made algorithms, and the human is in the loop to analyze the outputs from this analysis. This is a kind of human-machine reading. If further refinements are needed for the data analysis, the person may apply various computational tools in a variety of ways to extract meaning.

A Simplified Non-Linear Reading

In a sense, the machine reading of text provides a snapshot of the writing. In a rough-cut way, the frequency of mentions of words, phrases, symbols, equations, and other “strings” are equated to importance and focus. Said another way, if something is important, it is mentioned a lot. (That’s the intuition anyway.) Apparently, systems may take into account anaphora—parts of speech that refer to antecedents or postcedents for that meaning. While such referents may have to be disambiguated manually—by human intervention—my sense is that computers can take some pretty accurate guesses at the referents based on syntax and placement of words and proximity.) Semantic words may be expressed visually as word “clouds” and text frequency tables.

Another approach involves cluster analyses. These may be based on associations between words such as their usage in close proximity (based on n number of words in front or in back of the focal term). Clusters may also be created based on other types of relatedness such as co-use. These relationships may be expressed visually as tree maps, text graphs, and node-link diagrams.

There are various types of text similarity measures between corpuses based on shared words, so there may be comparisons between various articles to see how closely they relate to the other.

Then there are word or phrase or other alphanumeric text searches, which resolve to dendrograms. The various texts represented on each branch are considered related to each other more-so than to the other separate branches. These show the words used in direct adjacent proximity to the target word. These would include phrases leading up to and away from the target word. Such text proxemics capture a sense of gist (direction of opinion).

Then there is machine-based sentiment analysis, with the capturing of particular terms (word choices, depictions, advocacy positions, and others) that may indicate direction of opinion, value judgment, and emotion.

These writing snapshots have been used for stylometry, the quantitative analysis of writing style that may be fingerprinted to an author.

Essentially, computers may be used to provide textual auto-summaries across broad swaths of texts. For the desktop computer, and for the common researcher, a fair amount of research may be done with open-source tools as it is.

Stopwords or Delete Lists

Clearly, before machines can effectively crunch through heterogeneous textual data, there are pre-sets and ways to scrub data. As has been said by others, computers are very good at counting and listing…they are also very good at relating… they can be very good at finding patterns. Various text analysis tools that are broadly used by researchers harness these capabilities in practical ways.

The point is to focus on parts of a text only, the parts that may be revelatory of information from a particular angle. The rest (punctuation, helping verbs, and repeated information, among others) is filler. It would help to note that the aesthetic pleasure of good writing is not something theoretically capturable. Remember that style boils down to counts of word forms and unique phrases and various types of “tells” or indicators of authorship. Context is not truly relevant except as textual / symbolic proximity to each other. In machine-analysis, every piece of text is broken down to various “grams” or units of consecutive text. A full manuscript is a large ngram unit (theoretically), and it is made up of various units of words and phrases. (The Google Books Ngram Viewer gives a bit of a sense of this.)

Arguing for Close (Human) Readings

Of course, there are those who take offense at the uses of such machine-based text analysis approaches. The argument is that a close human reading of a work is needed for actual deep analysis and fresh human insights.

(Much work has been done in this area already, and as usual, I’m just a latecomer who has only recently discovered some of the tools. Such analyses are sufficiently sophisticated to apply across a wide range of languages, with nuanced tweaks to extract the required information.)

Suffice it to say that there’s a machine-analytic benefit to a high-level view of texts—whether static repositories or dynamic floods of text data (such as in active microblogging feeds). There is also very much a place for close-in readings of text by trained individuals with creativity and insights.