Technology at ICPSR

A look back to 2000 (part 1)

2012-03-05T06:30:00.000-05:00

I was looking through the Internet Archive's Wayback Machine the other day, trying to find an electronic copy of a printed document I have. I wasn't able to find the document I wanted, but I did stumble upon a copy of the 1999-2000 ICPSR annual report. This appears to have been published during the interregnum between Richard Rockwell and Myron Gutmann. Myron took over as the Executive Director of ICPSR in 2001, and hired me in 2002 to lead the Computer and Network Services team.

As I read the report it occurred to me that some parts of it were just as relevant today as 10+ years ago. In some cases the issues and observations are exactly the same; only the players and the technologies have changed.

I thought it might be interesting to compare excerpts from the Computer and Network Services section of that report, and compare them to today's ICPSR. There's quite a bit of text in that old annual report, and I think it will make sense to break it down into smaller pieces for easier reading and commentary.

I also found a wonderful array of old photographs of my colleagues at ICPSR. Some of them have left ICPSR, and some of them still work here today. I'll include one of them in each of the follow-up posts too. If you mouse over the picture, it should identify the person.

TRAC: A5.2: Transfer of rights

2012-03-02T05:00:00.000-05:00

A5.2 Repository contracts or deposit agreements must specify and transfer all necessary preservation rights, and those rights transferred must be documented.

Because the right to change or alter digital information is often restricted by law to the creator, it is important that digital repository contracts and agreements address the need to be able to work with and potentially modify digital objects to keep them accessible. Repository agreements with depositors must specify and/or transfer certain rights to the repository enabling appropriate and necessary preservation actions for the digital objects within the repository. (This requirement is linked to A3.3.)

Because legal negotiations can take time, potentially preventing or slowing the ingest of digital objects at risk, it is acceptable for a digital repository to take in or accept digital objects even with only minimal preservation rights using an open-ended agreement and then deal with expanding to detailed rights later. A repository’s rights must at least limit the repository’s liability or legal exposure that threatens the repository itself.

Evidence: Contracts, deposit agreements; specification(s) of rights transferred for different types of digital content (if applicable); policy statement on requisite preservation rights.

The ICPSR deposit agreement is pretty clear about this: The depositor is granting us non-exclusive rights to preserve and disseminate the content, and in order to perform these tasks effectively, we reserve the right to transform the content.

In fact, unlike a more conventional archive where the goal might be to preserve the item "as is" indefinitely, much of our work - particularly in the topical archives - involves changing the content in significant ways.

Sixteen products or one?

2012-02-29T06:30:00.000-05:00

A recent conversation with Nathan Adams, ICPSR's Assistant IT Director for Software Development got me thinking about this....

It's no secret that ICPSR uses a package called Survey Documentation and Analysis (SDA) from UC Berkeley as our on-line analysis system. But people may be surprised to learn that this one product forms the underpinnings of more than a dozen closely related ICPSR on-line analysis products.

One, Anonymous Analysis : This is where we make a dataset available via SDA and there is no authentication allowed.

Two, Authenticated Analysis : One must authenticate using MyData, Google, or Facebook.

Three, Member Analysis : One must authenticate and also be using a computer located on the campus (even virtually) of a member institution.

Four, Private Analysis : One must authenticate and the identity used must be a member of a previously created group of identities.

Five through eight, Secure Analysis : Like any of the options above, but where the raw, proprietary, binary data files reside on a separate server, and where the ICPSR web server accesses the content via HTTPS rather than through the filesystem.

Nine through Sixteen, Non-disclosed Analysis : Like any of the eight options above, but where SDA's disclosure.txt controls have been used to attempt to prevent unintentional disclosure.

So sixteen different combinations! And it is easy to imagine even more cropping up in the months ahead.

My experience is that one ends up with sixteen different online analysis "products" when things grow organically over time. When things evolve due to a small tweaks in response to requests like, "Hey, could we use SDA for this, but with just one small change ..... ?"

It is easy to see how it happens. But when things grow over time like this, they end up suffering from a profound lack of design, and end up costing more to maintain. They are fragile. They break when you change things, like the hardware. Or the OS. Or the NAS. Or the authentication scheme. Or the oil in your car.

So probably time to pull back a bit, pull together a team of content owners, and start asking some questions.

If we were going to start fresh today with an on-line analysis system, what should we build?

What sort of access controls are needed to prevent bad guys from using it?

What sort of disclosure mitigation capabilities are required to prevent accidents from happening?

To which populations might we need to restrict access?

What does the user experience look like? Is this geared for the novice or for expert-in-a-hurry? Or do we have multiple audiences and so need to build more than one experience?

Time to design.

TRAC: A5.1: Deposit agreements

2012-02-24T04:30:00.000-05:00

A5.1 If repository manages, preserves, and/or provides access to digital materials on behalf of another organization, it has and maintains appropriate contracts or deposit agreements.

Repositories, especially those with third-party deposit arrangements, should guarantee that relevant contracts, licenses, or deposit agreements express rights, responsibilities, and expectations of each party. Contracts and formal deposit agreements should be countersigned and current.

When the relationship between depositor and repository is less formal (i.e., a faculty member depositing work in an academic institution’s preservation repository), documentation articulating the repository’s capabilities and commitments should be provided to each depositor.

Repositories engaged in Web archiving may find this requirement difficult because of how Web-based information is harvested/captured for long-term preservation. This kind of data is rarely acquired with contracts or deposit agreements. By its very nature, digital information on the Web is perceived to belong to “everyone and no one.” Some repositories capture, manage, and preserve access to this material without written permission from the content creators. Others go through the very time-consuming and costly process of contacting content owners before capturing and ingesting information. Regardless of process, repositories harvesting and ingesting Web-based materials must articulate their rights issues within publicly accessible policies, and have mechanisms to respond to content owners if the repository’s rights to collect and preserve certain information are challenged.

Ideally, these agreements will be tracked, linked, managed, and made accessible in a contracts database.

Evidence: Deposit agreements; policies on third-party deposit arrangements; contracts; definitions of service levels; Web archiving policies; procedure for reviewing and maintaining agreements, contracts, and licenses.

The ICPSR Deposit System implements this TRAC requirement. The system makes the terms of deposit clear, and collects an electronic signature from the depositor. ICPSR keeps the agreements in perpetuity, even if the depositor later decides that s/he would rather not use ICPSR as a repository.

Going Google

2012-02-22T11:04:00.004-05:00

The University of Michigan is rolling out Google Apps for Education throughout 2012. A few of us are in the early (1.0) pilot population, and this group made the jump from a variety of legacy University of Michigan email and calendar systems on January 16, 2012. I first reported on this new initiative late last year, and it's now time for an update.

I should note that I have been using Google's productivity tools outside my professional life for many years, and so there is not much of a learning curve. I think this will also be true of some of the more broad population at UMich, but will not be true universally. And I should also note that I had been using a second Gmail account for my professional life too for the past 2-3 years. The main driver for me was storage space. While I'm not a huge fan of Outlook and Exchange, the service operated by ICPSR's parent organization - the Institute for Social Research - was always solid. However, the killer was that the allowable quota for mail was very low (400MB by default), and so I found it frustrating to always be shuffling email off into either the Trash Can or into PST mailboxes. It was especially rough when it came time to search for something.

And so the move from a consumer Gmail account that I use for work to a Google Apps Gmail account that I use for work has been a small change. The change from Exchange to Google Calendar for managing meetings has been a bigger change. On the plus side I'm finding it much easier to manage a single, coherent picture for meeting invitations; I had been trying to manage everything inside of Exchange before. However, I probably receive 100 meeting invitations for every one I generate myself, and so I haven't had to spend much time and effort ensuring that meetings I create on my Google Calendar are ending up on the ISR Exchange server intact. In fact, most of the headaches I experience with calendaring are related to cases where someone generates an invite within the ISR Exchange server, but does not include anything in the "body" of the invite. If I try to "read" the invite on a mobile device (e.g., Safari on an iPad), the meeting invite shows up as an empty message. And so I then track down a "real" computer to see what the meeting invite is all about.

My main take-away so far is that moving from Exchange to Google would best be done (1) quickly, and (2) all at once. My sense is that we early adopters will continue to face a few headaches like above until the rest of the organization moves to Google in 3-6 months.

Archival Storage @ ICPSR

2012-02-20T04:00:00.000-05:00

I gave this presentation to a group of students who came to ICPSR to hear our Life of a Dataset show.

Archival Storage @ ICPSR on Prezi

It contains a couple of pictures from the old ICPSR data warehouse (literally a warehouse) which has been torn down and replaced with a sprawling Costco complex.

And there is also a nice little graphic that shows where we're making copies of things we move into Archival Storage.

TRAC: A4.5: Managing revenue

2012-02-17T03:21:00.000-05:00

A4.5 Repository commits to monitoring for and bridging gaps in funding.

The repository must recognize the possibility of gaps between funding and the costs of meeting the repository’s commitments to its stakeholders. It commits to bridging these gaps by securing funding and resource commitments specifically for that purpose; these commitments can come either from the repository itself or parent organizations, as applicable. Even with effective business planning procedures in place, any repository with long-term commitments will likely face some kind of resource gap in the future. The repository must provide essentially an insurance buffer as a first—and hopefully effective — line of defense, obviating the need to invoke a succession plan except in extreme situations, such as the repository ceasing operations permanently.

Evidence: Fiscal and fiduciary policies, procedures, protocols, requirements; budgets and financial analysis documents; fiscal calendars; business plan(s); any evidence of active monitoring and preparedness.

This is another short entry since this requirement is pretty far beyond the scope of IT...

ICPSR's organizational documents require us to maintain something like a 90-day window of cash flow in our "checking account." Every so often these funds will be referenced, particularly during the annual budgeting process, and my sense is that these are exactly the types of monies referred to above.

And, again, a lot of ICPSR's fiscal discipline and operations are rooted in being part of the University of Michigan and the Institute for Social Research.

Stop using Word!

2012-02-15T05:00:00.000-05:00

I felt a rant come over me this week....

Hey, you! Yes, you! You know I'm talking to you.

I tried to look at that meeting agenda you sent me. You know, the one that you sent as an email attachment, and where the attachment is a Word document. But because I was using a web browser to access my Exchange email account, the browser won't show me the attachment unless I was to right-click on it and save it first.

And I was reading this on an iPad! Sheesh.

So, you know, that meant that I didn't look at the agenda until this morning when I got back into the office. Oh sure I could have made the time to boot up a Windows machine to try to get to web mail that way, but really.

So when I did open this thing, I found this agenda:

Introductions
Problem statement
Brainstorm solutions

And this couldn't just go into the message as text?

Oh, wait, here comes another one.....

You over there! Yes, you, the guilty-looking one.

I was looking for the policy on Widget Transformations yesterday on our CMS. I knew we had a policy because you made me go to all of those policy meetings. (I wasn't sure at the time that we even needed such a policy, but there you go.)

So I was searching and browsing and clicking and scrolling. And searching. And searching. No luck.

So I finally got so frustrated I asked Fred if he knew where it was. He found it right away. (He had it bookmarked. Good ol' Fred.)

It was a Word document.

Don't you know that our CMS is really lame and it doesn't search these things?

But the worst part is when I opened the policy document. I figured that it must have been done in Word since it had a lot of extra fancy content. But here's what I found:

Widget Transformation Policy

It it the policy of ICPSR that no widgets should ever be transformed.

And that was it.

This couldn't just go into the CMS as text?

Whew. Feeling much better now.

But I wish I had a nickel for every time I couldn't find something or read something because it was in Word, and where it was something very plain and very simple.

SSLv2 and ICPSR

2012-02-13T06:00:00.000-05:00

According to our web server logs, there aren't many of you who have been using SSLv2 when navigating to a URL that begins with "https" on the ICPSR web site. And that's good. Because this is a protocol which has known flaws for protecting information.

The ICPSR technology team disabled SSLv2 on its production web server last week. Based on the logs we keep, it looks like this change will affect few, if any, web site visitors.

ICPSR network maintenance

2012-02-12T09:06:00.003-05:00

This morning's network maintenance at the University of Michigan is running longer than expected. ICPSR staff rolled service from the production systems to our replica in Amazon's cloud at 4:55am EST today. The U-M data networking team now estimates that the work will be completed by 10:30am EST. Once ICPSR confirms that the maintenance is complete, we will roll service back to the production systems.

You can track the maintenance using the same link we are - http://status.its.umich.edu/outage.php?id=73819

TRAC: A4.4: Managing investments and risk

2012-02-10T04:18:00.000-05:00

A4.4 Repository has ongoing commitment to analyze and report on risk, benefit, investment, and expenditure (including assets, licenses, and liabilities).

The repository must commit to at least these categories of analysis and reporting, and maintain an appropriate balance between them. The repository should be able to demonstrate that it has identified and documented these categories, and actively manages them, including identifying and responding to risks, describing and leveraging benefits, specifying and balancing investments, and anticipating and preparing for expenditures.

Evidence: Risk management documents that identify perceived and potential threats and planned or implemented responses (a risk register); technology infrastructure investment planning documents; cost benefit analyses; financial investment documents and portfolios; requirements for and examples of licenses, contracts, and asset management; evidence of revision based on risk.

I don't think we spend enough time and focus at ICPSR paying attention to this sort of analysis. I think this is true for all areas, including my own technology area.

I think the reasons we don't do this more are not unique to ICPSR. People are too busy working on the next deliverable to ship, and are so caught up in the fray, there isn't time to step back, take a breath, and look at the big picture. Technology assets are parceled out to a variety of grants and contracts, each with wildly different purposes, tasks, and deliverables. And so while the management team can ensure that there is some consistency in design, development, and operating environment, it can feel daunting to try to figure our the Future World for such a disparate set of activities.

The culture, too, is very geared to nailing down the next grant or contract, and so there is usually a process for fitting the new work into the overall framework of the organization rather than seeking out specific projects that will fit an already identified in a plan. That said, if the goal is very broad ("build muscle with video assets") then there is indeed an effort to find those projects that help meet that goal.

It may be the case that it is time again for ICPSR to take a crank on a very forward-looking mission statement and strategic plan. And maybe a follow-on to such an activity would be deploying a process that regularly (quarterly? monthly?) revisits those documents to measure risk, benefit, investment, and expenditure rather than activity, accomplishments, and the like.

January 2012 deposits at ICPSR

2012-02-08T07:30:00.000-05:00

January looks like it was a very busy month at ICPSR:

# of files	# of deposits	File format
20	2	F 0x07 video/h264
172	10	application/msword
7	4	application/msword application/msword
1704	4	application/octet-stream
143	31	application/pdf
4	3	application/vnd.ms-excel
4	3	application/vnd.wordperfect
33	1	application/x-dosexec
8	2	application/x-empty
35	6	application/x-sas
32	21	application/x-spss
28	3	application/x-stata
1	1	application/x-zip
2	1	image/bmp
1696	1	image/jpeg
4	1	image/png
7	7	message/rfc8220117bit
3	3	text/html
9	2	text/plain; charset=iso-8859-1
142	17	text/plain; charset=us-ascii
2	1	text/plain; charset=utf-8
6	5	text/rtf
1	1	text/x-c; charset=us-ascii
3	2	text/x-mail; charset=unknown
17	3	text/xml

Two items are noteworthy.

One is that we moved a few key systems from older 32-bit machines running older versions of RHEL to new 64-bit machines running RHEL 6. As it turns out the magic database that file uses on RHEL 6 is in a new format, and did not work well with our local additions (aka localmagic and localmagic.mime for Linux folks). So my belief is that our file-based format detector threw up its hands more often than usual, and this accounts for the over 1700 unknown (application/octet-stream) format types last month. I think these are good candidates for a follow-up scan to correct the results.

Two, lots of images. I know that we are getting a lot of video and images as part of our Bill and Melinda Gates Foundation MET and MET Extension projects, but I also know that none of the files above is from that project. So where is all of this coming from? One big deposit....

Network maintenance - Sunday morning (EST) Feb 12, 2012

2012-02-08T05:00:00.000-05:00

The University of Michigan central IT organization, ITS, will be upgrading the network gear that connects ICPSR's building to the campus data network. The work is scheduled to start at 6am (EST) on the morning of Feb 12, 2012 and should take between one and two hours.

The ICPSR IT team will redirect traffic from the production system to our cloud replica during the maintenance period. The replica runs in Amazon's cloud and features services such as search, analyze, and download, but purposefully does not enable features such as deposit.

Between the cut-over to the replica, the network maintenance, and the fallback to the production system, access will likely be a little rocky next Sunday morning. Like a freeway during construction, if it is possible to take a detour around ICPSR's web site on Sunday morning, that's the safest route. But if you find that you need to download some data or use the site, the replica will be available.

Job posting - again

2012-02-06T05:30:00.000-05:00

We're posting a job description for a senior software developer for a third time. If there is a recession in the IT business in SE Michigan, somebody forgot to tell our pool of potential applicants.

This position is very much like other software developer positions at ICPSR.

In practice there is a blend of business analysis, system design, software development, and second-line on-going support for the stuff you write. Building stuff in java to run under tomcat is a must. Experience with Oracle, Eclipse, and one or more frameworks is very useful.

The main support for this position is our Bill and Melinda Gates Foundation MET Extension project. This is a two year grant to build systems that will delivery video content to researchers in the social sciences and education. We've had a lot of success - like a 0% failure rate - at hiring people to work on project X, and then moving them over to project Y a few years down the road. Project X is in this case is MET Extension; project Y is unknown. But so was the MET Extension project 10 months ago...

If you have interest in the position and would like more info, please feel free to drop me a note. And here's a short-lived, but direct, link to the job http://umjobs.org/job_detail/66372/software_developer_senior

TRAC: A4.3: Transparency

2012-02-03T04:16:00.000-05:00

A4.3 Repository’s financial practices and procedures are transparent, compliant with relevant accounting standards and practices, and audited by third parties in accordance with territorial legal requirements.

The repository cannot just claim transparency, it must show that it adjusts its business practices to keep them transparent, compliant, and auditable. Confidentiality requirements may prohibit making information about the repository’s finances public, but the repository should be able to demonstrate that it is as transparent as it needs to be and can be within the scope of its community.

Evidence: Demonstrated dissemination requirements for business planning and practices; citations to and/or examples of accounting and audit requirements, standards, and practice; evidence of financial audits already taking place.

ICPSR publishes an annual report which contains a financial statement, and it also completes many, many regular reports because it is part of the Institute for Social Research at the University of Michigan.

January 2012 web availability

2012-02-01T07:58:00.002-05:00

Ick.

If you click on the image, it'll expand into something more easily read. But don't.

January 2012 was not a good month for ICPSR's web delivery system, We missed our monthly goal of 99%+ availability. Not by a lot, but still missed it.

Many of the problems this month are related to an upgrade we made to our delivery infrastructure. We replaced a five-year-old 32-bit machine with a new 64-bit machine with significantly more memory and processing power. And we upgraded from an older version of tomcat to the latest release.

The maintenance itself accounted for less than an hour of downtime, but various issues related to the upgrade brought additional short periods of downtime. A "death by a thousand cuts" sort of thing.

The good news if that the reliability and stability of the system now seems better, and the upgrade has exposed a couple of software flaws that we have been able to correct. It also has eliminated the last of the 32-bit systems from ICPSR, and so we are less likely to run into problems where we (accidentally) try to run 64-bit binaries and libraries on 32-bit machines.

Customer service, Zingerman's style

2012-01-30T06:51:00.000-05:00

Our parent organization, the University of Michigan's Institute for Social Research (ISR), is working with the training component of the Zingerman's family of companies - ZingTrain - to build a customized training module for use at the ISR. The focus is, of course, on delivering excellent customer service, and I had the opportunity to attend a session led by two ZingTrain consultants.

I don't want to give away too much of their "secret sauce" but I found their interaction with the group engaging and informative. I almost used the word "presentation" but that feels wrong; it really isn't a monologue whatsoever. And there are no Powerpoint slides in sight. As you might expect the ZingTrain folks shared some tips and techniques about how they build the right culture and right processes. And they brought goodies from the Bakehouse!

I started to think about some of the tips and techniques I've learned to use in the technology business over the years. In this realm an awful lot of the interaction with others takes place electronically, and so one doesn't have all of the visual cues and tonal cues one normally can use in conversation. For example, how do you let someone know that if the solution you have offered does not work, you want and expect the person to let you know so that you can keep trying to solve the problem? How do you let them know that you will own the problem until it is solved?

One easy way, of course, is to be explicit.

If that doesn't do the trick, please let me know. I have a few other ideas we can try.

By asking the person to return and letting them know that "we" can try some other things, it shows that one is engaged. It lets them know that this is the start of a conversation, not the end of one.

On the other hand, I will often see people write this instead:

Hope this helps.

I know people often write this with the best of intentions, but consider how people may read it. It sounds like the conversation is over. "Here, try this. I hope it works. But if it doesn't, it's your problem, not mine." There's no invitation to come back for more advice, more assistance, more analysis if the issue hasn't been resolved.

And that's my customer service tip for the month.

Hope it helps. :-)

TRAC: A4.2: Charting and changing a course

2012-01-27T07:41:00.000-05:00

A4.2 Repository has in place processes to review and adjust business plans at least annually.

The repository must demonstrate its commitment to proactive business planning by performing cyclical planning processes at least yearly. The repository should be able to demonstrate its responsiveness to audit results, for example.

Evidence: Business plans, audit planning (e.g., scope, schedule, process, and requirements) and results; financial forecasts; recent audits and evidence of impact on repository operating procedures.

It certainly is the case that ICPSR undertakes a great deal of what I will describe as "tactical" business planning, and does so on a regular basis. For example, this is the time of the year when ICPSR begins building its draft budgets for its next fiscal year (July 1 - June 30). For my team this means that we need to look into our crystal balls and guess what our technology equipment and software expenses might be for the following year (7 to 19 months hence) and also allocate our team across two dozen different projects (e.g., 20% of Bob will be on Project X, 10% of Project Y, and the rest of Project Z). My experience is that we almost always create a budget which accurately reflects the needs and direction of the organization, and..... is almost always missing some major initiative that only appears much, much later in the calendar year. And so in practice the technology team aligns about 80-90% of its resources with the plan in the budget, and 10-20% end up doing something unplanned and unbudgeted. This keeps things exciting.

Many of the other workgroups at ICPSR are much smaller than the technology team, and they have relatively long-lived contracts and grants where the budget, deliverables, work scope, etc are reasonably well defined. I'm sure they also encounter their fair share of curveballs from their funding sources, and also review and tweak their project-year budgets at the tactical level.

So maybe a B+ or even an A- overall for ICPSR on "regular, tactical business planning." Not bad.

Many organizations do a much poorer job of looking more deeply into their crystal balls,attempting to peer three, four, maybe even five years down the road. And ICPSR is no exception. I call this "strategic" business planning, and I view its role as complementary to "tactical" business planning.

If "tactical" business planning helps you figure out what you're going to do the next year, and how you're going to get it done, "strategic" business planning helps you figure out what you're NOT going to do any time soon, and how you've going to avoid heading down the wrong roads.

The output of this type of planning isn't a spreadsheet or a list of tasks. It isn't necessarily even a list of goals. Instead it is a shared vision for who you are, what business you are in, and where you think you need to be in three, four, maybe even five years down the road. Here's an example:

Today we're the largest archive of survey research data in the world. Our operations are geared to finding, collecting, curating, preserving, and delivering survey data. We think we are the best at these activities.

In four years, however, we believe that survey data will account for only a tiny portion of our business. We believe that video and social media content will be the new core research content of the future, and this content requires expertise and systems very different than we have today. Our intent is to limit the amount of time we spend growing our staff and growing our systems to support survey data; we will maintain, but not enhance. We will seek out grants and contracts that allow us to build infrastructure and expertise in these areas. And we will begin to invest in our people and our systems for video and social media data.

This may or may not be the right vision and the right "strategic" business plan to make, but it illustrates the idea that the organization has charted a course. It lets people know where they are heading. It does NOT tell them how they will get there. (That needs to happen eventually too, of course.) It tells people what they are NOT going to do.

It can be really tough to step outside of the fray and the demands of the day-to-day job to think about the longer term, but it's crucial. Otherwise an organization just keeps heading down the same road instead of looking at other available roads that may lead to better places.

Tech@ICPSR talks about the cloud @ LA2M

2012-01-23T07:34:00.000-05:00

Tech@ICPSR will be giving a talk on cloud computing at the February 1, 2011 LA2M meeting. We'll be talking about the cloud; kind of a high-level overview of what different folks say the cloud is, and some of the consumer- and business-oriented services and systems that live in it.

I'll add a link to the materials shortly after the talk, and, if LA2M adds the video of the talk to their archive, I'll add a link to that as well.

TRAC: A4.1: Business planning

2012-01-20T06:30:00.000-05:00

A4.1 Repository has short- and long-term business planning processes in place to sustain the repository over time.

The repository must demonstrate that it has formal, cyclical, proactive business planning processes in place. A brief description of the repository’s business plan should show how the repository will generate income and assets through services, third-party partnerships, grants, and so forth. As for A1.2 (succession/ contingency/escrow planning), the repository must establish these processes when it is viable to avoid business crises. These questions may be pertinent to this requirement:

Under this plan, to what extent is the repository supported, or expected to be supported, by revenue from content-contributing organizations and agencies, such as publishers?
To what extent is the repository supported, or expected to be supported, by revenue from subscribers or subscribing institutions?
What measures are in place, if any, to limit access by nonsubscribing stakeholders?
What financial incentives are offered, if any, to discourage subscribers from postponing their investment in the repository? From discontinuing investing in the repository?
To what extent is the repository supported, or expected to be supported, by other kinds of parties?
How will major future costs, such as migrations, capital improvements, enhancements, providing access in the event of publisher failure, etc., be distributed between publishers, subscribers, and other supporting parties?
What contingency plans are in place to cover the loss of future revenue and/or outside funding?
In the event of a catastrophic failure, are reserve assets sufficient to ensure the restoration of subscriber access to content reasonably quickly?
If this is a national or government-sponsored repository, how is it insulated from political events, such as international conflicts or diplomatic crises, that might affect its ability to serve foreign constituencies?

Evidence: Operating plans; financial reports; budgets; financial audit reports; annual financial reports; financial forecasts; business plans; audit procedures and calendars; evidence of comparable institutions; exposure of business plan to scenarios.

An awful lot of this TRAC requirement is met simply by being a unit of the the Institute for Social Research at the University of Michigan. (This will be a recurring theme across a lot of TRAC A4.) The U-M, like most large universities, has an elaborate bureaucracy for managing budgets and expenditures, and producing a cornucopia of reports, statements, and other fine documents.

In addition, like a lot of non-profits across the US, ICPSR produces an annual report that it makes available publicly, and this contains high-level documentation for both planning and reporting. For example, the ICPSR annual report includes a breakdown of revenues and expenses across major areas, and usually contains several essays on recent accomplishments, upcoming initiatives, and areas of focus for the upcoming fiscal year.

Disaster Recovery v. High Availability

2012-01-18T07:10:00.000-05:00

A question I often receive from customers and colleagues is: If ICPSR has a replica of its production delivery system in Amazon's cloud, why is it that the web site is sometimes down due to scheduled maintenance or unplanned outages?

The short answer is: ICPSR's cloud replica serves a disaster recovery (DR) purpose, but not a high availability (HA) purpose. Of course, more often than not, this generates a look that falls somewhere between Bah! and This sounds like some made-up IT nonsense! However, it really is the answer. But that begs the question: What's the difference between DR and HA? But first a trip back in time....

As some long-time ICPSR clients may recall, the ICPSR delivery system was off-line for nearly a week during the holiday break between 2008 and 2009. The root cause was a long power outage due to a major ice storm in the Midwest which knocked out power to many homes and businesses, including many in Ann Arbor. And because ICPSR resides in a building just a little bit off the University of Michigan's central campus, we're just like any other home or business that waits for DTE Energy to restore power.

As one might expect both myself and the ICPSR Director at the time, Myron Gutmann, were quite anxious for the power to be restored. The storm had caused so much damage that it wasn't at all clear when the building's power would be restored. And, after the first few days without power - and heat - the building's pipes were in danger of bursting. Things were looking pretty bad.

However, as it turned out we had been experimenting with Amazon's new computing and storage cloud just prior to the storm. It would be pretty easy to stand up a minimal web server in Amazon's cloud, something that would basically say Yes, we know our delivery system is down, and we're sorry about that. And here's the best guess from the local power company about when power will be restored. We then worked with some of our colleagues at the University of Michigan and the San Diego Computing Center to update the system that maps names (like www.icpsr.umich.edu) to network addresses so that ICPSR's URLs for its web site would point to this new, minimal web server in Amazon's cloud. That didn't fix the problem, of course, but it let people know that ICPSR knew there was a problem, and shared the best information we had about the problem.

Once power was restored and the main delivery system came back on-line, I had a long conversation with Myron about how we wanted to position ICPSR for any future problem like this. What if the building lost power again for an extended period? What if a tornado knocked down the whole building? What if the U-M suffered some catastrophic problem with its network?

One option was to change the architecture of ICPSR's delivery systems. Rather than having a complex series of simple web applications, we could redesign and rebuild the whole system so that it would also contain a middle layer of technology that would catch and route incoming requests to one of many delivery system components. And rather than having a single production system at the University of Michigan, we would build a multi-site production system spread across multiple network providers and service providers so that no single problem would disrupt services. This is essentially the high availability (HA) version of ICPSR's delivery system. It would have the virtue of providing true 99.99%+ reliability, but would cost plenty of money to design, build, and operate. If you are running IT systems for a bank or a hospital or an aircraft carrier, you build them with HA. But what about a data archive?

Another option was to keep the ICPSR delivery architecture the same, but replicate it somewhere off-site. Automated jobs could keep the web content, data content, and web applications synchronized. And an easy - but manual - process could be used to redirect traffic to the replica when needed. In this world there would still be plenty of times where a component of ICPSR's delivery system might be off-line due to maintenance or a fault, but if the maintenance or fault was long-lived, then the replica could be pressed into service. This type of solution would be inexpensive to design, deploy, and operate, and would deliver a credible disaster recovery (DR) story, but would probably only give us uptime somewhere between 99.0% and 99.9%. Would that be good enough?

In the end, of course, we decided that the best use of resources would be to build a system that would still have some outages from time to time, but which would never again be off-line for an entire week. We set an availability goal of 99.5% for each month across all components. That is, every time a single component faults - search, download, online analysis, and so on - it counts against the uptime of the WHOLE system. And we would leave it up to the judgement of the on-call engineer to decide when a problem was likely to be long-lived enough to warrant a switch to the replica.

So we chose DR instead of HA.

Looking back, my sense is that we made the right decision. In practice we seem to hit our 99.5% availability goal most months, and because we did not tie up our software and systems development resources on rebuilding the delivery system to guarantee HA, we were able to design and build systems like our Restricted Contract System, Secure Data Environment, and Virtual Data Enclave. Of course, when we need to perform a major bit of maintenance like last weekend where it is important that we continue to point www.icpsr.umich.edu at the production system rather than the replica, it always makes me wonder about the HA alternative.

Provenance metadata and the OAIS Receive Submission

2012-01-16T05:32:00.000-05:00

The FLAME (File-Level Archival Management Engine) project continues to articulate functional requirements for the software system. So far the process has looked something like this:

Select one of the functional areas of OAIS
Drill down into one of the sub-functions within that area
Enumerate a list of high-level statements that should be true for that sub-function
Translate those high-level statements into medium-level specifications for the software

For example, we tackled one such area before the recent holiday break:

Ingest
Receive Submission
The producer provided basic provenance information at deposit

Of course, this raises the question: What do we consider "basic provenance information" at the time of deposit? What information can we collect from the deposited content, and what information do we need to collect from the person performing the deposit?

Here is the draft list we created:

a. FLAME should enable the ability to transfer digital content to ICPSR through web-based file upload
i. Uploading files should NOT require MyData authentication
ii. The act of uploading files serves as a signature for the transfer
b. FLAME should enable the ability to document receipt of content transferred to ICPSR through non-electronic means
i. Date package arrived (required)
ii. Shipping company (required)
iii. Tracking ID number (required)
iv. Other details about the shipment (optional)
c. FLAME should capture the following provenance information from the content provider:
i. Self-reported identity of the content provider, or identity from MyData profile (for electronic transfer)
1. Name of depositor (required)
ii. Self-reported contact information for the depositor, or from MyData profile
1. Email-address (required)
2. Telephone number (optional)
3. Mailing address (optional)
iii. Self-reported descriptive provenance information from the depositor
1. Name or title of deposit (required)
2. Summary or description of the deposit (optional)
3. Name of organization that sponsored the research, or "not applicable" (required)
4. Number of ID of the grant or contract, or "not applicable" (required)

d. FLAME should capture the following provenance information from the files after each content transfer:
i. Date and time at which each file is received
ii. Checksum of each file
iii. MIME type of each file
iv. Original name of each file
v. Packaging information (e.g., file was part of a Zip archive)

What do you think basic provenance information should include? Does our list look like it captures everything one could reasonably expect to collect at the time of deposit?

TRAC: A3.9: Self-assessment and certification

2012-01-13T05:00:00.000-05:00

A3.9 Repository commits to a regular schedule of self-assessment and certification and, if certified, commits to notifying certifying bodies of operational changes that will change or nullify its certification status.

A repository cannot self-certify because an objective, external measurement using a consistent and repeatable certification process is needed to ensure and demonstrate that the repository meets and will likely continue to meet preservation requirements. Therefore, certification is the best indicator that the repository meets its requirements, fulfills its role, and adheres to appropriate standards. The repository must demonstrate that it integrates certification preparation and response into its operations and planning.

Evidence: Completed, dated audit checklists from self-assessment or objective audit; certificates awarded for certification; presence in a certification register (when available); timetable or budget allocation for future certification.

Like a few of the other A-group TRAC requirements, this one really operates at the uppermost level of the organization, and so it is difficult to address it from the IT perspective.

HOWEVA..... One barrier to implementing a regular certification cycle are some fundamental questions:

Where do I find a list of consultants or analysts that can grant "TRAC certification" to my repository?

Which organization sanctions those consultants and analysts?

What does it mean - precisely - to be "TRAC certified?"

Are there different levels of TRAC certification, much like FISMA levels?

If I'm already FISMA certified, does that automatically grant TRAC certification for certain items (especially in section C)?

And so on.

It seems like there is a business opportunity here.

For instance, if ICPSR asserted that it was now in the business of reviewing TRAC requirements for organizations, and a team of ICPSR analysts would either certify your data archive as TRAC compliant or would identify clear action items required to become compliant, would that be a useful thing? Or would other organizations rise up to say, "Hey, who are you, ICPSR, to be granting certifications?"

How should this work?

December 2011 deposits at ICPSR

2012-01-11T11:38:00.000-05:00

Chart.

# of files	# of deposits	File format
98	13	application/msword
67	4	application/octet-stream
207	25	application/pdf
192	8	application/vnd.ms-excel
44	5	application/x-sas
86	13	application/x-spss
4	4	application/x-zip
2	1	image/jpeg
2	1	image/tiff
12	1	message/rfc8220117bit
7	7	text/html
23	2	text/plain; charset=iso-8859-1
69	5	text/plain; charset=unknown
468	13	text/plain; charset=us-ascii
7	2	text/rtf
1	1	text/x-c++; charset=us-ascii
1	1	text/x-c; charset=unknown
10	3	text/x-c; charset=us-ascii
4	1	text/xml

Lots of plain text and Excel this month, and not so much from the conventional stats packages. The usual set of C and C++ bogons that are undoubtedly plain text. And a large number of files where we could not identify the content (octet-stream) which tells me that we either received lots of binary data, or we are starting to see a new format that our MIME detector can't figure out.

ICPSR web availability through December 2012

2012-01-09T05:47:00.000-05:00

Web availability in December was looking very, very good through most of the month. We had seen only a single noteworthy event the entire month, and that resulted only in a few minutes of downtime. (As happens from time to time, a member site was scraping our web pages, presumably to collect the metadata we publish. And while professional scrapers like Google, Yahoo, and the other search engines scrape gently and non-intrusively, this is not often the case with less experienced scrapers.)

Of course, December is always a tricky month here at ICPSR. Snow storms. Ice storms. Power outages. I can't remember the last time that my entire team was able to take off the entire week between Christmas and New Years (like the rest of the U-M) without having to come into the office to troubleshoot a problem.

And this year was no different.

We started to see sporadic up/down alerts from the U-M network monitoring system on the morning of December 30. It looked like our production web server was working OK overall, but having some problems. When we tried to load the home page from home, the page wouldn't load. And when we tried to login (via ssh) from home, the connection timed out. It looked as if everything was down even though the monitoring system said it was OK.

We found we could log into other systems on campus, and then use those as a launch pad to get to ICPSR. All of our systems were up, but none seemed reachable from systems off campus. This explained why the U-M monitoring system didn't through more alarms earlier.

Then we noticed this:

http://status.its.umich.edu/outage.php?id=73300

(I think this link works even from off-campus.)

We then worked with the campus network engineers to draw their attention to the problem that was affecting us. Unfortunately it was kind of helpful to have the ICPSR web site be unavailable from off-campus as a test case; we would know the network was fixed when the web site was available again.

All in all not a horrible month for availability, but we moved from 99.9% on Dec 29 to 99.5% by the end of Dec 30.