During BVE last week I spent time chatting with Larry Jordan for the ‘The Digital Production Buzz’ podcast talking about:

• The two maxims .. now three…
• Highly secure protection of file-based rushes/dailies
• MatrixStore as nearline or parking space
• Keeping legacy assets online in order to re-use or re-purpose

We also touched on the integrations with CatDV/Final Cut Server and disk based archiving in general.

Click on the image to to to the website:

Larry  is an internationally-renowned consultant and Apple-Certified trainer in digital media with over 25 years experience as a television producer, director and editor with national broadcast and corporate credits. Based on Los Angeles, he’s a member of both the Directors Guild of America and the Producers Guild of America. Jordan is the Founder and Editor-in-Chief of Edit Well, the rich-media newsletter for Final Cut Studio, published by Peachpit Press. He is the author of hundreds of hours of online training, published by Lynda.com and two books on Final Cut Pro, published by Peachpit Press. Visit his website at www.larryjordan.biz.

The View From OM

I would like to thank Larry for once again giving us the opportunity to talk about MatrixStore and it position in video workflow. I will post about other nice stuff at BVE in another post.

TCO Analysis Document (TCOAnalysis)

An overview of the elements one needs to consider when putting together a TCO model. The elements in this document were garnered with the help from the wider storage community and creative professionals. The content under those elements is our work and we hope to have avoided FUDlike comment.

TCO Spreadsheet (Disk Vs Tape TCO v2.1.1 2009 (version 1))

Open to scrutiny and allows the user to enter their own values. We have only put in one tape library that has been spec’d to grow at 50TB p.a as, surprise surprise, we have been unable to get accurate pricing on a solution like the Scalar i2000 which can scale to 4500 tape slots (just under 5PB). If anyone out there has that pricing and can plug it in please do so and report back here in the comments.

In the spreadsheet we have place holders for showing the benefit of having the data available online and available. When you start to input the manual costs of retrieving content the TCO swings further towards clustered disk.

The View From OM

Please take these docs, scrutinise them, love them, hate them, change them, share them, add products and pricing. It is one way as a community we can create a living TCO model.

Being a vendor of disk based solutions we are of course confident the numbers will stay in favour of disk. We more than happy to have a conversation around this subject and potentially walk through the documents in order to ensure you get the best out of them for your needs. Get in touch!

Previous Posts on TCO:

http://www.matrixstore.net/2009/09/17/defining-an-up-to-date-tco-model/

http://www.matrixstore.net/2008/03/03/problems-with-total-cost-of-ownership/

So that’s what we did. We built security and compliance functionality into MatrixStore from day one. I would like to state now (in the way one of our percieved competitors seems to do quite often .. wow .. geee, aren’t we the bees knees, woot!) that it was our incredible foresight that saw this functionality being 100% applicable to the media and entertainment space… yeehaa. However, the reality is that as a talented team of geekoids we decided to build Rome (compliance, distributed search, seamless scaling, replication .. yada yada yada) and wait to people to come and buy it. If you build it they will come?

Anyway, back to the point of this blog post. How is it that the functionality (included free don’t you know) built for the protection of cheque images and MRI scans is applicable to companies with video workflows or legacy assets? I have covered some of the key points below and at the bottom of this post is a 2 pager on how MatrixStore helps with organisations under the regulatory cosh.

Authenticity

Producers, Facility managers want to know the content that has come off the camera and going into post is authentic. Guaranteed. MatrixStore generates a hash (digest, digital fingerprint) of the content on the client and also on the two MatrixStore nodes the content lands on. Three different places where the content is checked ensuring the content in MatrixStore is bit for bit the same as came off the P2 or SxS card.

Because MatrixStore keeps that digest/fingerprint as metadata with the content we are able to automatically, periodically and systematically (insert line from Grease here) ensure the content on disk matches the digest kept with it. Forever.. and ever.

Retention

Generating video content and adopting creative workflows is no longer restricted to post-production facilities or broadcasters. There is a bucket load of content being generated daily within heavily regulated industries. Hospitals, banks, police forces, utility companies. All producing video and some of them required to adhere to regulations around the retention and authenticity of that video content. Banks for instance may face FSA regulations that require content to be kept protected and available for 1 year from creation. Another factor is that a lot of these organisations are managing the creative process in-house which means they are turning to video workflow solutions over the less appropriate IT based tools. Solutions integrated into those workflows will win.

MatrixStore provides the option to keep content in vaults that can have retention periods set upon them. No deletion or modification before the retention date. End of. This helps not only for the long term retention of content but also as a highly secure ingest vault for rushes/dailies coming into the post process. Set a one month retention policy on an ingest vault and that content will be secure from the accidents that do tend to happen.

Mind you, ask most content owners how long they want to keep their completed works for and the answer will be forever. Good recurring support revenue/business if you can get it.

Access & Availabilty

Not providing an email for a court case has cost some companies a pretty penny. Broadcast or post facilities not being able to serve up content in a timely fashion can be equally costly. Data on demand is key functionality even if components fail.

Find is also key here, the magical eDiscovery. If you can’t find it, you don’t have it (Maxim no. 2). Metadata coupled closely to the content is very important if you are going to stand a hope of getting the proverbial needle out of the 500TB haystack. MatrixStore has that covered.

Equally important is restricting access to content from those who are not authorised to do so. MatrixStore allows content to be kept in vaults that can only be accessed by personnel who have the correct security credentials. Whilst using the same underlying hardware the vaults are completely distinct.

Auditing and Governance

Being able to provide a full audit trail of the actions performed upon a piece of content is crucial in the world of regulations and is also a nice feature for organisations with video workflow (and can actually be very useful where elements of the workflow require repeatable results). MatrixStore provides a built-in full audit trail of all actions, both management and data operations, performed against a MatrixStore cluster. Who archived the content? Who deleted the content? Who TRIED to delete the content? Who retrieved the content and put it on YouTube just before it was officially released? That sort of useful.

The view From OM

Organisations across the globe that are subsiduaries of US companies are pretty much tied to keep in line with US regulations such as Sarbanes Oxley. Even if video data is not a concern of Sarbanes Oxley the internal compliance departments within organisations tend to want all business data treated the same. It has to be protected, authentic and available when required.

We have provided this functionality as part of the solution, at no extra license or cost. As it should be. here is the sheet for your perusal:

The OM  team will demonstrate how The MatrixStore can protect content from ingest to archive:

- Guaranteed ingest with metadata harvesting (Panasonic P2)

- Secure Ingest Vaults (Lock down from accidental deletion)

- Production/Project parking to a Nearline Vault in AVID and Final Cut Workflows (one click operation)

- Archiving Legacy assets to an online Library Vault using DropSpot, Final Cut Server and CatDV

Implementing a tapeless or file based workflow has been highlighted as one of the top challenges facing creative organisations in 2010. With no tape to put on the shelf broadcasters, post-production facilities and creative agencies are looking for a solution to keep their assets secure and more importantly available throughout the production process. A single MatrixStore cluster acts as pre-production, nearline or local archive allowing content to move seamlessly through any digital workflow from ingest to delivery.

The View From OM

We are very much looking forward to the show so come along and see us on stand C10!

One sweet spot for MatrixStore technology is in the creative space. Creative organisations are increasingly being subjected to change driven by external forces such as changes in the way content is captured and our insatiable appetite for HD or 3D content (see The Big Broadcast Survey published in 2009). The industry is one that is used to adapting to whatever is thrown at it but it also relies on solutions that just work.

Below is a quick summary of the challenges they currently face and how MatrixStore is positioned to help. I will elaborate on each point in later posts :

1. File Based/Tapeless Ingest - Some call it pre-production archiving, some call it prechiving, the problem is the same, more and more content is being captured on disk or flash. There is no tape to put on the shelf. Protecting content that enters the building in an audited manner is a key area where MatrixStore is helping. Integration into Final Cut and Avid workflows, digital fingerprints on ingest, locked vaults to protect from deletion and metadata harvesting from P2, SxS etc are some of the benefits.

2. Parking/Nearline – Production disk, the online, the SAN should be kept lean and mean. Some projects can take years to complete so content needs to be moved off and onto the expensive disk repeatedly over a prolonged period. Tape based technologies do not cut the mustard. Organisations are also looking to ‘rightsize’ their expensive disk by offloading older or less-used content that should not sit clogging up the SAN to a more appropriate tier of storage. Seamless one touch capacity expansion, DropSpot (metadata support/search) and performance make MatrixStore a natural choice.

3. Legacy/Local Archive – organisations who own content need to protect their content or they will not be able to re-use or monetise that content. The two maxims come into their own here. Integration with DAM/MAMs, Self-Management,  redundancy of data, retention policies (prevent change or deletion), metadata and search goodness are key here.

4. Content Manipulation/Distribution Repository - If you own the content and have it protected and available on disk then what can you do with that data? Clustered/Grid architecture, high availability, metadata and search again key.

5. Regulations - More and more financial institutions are using video in the way they communicate with staff or shareholders, that video needs to be protected and stored for a pre-determined period in accordance with global regulations. MatrixStore has built in characteristics that satisfy these compliance requirements.

The View From OM

Even medium to large creative organisations will not have dedicated I.T staff to work on the video side of the business so they need solutions that just work and do not take them out of their comfort zone. The MatrixStore is the storage geek so they don’t need one.

Cardiff, UK – Object Matrix, a developer of secure asset archive and retrieval software, has
today supplied Cardiff based post-production facility Mwnci with its pre-production rushes archive and nearline archive solution, The MatrixStore. The multi-purpose data management
solution provides a secure archive for client owned content that enters the facility and also acts as a secure parking location for ongoing projects.

Rich Moss, Managing Director of Mwnci stated, “More and more of the projects coming into Mwnci are file based and as such we need to ensure we have the best infrastructure in place to cope with not only the incredible amount of content but also the wide variety of the content formats that come into the facility. The DropSpot software from Object Matrix understands the Panasonic P2 and Sony SxS formats allowing metadata entered on the camera to be harvested and attached to the content in our MatrixStore cluster. We made a decision to keep our production SAN at an optimal size for both performance and throughput whilst moving older content that still needs to be accessed to a more appropriate platform.

The MatrixStore fits perfectly as both the pre-production and the nearline archive in our Avid based workflow.”

Mwnci uses mostly Avid technology in the post production of television programming for clients such as the BBC and S4C.

More here….

The View From OM:
For more information on the use of MatrixStore in an AVID workflow go here..

Happy Days. A great end to 2009. Merry Christmas and wishing you a prosperous new year!

Adding to our proven Apple Final Cut Server and CatDV workflows we’ve made some recent improvements to using MatrixStore as both a Pre-Production Archive (ingest location) and a Nearline Archive in Avid workflows.

The workflow shown in this blog post uses Avid AMA and Avid Unity together with MatrixStore at a post-production house here in the UK.

The Phase 1 setup below was fully tested during whilst the items in phase 2 were left to be tested at a later stage (although they are assumed to be working and will be tested as required).

Phase 1 Usage

• Footage and metadata were ingested from P2 cards and local disks
• Windows clients (Vista 64 bit, XP 32-bit)
  
• 3-node MatrixStore Nearline cluster accessed from clients on a Gigabit network
• Avid AMA clients on Windows Vista 64-bit machines
• DropSpot (MatrixStore Client data browser application on Windows and Mac)
• Avid Unity on a FC network

Phase 2 (not tested but to be tested when required)

• Sony SxS cards
• Avid AMA Mac clients
• Panasonic P2 Viewer updating to MatrixStore
• Avid AMA building sequences to MatrixStore
• LTO tape writers running off MatrixStore via MXFS

The following diagram shows the above set-up:

Commentary and Results

The MatrixStore cluster and clients were installed, physically connected to the network and were usable within 3 hours of arriving onsite, with the main delay being working out which clients to use for which stages of the workflow.

Ingest media from P2 or disk was imported via the DropSpot application using the following steps:

1. Data folders to move the footage to were created in DropSpot.
2. Data was imported via DropSpot to the MatrixStore.
3. Multiple clients could Ingest in parallel.

During the Ingest of data, the metadata from the P2 cards was examined and added to the footage itself. This allows the footage to be searched for and also means that the metadata will be kept securely with the footage for as long the footage is kept in the MatrixStore Pre-Production/Nearline archive.

DropSpot runs on Mac, Windows and Linux, and MatrixStore was also found to be useful as a shared storage location between clients.

The resulting data movement occurred as fast as the data could be read. In the case where P2 cards were used this was approximately 50MB/s to 60MB/s, and in a test where data was read from a fast direct attached storage device, data moved at the full network card interface speed available.  Sony SxS cards are expected to ingest at a slower rate where they are using USB technology.

For Edition, Avid AMA clients were set-up on Windows Vista 64bit and XP 32 bit machines and used the MatrixStore “MXFS” virtual filesystem to view and update ingested data as if it were on an attached SAN device. This was achieved in two stages:

1. The folders of Ingested data were scanned by selecting the Avid AMA import volume option. Scanning a folder takes around 20-30 seconds per P2 card whilst Avid software browsed and created stubs.
2. Avid AMA was then to consolidate the data onto the Unity platform. Where consolidating didn’t transcode the data moved from the MatrixStore to Unity at full speed with the client CPU utilized seen to be approximately 10%. Where consolidation was involved transcoding the data moved at 25-30MB/s and the 8 cores of CPU were at 90%+ (ie the bandwidth was limited by the transcoding CPU speed).

Multiple clients can consolidate/send data to Unity in parallel.

One typical usage scenario of the MatrixStore Nearline used was that prior to ingest of data, the user would open up one to three data streams in an Avid AMA client to view the content. So long as the client could cope with the streams (e.g., the graphics card could cope) this worked very well.

Conclusion

• The set-up of the Nearline archive and clients was straightforward and resulted in the Nearline archive being ready to be used on the day that it arrived, validating the Object Matrix “plug and play” analogy.
• The main bottle-necks in the workflow were the speed of the cards being read from and the time taken to transcode data (the users were very happy to see their dual quad core clients being used to their capacity!).
• The new MatrixStore MXFS connection from Windows and Mac was straightforward to set-up and provided up to full bandwidth across the network.
• MatrixStore acted as a first class Nearline archive keeping data safe, secure and available throughout.

Future plans for Object Matrix include forthcoming direct support for Avid Interplay and further support for metadata extraction and handling, but if you’d like to know more about our reference sites and plans for future workflow support let us know!

INGEST:  The Space Box receives and delivers high speed content.

PRODUCTION MEDIA MANAGER:  CatDV catalogues the media allowing you to add and search for metadata

ADOBE NATIVE FILE BASED WORKFLOW: Without Rewrapping  or transcoding Red, P2, XDCAM & AVCHD files

ADOBE PROJECT CREATION & OUTPUT: Speech search, Effects, XML exchange with Avid & FCP

OUTPUT: Blu-Ray authoring with Encore and P2 Archiving with Rimage.

ARCHIVE: MatrixStore – pre-production and nearline storage for tapeless workflows

View Invites

• Cardiff (6pm 17th Nov 2009): adobe-cardiff-seminar
• Bristol (6pm 18th Nov 2009): 0909-qls-adobe-seminar-bristol

Come along and say hello!

Broadcast Engineering Article on Tapeless Digital Workflow

http://broadcastengineering.com/automation/what-digital-workflow-mean-creative-industry-1028/

Article on Data Centre Inefficiencies.

http://www.bcs.org/server.php?show=conWebDoc.32449 (BCS – Chartered Institute for IT)

http://www.businesscomputingworld.co.uk/?tag=energy-consumption (Business Computing World)

Cloud Computing Opinion Piece

http://www.launchlab.co.uk/article/Business-tech/What-is-Cloud-Computing-and-what-are-the-benefits/892

TV-Bay article on Panasonic P2 Live Centre and also piece on MatrixStore as pre-production and nearline archive.

http://www.tv-bay.com/imag/issue_034/pageflip.htm

That DAM Show (Audio Clip)

http://www.thatdamshow.com/latest/?currentPage=6

Digital Production Buzz

http://www.digitalproductionbuzz.com/Archives/buzz_09_05_14.m4a